i-FRAME : proposed methodological framework to asses the impacts of social policy pnnovations promoting social investment in the EU.
Date: 2017-11-20
Institutions: Joint Research Centre
Subjects: EU research policy, Internet, digital technology, innovation, investment, research report, simulation, social impact, social policy

i-FRAME – Assessing impacts of 
social policy innovation in the EU 

Proposed  methodological  framework 
to evaluate socio-economic returns on 
investment of social policy innovations 

Gianluca Misuraca, Luigi Geppert, Cristiano Codagnone  

2017  

EUR 28824 EN 

 
 
 
 
This publication is a Science for Policy report by the Joint Research Centre (JRC), the European Commission's 
science  and  knowledge  service.  It  aims  to  provide  evidence-based  scientific  support  to  the  European 
policymaking  process.  The  scientific  output  expressed  does  not  imply  a  policy  position  of  the  European 
Commission.  Neither  the  European  Commission  nor  any  person  acting  on  behalf  of  the  Commission  is 
responsible for the use that might be made of this publication. 

Contact information 

Gianluca Misuraca 

JRC Unit B4, Edificio Expo, C/ Inca Garcilaso, 3 – 41092 Seville, Spain 

E-mail: gianluca.misuraca@ec.europa.eu 

Tel.: (+34) 954488718 

JRC Science Hub 
https://ec.europa.eu/jrc 

JRC108078 

EUR 28824 EN 

PDF 

ISBN 978-92-79-74271-2 

ISSN 1831-9424 

doi:10.2760/83089 

Luxembourg: Publications Office of the European Union, 2017 

© European Union, 2017 

The  reuse  of  the  document  is  authorised,  provided  the  source  is  acknowledged  and  the  original  meaning  or 
message of the texts are not distorted. The European Commission shall not be held liable for any consequences 
stemming from the reuse. 

How  to  cite  this  report:  Misuraca,  G.,  Geppert,  L.,  Codagnone,  C.,  (2017)  i-FRAME  –  Assessing  impacts  of 
social  policy  innovation  in  the  EU,  European  Commission's  Joint  Research  Centre,  JRC  Science  for  Policy 
Reports Series; EUR 28824 EN; doi:10.2760/83089 

All images © European Union 2017 except cover page (Credits: Oxford University Press) 

This report is based on the results of the IESI Project conducted under the Administrative arrangement between 
JRC and DG EMPL (№ 33268-2014-01) for a multi-year research on 'ICT Enabled Social Innovation to support 
the implementation of the EU Social Investment Package' – https://ec.europa.eu/jrc/en/iesi 

Title: i-FRAME – Assessing impacts of social policy innovation in the EU: Proposed methodological framework to 
evaluate socio-economic returns on investment of social policy innovations  

Abstract 
This  report  presents  the  final  proposal  for  developing  a  methodological  framework  to  assess  the  impacts 
generated  by  social  policy  innovations  which  promote  social  investment  in  the  EU,  in  short  i-FRAME.  This 
framework has the objective to provide a structured approach  that shall serve as a comprehensive framework 
for conducting analysis of the economic and social returns on investments of social policy innovations. It also 
aims to act as a guide to gather insights into replicability and transferability of initiatives which promote social 
investment  across  the  EU.  The  report  outlines  the  reviewed  and  improved  theoretical  and  methodological 
approach  developed  by  the  JRC  with  help  from  external  experts,  and  validated  by  testing  the  operational 
components  proposed  on  a  number  of  case  studies  and  scenarios  of  use.  After  outlining  the  conceptual  and 
methodological  approach  underpinning  the  i-FRAME  (V1.0),  the  report  discusses  the  proposal  for  building  its 
operational  components  according  to  a  structured  theoretical  framework  of  a  dynamic  simulation  model  for 
social  impact  assessment  (V1.5).  The  final  proposal  for  i-FRAME  (V2.0)  and  an  overview  of  the  operational 
components for its implementation are then presented discussing the key elements that should be developed to 
build a comprehensive i-FRAME Web-Platform and simulator for social impact assessment. Conclusions are then 
offered in terms of implications for policy and directions for future research. These were drawn after consulting 
experts  from  different  research  disciplines,  practitioners  and  representatives  of  relevant  stakeholders  and 
policymakers, and they include .recommendations for further developing the operational components proposed, 
paving the way towards building the i-FRAME (V3.0) and beyond.  

Table of contents 

Acknowledgements ................................................................................................ 1 

Executive Summary ............................................................................................... 2 

1  Introduction ...................................................................................................... 6 

1.1  Background................................................................................................. 6 

1.2  Objectives .................................................................................................. 9 

1.3  Outline ..................................................................................................... 10 

2  Methodology ................................................................................................... 11 

2.1  Co-designing the i-FRAME ........................................................................... 11 

2.2  State of the art and literature review ........................................................... 13 

2.3  Testing and validation ................................................................................ 18 

3  i-FRAME 1.0: piecing the puzzle together ............................................................ 19 

3.1  The quest for and challenges of social impact assessment ............................... 19 

3.2  Addressing the complexity of digital social policy innovation ............................ 21 

3.3  Outlining a 'meta-framework' for social impact assessment ............................. 22 

4  i-FRAME 1.5: the potential of dynamic simulation modelling .................................. 28 

4.1  Deconstructing social policy innovations as complex systems .......................... 28 

4.2  i-FRAME Decalogue: experimenting with system dynamics .............................. 33 

4.3  Insights from testing, validation and review of i-FRAME 1.5 ............................ 39 

5  i-FRAME 2.0: pluralistic evidence-informed policy-making ..................................... 46 

5.1  Conceptualising and contextualising social policy innovation ............................ 46 

5.2  Improved meta-framework and operational components ................................ 53 

5.2.1  The revised meta-framework ............................................................... 53 

5.2.2  Operational components of the i-FRAME 2.0 .......................................... 57 

5.2.3  Prototyping a computer-based simulation model for i-FRAME .................. 65 

5.3  Less is more: from a 10 to a 6-step approach beyond i-FRAME 2.0 .................. 68 

5.3.1 

i-FRAME 2.0 approach for evidence informed social policy innovation ....... 68 

5.3.2  Evidence Informed Social Policy Innovation warehouse ........................... 69 

5.3.3 

i-FRAME 2.0 extended modules of the K+S Macro Agent-Based Model ...... 70 

6  Conclusions: paving the way to i-FRAME 3.0 and beyond ...................................... 78 

6.1  Beyond reductionist and positivist evidence-based policy ................................ 78 

6.2  Future research directions: towards i-FRAME 3.0 ........................................... 80 

6.3  Policy implications ..................................................................................... 82 

Annexes ............................................................................................................. 84 

Bibliography...................................................................................................... 116 

i 

List of figures 

Figure 1: i-FRAME 1.0 Outline ............................................................................................... 3

Figure 2: Diamond for Evidence-Informed Social Policy Innovation ............................................ 4

Figure 3: Methodological approach for developing and validating the i-FRAME ............................ 11

Figure 4: Methodological framework for systematic review ...................................................... 15

Figure 5: Ecosystem of ICT-Enabled Social Innovation promoting social investment: generic 
stylized modelling environment and its dynamics ................................................................... 21

Figure 6: i-FRAME Outline – meta-framework ........................................................................ 25

Figure 7: Conceptual model underpinning the i-FRAME methodological approach ........................ 31

Figure 8: Example of application of the conceptual model underpinning the i-FRAME .................. 32

Figure 9: Example of a System Dynamics model of 'relevant population' ................................... 35

Figure 10: Example of 'domain-related sub-models of a System Dynamics model ....................... 36

Figure 11: Example of a dynamic simulation model (in a CLD) assessing the impact of ICT 
investments on ageing ........................................................................................................ 38

Figure 12: Domain–related sub-models of a dynamic simulation model (in CLD) assessing the 
impact of ICT investments on ageing .................................................................................... 38

Figure 13: CLD model of simplified PASS-like case .................................................................. 40

Figure 14: Stock and Flow diagram of PASS-like model ........................................................... 41

Figure 15: Output of simulation modelling of PASS ................................................................. 42

Figure 16: Integrated social policy and service innovation in practice ........................................ 54

Figure 17: i-FRAME 2.0: Diamond for Evidence-Informed Social Policy Innovation ...................... 55

Figure 18: Support Factors - Reading Score Example .............................................................. 57

Figure 19: LTU problem tree ................................................................................................ 58

Figure 20: LTU intervention support factors ........................................................................... 58

Figure 21: LTU intervention simplified causal depiction ............................................................ 59

Figure 22: Example of 'process tracing' approach on World Bank Social funds ............................ 60

Figure 23: Support Factors for Social Funds ........................................................................... 60

Figure 24: Quick Exit Tree for Work Ready Policy for Problem Drug Users (PDU) ........................ 61

Figure 25: LTU Intervention Logic Model ............................................................................... 62

Figure 26: Generic micro-level measurement tool ................................................................... 63

Figure 27: IESI Online Knowledge Base and Portal ................................................................. 65

Figure 28: Preliminary proposal of prototype of i-FRAME 2.0 Simulator Web-Platform ................. 66

Figure 29: Interactive and dynamic warehouse for evidence-informed social policy innovation ..... 70

Figure 30: The K+S model with the labour-market extension ................................................... 71

Figure 31: Labour market structural reforms: impact on unemployment and vacancy rates ......... 73

Figure 32: Labour market structural reforms: impact on inequality ........................................... 73

Figure 33: Labour market structural reforms and the emergence of hysteresis ........................... 74

Figure 34: SD, Discrete Events, and ABM .............................................................................. 85

Figure 35: Schematic presentation of the K+S model .............................................................. 87

Figure 36: Analysis of simulation output ................................................................................ 90

Figure 37: Counterfactual causation.................................................................................... 100 

ii 

List of tables 

Table 1: Social innovation in the Europe 2020 strategy 

Table 2: Social Innovation in EU initiatives 

Table 3: Typology of social policy and service innovation 

Table 4: LTU intervention monitoring indicators (exemplificative only) 

Table 5: Comparison of models 

Table 6: SD and ABM compared 

Table 7: SD, ABM, and their hybridisation compared 

Table 8: Database overview 

Table 9: Systematic review - Search strings 

Table 10: Systematic review – full results 

Table 11: Systematic review – Employment 

Table 12: Systematic review – Social inclusion 

Table 13: Systematic review – Long-term care 

Table 14: Relevant statistics and datasets 

16 

17 

47 

64 

84 

85 

86 

91 

92 

96 

107 

112 

113 

114 

iii 

 
 
 
Acknowledgements 

This  report  is  one  of  the  outcomes  of  the  research  on  'ICT-Enabled  Social  Innovation  to 
support  the  implementation  of  the  Social  Investment  Package'  (IESI),  conducted,  by  the 
European  Commission's  Joint  Research  Centre  in  collaboration  with  DG  Employment,  Social 
Affairs and Inclusion (DG EMPL) between 2014 and 2016. 

The  report  has  been  elaborated  under  the  scientific  leadership  of  Gianluca  Misuraca,  by  the 
IESI  Team  which  was  part  of  the  Information  Society  Unit  of  the  Institute  for  Prospective 
Technological Studies (IPTS), and now within the JRC Directorate for Growth and Innovation.  

However, it is the result of  extensive collaboration, as it incorporates  the findings of different 
support  studies  conducted  by  external  contractors  under  the  supervision  of  JRC.  It  also 
integrates inputs received by many  experts  who acted as reviewers or participated in various 
consultation workshops.  

Therefore,  we  are  grateful  to  all  the  experts  and  researchers  that  provided  support  to  the 
conceptualisation,  design,  testing  and  validation  of  the  i-FRAME  proposal,  and  we  hope  to 
cooperate further with each of them in the future.  

In particular, we are thankful to Karl Richter and the teams of EngagedX and the University of 
Kent  for  supporting  in  reviewing  the  state  of  the  art  of  social  impact  assessment  and  to 
contribute to the development of the i-FRAME V1.0 in the preliminary exploratory phase of the 
research  over  the  course  of  2015.  We  are  also  thankful  for  the  work  carried  out  by  the  Fair 
Dynamics  research  team,  under  the  scientific  coordination  of  Luigi  Geppert.  This  team 
produced  a  study  which  developed  a  proposal  for  an  operational  framework  to  assess  the 
impacts  generated  by  ICT-enabled  social  innovation  initiatives  which  promote  social 
investment in the EU (i-FRAME V1.5).  This study, conducted during 2016, allowed the JRC to 
test  a  number  of  operational  approaches  to  simulation  modelling  with  contributions  from 
external  experts,  including  Prof.  Pal  Davidsen  of  the  University  of  Bergen,  and  Douglas 
McKelvie from the Symmetric Partnership, UK.  

The  current  final  proposal  for  the  development  of  i-FRAME  V2.0,  includes  the  preliminary 
results of the support study carried out by a consortium composed of Open Evidence, Ernst & 
Young  and  the  Sant'Anna  School  of  Advanced  Studies  and led  by  Cristiano  Codagnone  (State 
University  Milan),  Francisco  Lupianez-Villanueva  (Open  University  of  Catalonia),  and  Andrea 
Roventini from the Sant'Anna School of Advanced Studies in Pisa.  

The  final  proposal  for  the  development  of  i-FRAME  also  builds  on  comments  and  suggestions 
from independent external reviewers on intermediate pieces of the research. Special thanks go 
to  Gwendolyn  Carpenter  of  the  Danish  Technological  Institute  (DTI),  and  Simona  Milio  of  the 
London  School  of  Economics  (LSE),  who  reviewed  the  first  proposal  for  i-FRAME  (V1.0);  to 
Mehtap Agkuc, of the Centre for European policy Studies (CEPS) and Bruce Edmonds, Director 
of  the  Centre  for  Policy  Modelling  at  the  Manchester  Metropolitan  University  and  Petra 
Ahrweiler, Director  of the European Academy  of Technology and  Innovation Assessment,  who 
reviewed the version 1.5 of the i-FRAME proposal and provided comments for the version 2.0.  

Furthermore,  the  development  and  validation  of  the  i-FRAME  benefited  from  inputs  from 
participants  in  various  events  where  it  was  presented  and  discussed.  More  specifically,  we 
would like to thank all the experts and representatives of stakeholders who attended the IESI 
Experts  and  Stakeholders  Workshops  (Brussels  on  4th  November  2014  and  24-25  February 
2015 and Seville in 2016 on 7-8 July, 6th April and 25th October), the Final IESI  Conference 
(Brussels, 14-15 March 2017), and several other policy events. At these  events,  the i-FRAME 
was discussed with practitioners, policymakers and researchers from various disciplines. Their 
valuable insights and suggestions have inspired the development of the i-FRAME to date, and 
will be taken into consideration to set the future research directions.  

Finally,  we  are  indebted  to  our  colleagues  of  DG  EMPL  for  their  continuous  guidance 
throughout the 'co-development' of the proposal for the i-FRAME since its inception in 2014. 

1 

 
 
Executive Summary 

This report presents the final proposal for developing a methodological framework 
to  assess  the  impacts  generated  by  Social  Policy  Innovations  which  promote 
social investment in the EU – i-FRAME. This framework has two objectives: 

1.  To  provide  a  structured  approach  to  the  analysis  of  initiatives  collected  through 

mapping social policy innovations in the EU.  

2.  To serve as a comprehensive framework for conducting analysis of the economic 

and social returns on investments of social policy innovations.  

In  doing  so,  it  aims  to  act  as  a  guide  to  in-depth  analysis  to  gather  insights  into 
replicability  and  transferability  of  initiatives  which  promote  social  investment  across  EU 
Member States. 

The  conceptual  framework  and  methodological  approach  underpinning  the  i-FRAME  was 
outlined by the JRC in 2015 and further described in the proposal for i-FRAME (V1.0). 
This  was  then  further  elaborated  and  tested  with  support  from  external  experts,  by 
applying  it  to  selected  scenarios  of  use  drawn  from  case  studies  carried  out  across  the 
EU.  This  allowed  the  JRC  to  develop  a  proposal  for  a  theoretical  and  methodological 
approach to building the operational components of the i-FRAME (V1.5) according to a 
structured theoretical framework of a simulation model for social impact assessment. 

The  final  proposal  for  developing  i-FRAME  (V2.0)  outlines  a  reviewed  and  improved 
theoretical and methodological approach. It aims to develop a comprehensive framework 
for  evaluating  the  social  and  economic  impacts  of  social  policy  innovations 
promoting social investment, where ICTs play an important role. In this respect, 
whereas social investment captures the 'congeries of ideas about the objectives, areas 
of intervention and instruments', social innovation 'represents the enablers and drivers 
for  social  change,  more  equal  economic  development  and  possible  shared  prosperity'. 
Social  innovation  can  improve  the  efficiency  of  social  policies  and  their  effectiveness  in 
addressing societal challenges and facilitate life-long investment in human capital.  

However,  evidence  to  inform  decisions  is  often  limited  or  produced  too  late,  thus 
becoming  ineffective,  or  insufficient  to  support  policy-making  in  a  structured  manner. 
The  limits  of  traditional  approaches  are  even  more  evident  in  the  case  of  social 
innovation, as the concept itself is ill-defined and difficult to measure. The review of the 
state of the art thus pointed to the  need to define a methodological  approach and 
develop  a  meta-framework  capable  of  assessing  the  social  and  economic 
returns  of  initiatives  promoting  social  investments.  This  need  is  even  more 
pressing  when  it  comes  to  assessing  impacts  generated  by  initiatives  that  address 
complex phenomena such as innovations in social policy and social services delivery. 

Social policy innovation is conceived as an ecosystem - a complex adaptive system 
in  which  different  phenomena  are  interconnected.  In  this  ecosystem,  people  act  in 
partnerships  and  networks,  while  integrated  programmes  are  implemented  within  a 
system  of  multi-level  governance.  Like  other  complex  systems,  it  presents  causal 
relationships which cannot be completely controlled or predicted in advance. 

The  dimensions  of  this  social  policy  innovation  ecosystem  laid  the  foundations  for  the 
initial  proposal  for  i-FRAME  1.0.  This  included  the  elements  to  consider  for  defining 
data and variables that serve to structure and  'feed' the impact assessment framework. 
One of the crucial dimensions of this ecosystem is linked to innovation process dynamics 
and the level of maturity of the social innovations. Knowing a social innovation's 'stage 
of  maturity'  may  be  important  for  tailoring  better  support  measures  and  adequate 
funding  structures.  This  is  fundamental  for  those  social  innovations  where  ICTs  play  a 
'game-changing' role. For these innovations, it may be more difficult to gather scientific 
evidence of impact with traditional evaluation techniques or modelling approaches. 

2 

 
 
The  i-FRAME  therefore  aims  to  capture  the  direct  effects  and  indirect  consequences  of 
policy interventions and to understand how these affect beneficiaries, organisations and 
possible  intermediaries,  as  well  as  the  social  innovation  eco-system,  and  the  welfare 
system in which such initiatives are embedded.  

The  i-FRAME  can  include  existing  methodologies  and  approaches  that  can  be  combined 
depending  on  the  specific  needs  and  levels  of  analysis.  However,  traditional  policy 
evaluation methods have shown their limitations when dealing with complex issues, and 
thus the proposed meta-framework is meant to overcome these limits. 

Figure 1: i-FRAME 1.0 Outline 

Source: Misuraca et al., 2015, JRC Technical Report, IESI D2.1 – i-FRAME 1.0 

On  the  one  hand,  the  meta-framework  encompasses  both  objectives  of  helping 
'Social  Policy  Innovation 
policymakers  and  practitioners  better  understand  the 
Ecosystems'  and  of  assessing  the  conditions  in  which  where  their  interventions  take 
place. On the other hand, the operational components focus on pragmatic micro-level 
measurement  tools,  computer-based  instruments  for  data  gathering  and  analysis,  and 
macro-level simulation modelling approaches rooted in complex systems theories. These 
tools support the design and evaluation (ex-ante, in-itinere, and ex-post) of social policy 
innovations through a modular approach for development, testing, and validation.  

Thus,  as  a  result  of  the  development  and  testing  phase  of  the  research,  the  i-FRAME 
methodology  was  designed  as  a  structured  approach  which  identifies  the  actions  that 
should be followed to  shape a dynamic simulation model of the impacts  of  ICT-enabled 
social innovation initiatives which promote social investment, in an attempt to link micro, 
meso  and  macro  level  effects.  This  approach  can  help  stakeholders  to  cope  with 
innovation-related uncertainties and contribute to a better understanding of the various 
factors  which  influence  the  evolutionary  process  related  to  social  policies  and  their 
innovation.  They  also  help  to  define  favourable  conditions  by  considering  alternative 
development paths and outcomes. 

In  this  perspective,  as  demonstrated  by  the  findings  of  the  literature  reviews,  it  was 
necessary  to  deconstruct  and  interpret  social  policy  innovation  ecosystems  using 
complexity  thinking.  In  some  cases,  this  exercise  could  be  conducted  by  using  more 
formalised  techniques  such  as  modelling  simulations  that  include  Agent-Based  Models 
(ABMs),  System  Dynamics  (SD),  AMB/SD  hybridisation,  and  Social  Network  Analysis 
(SNA), all of which can be informed by behavioural insights.  

3 

 
 
 
When  this  is  not  possible,  ecosystems  could  be  reconstructed  through  in-depth 
qualitative case studies, which in turn fed into, and support, the micro- and macro-level 
operational tools.  

In this respect, the final version of the proposal for developing a methodological 
framework to assess social policy innovations which promote social investment 
-  i-FRAME  2.0  keep  both  quantitative  and  qualitative  dimensions  together,  by 
developing  and  piloting  the  various  operational  components  in  diverse  stages  of 
development and application domains, as well as different timeframes.  

The  i-FRAME  2.0  is  thus  presented  as  a  generic  meta-framework  applicable  at  different 
levels: policy, service, programme and project. In addition to specific impact assessment 
methods, it envisages a phase of Deliberation and Design which will shape the design 
of interventions and eventually inform the three phases of evaluation in a loop that will 
produce  gradual  but  constant improvements  across  the  cycle  in  what  we  have  depicted 
as the Diamond for Evidence-informed Social Policy Innovation (EISPI). 

Figure 2: Diamond for Evidence-Informed Social Policy Innovation 

Source: Misuraca et al., 2017, JRC Technical Report, IESI D2.3 – i-FRAME 2.0 

The  operational  components  of  the  i-FRAME  2.0  for  impact  assessment  then  centres 
on a number of tools according to the three evaluation phases:  

i)  Ex-ante:  traditional  Impact  Evaluation  methods  and  tools  (e.g.  Intervention  logic, 
measurement indicators etc.); as well as the proposal of an experimental application 
and extension of a simulation approach using Macroeconomic Agent-Based-Modelling 
(MABM);  

ii)  In-itinere:  traditional  measurement  and  monitoring  tools  and  methodologies  for 

theory-based evaluation; and  

iii)  Ex-post:  experimental  and  quasi  experimental  techniques;  structured  system  of 

measurement indicators; and theory-based impact evaluation.  

This ensemble of tools can be used in interactive workshops in policy lab sessions and 
also  in  more  traditional  capacity-building  exercises.  To  this  end,  a  number  of  relevant 
tools  are  being  developed  and  will  be  integrated  into  the  prototype  of  the  i-FRAME  2.0 
computer-based  simulation  to  be  further  elaborated  as  part  of  an  interactive  and 
dynamic  warehouse  for  Evidence-Informed  Social  Policy  Innovation  within  the 
i-FRAME Web Platform.  

4 

 
 
 
These tools would include a computer-based problem tree, and an adapted version of an 
electronic toolkit for impact measurement previously developed by JRC, to also  support 
other methodologies and techniques, such as  ex ante failure scenarios with a simplified 
causal  model;  step-by-step  and  backward  theory-based  evaluation  thinking;  quick  exit 
tree  approach  part  of  the  Deliberation  and  Design  phase.  In  addition,  an  Interactive 
Support Tool to funnel users to the 'i-FRAME Simulator' should be developed as part of 
the i-FRAME Platform. Direct users will be able to take a flexible approach, according to 
their  degree  of  expertise  and  prior  knowledge,  to  alternative  or  complementary 
simulation modelling that could be useful to address specific problems or policy issues.  

The  i-FRAME  2.0  proposal  thus  lays  the  foundations  for  both  system-oriented 
formalised  modelling  simulations  and  for  theory-based  approaches.  It  also 
suggests  defining  a  blueprint  for  conceptual  modelling  and  the  further  development  of 
the  proposed  operational  components  in  line  with  the  vision  for  future  implementation, 
paving the way to building the i-FRAME 3.0 and beyond.  

In  this  vein,  it  is  our  opinion  that  the  proposed  i-FRAME  can  make  significant 
progress  towards  addressing  some  important  issues  in  the  area  of  supporting 
the  evaluation  of  Social  Policy  Innovation  which  promotes  social  investment. 
However  much  more  needs  to  be  done.  In  a  future  perspective,  the  i-FRAME  has  the 
potential for making a important contribution, especially in setting standards for the use 
of models for the evaluation of impacts of social policy initiatives.  

The  further  refinement  and  operationalisation  of  the  i-FRAME  should  build  on  the  rich 
knowledge  base  for  social  innovation  initiatives  that  has  been  developed  by  JRC.  A 
number  of  proof-of-concept  use  cases  should  be  defined  and  data  from  the  knowledge 
repository  could  be  used  for  calibrating  simulation  experiments  for  diverse  scenarios  of 
use.  Results  of  test  validation  will  be  then  discussed  with  policymakers  to  assess  their 
usefulness.  To  this  end,  connecting  to  other  initiatives  and  activities  using  a 
complex systems approach to support policy making and evaluation is crucial.  

To  achieve  these  objectives  and  carry  out  this  challenging,  but  very  much  needed  and 
timely  work,  an  ambitious  research  project  shall  be  envisaged,  both  to  reinforce 
JRC  internal  capacities  and  make  available  appropriate  resources.  This  should  be 
embedded  into  a  specific  high-level  science  for  policy  agenda.  Experts  and 
representatives  of  stakeholders  shall  be  closely  involved  on  an  ongoing  basis: 
researchers  from  relevant  scientific  communities,  and  practitioners  and  policymakers 
shall be called on directly to address concrete and specific complex policy challenges. 

The positive results of the preliminary application of the i-FRAME approach open the door 
implementation  of  the  proposed 
to  a  more  extensive  and  systematic 
methodology  at  policy  level.  This  could  involve  building  a  knowledge  repository  of 
simulation  models  based  on  a  portfolio  of  cases  analysed  as  part  of  i-FRAME 
development  and  further  enriched  with  new  application  examples  and  scenarios  of  use 
across the EU and in different policy domains.  

At the same time, it would require the development of a fully-fledged dynamic electronic 
toolkit  to  support  policymakers  in  modelling  and  simulating  in  real-time  policies  and 
programme  interventions  included  in  the  i-FRAME  knowledge  repository.  To  this  end, 
large-scale  computational  modelling  and  systems  simulation  tools  could  be  exploited. 
These are able to capture not only predictable human behaviour through linear top-down 
forecasting techniques, but also unplanned outcomes of complex interactions,  by taking 
advantage of data analytics and computer-based policy modelling.  This  would  lay the 
foundations for  what  could  prospectively  be  called  Data-powered  i-FRAME  4.0, 
which would include real-time structured data inputting from initiatives gathered through 
the i-FRAME Web-Platform to the i-FRAME social policy innovation simulator. This Data-
powered  i-FRAME  could  help  the  European  Commission  and  EU  Member  States  to 
monitor  the  implementation  of  actions  intended  to  revamp  the  'Social  Union',  and  thus 
shape a better future for Europe. 

5 

 
 
1  Introduction 

Summary of content of Chapter 1 

This chapter introduces the report as follows: 

§1.1 provides an overview of the policy context and research background. 

§1.2 presents the objectives of i-FRAME within the context of IESI. 

§1.3 outlines the specific aims and structure of this report. 

1.1  Background 

The notion that government is not the only provider of public services or the only actor 
that can deal with social problems and challenges is not new. The history of government 
is not a history of progressive interventionism, nor is it one of progressive retreat (e.g. 
privatisation, outsourcing, horizontal subsidiarity of part of welfare services passed on to 
NGOs and charities). Instead, it is an oscillation between these two extremes.  

The main legacy of the last century is the development of the welfare state, started after 
the  Second  World  War.  However,  according  to  the  literature,  since  the  1980s,  the 
western world - and in particular European economies - saw a resurgence of privatisation 
and  other  processes  of  state  hollowing  out.  In  many  domains,  the  importance  of 
horizontal  exchange  and  collaboration  has  been  acknowledged.  Tacit  knowledge 
exchanged  between  workers  in  communities  of  practice  is  a  factor  of  economic 
competitiveness (Nonaka and Hirotaka, 1995). Peer tutoring between students improves 
their  performance  (Merrell  and  Tymms,  2011).  Examples  of  shared  value  creation 
through  close  collaboration  amongst  private  and  public  sectors  and  the  citizens  show 
how  this  new  way  of  producing  goods  and  services  can  be  profitable  and  at  the  same 
time sustainable (Porter and Kramer, 2011). 

Equally,  dire  social  challenges  and  problems  that  must  be  faced  by  societies  and 
humanity as a whole are not new either. Certainly since the publication of the often-cited 
Meadows report on the state of humanity at the Club of Rome (Meadows  et al., 1972 – 
revisited  in  2004),  if  not  earlier,  the  limits  to  permanent  and  exponential  growth  in  a 
confined system, and the considerable role technological development has played in this 
context,  have  been  discussed.  Even  though  Meadows  takes  a  non-oppositional  stance 
towards technology, he suggested that the use of technological measures did not resolve 
the  world's  central  problems.  Instead,  they  tended  to  intensify  them.  He  believed  that 
unforeseeable social side effects and new social problems were generally associated with 
even very useful new technologies and that there were no technical answers to the most 
significant social problems in the modern world.  

This  prompted  a  discussion  regarding  the  need  for  a  different  way  of  living  and  a 
different  economy,  particularly  in  affluent  industrial  economies,  as  pointed  out  by  the 
Vienna  Declaration  (2011)1,  which  can  be  considered  a  sort  of  Manifesto  for  Innovation 
in  Social  Services.  It  aimed  to  identify  the  most  important  topics  in  social  innovation 
research which address societal challenges. These were perceived as even more pressing 
in  the  current  period  of  long-lasting  world-wide  economic  crises.  In  addition,  a  general 
rethinking  of  the  current  dynamics  between  the  market  and  the  State  has  been 
advocated  by  many  well-known  scholars.  For  instance,  Jacobs  and  Mazzucato,  (2016) 
make  the  case  for  rethinking  capitalism,  and  propose  a  new  way  of  conceiving 
'economics and policy for sustainable and inclusive development'.  

1   Declaration signed by key stakeholders in the field of social innovation during the 'International Conference 
on Indicators and Concepts of Innovation', 19-21 September, Vienna, organised by the EC-funded project 
NET4ALL society and available at http://www.socialinnovation2011.eu  

6 

 
 
                                           
This is in line with what was advocated by the  French economist Jean-Paul Fitoussi and 
the  Nobel  Laureates  Joseph  Stiglitz  and  Amartya  Sen  back  in  2009,  immediately  after 
the  financial  downturn.  They  argued  that  there  was  a  need  to  redress  what  engineers 
would call 'system failure' in order to fix social and economic inequalities.  

Recently, more attention has been paid to the need to rebalance the social and economic 
dimensions  of  our  societal  well-being.  This  notion  has  entered  the  broader  public  and 
political  debate  through  the  emergence  of  two  quasi  concepts:  social  investment  and 
social innovation. These are considered complementary and mutually reinforcing (see for 
instance Jenson, 2012).  

Whereas social investment captures the congeries of ideas about the objectives, areas 
of intervention and instruments,  social innovation, and ICT-Enabled Social Innovation 
in  particular,  'represents  the  enablers  and  drivers  for  social  change,  more  equal 
economic  development  and  possible  shared  prosperity'.  In  fact,  social investment  relies 
on  social  innovation  to  provide  solutions  that  produce  better  results  than  existing 
solutions  or  the  status  quo.  Social  innovations  can  improve  the  efficiency  of  social 
policies  and  their  effectiveness  in  addressing  societal  challenges  and  also  facilitate  life-
long investment in human capital.  

A more inclusive economy could reduce income and wealth inequalities and contribute to 
the common European effort to combat social exclusion, thus renewing citizens' trust in 
the  European  integration  project.  For  many  years,  the  European  Union  has  therefore 
been devising policies which promote – directly or indirectly - social innovation and social 
investment.  Many  research  projects  which  address  social innovation  and  social  services 
reform  have  been  funded  under  the  FP7  or  H2020  programmes.  An  example  worth 
mentioning  is  the  European  Platform  against  Poverty  and  Social  Exclusion  (EPAPSE) 
which aims to design and implement initiatives to promote social innovation for the most 
vulnerable in society.2 Another is the  'Innovation Union' flagship policy, which sets new 
conditions  to  improve  access  to  finance  for  social  enterprises3.  It  includes  the  pilot 
project  'Social  Innovation  Europe',  which  has  established  a  virtual  hub  for  social 
entrepreneurs,  public  and  third  sector  organisations4  to  be  further  developed  under  the 
growing  Social  Economy  policy  support  agenda5.  Other  important  policy  initiatives  that 
focus on social innovation were included in the legislative package on cohesion policy for 
2014-2020. For example, the European Social Fund (ESF) supports initiatives that aim to 
scale  up  and  build  capacity  for  social  innovation.  Another  example  is  the  European 
Regional  Development  Fund  (ERDF)  which  supports  innovation  in  sustainable  urban 
development.  More  specifically,  the  Employment  and  Social  Innovation  programme 
(EaSI) is a EU financing instrument that promotes high quality sustainable employment, 
guarantees adequate and decent social protection, combats social exclusion and poverty 
and improves working conditions.6 

In this context, in February 2013, the European Union adopted the  Social Investment 
Package (SIP)7 to contribute to the economic growth of Europe, to protect people from 
poverty  and  to  act  as  an  economic  stabiliser  from  inequalities.  It  stresses  that  welfare 
systems have to fulfil three key complementary functions: 1) social investment; 2) social 
protection and 3) stabilisation of the economy. The social investment approach strongly 
relies  on  the  assumption  that  social  and  economic  policies  are  mutually  reinforcing. 

2   http://ec.europa.eu/social/main.jsp?catId=961 
3   http://ec.europa.eu/research/innovation-union/index_en.cfm 
4   https://webgate.ec.europa.eu/socialinnovationeurope/it 
5   See  for  instance  the  Bratislava  Declaration  on  the  Social  Economy  as  a  key  player  in  providing  effective 
answers  to  current  societal  challenges  in  the  EU  and  the  world  adopted  in  Bratislava  on  1st  December 
2016, https://coopseurope.coop/sites/default/files/Bratislava%20declaration.pdf  

6   See http://ec.europa.eu/social/main.jsp?catId=1081  
7   Communication  from  the  Commission  to  the  European  Parliament,  the  Council,  the  European  and  Social 
Committee  and  the  Committee  of  the  Regions:  Towards  Social  Investment  for  Growth  and  Cohesion  – 
including 
final.  See: 
http://ec.europa.eu/social/main.jsp?catId=1044 

the  European  Social  Fund  2014-2020.  COM(2013)  83 

implementing 

7 

 
                                           
Social policies that form part of a social investment perspective are considered 
a precondition for future economic and employment growth.  

Social  investment,  as  outlined  in  the  SIP,  is  thus  the  set  of  policy  measures  and 
instruments  that  consist  of  investment  in  human  capital  and  enhancement  of  people's 
capacity  to  participate  in  social  and  economic  life  and  in  labour  market.  Social 
investment  involves  strengthening  people's  current  and  future  capacities.  In  other 
words,  social  investment  policies  have  both  immediate  and  long-term  effects,  because 
they  offer  economic  and  social  returns  over  time,  notably  in  terms  of  employment 
prospects or labour market incomes.  

The  SIP  has  proposed  a  radical  change  in  the  approach  to  social  services  design  and 
delivery:  a  citizen  centric  perspective  and  public  services  transformation  and 
modernisation  are  the  key  interrelated  elements  of  the  new  and  more  sustainable 
welfare systems in Europe. This is clearly evident in the way the SIP has been conceived. 
On the one hand, policy interventions should be designed with a life-course perspective 
(i.e. they should represent a continuum of measures that accompany people through the 
key  stages  of  their  lives:  childhood,  working-age,  parenthood,  and  old  age8).  On  the 
other  hand,  measures  related  to  the  various  policy  areas  should  be  contemporary  and 
mutually reinforcing. In other words, the development of institutional complementarities 
is  a  necessary  condition  for  the  implementation  of  successful  social  investment 
strategies. In particular, the availability of quality enabling social services has a key role 
to play in ensuring the integration of policy measures.9 Moreover, changing the focus to 
address  the  needs  of  individuals  throughout  their  lives  also  implies  that  innovative 
approaches  to  social  services  design  and  delivery  are  expected  to  contribute  to  the 
modernisation of European welfare systems.  

The SIP Communication urges the Member States to prioritise social investment and the 
modernisation of their welfare systems. They must address unemployment, poverty and 
social  exclusion  challenges  brought  about  by  the  economic  crisis  and  sustainability 
challenges  posed  by  the  ageing  population  trends.  The  SIP  emphasises  that  social 
innovation's  potential  is  further  increased  by  the  growing  range  of  available  innovative 
solutions based on Information and Communication Technologies (ICTs). It is clear that 
ICT-Enabled Social Innovation plays an important role in promoting social investment 
policies as ICTs help to fully digitalise social services processes, to reduce social services 
fragmentation  and  duplication  across  organisations  and  countries,  and  to  contribute  to 
making  the  services  more  proactive  and  closer  to  the  point  of  need.  It  is  also  an 
opportunity  to  directly  engage  citizens  in  the  whole  social  services  process  design  and 
management.  Moreover,  it  can  activate  continuous  improvement  processes  to  make 
social protection systems more inclusive and self-sustainable in the long term.  

There  is  a  need  to  better  understand  how  ICT-enabled  social  innovation  can  support 
social investment policies and provide recommendations to assess the implementation of 
the  EU  Social  Investment  Package  (SIP).  The  European  Commission's  Joint  Research 
Centre and DG Employment, Social Affairs and Inclusion (DG EMPL) have thus launched 
a multi-year research project entitled 'ICT-enabled social innovation to support the 
implementation of the Social Investment Package' (in short IESI).10 

8   Jon  Kvist  argues  that  to  take  full  advantage  of  social  investments,  the  SIP  needs  a  more  coherent 
framework  that  takes  into  account  the  dynamic  and  multidimensional  nature  of  social  issues  and  social 
investments.  He  suggests  that  such  a  framework  should  consist  of  generational,  life  course  and  gender 
perspectives  on  social  investments.  See  Kvist,  J.,  'A  framework  for  social  investment  strategies: 
Integrating  generational,  life  course  and  gender  perspectives  in  the  EU  social  investment  strategy', 
Comparative European Politics, Vol. 13, No. 1, pp. 131-149, 10.1057/cep.2014.45. 

9   It  is  important  to  note  that:  the  central  role  of  the  individual,  its  life  course  approach  and  the 
contemporary use of various policy measures are central element of the SIP and therefore they are crucial 
in  the  development  of  the  i-FRAME  which  entails  an  approach  encompassing  a  micro-meso-macro 
perspective.  

10   For a more detailed presentation of the IESI conceptual and analytical framework developed as part of the 
IESI  research  see:  Misuraca  G.,  et  al..  (2015),  JRC  Science  for  Policy  Report,  and  other  deliverables 
accessible through the JRC Science Hub's IESI Project's webpage https://ec.europa.eu/jrc/en/iesi  

8 

 
 
                                           
1.2  Objectives 

The objectives of the IESI research are to: 

•  provide  a  deeper  understanding  of  how  EU  Member  States  can  make  better  use  of 

ICT-Enabled Social Innovation to implement the actions suggested in the SIP;  

•  develop  a  proposal  to  build  a  methodological  framework  of  analysis  of  the  impacts 
generated  by  ICT-Enabled  Social  Innovation  initiatives  which  promote  social 
investment;  

•  contribute  to  building  evidence-based  input  to  social  policy  innovation,  providing 
results  of  a  structured  analysis  of  initiatives  and  sharing  successful  experiences 
implemented in EU Member States. 

The starting point for the analysis was to address innovative delivery of Personal Social 
Services  of  General  Interest  (PSSGI)  (i.e.  the  services  that  respond  to  vital  human 
needs,  contributing  to  non-discrimination  and  creating  equal  opportunities).11  The 
intended  aim  was  to  better  understand  how  ICT-enabled  social  innovation  initiatives 
which  promote  social  investment  can  contribute  to:  simplifying  administrations; 
integrating services; improving  the management, provision and coordination of  delivery 
mechanisms;  designing  high-quality  and  cost-effective  services  that  meet  the  needs  of 
citizens;  and  supporting  access  to  and  take-up  of  social  services.  In  other  words,  can 
ICT-enabled social innovation initiatives achieve a systemic effect and ensure that social 
policy and/or service delivery and implementation have a sustainable impact?  

One  of  the  key  components  of  the  IESI  research  was  to  develop  a  proposal  for 
building  a  methodological  framework  to  assess  the  impacts  of  ICT-Enabled 
Social Innovation initiatives promoting social investment in the EU. 

This methodological framework, in short the i-FRAME12, aims to: 

•  Provide  a  structured  approach  to  analyse  the  initiatives  collected  through  mapping 
ICT-enabled social policy innovation in the EU. This approach aims to act as a guide 
to  in-depth  analysis  of  case  studies,  and  provide  insights  into  the  replicability  and 
transferability of initiatives which promote social investment across the EU. 

•  Serve  as  a  comprehensive  framework  for  conducting  analysis  of  the  economic  and 
social  returns  on  investment  in  which  ICT-enabled  social  innovation  is  a  key 
component. This shall allow the JRC to make recommendations on how the European 
Commission  and  Member  States  could  analyse  (ex-ante,  in-itinere  and  ex-post)  the 
impact of ICT-enabled social innovation initiatives which promote social investment.  

The conceptual framework and methodological approach underpinning the i-FRAME were 
outlined by the JRC in February 2015 and then further described in the original proposal 
for  i-FRAME  (V1.0).  An  initial  proposal  of  how  to  build  the  operational  components  of 
the framework was formalised in June 2015, and was then further elaborated and tested 
with  support  from  external  experts,  by  applying  it  to  selected  scenarios  of  use  drawn 
from case studies carried out across the EU in 2015. This allowed the JRC to develop a 
preliminary  proposal  for  a  theoretical  and  methodological  approach  to  build  the 
operational  components  of  the  i-FRAME  (V1.5)  and  a  revised  and  improved  proposal 
for  building  the  i-FRAME  (V2.0)  according  to  a  structured  theoretical  framework  of  a 
simulation model for social impact assessment.  

11   Since the first phases of the research, these have been classified according to the following 10 analytical 
types: (1) Childcare; (2) Education and training; (3) Social assistance; (4) Social care; (5) Social housing; 
(6)  Employability;  (7)  Employment;  (8)  Social  inclusion/participation;  (9)  Civic  engagement;  (10)  Active 
and  healthy  ageing  and  long-term  care  (For  the  scope  of  this  research,  this  area  is  further  divided  into 
three sub-themes according to the main EC policy objectives, namely: a) prevention, health promotion and 
rehabilitation; b) integrated care; and c) independent living). 

12   The  acronym  i-FRAME  has  been  suggested  to  stand  for  Impact  Framework  for  Real  and  Meaningful 
Evaluation at the 2nd IESI Experts & Stakeholders Consultation Workshop, Brussels, 24-25 February 2015. 

9 

 
                                           
1.3  Outline 

This  report  presents  the  final  proposal  for  developing  the  i-FRAME.  It  outlines  a 
reviewed and improved theoretical and methodological approach, which aims to develop 
a  comprehensive  framework  for  evaluating  the  social  and  economic  impacts  of  ICT-
enabled social innovation initiatives which promote social investment in the EU. 

The  final  proposal  for  building  the  i-FRAME  was  elaborated  by  the  JRC  with  help  from 
external experts.  It has been validated by testing the methodological approach and the 
operational components proposed on a number of case studies and scenarios of use. In 
addition,  experts  drawn 
from  different  research  disciplines,  practitioners  and 
representatives of relevant stakeholders and policymakers were consulted. 

This report addresses the following research objectives:  

•  To  review 

the  proposed 

theoretical  and  methodological  approach 
underpinning  the  i-FRAME,  so  that  the  social  and  economic  impacts  of  ICT-enabled 
social  innovation  initiatives  promoting  social  investment  can  be  evaluated  (ex-ante, 
in-itinere and ex-post); 

•  To  design  the  structure  of  the  operational  components  of  the  i-FRAME  as  a 
computer based simulation model. This model should encompass different levels 
of  analysis  by  using  the  same  structural  environment,  to  facilitate  the  collection  of 
data  through  the  definition  of  measurable,  relevant  and  coherent  indicators.  It  shall 
also  take  into  account  possible  counter-intuitive  behaviour  and  allow  a  flexible 
approach  to  re-calibrating  the  model  as  a  consequence  of  ex-post  analysis  or 
changes  in  the  theoretical  assumptions/causal  relationships  and/or  dynamics 
hypothesis underlying the framework and its operational components; 

•  To  provide  recommendations  on  how  to  further  develop  the  i-FRAME  as  a 
'social policy innovation simulator'. Scenarios of use and case studies have been used 
to  test  and  validate  the  relationships  established  and  the  dynamic  hypothesis 
underlying  the  theoretical  and  methodological  approach  to  building  some  of  the 
operational components of the i-FRAME that are proposed for future development. 

The remainder of this report is structured as follows: 

•  Chapter 2 presents the methodological approach followed to develop the i-FRAME. 

•  Chapter  3  discusses  the  rationale  which  underpins  the  i-FRAME  as  regards  social 
impact  assessment.  It  discusses  findings  from  the  review  of  the  state  of  play  and 
outlines the initial proposal of meta-framework - i-FRAME (V1.0).  

•  Chapter  4  presents  a  structured  proposal  for  a  theoretical  and  methodological 
approach to build the operational components of the i-FRAME (V1.5). To this end, it 
discusses  the  potential of  dynamic  simulation modelling  to  support  innovative  social 
policy. 

•  Chapter 5 outlines the final proposal for developing a methodological framework for 
assessing the social and economic impact of social policy innovations which promote 
social  investment,  i-FRAME  (V2.0).  This  includes  an  overview  of  the  operational 
components of the i-FRAME, and anticipates some of the key elements that should be 
developed  to  build  a  comprehensive  i-FRAME  Web-Platform  for  social  impact 
assessment. 

•  Chapter  6  draws  some  conclusions  and  explains  the  implications  these  have  for 
policy. Directions for future research are also provided suggesting recommendations 
for  further  developing  the  operational  components  of  i-FRAME  2.0  and  paving  the 
way towards building i-FRAME (V3.0) and beyond.  

•  The  Technical  Annexes  contain  details  of  the  findings  from  the  systematic  review 
conducted to further validate the previous development phases of the i-FRAME and to 
lay the foundations for the final proposal for the development of the i-FRAME. 

10 

 
 
2  Methodology 

Summary of content of Chapter 2 

This chapter summarises the methodology followed for the development and validation 
of the final proposal for the development of i-FRAME (V2.0). It is organised as follows: 

§2.1 outlines the overall approach adopted for co-designing the i-FRAME through peer 
review and consultation with experts. 

§2.2  provides  an  overview  of  the  methodology  used  for  the  literature  reviews  and 
analysis of the state of the art. 

§2.3 presents the activities conducted for testing the operational components of the i-
FRAME and for validating the theoretical and methodological approach proposed. 

2.1  Co-designing the i-FRAME 

The methodological approach followed for developing the proposal for i-FRAME consisted 
of  a  number  of  sequential  activities.  These  unfolded  during  the  implementation  of  the 
IESI project and ran in parallel with the other components  of the  research (namely the 
Systematic  mapping  and  analysis  of  ICT-enabled  social innovation  initiatives  in  the  EU; 
and  the  Thematic  analysis/case  studies:  focusing  on  the  role  of  ICT-enabled  social 
innovation promoting social investment to support the modernisation of social protection 
systems  in  the  EU).  Each  activity  received  input  from  the  other  components  of  the 
research,  which  also  provided  specific  contributions 
for  the  development  and 
implementation of the proposal for the i-FRAME. 

Figure 3: Methodological approach for developing and validating the i-FRAME 

Source: own elaboration 

11 

 
 
 
 
As  illustrated  in  Figure  3,  the  methodological  approach  adopted  is  iterative  and  based 
upon:  

•  Desk research to review the state of the art on social impact assessment in the policy 

areas under investigation and focusing on methodological approaches adopted;  

•  Conceptual work to outline and structure the proposed i-FRAME meta-framework and 

related operational components; and  

•  Consultation  with  experts  from  different  research,  practice  and  policy  communities, 
including  representatives  of  key  relevant  stakeholders  and  policymakers  at  local, 
regional, national and international levels.  

A  first  review  of  the  state  of  the  art  and  preliminary  conceptualisation  work  was 
conducted  in  2014  and  allowed  JRC  to  define  an  initial  outline  of  the  methodological 
framework and to propose an approach for building the operational components. 

Based  on  feedback  received  from  expert  consultations  and  integration  of  the  results  of 
the  analysis  of  the  review  of  the  state  of  the  art,  the  initial  proposal  for  the  i-FRAME 
(V1.0) was developed and presented in June 2015.  

From  July  2015  to  January  2016,  with  support  from  a  team  of  external  experts  JRC 
further developed the initial proposal for the development of i-FRAME (V1.0) following a 
structured  theoretical  framework  of  a  simulation  model  for  social  impact  assessment. 
The resulting revised proposal for i-FRAME (V1.5) was then finalised in February 2016 
and further elaborated after testing on a number of case studies and scenarios of use. 

for  a 

light  the  need 

The  last  phase  of  the  research  built  on  additional  desk  research  and  consultation  with 
experts, as well as insights from  external independent reviewers.  This exercise brought 
to 
further  analysis  of  the  theoretical  orientations  and 
conceptualisations on which the initial versions of the i-FRAME are based upon. In other 
words, the complexity which underpins evidence-based social policy evaluation needs to 
be  better  addressed  in  order  to  understand  what  dynamic  simulation  modelling  could 
offer and the 'game-changing' role ICTs could play. Further analysis was also required in 
order  to  outline  the  proposed  i-FRAME  as  a  meta-framework  for  impact  assessment  of 
social  policy  innovation  and  to  advance  recommendations  for  practical  use  and  further 
development of the i-FRAME beyond the current proposal. 

The  final  proposal  for  the  design  of  the  i-FRAME(V2.0)  presented  in  this  report, 
thus  outlines  a  comprehensive  approach  for  defining  a  common  methodological 
framework  with  recommendations  for  its  further  development  as  a  computer-based 
simulation model.  

The  proposed  framework  is  structured  according  to  a  series  of  operational  components 
that  shall  be  further  defined  according  to  different  typologies  of  initiatives  and 
stakeholders  involved.  It  should  be  stressed,  in  fact,  that  while  the  logic  model 
underpinning the i-FRAME is necessarily generic in order to address the broad spectrum 
of social policy initiatives, the operational components of the i-FRAME are structured so 
as to address the potential impact of social policy innovations in a specific policy area.  

The development of the initial proposal for i-FRAME (V1.0) proposed in 2015 was based 
on a categorisation of the various typologies of initiatives labelled as ICT-enabled social 
innovation  promoting  social  investment.  The  related  impact  dimensions  and  variables 
considered  in  modelling  the  i-FRAME  were  further  defined  in  the  following  phase  of  the 
research,  and  served  to  advance  the  structured  methodological  framework  of  the  i-
FRAME (V1.5).  

These impact dimensions and variables served as a guide for designing the final proposal 
for developing the i-FRAME (V2.0) and for perfecting the operational components of the 
modelling  and  simulation  approach  to  be  developed  for  assessing  the  effects  of 
initiatives.  Thus  a  comprehensive  picture  can  be  provided  of  the  potential  impacts  that 
broader  social  policy  innovations  may  generate.  This  would  lay  the  foundations  for 
further developing specific indicators, methods and tools to measure and evaluate them.  

12 

 
2.2  State of the art and literature review 

In  order  to  conceptualise  the  overall  proposal  for  i-FRAME  as  a  'meta-framework'  for 
assessing  the  economic  and  social  impact  of  social  policy  innovations  which  promote 
social investment, and to define and further develop the operational components  of the 
i-FRAME  simulation  model,  an  extensive  analysis  of  the  state  of  the  art  based  on 
sequential specific reviews of the literature was considered crucial.  

These  reviews  aimed  to  provide  a  comprehensive  picture  of  the  various  domains  to 
which social investment policies might apply. They also appraised the methodologies that 
could  be  used  to  evaluate  the  impacts  of  these  interventions,  taking  into  account  the 
specificities of different approaches to social policy innovation.  

In 2014 – 2015, given the exploratory nature of the research, the review of the state of 
the  art  focused  on  making  sense  of  the  broad  debate  on  measurement  and 
evaluation  of  social  impact.  Though  this  topic  is  not  new,  it  has  gained  increasing 
importance  recently  under 
the  umbrella  of  social  performance  and  outcome 
measurement generated by non-profit organisations and social ventures. This trend has 
been  discussed  with  growing  intensity  in  evaluation  research  (e.g.  Stufflebeam  and 
Shinkfield,  2007)  as  many  stakeholders  in  the  non-profit  sector  have  begun  to 
increasingly shift their operations towards a market and profit-oriented way of thinking. 
Social entrepreneurs and venture philanthropists, in some cases backed by funding from 
foundations,  have  started  applying  well  or  less  well-known  concepts  of  outcome  and 
impact analysis and measurement of their own accord.  

At the same time, the research community started to pay attention to currently popular 
concepts  of  social  impact  measurement  and,  in  particular,  to  Social  Return  on 
Investment  (SROI)  analysis  as  an  alternative  to  mainstream  concepts  of  (economic) 
evaluation that have been known and applied for a long time. In this respect, pragmatic 
evaluation  approaches  became  popular,  such  as  for  instance  the  logic  model  or  the 
impact  chain,  also  known  as  'theory  of  change'.  These  approaches  are  playing  an 
increasingly important role in evaluation practice. At the same time,  policymakers have 
started  considering  social  impact  assessment  techniques  weighting  advantages  vs. 
disadvantages, alongside more traditional cost benefit analyses of economic evaluation.  

Thus,  the  preliminary  review  of  the  state  of  play,  conducted  as  the  basis  for  the  initial 
proposal  for  the  development  of  i-FRAME  1.0,  generally  addressed  the  topic  of  impact 
measurement  in  two  ways.  Firstly,  it  analysed  broader  approaches for  social  impact 
assessment (e.g. Maas and Liket, 2011; Epstein and Yuthas, 2014; Mildenberger et al. 
2012;  Rauscher,  2012).  Secondly,  it  also  looked  at  various  methods  for  outcome 
measurement  and  impact  evaluation.  These  have  a  significantly  longer  tradition  in 
evaluation  research  (see  Rossi  et  al.  1994,  2004;  Stockmann  et  al.  2006  and  for  a 
focused policy and practice-oriented review, Misuraca et al, 2014). 

The  orientations  set  out  in  the  original  proposal  for  i-FRAME  1.0  identified  that  'when 
complexity  is  one  of  the  dominant  aspects  of  a  phenomenon  (or  a  series  of  interlinked 
phenomena),  the  most  appropriate  way  to  analyse  it  as  a  complex  system  is  by  using 
modelling at first and then possibly simulation in order to understand its behaviour and 
attempt predicting its evolution'.  

Then,  in  2015  –  2016,  as  part  of  the  research  activities  to  develop  the  i-FRAME  1.5,  a 
further review of literature was conducted. This review had two objectives. First, it aimed 
to  compare  different  methodological  approaches  that  could  be  applied  to  the  modelling 
and  simulation  of  complex  systems,  particularly  in  the  field  of  ICT-enabled  social 
innovation promoting social investment. Second, it aimed to review existing applications 
of  those  approaches  identified  as  most  suitable,  either  on  their  own  or  in  combination. 
This literature  review  identified  examples  of  applications  of  modelling and  simulation  to 
assess  the  impacts  of  ICT-enabled  social  innovation  initiatives  and  to  investigate  their 
key characteristics, in order to understand how to use them to shape the i-FRAME. 

13 

 
The  literature  review  addressing  the  first  objective  consisted  of  a  cross-analysis  of 
modelling  and  simulation  capabilities  of  the  following  modelling  methods:  General 
Equilibrium;  System  Dynamics;  Markov;  Agent  Based;  Discrete  Event.  These  represent 
the  most  common  methodological  approaches  to  modelling  and  simulation  applied 
worldwide. In order to conduct this cross-comparison, the search concentrated on recent 
papers  which  discussed  the  advantages  and  disadvantages  of  the  different  methods  of 
modelling and simulation listed above. The results of this comparison provided evidence 
of  how  Dynamic  Simulation,  and  in  particular  Systems  Dynamics  (SD),  Agent-Based 
Modelling  Simulation  (ABMS)  and  their  combination  through  an  Dynamic  Simulation 
Hybrid  Model  (DS-HM)  seem  to  be  the  most  suitable  approaches  to  the  problem 
addressed by social policy innovations. These modelling techniques are characterised by 
nonlinearity  and  retroactions  that  cannot  be  easily  represented  by  other  classes  of 
modelling and simulation methods.13 

The  review  addressing  the  second  objective  –  i.e.  of  applications  of  modelling  and 
simulation  approaches  -  was  based  on  an  extensive  and  systematic  search.  This 
search  aimed  to  identify  clear  examples  of  applications  of  SD  and  ABMS  Modelling  and 
simulation  which  assess  the  impacts  of  ICT-enabled  social  innovation  initiatives 
promoting social investment. The search focused specifically on Personal Social Services 
of  General  Interest  (PSSGI).  The  research  team  initially  identified  953  papers  that 
potentially  addressed  the  use  of  dynamic  simulation  in  the  fields  of  analysis.  A  further 
examination of the abstracts of these papers in order to select only those which explicitly 
contained  a  description  of  dynamic  simulation  models  in  relevant  fields,  led  to  the 
selection of a subset of 137 papers. A full reading of their contents further reduced the 
number of papers to 65. 

Modelling  ecosystems  as  complex  as  those  of  social  policy  innovation  applied  to  PSSGI 
require  a  close  connection  between  data  and  model  (Ahrweiler,  2017;  Ahrweiler  et  al. 
2015).  Data  availability  for  calibrating  a  model  of  the  social  innovation  ecosystem  as 
'target' is crucial for the relevance and utility of any simulation results. This is, obviously, 
one of the limitations that must be flagged up in the development from i-FRAME 1.0 to i-
FRAME 2.0. The time and resources available did not allow the research team to map all 
possible  data  against  all  possible  models  to  undertake  this  calibration  work.  This  very 
demanding and time consuming work will, however, have to be conducted in the coming 
years,  if  the  i-FRAME  is  to  be  implemented  and  realise  its  promise  as  a  practical 
instrument for policymakers. 

In 2016, a supplementary systematic literature review was conducted in order to 
further  review  the  theoretical  foundations  which  underpin  the  design  for  the  proposed 
methodological  framework  (i-FRAME  2.0).  Thus,  the  design  and  set  of  operational 
components  advanced  was  improved,  and  the  area  under  analysis  was  broadened  (see 
Chapter  5).  The  review  screened  the  existing  literature  in  the  field  of  social  impact 
assessment and social policy innovations which promote social investment, by combining 
impact  evaluations  and  assessments  of  social  innovation  in  six  domains:  Employment 
and  training,  Child  care,  Long-term  care,  Social  Inclusion,  Social  Housing,  Social 
Assistance14,15.  

13   A summary with details on the results of the comparison of modelling approaches is reported in Annex I. 
14   Building  on  the  approach  advanced  as  part  of  the  IESI  research  the  review  considered  what  constitute 
Social  Services of General  Interest  (SSGI)  according  to  the  first  Communication  on  this topic  released  in 
2006  by  the  European  Commission  which  identified  two  broad  types  of  services:  (1)  Statutory  and 
complementary social security schemes covering the main risks of life; and (2) Services provided directly 
to  the  person,  such  as  social  assistance  services,  employment  and  training  services,  childcare,  social 
housing  or  long-term  care  for  the  elderly  and  for  people  with  disabilities.  The  sub-category  (2)  can  be 
considered as broadly corresponding to the concept of PSSGI and the mentioned communication included 
five  main  areas:  employment  and  training  services;  Social  housing;  Child  care,  Long-term  care,  social 
assistance  services.  To  these  five,  Social  inclusion  services  was  added  given  their  relevance  for  the  i-
FRAME.  These  6  categories  comprehend  the  analytical  types  defined  in  the  IESI  research  scope  (see 
Misuraca et al., 2015) and have been used to test the i-FRAME on broader social services areas of impact. 

15   Details on methods and sources of the systematic review are provided in the Annexes III, V and VI. 

14 

 
                                           
This supplementary analysis had two objectives. One was to further confirm and validate 
choices  made  in  the  design  of  the  i-FRAME,  following  the  findings  of  the  previous 
literature reviews and analysis of the state of the art. The other was to expand the scope 
of  the  review  to  include  the  broader  concept  of  social  policy  innovation  promoting 
social  investment.  ICTs  are,  of  course,  an  important  component  of  this  kind  of 
innovation,  but  the  research  team  did  not  want  to  limit  the  analysis  of  innovation  in 
social policy and social services to  only those involving digital technology. This decision 
was made as a consequence of the findings from previous reviews and consultation with 
experts  and  representatives  of  stakeholders.  In  addition,  widening  the  scope  of  the 
possible  analysis  provided  the  opportunity  to  better  define  recommendations  to  go 
beyond the exploratory phase of the IESI research and to extend it to other policy fields. 

The  analysis  considered  existing  reviews  or  meta-analyses  and  adopted  a  structured 
approach  to  finding  and  choosing  the  relevant  articles.  This  selection  could  not  be  too 
broad, as we had to have a manageable number of papers, nor could it be too restricted, 
if  all  the  existing  relevant  evidence  was  to  be  included.  The  selection  of  the  relevant 
articles followed three steps:  

(i) Setting the scope, the search string, and the inclusion criteria;  

(ii) Creating the database, identifying studies; and  

(iii) Selecting the studies to be included in the review.  

Figure 4 summarises the approach adopted for the systematic review.  

Figure 4: Methodological framework for systematic review 

Source: own elaboration 

This  supplementary  search  was  conducted  in  November  2016  in  four  electronic 
databases:  IDEAS  –  RePEc  (Research  Papers  in  Economics),  SSRN  (Social  Science 
Research Network), ISI Web of Science, and Scopus. After in-depth screening, the final 
selection consisted of 219 results, made up of 155 meta-analyses and 64 evaluations. In 
parallel to the analysis of the scientific sources search, an extensive desk review of grey 
literature  was  carried  out,  to  gather  sources  complementary  to  those  identified  in  the 
systematic  review.  These  sources  included  EU  policy  and  guidance  documents,  funding 
programmes, individual projects and one-off initiatives. At policy level, the starting point 
was  the  Social  Investment  Package  (SIP)  which,  importantly,  underlines  the  added 
value in focusing on innovative social policies and on embedding innovation in evidence-
based  policy-making.  In  addition,  the  seven  Europe  2020  flagship  initiatives  were 
also reviewed to assess their contribution to and consideration of social innovation within 
the broader EU policy framework (Table 1). 

15 

 
 
 
 
Table 1: Social innovation in the Europe 2020 strategy 

Flagship 
Initiative 

Social Innovation uptake 

A resource-
efficient Europe 

Energy Efficiency Directive 
Eco-design Directive 
Citizen Energy Forum's Vulnerable Consumers Working Group 
Resource efficiency indicators 

An industrial 
policy for the 
globalisation era 

Digital Agenda 
for Europe 

Innovation 
Union 

Single Market Act I 
Single Market Act II 
Social Business Initiative 
COSME 
Workplace innovation 

Digital skills and jobs 
Regulatory environment, protection of property 
Digital infrastructure 

Social Innovation Europe virtual hub 
Innovation in social policy (ESF, PROGRESS) 
Dedicated FP7/H2020 calls 
European Public Sector Innovation Scoreboard 
European Innovation Partnerships 

Youth on the 
Move 

Youth participation in policy design 
Workplace and entrepreneurial experience 
University modernization 

The agenda for 
new skills and 
jobs 

Labour market reforms in the social economy 
Job quality and skills development 
Measures to improve education and training systems 

The European 
Platform against 
Poverty and 
Social Exclusion 

Source: own elaboration 

Developing an evidence-based approach to social innovations and 
reforms 
Promoting a partnership approach to the social economy 

At  the  programme  and  project  level,  the  review  focused  on  the  EaSI  programme,  its 
predecessor  PROGRESS,  on  Regional  Policy,  and  on  the  uptake  of  social  innovation  by 
the  European  Structural  and  Investment  Funds  (ESIF).  It  also  looked  at  the  many  EU-
funded  research  projects,  such  as  FP7,  that  more  or  less  directly  deal  with  social 
innovation.  In  addition,  EU-backed  one-off  initiatives  for  policymakers  and  social 
innovators were also reviewed. See Table 2, and details in Annex. 

In  addition,  an  analysis  of  scientific  and  grey  sources  was  performed  to  help 
contextualise the refinement and improvement of the proposed operational components 
of  the  i-FRAME.  This  looked  first  at  scientific  literature  that  discusses  descriptively  or 
prescriptively  past  and  current  reforms  which  aimed  to  establish  'One  stop  shop' 
approaches  (i.e.  integrated,  joined  up,  etc.)  to  the  delivery  of  social  services  (Angers, 
2011;  Askim  et  al.,  2011;  Budapest  Institute,  2014a,  2014b;  Champion  and  Bonoli, 
2011;  Christensen  and  Lægreid,  2007;  Clasen  and  Clegg,  2011;  Clasen  et  al.,  2001; 
Ditch  and  Roberts,  2002;  Lindsay  et  al.,  2008;  McQuaid,  2010;  McQuaid  et  al.,  2007; 
Minas, 2014; Mosley, 2011; Munday, 2007; OECD, 2003; Øverbye et al., 2010; Scharle 
et al., 2014; Struyven, 2004; van Berkel et al., 2012; Wiggan, 2007).  

16 

 
One  stop  shops  are  clearly  relevant  to  the  discussion  of  service  integration,  which  has 
been  a  specific  dimension  of  the  IESI  research  since  its  inception  (see  Misuraca  et  al., 
2015). This review confirmed that there is little evaluative evidence on the effectiveness 
of  this  approach.  It  did  however  highlight  the  administrative,  organisational,  and 
governance complexity of integrated social service delivery.  

The  grey  literature  sources  address  precisely  the  topic  of  social  services  complexity. 
To some extent, these sources express a critique of the indiscriminate and deterministic 
application  of  Evidence-Based  Policy  prescriptions  and  hierarchies  of  evidence  to  social 
policies and social service delivery.  

This  literature  was  selected  from  the  growing  body  of  academic  critical  appraisals 
(Cartwright & Hardie, 2012; McMillin, 2012; Munro, 2014; Pawson, 2002; Pawson, 2006; 
Petticrew  &  Roberts,  2003;  Sanderson,  2011;  Strassheim  &  Kettunen,  2014;  Torriti, 
2010), also looking at official EBP prescriptions and evaluations (Allen, 2011; Balshem et 
al.,  2011;  Bohrnstedt  &  Stecher,  2002;  Cabinet  Office,  1999;  Davies  et  al.,  2000;  EFF, 
2005; GLA, 2010; Haynes et al., 2012; Munro, 2011;UKDPC, 2008; UNICEF, 2008). 

Table 2: Social Innovation in EU initiatives 

EU initiative 

Social Innovation uptake 

EASI 
programme 

Design,  evaluation  and  larger-scale  implementation  of  new  social 
policy initiatives in line with the SIP approach 
Exploring  the  role  of  PPPs  in  welfare  reforms  and  investment  in 
human capital 
Award schemes for social entrepreneurs 
Social policy experimentation in support of SIP actions: housing first, 
one-stop-shops, and work-in-stations 

Research 
Projects 
funded  by  EC 
Framework 
Programmes 
(FP7, H2020) 

the 

(Enabling 

flourishing  and  evolution  of  social 

CRESSI (Creating Economic Space for Social Innovation) 
EFESEIIS 
entrepreneurship for innovative and inclusive societies) 
ImPRovE (Poverty Reduction in Europe: Social policy and innovation) 
ITSSOIN (Social Innovation and Civic Engagement) 
SEFORIS  (Social  Enterprise  as  Force  for  more  Inclusive  and 
Innovative Societies) 
SI-DRIVE (Social Innovation: Driving Force of Social Change) 
SIMPACT (Boosting the Impact of Social Innovation in Europe through 
Economic Underpinnings) 
SocIEtY (Social Innovation - Empowering the Young for the Common 
Good) 
TEPSIE  (Theoretical,  Empirical  and  Policy  Foundations  for  Social 
Innovation in Europe) 
Third Sector Impact (The Contribution of the Third Sector to Europe's 
Socio-economic Development) 
TRANSIT (Transformative Social Innovation Theory) 

European 
Structural  and 
Investment 
Funds (ESIF) 

European Social Fund (ESF) 
European  Regional  Development  Fund  (ERDF),  in  particular  actions 
for the development of ICTs and social entrepreneurship 
European Agricultural Fund for Rural Development (EAFRD)  

Other 
initiatives 

COSME programme 
Erasmus for Young Entrepreneurs 
Erasmus+ 
Creative Europe 
Eco-Innovation Action Plan 

Source: own elaboration 

17 

 
2.3  Testing and validation 

In  order  to  test  and  validate  the  theoretical  and  methodological  approach  which 
underpins  the  i-FRAME,  three  complementary  activities  were  carried  out  during  the 
research. These activities are being further developed as part of current research, which 
has continued beyond the exploratory phase of the IESI project. 

First  of  all,  the  i-FRAME  methodology  was  applied  qualitatively  to  a  number  of  Case 
studies  which  were  drawn  either  from  research  conducted  by  the  JRC  as  part  of  a 
complementary component of the IESI project, or selected  ad hoc for their specificities. 
These  case  studies  address  the  implementation  of  ICT-enabled  social  innovation 
initiatives  across  the  EU.  They  involve  various  policy  domains  that  integrate  offer  of 
various  PSSGI.  They  therefore  constitute  an  interesting  testing  environment  to  validate 
the possibility of using the i-FRAME methodology to dynamically simulate the impacts of 
these  innovations  in  the  context  of  social  policy  reforms  in  the  EU.  This  kind  of  testing 
and  validation  activity  is  being  further  developed  as  part  of  the  current  research 
activities of JRC to pilot and validate the i-FRAME 2.0 (see Chapter 5). 

In  addition,  a  quantitative  validation  of  the  degree  of  applicability  of  the  i-FRAME 
methodology  was  done  using  two  Scenarios  of  use  related  to  the  implementation  of 
ICT-enabled  social  innovation  in  different  programmes  and  contexts  of  social  services 
delivery.  This activity aimed to show, in practical terms, how the proposed approach to 
dynamic  simulation  could  be  applied  to  simulate  the  impact  ICT-enabled  social 
innovation  initiatives  could  have  on  various  social  service  delivery  processes.  The 
experimentation  of  the  operational  components  for  dynamic  simulation  modelling  using 
scenarios of use, as proposed for the  i-FRAME 2.0, is part of the JRC's current research 
activities in piloting and validating it (see Chapter 5). 

Finally, consultation with experts and representatives of stakeholders allowed the 
IESI  research  team  to  gather  insights  from  researchers,  practitioners  and  policymakers 
in order to better define the characteristics of the approach proposed. This included the 
organisation  of  dedicated  'Group  model  building'  sessions  at  the  second  IESI  Workshop 
in Brussels on 24-25 February, the third IESI Workshop in Seville on 7-8 July 2015 and 
some ad hoc i-FRAME Workshops16. 

In  2015,  The  i-FRAME  proposal  was  also  presented  and  discussed  in  various  scientific 
and policy events: the Social Innovation Research Conference (SIRC) in Shanghai, on 
21  May  2015;  the  SIMPACT  Workshop  in  Brussels  on  25  June  2015;  the  International 
Conference  on  Complex  Systems  in  Turin  on  14  October  2015,  the  ICT  Event  in  Lisbon 
on  21  October  2015.  In  2016,  it  was  presented  and  discussed  at  the  1st  European 
Conference on Digital Ecosystems for Social Services, Rome, 20 May 2016; the European 
Social Services Conference (ESSC), The Hague, 20-22 June 2016; the DSI4EU event on 
Shaping  the  Future  of  Digital  Social  Innovation  in  Europe,  Brussels,  29  June  2016;  the 
Experts Hearing on the Future of Welfare systems as part of the consultation on the EU 
Pillar  on  social  Rights,  Brussels  30  June  2016,  and  the  launch  of  the  Social  Innovation 
Community (SIC) in Brussels on 26 September 2016. 

Finally, the i-FRAME 2.0 proposal was discussed at a restricted IESI Experts Workshop in 
Seville  on  25  October  2016,  which  was  attended  by  modellers  and  science-for-policy 
advisers.  The  final  proposal  was  also  presented  to  relevant  stakeholders  at  the  DSIFair 
2017 in Rome on 2 February 2017; to experts at the FuturICT2.0 workshop in Rome on 
3 February 2017; and at the 1st International Conference on Synthetic Population (ICSP) 
in Lucca on 22-23 February 2017.  

The  Final  IESI  Conference  held  in  Brussels  on  14-15  March  2017  allowed  further 
discussion  with  experts,  representatives  of  key  stakeholders  and  policymakers  to  plan 
future developments and the way forward for research (see Chapter 5 and 6). 

16  i.e.  the  ESN  Seminar  on  Evidence  Based  Policy,  Manchester,  4  November  2015;  the  EIPA  Public  Sector 
Innovation Lab, Barcelona, 10 December 2015, and the eGov Conference, Guimaraes, 6 September 2016. 

18 

 
 
                                           
3  i-FRAME 1.0: piecing the puzzle together 

Summary of content of Chapter 3 

This  chapter  provides  an  overview  of  the  initial  proposal  of i-FRAME  (V1.0).  It  is 
organised as follows: 

§3.1 discusses the quest for social impact assessment; 

§3.2  elaborates  on  the  need  to  tackle  the  complexity  of  the  digital  social  policy 
innovation ´ecosystem´ 

§3.3  outlines  the  initial  proposal  of  i-FRAME  as  a  ´meta-framework´  for  social 
impact assessment. 

3.1  The quest for and challenges of social impact assessment 

The review of the state of the art conducted during the first phase of the IESI research 
led to the conclusion that: 

• 

It  is  widely  recognised  that  social  impact  assessment  is  still  under-researched 
and evaluation approaches undertaken are methodologically weak. 

•  Though  social  impact  assessment  is  still  largely  perceived  as  'nice  to  have',  it  is 

generally not included in the design of interventions. 

•  There is a lack of accepted and tested methods, tools and indicators to assess 
the social and economic impact of ICT-enabled social innovation initiatives in general 
and of those promoting integrated approaches to social services in particular. 

However,  the  review  confirmed  that  although  Social  Impact  Assessment  (SIA)  is  less 
developed  than  the  assessment  of  economic  and  financial  impacts,  it  is  gaining 
momentum in both research and policy.  

In terms of policy, this is clearly demonstrated by the strong commitment expressed by 
the President of the European Commission, Jean-Claude Juncker. The Political Guidelines 
for  the  European  Commission,  entitled  'A  New  Start  for  Europe:  My  Agenda  for  Jobs, 
Growth, Fairness and Democratic Change' (European Commission, 2014e), said that:  

'…in  the  future,  any  support  and  reform  programme  goes  not  only  through  a  fiscal 
sustainability  assessment  but  through  a  social  impact  assessment  as  well.  The  social 
effects of structural reforms need to be discussed in public'. 

From a 'practice-oriented' perspective, this momentum is reflected in the KPMG report 'A 
New  Vision  of  Value',  which  argues  that  societal  and  corporate  values  are  inter-
connected. Methodologies for measuring impact should shift from financial-driven impact 
assessments  to  taking  into  account  societal  values  when  assessing  the  outcomes  of  an 
initiative (KPMG, 2014a). In the report, it is advocated the development of a 'true value 
method  of  analysis'.  This  method  would  aim  to  provide  a  model  based  on  case  study 
analysis  to  explore  the  potential  of  monetising  impacts,  taking  into  account  both  the 
positive and negative externalities of social investments.  

This  need  of  combining  financial  returns  and  social  value,  finding  a  positive  correlation 
between the two, is also referred to as '(Social) Impact Investing' (e.g. Bugg-Levine 
and Emerson, 2011).  

The issues related to social impact assessment have been debated for several years now. 
However,  the  analysis  of  the  state  of  the  art  still  points  to  the  need  to  define  a 
methodological  approach  and  develop  a  meta-framework  capable  of  assessing 
the social and economic returns of initiatives promoting social investment.  

19 

 
 
This need is even more pressing when it comes to assessing impacts generated by ICT-
enabled  solutions  that  address  complex  phenomena  in  the  broad  domain  of  social 
policy innovation which promote social investment. 

As anticipated in Chapter 1, these require that:  

•  Policy  interventions  should  be  conceived  from  a  life-course  perspective  (i.e.  they 
should represent a continuum of measures which accompany people through the key 
stages of their lives: childhood, working-age, parenthood, and old age; and  

•  Measures  related  to  the  various  policy  areas  should  be  contemporaneous  (i.e. 
occurring  at  the  same  time)  and  mutually  reinforcing.  In  other  words,  the 
development  of  institutional  complementarities  is  a  necessary  condition  for  the 
implementation of successful social investment strategies. 

From  this  perspective,  the  review  of  literature  and  practice  demonstrates  that 
traditional  econometric  and 
innovation  models  have  not  provided  a 
comprehensive  approach.  They  do  not  integrate:  a)  social,  economic  and 
environmental  concerns;  or  b)  heterogeneous  agents'  behaviours,  roles  and 
relationships.  Therefore,  along  with  interdisciplinary  research  to  address  the  multiple 
dimensions  which  characterise  social  policies,  alternative  methods  to  complement  more 
traditional evaluation techniques are needed. Foresight, scenario building, and modelling 
and simulation techniques can all be used to help stakeholders to cope with innovation-
related uncertainties. These methods contribute to a better understanding of the various 
factors  which  influence  the  evolutionary  process  related  to  social  policies  and  their 
innovation.  They  also  help  to  define  favourable  conditions  by  considering  alternative 
development paths and outcomes. 

When complexity is the defining aspect of a phenomenon, the most appropriate 
way  to  analyse  it  as  a  'complex  system'  is  by  using  modelling  first.  Then 
simulation can be used to understand its behaviour and to attempt to predict its 
evolution.  If,  for  example,  the  situation  modelled is  a  socio-economic issue,  modelling 
and  simulation  can  give  a  numerical  indication  of  the  resources  needed,  the  time 
required  to  achieve  an  objective,  the  duration  of  an  initiative,  and  so  on.  This  gives  a 
detailed  qualitative  idea  first.  Then  quantitative  values  can  be  obtained  when  the 
numerical  simulation  is  applied  to  the  analysis  and  an  assessment  of  all  the  possible 
performance  indicators  is  made.  Consequently,  modelling  can  provide  evidence  to 
highlight the ideas, doubts, and intentions of the policy and decision makers involved in 
possible structural, operational and organisational changes (Misuraca et al., 2016)17. 

Within  this  context,  a  'model'  is  defined  as  'a  physical,  mathematical,  or  otherwise 
logical  representation  of  a  system,  entity,  phenomenon,  or  process',  and  the  act  of 
modelling as the 'application of a standard, rigorous, structured methodology to create 
and  validate  a  physical,  mathematical,  or  otherwise  logical  representation  of  a  system, 
entity, phenomenon, or process' (USA, DoD, 1998 and 2010).  

Under  this  perspective  a  model  can  be  considered  as  a  tool  for  aiding  in  the 
understanding  and  the  predicting  of  a  system's  behaviour,  while  simulation  can  be 
defined as 'the exercise (either statistically or over the time) of a model'. The process of 
modelling and simulation thereby consists in 'the rigorous process of conceptualising, 
developing, and if necessary testing the model, followed by the exercise of that model to 
study its behaviour' (Sterman, 2000 and 2006). 

17   Approaches based on modelling of complex systems, although applied often in the health sector are rarely 
used  to  address  social  care  or  social  work.  However,  social  work  theorists  have  drawn  extensively  on 
'Systems thinking' and simulation, thus methodological approaches such as Systems Dynamics appears to 
be  appropriate  to  illustrate  the  variety  of  phenomena  that  can  describe  the  complexity  of  social  services 
delivery (McKelvie, 2013). 

20 

 
 
 
 
                                           
3.2  Addressing the complexity of digital social policy innovation 

Findings from the analysis of the state of the art show that -broadly speaking- there are 
the following three main types of innovation in social policies: 

•  Service innovation. Innovation in social services may derive from opposite drivers: 
individualisation  vs.  standardisation  (e.g.  a  new  concept  in  the  delivery  of  social 
services, common to several very different targets, is that of 'care at home is better'. 
For  instance,  the  concept  of  deinstitutionalisation  and  community  care  are  widely 
applied  to  care  of  mental  health  patients,  the  elderly  and  abandoned  children). 
Service  innovation  also  involves  the  concept  of  service  accessibility, i.e.  how  access 
to  services  is  organised,  the  costs  of  access  for  families  and  the  type  of  access 
(voluntary vs. compulsory). 

•  Governance  Innovation. 

Innovation 

includes  new  work 
arrangements,  and  the  involvement  of  new  stakeholders.  It  can  also  consist  of 
innovation  in  organisational  systems  such  as  the  search  for  new  'solutions'  through 
the 
implementation  of  new  methods  of  management  or  multi-stakeholder 
partnerships. These changes in governance introduce different relationships between 
stakeholders and different decision-making mechanisms. 

in  Governance 

•  Digital  innovation.  The  social  services  sector  is  centred  on  people  and  service 
delivery. Hence, ICTs can contribute a great deal to the personalisation of services by 
providing  a  targeted  service  and  personalised  experiences.  Moreover,  ICT-enabled 
innovation  in  social  services  also  influences,  and  is  influenced  by,  the  interaction 
between  different  actors.  This  interaction  affects  the  roles  played  by  stakeholders 
and end users and the relationships between them, and also the way social networks 
are structured. New forms of interaction/cooperation can emerge thanks to ICTs, as 
can new approaches to acquiring funding, and monitoring and assessing results. 

These key drivers of innovation in social policy provide the basis for a generic 'modelling 
environment',  which  can  be  defined  as  the  'Ecosystem  of  ICT-Enabled  Social 
Innovation' that promotes social investment see Figure 5. 

Figure  5:  Ecosystem  of  ICT-Enabled  Social  Innovation  promoting  social 
investment: generic stylised modelling environment and its dynamics 

Source: Misuraca et al., 2015, p. 44 

21 

 
 
In line with the scope of the IESI research, this ecosystem is represented in Figure 5 as 
having an outer ring with macro-meso level contextual variables, and an inner ring with 
the  key  components  of  social  services  deployment/implementation  and/or  functioning 
(meso-micro  level).  The  outer  variables  in  combination  with  the  specific  dimensions  of 
each ICT-enabled social innovation initiative shape the attributes of the PSSGI provided. 
The  level  of  deployment/diffusion  and  the  outcomes  produced  by  each  initiative 
correspond to the level of governance adopted by different types of PSSGI considered18. 

In  the  case  of  ICT-enabled  social  innovations,  institutional  settings  generally  involve 
what is legitimate and possible/desirable according  to political norms, laws, regulations, 
and  policies,  and  access  to  funding  mechanisms.  These  settings  influence  the 
governance  of  the  ICT-enabled  social  services  and  its  ecosystem  network  configuration 
(equivalent to what in business terms would be the value chain) both:  

• 

Indirectly, by way of available funding and policy at the macro-meso-micro level, as 
institutional  settings  also  shape  the  level  of  systemic  eReadiness  (i.e.  availability, 
usage  and  cost  of  ICTs)  in  the  specific  context  (e.g.  local  community,  region, 
country);  

•  And  directly,  given  the  funds  made  available  for  social  services  and  the  scope  of 
action  defined  by  regulation  and  policies,  including  the  institutional  settings  which 
influence  the  aim  and  scope  of  the  social  services  and  the  type  of  ICTs  deployed. 
Although it is not possible to insert institutional settings in the graph, they (especially 
access  to 
long-term 
sustainability. 

funding)  have  considerable 

impact  on  an 

initiative's 

The  socio-economic  conditions  in  a  given  community  make  some  social  needs  and 
challenges  more  pressing  than  others.  Although  these  conditions  are  not  visible  in  the 
graph, they also influence the laws, regulations, and policies defining the scope of social 
services.  The  community's  needs  and  challenges  evidently  trigger  the  actions  of  the 
adopters  of  ICT-enabled  social  innovations  and  shape  their  motivations  and  attitudes. 
They also affect organisational capabilities and how they are reflected in the definition of 
the aims, scope and breadth of the services. 

The  notion  of  social  capital  and  collaborative  networks  plays  an  important  role  in  the 
context in which social services are implemented. The existence of vibrant social capital 
and  community  networks  clearly  drives  success,  whereby  the  configuration  of  social 
capital and network influences and is influenced by the level of systemic eReadiness. 

3.3  Outlining a 'meta-framework' for social impact assessment 

The dimensions of the ecosystem of ICT-enabled social innovation which promotes social 
investment  that  we  briefly  described  above  laid  the  foundations  for  the  initial  proposal 
for i-FRAME 1.0. This included the elements to consider for defining data and variables 
that would serve to structure and 'feed' the impact assessment framework.  

One  of  the  crucial  dimensions  of  the  ICT-enabled  social innovation  ecosystem  is  clearly 
linked to innovation process dynamics and the level of maturity of the social innovations. 
Knowing  a  social  innovation's  'stage  of  maturity'  may  be  important  for  tailoring  better 
support  measures  and  adequate  funding  structures.  This  is  more  important  for  those 
social  innovations  where  ICTs  play  a  'game-changing'  role  (see  Misuraca  et  al.,  2015). 
For  these  innovations,  it  may  be  more  difficult  to  gather  scientific  evidence  of  impact 
with traditional evaluation techniques or modelling approaches. 

18   For  the  sake  of  simplicity,  the  ecosystem  depicted  in  Figure  5  does  not  report  all  possible  interactions 

among all the variables within each ring and between the two rings. These interactions are implied. 

22 

 
 
 
                                           
We are in an exciting period of innovation characterised by schemes based on traditional 
and  emerging  ICTs,  new  funding  models,  and  a  more  dynamic  relationship  between 
governments, citizens, and service providers from the private and not-for-profit sectors.  

Social innovation  - and more specifically ICT-enabled social innovation  - can thus make 
an important contribution to social policy reform by providing new/better/different ways 
of  integrating  the  provision  of  social  services.  But  evidence  to  inform  decisions  is  often 
limited  or  produced  too  late,  thus  becoming  meaningless,  or  insufficient  to  support 
policy-making in a structured manner.  

The other important dimension underlying the conceptual framework advanced as part of 
the  IESI  research,  refers  to  'The  increased  coordination  of  operations  across  traditional 
functional  units  in  the  public  sector,  and  also  across  other  non-public  sector  providers, 
the aim being to put the final users/beneficiaries (including service intermediaries) in the 
centre and treat their needs holistically' (Misuraca et al., 2015). 

The  focus  on  integrated  approaches  to  social  services  is  due  to  the  fact  that  they  can 
generally  improve  efficiency  of  social  service  systems.  In  addition,  it  is  assumed  that 
they  can  also  better  solve  what  are  known  as  'wicked  problems'  –  for  example,  the 
provision of social services in the current context of economic and social turmoil. These 
problems  are  related  to  many  causes  and  effects,  which  overlap  and  intertwine.  They 
require multifaceted solutions and multi-service provision. Social services integration can 
be considered as an answer to wicked problems, and ICT-enabled social innovation as a 
'change' factor associated with it.  

Hence, when the i-FRAME was designed, the main aim was to develop a multi-level and 
multi-dimensional approach from both: 

•  An operational perspective, as the integration of services enhances organisational 
performance  and  the  effectiveness  of  services  in  terms  of  improved  outcomes, 
efficiency  and  reduced  costs.  It  also  increases  capacity  and  value  for  money, 
strengthens  strategic  planning  and  system  integrity,  and  reduces  demand  for  crisis 
services.  Moreover,  from  the  beneficiary's  perspective,  it  provides  simplified  access, 
holistic and customised support, faster response times, and improved outcomes and 
user experience. 

•  A system perspective, as a service does not have an autonomous existence in the 
same  way  that  a  physical  thing  with  technical  specifications  does  (Crepaldi  et  al., 
2012).  It  is  a  social  construction  (with  its  world  of  reference),  which  fits  into  time 
frames  in  different  ways  (time  horizon)  and  into  matter  (degree  of  materiality) 
(Djellal and Gallouj, 2001). Thus the relational dimension plays a central role, as the 
relationship between the user and the service  provider is direct (Bandt and Gadrey, 
1994; Gadrey, 2003; Laville, 2011). In this context, the technological dimension can 
play  an  important  role  in  social  services  innovation  processes  and  can  contribute 
positively  to  the  quality  and  productivity  of  services  with  new  solutions  to  policy 
challenges (Randle and Kippin 2014).  

Therefore, to address the  first  objective of the  i-FRAME (i.e. to provide a structured 
approach  for  analysing  the  initiatives  collected  through  the  mapping  of  ICT-enabled 
social  policy  innovation  in  the  EU),  an  analytical  framework  for  mapping  and  analysing 
initiatives  of  ICT-enabled  social  innovation  in  integrated  approaches  to  social  services 
delivery was developed. The framework elaborates on the interrelationship between four 
main dimensions of analysis, namely: 1) ICT-enabled innovation potential; 2) levels 
of  governance  of  service  integration  3)  elements  of  social  innovation;  and  4) 
types of  service  integration.  In addition, it serves to analyse the initiatives collected 
as  part  of  the  research,  positioning them in the  'IESI  Knowledge  Map'  (see  Misuraca  et 
al., 2015).  

23 

 
 
Building  on  the  IESI  analytical  framework,  a  preliminary  proposal  of  operational 
components  for  assessing  outcomes  and  possible  impacts  of  ICT-enabled  social 
innovation  initiatives  was  developed.  This  analysis  is  mainly  at  the  micro-level,  but  it 
also has implications for the meso level (See Misuraca et al., 2015).  

This choice of operational components, which mainly include the key dimensions of direct 
and  indirect  outcomes  and  possible  indicators  for  measuring  outputs  and  outcomes,  is 
based  on  an  intervention  logic  model.  This  is  a  representation  of  how  a  policy,  a 
programme  or  an  initiative  functions  theoretically  under  specific  conditions  to  achieve 
the desired target objectives.  

In  simple  terms,  the  conditions  of  an  intervention/programme/policy  are  understood  as 
the factors from which the initiative, programme or policy starts. These are normally: 

•  The  general  conditions,  such  as  economic,  political  or  social  circumstances  (i.e.  the 

context in which an intervention takes place);  

•  The  target  group  specifications  (i.e.  attitudes,  knowledge,  needs  and  compliance  of 

the target group members must be taken into account);  

•  The financial, human and material resources (input); and  

•  The  characteristics  of  the  'programme  sponsor',  such  as  its  legal  form  or  financing 
structure.  These  characteristics  also  include  how  the  parties  responsible  for  the 
programme's  implementation  define  when  targets  should  be  achieved,  with  which 
target group, through which activities, etc.  

During  the  process,  the  measures  intended  to  achieve  the  target  are  implemented. 
Those contributions that are directly provided by the intervention, programme  or policy 
are referred to as outputs (which are directly measurable results). Outcomes represent 
the  desired  conditions  for  the  members  of  the  target  groups  after  completion  of  the 
activities. The outputs are meant to produce the desired outcomes and contribute to the 
achievement  of  specific  (i.e.  directly  linked  to  the  initiative,  programme  or  policy),  or 
broader  global  impacts  (i.e.  affecting  socio-economic  conditions  of  the  context  of 
reference) in a way that is logically, theoretically or empirically substantiated. 

However, while it is assumed that the impacts of policies, programmes or initiatives can 
be logically derived and estimated through a logic model based on theory of change, the 
effective cause-effect relationships cannot be substantiated by this approach.  

Therefore, in order to address the second objective of the i-FRAME (i.e. to serve as a 
comprehensive framework for conducting analysis of the economic and social returns on 
investment  in  which  ICT-enabled  social  innovation  is  a  key  component),  more  in-depth 
investigation  is  required  into  both  the  direct  and  indirect  impacts  of  ICT-enabled  social 
innovation  on  individuals  (and  their  wellbeing  and  quality  of  life,  for  instance),  and  the 
'systems effects' that can be promoted through this process.  

It was also necessary to consider the unintended consequences of ICT-enabled 
social  innovation  and  the  network  effects  that  can  be  generated,  though  these 
are  difficult  to  capture.  Single  initiatives  cannot  alone  explain  the  innovation  dynamics 
triggered by the complex and multi-network processes inherent in the phenomena under 
investigation,  in  which  ICTs  contribute  as  both  enabling  and  game-changing  factors. 
Instead, a systems approach should be considered, which integrates a complexity theory 
perspective  (e.g.  Lane  et  al.,  2007)  and  a  multi-level  and  dynamic  approach  to 
innovation (e.g. Padgett and Powell 2012).  

For  this  purpose,  we  developed  the  proposal  for  the  i-FRAME  as  a  meta-framework, 
which  comprises  several  methodologies  and  approaches.  These  can  be  applied  at 
different  levels  of  analysis  where  and  when  appropriate,  depending  on  the  conditions 
available and the specific degree of detail required.  

24 

 
 
The  initial  version  of  the  i-FRAME  (1.0)  is  outlined  in  Figure  6.  It  was  originally  put 
forward for discussion at the 2nd IESI Experts and Stakeholders Consultation Workshop 
held in Brussels on 24-25 February 2015.  

Figure 6: i-FRAME Outline – meta-framework  

Source: Misuraca et al., 2015, JRC Technical Report, IESI D2.1 – i-FRAME 1.0 

The review of the state of the art found that current approaches to assessing the social 
and  economic  impacts  at  micro-meso-macro  levels  of  ICT-enabled  social  innovation 
initiatives  promoting  social  innovation  are  limited  in  scope.  It  was  therefore  deemed 
necessary  to  set  up  a  specific  methodological  framework.  The  i-FRAME  can  however 
benefit from existing methodologies and approaches that can be combined depending on 
the specific needs and levels of analysis.  

The  i-FRAME  should  be  considered  as  a  meta-framework  which  provides 
guidelines  for  using  different  methodologies  and  for  building  a  comprehensive 
(computer-based) simulation model. This model, which starts from micro-simulation 
analysis and continues  to a meso/system  level elaboration, could help policy makers  to 
devise appropriate macro/policy indicators. 

To  do  this,  the  i-FRAME  must  be  able  to  capture  the  direct  effects  and  indirect 
consequences  of  'initiatives'  (i.e.  policy/programme/project)  and  understand  how  these 
impact on beneficiaries, organisations, possible intermediaries, the social innovation eco-
system  and  the  welfare  system  in  which  these  initiatives  are  embedded.  At  the  same 
time,  the  socio-economic  effects  on  individuals,  organisations  and  the  context  of 
reference should be studied and related to the social service delivery models and welfare 
systems in which they operate.  

The specific role of ICT-enabled innovations and the social nature of the initiatives under 
investigation should also be factored into the analysis, possibly by using quantified (and 
when possible monetised) indicators and variables. 'Proxy-indicators' may be used when 
data are not available or, alternatively, data could be retrieved from qualitative  analysis 
and interviews with stakeholders and beneficiaries.  

The  initial  outline  for  i-FRAME  1.0  also  included  a  preliminary  proposal  of  what 
operational  components  and  techniques  could  be  used  –or  should  be  developed-  to 
assess impacts at a higher level of abstraction (meso and macro).  

25 

 
 
 
 
These, through the link with the operational tools developed to gather data and estimate 
effects at micro-meso level, have allowed  the  design of a theoretical model  to estimate 
the  social  and  economic  return  on  investment  of  ICT-enabled innovation  initiatives  (i.e. 
policies/programmes/projects). A first set of indicators has also been proposed. 

This framework could be used by policymakers at various levels (European Commission, 
Member States, regions, municipalities, etc.) to analyse (ex-ante, in-itinere and ex-post) 
the  possible  consequences  of  policy  interventions  based  on  ICT-Enabled  Social 
Innovations promoting social investment.  

At  the  micro-meso  level,  this  implies  that  an  analysis  of  the  diffusion  and  net-
contribution  of  ICT-enabled  social  innovation  in  social  services  should  provide  an 
estimation  of  their  direct  impact  on  perceived  social  value.  It  should  also  provide  an 
estimate of the performance of the social innovation ecosystem in which an initiative (be 
it a policy, a specific programme of intervention, or a group of projects) is implemented. 

For this purpose, a number of methodologies and evaluation techniques can be applied. 
First of all, baseline data and benchmark indicators need to be identified where possible 
through  literature  reviews  and  thematic  analyses.  The  review  of  the  state  of  the  art 
provided  an  overview  of  the  existing  evidence.  But  this  should  be  further  developed  ad 
hoc and contextualised, looking at specific case studies and scenarios of use.  

The  availability  of  sufficient  quality  data  is  a  crucial  aspect  for  the  analysis  of  ICT-
enabled  social  innovation  initiatives  and  their  contribution  to  social  services  reform. 
However,  the  review  of  the  state  of  the  art  confirmed  a  lack  of  data  for  measuring 
performance and impacts of social services initiatives in general and in social innovation 
in particular. The development of a systematic collection process to gather micro-data is 
therefore essential. These data will be necessary for analysing impacts at micro level and 
also for 'feeding' the simulation modelling for assessing impacts at meso/macro level. 

The IESI project has made great advances to set up an online platform, which includes 
an interactive tool for data gathering. This has allowed us to build a structured sample of 
initiatives,  from  which  we  can  also  draw  case  studies  and  scenarios  of  use  for  testing 
some  of  the  operational  components  of  the  i-FRAME.  In  the  future,  however,  the 
possibility  of  further  expanding  this  data-collection  process  with  a  more  data-driven 
approach  should  be  explored.  This  may  involve  the  development  of  a  practical  data 
aggregation  perspective  for  collecting  and  exchanging  impact  data  through  a  machine 
readable  data  formatting  protocols  that  can  capture  the  key  data  generated  by  various 
impact reports and assessments, using data available on the web for context and other 
system's related variables. 

Additional  data  on  perceptions,  value  preferences,  relations  among  stakeholders,  and 
agents'  behaviours  also  need  to  be  captured  in  order  to  better  understand  the 
implications  and  effects  of  initiatives  on  the  social  innovation  ecosystem.  For  this 
purpose, a range of techniques including cost-benefit analysis and in-depth case studies, 
stakeholder  surveys  and  behavioural  analyses  -  including  in  particular  Agent-Based 
Modelling (ABM) - can be performed.  

Once  again,  how  data  are  collected  and  structured  is  pivotal  as  it  must  allow  the 
standardisation  of  information  for  modelling  and  simulation  purposes.  The  approach  for 
online data gathering developed for the IESI research could serve to collect information 
on key variables for specific evaluations and also for paving the way to more advanced 
simulation modelling techniques.  

In  addition,  to  elicit  specific  data  and  unknown  variables  of  direct  and  indirect  effects, 
specific  methodologies  based  on  a  counterfactual  approach  to  policy  evaluation  can  be 
adopted, in an attempt to achieve what is considered the 'holy grail' of impact evaluation 
(i.e.  to  demonstrate  causality  in  a  scientifically  indisputable  manner).  These  ways  of 
identifying  the  causal  effect  of  a  treatment  can  be  found  in  experimental  and  quasi-
experimental approaches.  

26 

 
An example is to carry out a Randomised Controlled Trial (RCT), which is considered 
the  best  evaluation  method  for  inferring  a  causal  relationship  and  generally  provides 
robust reliable evidence.19  

However,  the  cost  of  setting  up  such  a  robust  counterfactual  approach  to  measure  the 
causal impact of social policies and the related ICT-enabled social innovation initiatives is 
not only very high (when feasible), but potentially unethical.  

Another  way  of  gathering  unique  data  and  robust  evidence  of  impact  is  by  setting  up 
Social  Policy  Experimentations.  Social  policy  experimentations  in  fact  require  both 
designing  a  potentially  policy-relevant  intervention  and  measuring  its  actual  efficacy. 
They  test  the  validity  of  new  innovative  policies  by  collecting  evidence  about  the  real 
impact of measures on people. The main principles of social policy experimentations are: 
they  bring  innovative  answers  to  social  needs;  are  small-scale  probing  interventions  to 
test impact; are made in conditions where their impact can be measured; and they can 
be scaled up if the results prove convincing.20 

The  new  wave  of  EC-funded  social  policy  experimentations  initiated  as  part  of  the  the 
programme  for  Employment  and  Social  Innovation  (EaSI)  could  be  a  fertile 
ground for experimentation and testing the approach proposed by the i-FRAME. 

In addition to the identification of the causality relationships between interventions and 
related outcomes, one of the main difficulties of impact evaluation is to disentangle the 
complexity  of  the  links  between  interventions  at  the  micro-meso  level  and  the  broader 
effects at the macro level. 

These effects are normally estimated by macro-economic indicators such as contribution 
on  GDP,  employment  or  competitiveness  but  do  not  capture  the  multiple  and  diverse 
effects of complex interventions.  

Assessing  the  contribution  of  ICT-enabled  social  innovation  to  these  macroeconomic 
indicators would require a much higher level of abstraction.  

However, what the i-FRAME proposes at the operational level is to estimate these effects 
indirectly  through  the  estimation  of  indicators  of  social  value  impact  on  the  welfare 
systems.  

To do this, a simulation model could be designed, based on alternative scenarios of use 
that could estimate the effects of key impact variables using forecast methods. Examples 
of  these  methods  are  Systems  Dynamics  and  Agent  Based  Modelling  which  we  will 
discuss in more depth in Chapters 4 and 5.  

19   Randomised  Controlled  Trials  (RTCs)  are  'a  type  of  impact  evaluation  which  uses  randomised  access  to 
social  programmes  as  a  means  of  limiting  bias  and  generating  an  internally  valid  impact  estimate' 
(according to the Best Evaluation website). However, Cartwright and Hardie (2012), in a powerful critique, 
point  out  that,  by  itself,  a  given  RCT  can  only  'tell  you  that  a  policy  worked  there,  where  the  trial  was 
carried  out,  with  a  given  population'.  Their  conclusion  is  that  to  be  confident  that  a  policy  will  work 
somewhere  else,  with  a  different  population,  the  right  support  factors  must  be  in  place  and  the  causal 
roles must be understood correctly. As Chapter will describe, the limits of RCT especially to assess impacts 
of policy interventions are now being recognised, especially in the social policy field. 

20   See: http://ec.europa.eu/social/main.jsp?catId=1022 

27 

 
                                           
4  i-FRAME 1.5: the potential of dynamic simulation 

modelling 

Summary of content of Chapter 4 

This  chapter  discusses  the  structured  theoretical  and  methodological  approach 
proposed to building the operational components of the i-FRAME (V1.5).  

It is organised as follows: 

§4.1 discusses how simulation modelling can be a powerful means of  addressing 
the complexity of the problem of evaluating the impact of social policy innovation 
initiatives which promote social investment; 

§4.2  presents  an  overview  of  the  methodological  approach  proposed  for 
developing the operational components of the i-FRAME; 

§4.3  assesses  the  potential  of  the  approach  proposed  in  light  of  insights  from 
testing activities and peer-review. 

4.1  Deconstructing social policy innovations as complex systems 

The  long-term  objective  of  the  i-FRAME  is  to  become  a  methodological  reference 
framework  that  helps  policymakers  and  various  stakeholders  to  better  understand  the 
effects  of  social  policy  innovation  initiatives  which  promote  social  investment.  In  the 
original proposal, it was suggested that modelling and simulation approaches to complex 
systems could be adopted as tools to assess potential impacts of social policy innovation 
initiatives in an attempt to link micro, meso and macro level effects.  

The findings of the second round of literature review (see Chapter 2) helped us to build 
on this proposal. They emphasised that the difficulty of modelling complex phenomena in 
social  policies  is  further  increased  by  the  presence  of  human  activities,  which  have  an 
impact on the whole system and are difficult to predict and model.  

Like all complex systems in general, the ecosystem dynamics of social policy innovation 
may  lead  to  the  emergence  of  new  properties  that  do  not  belong  to  the  units  in  the 
system.  While  the  properties,  interacting  rules,  and  environmental  constraints  of  each 
individual  unit  are  usually  known  beforehand,  the  emergent  properties  of  a  complex 
system are unknown ex-ante, and manifest themselves only later in time (ex-post). 

Understanding the potential indirect effects of a policy measure is particularly important 
for  the  primary  reference  actors  of  the  i-FRAME,  i.e.  policy  makers  at  EC  and  Member 
State  level.  At  the  same  time,  various  stakeholders  with  different  roles  in  the 
implementation of these initiatives can also use the i-FRAME to design their social policy 
intervention logic. The i-FRAME will help them understand how to measure and improve 
intended outcomes and maximise the social impacts of their initiatives. 

However,  this  is  clearly  not  an  easy  task,  precisely  because  of  the  complexity  of  the 
context in which the social policy innovation initiatives are conceived. The assessment of 
their impacts requires a deep knowledge of the dynamics of causal relationships among 
relevant variables and their negative and positive interactions that are not usually linear.  

In  this  context,  simulation  modelling  could  help  us  understand  and  predict  future 
states.  According  to  Hughes  (1997),  simulation  modelling  is  useful  for  exploration, 
explanation or extrapolation. Exploration can be defined as the process of searching in 
order  to  discover.  Explanation  is  the  process  of  linking  causes  to  effects  in  order  to 
understand,  and  extrapolation  is  the  process  of  predicting  beyond  the  known  and 
observed.  

28 

 
 
 
This approach allows us to address only those aspects of a problem that we believe are 
particularly meaningful. It helps us to avoid introducing details that would be useless and 
confusing  at  the  level  of  abstraction  chosen  for  the  representation  of  the  problem.  In 
addition, it also reduces the structural complexity of the system.  

The  problem  can  then  be  depicted  as  a  simplified  framework  where  only  the  most 
important facts, properties, behaviours are included, organised and analysed over time. 

In  this  respect,  the  expression  'over  time'  is  a  crucial  aspect  of  a  model.  The  goal  of 
modelling  and  simulation  is  not  to  merely  represent  the  structure  of  cause-effect 
relationships  which  generate  the  problem.  Instead,  the  real  goal  is  to  catch  the 
relationship  between  the  main  variables  of  the  problem  and  to  include  appropriate 
hypotheses (e.g. through the use of feedback loops and/or state transitions charts) that 
assess the behaviour of the system (representing the problem) over time (i.e. including 
a 'dynamic' hypothesis that drives the evolution of the system's behaviour over time). If 
this  approach  is  accompanied  by  a  set  of  appropriate  indicators,  it  shows  how 
the system changes, adapts and evolves over time. 

Simulation modelling can perform the following tasks: 

• 

Identify critical functional and relational aspects of complex systems. 

•  Understand why a system behaves the way it does as a function of its structure. 

• 

Inform  decision-making  by  evaluating  intended  and  unintended  consequences  of  an 
intervention. 

However, this brings with it a number of challenges that fall into three main groups: 

•  Analytical:  the  definition  of  the  key  variables  and  the  cause-effect  relationships 
amongst them requires an extensive literature review and the involvement of experts 
in group-modelling sessions. This is especially true in social policy innovation, where 
limited evidence has been provided by scientific studies. 

•  Empirical:  the  lack  of  availability  of  data  to  test  the  model  may  hamper  its 
application. The deployment of a data gathering approach based on the development 
of  scenarios  of  use  that  are  close  to  reality  and/or  based  on  real  case  studies  from 
the model's initial design may be part of the solution. 

•  Methodological:  the  validation  of  the  theoretical  and  methodological  approach 
developed for a model based on systems thinking and simulation requires consensus 
on  what  kind  of  case  studies  should  be  selected  and  analysed  during  the  empirical 
activity.  This  can  be  achieved  by  involving  experts  and  stakeholders  and  by  using 
consolidated indicators or proxy variables as baselines when available. An additional 
complication can arise when typical behaviour at one level of aggregation changes in 
another21. 

To address these challenges, the second phase of the i-FRAME research (V1.5) aimed to: 

•  Develop  a  structured  theoretical  and  methodological  approach  to  building  the 

operational components of the i-FRAME  

•  Validate the relationships and the dynamic hypothesis underlying the theoretical and 
methodological approach proposed by testing it on a number of selected scenarios of 
use relevant to IESI's scope.  

•  Propose a possible approach to defining and developing the operational components 
of  a  computer-based  simulation  model  that  could  address  all  levels  of  analysis  by 
using the same structural environment. 

21   This  means  that  the  typical  behaviours  shown  at  micro  level  can  form  new,  sometimes  unknown 
behaviours at the upper levels, meso and macro. The way these can be connected in a dynamic manner 
requires complex modelling techniques based on computer simulation tools.  

29 

 
                                           
The  main  assumption  underlying  the  proposal  to  use  dynamic  simulation  modelling  as 
part  of  the  methodological  approach  for  the  i-FRAME  is  that  –  under  certain 
circumstances  -  it  can  overcome  the  key  problem  of  modelling  and  simulating  policy 
decisions  in  complex  social  systems.  Dynamic  simulation  modelling  enables 
policymakers  and  other  relevant  stakeholders  to  better  understand  the  impact 
of  a  given  policy.  It  can  therefore  prevent  them  from  being  'overconfident'  in  the 
effects specific policy designs can produce.  

This  is  particularly  important  when  policymakers  have  to  analyse  'Downstream'  and 
'Upstream'  interventions.  These  represent  two  opposite  policy  measures  that  can  be 
considered  when  addressing  social  problems  and  designing  policy  interventions, 
especially when they are related to social policy and service delivery and the introduction 
(or not) of elements of ICT-enabled social innovation. 

Downstream  measures  try  to  cope  with  the  consequences  of  harm,  after  it  has 
occurred. They focus on specific cases in an attempt to stop things getting worse (Coote, 
2012), i.e. these measures intervene in the 'effects' of a certain event.  

Upstream measures, instead, aim to prevent harm before it occurs. They intervene in 
the 'causes' of the problem, in order to try and stop it happening (Coote, 2012). This is 
particularly  relevant  for  social  investment  where  anticipating  the  possible  negative 
consequences of a problem is preferred to repairing the problem once it has occurred. 

Clearly,  Downstream  interventions  which  address  structural  changes  in  social  service 
delivery processes are more expensive and less effective than the preventive ones. This 
has been demonstrated in several cases (see for example Hirsch et al., 2004 and Maggio 
and Pi-Sunyer, 1997). By contrast, Upstream interventions are more difficult to evaluate 
without  any  evidence-based  approach  that  accounts  for  the  dynamical  changes  of  the 
context where the interventions are conceived (see e.g. Homer et al., 2007). 

Upstream interventions have the most potential for changing population behaviours and 
addressing complex social problems (Raine, 2010). They can also be less expensive than 
the  Downstream  interventions  when  addressing  restructuring  policies.  However,  if 
policymakers decide to adopt one type of intervention, this does not prevent them from 
also  using  the  other.  In  fact,  a  social  policy  that  provides  a  good  balance  between  the 
two measures produces more reliable results. 

Generally,  resources  for  the  modernisation  of  social  services  tend  to  be  scarce. 
Policymakers  therefore  try  to  design  policy  initiatives  that  achieve  the  best  trade-off 
between  upstream  and  downstream  interventions  in  order  to  minimise  costs  and 
maximise  the  expected  results  and  effects.  However,  the  balance  between  these  two 
types of measures depends on the behaviour of the stakeholders involved, and how they 
make their decisions is not easily modelled or simulated. 

Therefore,  dynamic  simulation  modelling  is  particularly  useful  for  analysing  policy 
initiatives because it helps policymakers and relevant stakeholders to better understand 
the consequences of their decisions before they happen.  

However,  for  the  model  to  be  an  effective  tool  to  support  policy-making,  several 
mutually  interrelated  perspectives  should  be  combined;  for  instance  taking  into 
consideration: 

•  Views and behaviours of individuals acting in isolation should be combined with those 

of the individual as member of a given community and of the society; 

•  Stakeholders perspectives and behaviours at different layers of interaction; 

•  Effects that can be achieved in different time periods over time; 

•  Different layers of analysis (micro-meso-macro). 

30 

 
 
The  complexity  of  this  situation  is  represented  in  Figure  7,  which  describes  the 
interaction  of  human  actions/behaviour  in  a  multidimensional  space.  The  horizontal 
arrow represents the time axis for an individual's health-related behaviour and actions as 
he/she moves from birth to death.  

During  this  time  period,  the  individual  interacts  with  a  multidimensional  space  that 
represents the characteristics of the person (i.e. his/her lifestyle and related risk factors) 
and  the  external  environment  (i.e.  the  context)  in  which  the  individual  lives  from  the 
micro level, to the meso and macro level of interaction. 

Figure 7: Conceptual model underpinning the i-FRAME methodological approach 

Source: adaptation from the work of Glass and McAtee, 2006 

Individual  behaviour  is  influenced  by  structured  contingency  within  the  social  and 
physical  environment  and  by  lifestyle  and  related  risk  factors.  Structural  contingencies 
are represented by opportunities and constraints that are generated by the environment 
in  which  the individual lives.  Lifestyles  and  risk  factors,  however,  are  mainly influenced 
by the actions and behaviour of the person. At the same time, lifestyle and risk factors 
can produce negative/positive effects on the individual's health status and they can also 
change his/her psycho-physical condition. 

For instance, if we apply the conceptual model in Figure 7 to life-long learning and the 
employment  status  of  a  person,  this  person's  knowledge  capital  evolves  throughout 
his/her  life  and  it  can  be  influenced  by  the  micro/meso/macro  characteristics  of  the 
context in which he/she lives.  

Cultural norms, the structure of the cultural offer of a given context, the local conditions 
where  he/she  was  born,  and  the  psychological  hazards  to  which  he/she  is  subject,  all 
determine  his/her  initial  accumulation  of  knowledge  capital.  The  psycho-physical  and 
health characteristics of the person are influenced on various levels by the context and, 
at  the  same  time,  they  influence  the  behaviour  of  the  person  and  his/her  choices  and 
ability  to  take  the  opportunities  offered  by  the  context  in  which  he/she  lives.  Thus,  a 
person's  life  course,  and  also  his/her  knowledge  capital  and  employment  status  are 
subject  to  change  due  to  human  behaviour  and  actions,  his/her  psycho-physical  and 
health  status  and  the  opportunities  and  constraints  which  characterise  the  context  at 
various levels of interaction.  

31 

 
 
 
 
These considerations are represented by the feedbacks loops in Figure 8. 

Figure  8:  Example  of  application  of  the  conceptual  model  underpinning  the  i-
FRAME 

Source: own elaboration 

This  representation  of  an  individual's  behaviour/actions  during  his/her  life  course,  the 
mutual  influence  between  individual  behaviour  and  the  context  -  at  various  levels  of 
interaction,  and  the  effects  of  this  influence  on  his/her  psycho-physical  and  health 
condition  is  well-documented  by  studies  that  demonstrate  the  sensitivity  effects  of 
certain  exposures.  For  example,  studies  of  work  and  health  show  that  employment 
status  and  working  conditions  exert  their  strongest  influence  during  mid-adulthood 
(Marmot et al., 2001).  

Exposure to social conditions, on the other hand, appears to be cumulative (House et al., 
1994).  Furthermore,  several  studies  have  consistently  observed  an  increased  risk  of 
disease due to economic and social disadvantage (see for instance Hallqvist et al., 2004; 
Singh-Manoux, et al., 2004). 

This is particularly important for the development of the i-FRAME, as  it aims  – when at 
regime – to support policy makers at European Commission and Member State level, by 
suggesting ways to assess impacts of social policy innovation initiatives promoting social 
investment.  At  the  same  time,  it  should  also  become  a  useful  practical  resource  for 
various  stakeholders  and  practitioners  involved  in  the  implementation  of  social 
innovations  in  social  services,  helping  them  to  design  their  intervention  logic  and 
understand how to measure outcomes and evaluate the social impacts of their initiatives. 

32 

 
 
 
 
4.2  i-FRAME Decalogue: experimenting with system dynamics 

The  example  discussed  above  (§4.1)  suggests  that  social  policy  innovations  promoting 
social  investment  can  be  studied  as  complex  systems22,  according  to  the  conceptual 
model presented. Complex systems can fairly represent all the situations in which  social 
policy innovations affect the welfare system, especially with regard to the design and/or 
delivery of PSSGIs, given the personal nature of these services. 

As  we  have  emphasised  above,  social  policy  innovations,  especially  when  ICT-enabled, 
can  act  directly  on  a  person  by  changing  his/her  psychophysical  and  health  conditions; 
these in turn can influence his/her behaviour and actions. At the same time, they can act 
indirectly on the person's behaviour, by modifying the context in which he/she lives, by 
changing opportunities and releasing constraints that, in-turn, can affect his/her actions 
and behaviour as well. All these situations take place during a person's life course. 

This  complexity,  which  contains  many  causal  relationships,  feedback  loops  and  non-
linearity  along  the  temporal  axis  can  be  modelled  and  simulated  by  using  dynamic 
simulation  models  such  as  System  Dynamics  (SD)  and  Agent-Based  Modelling  and 
Simulation  (ABMS).  This  is  confirmed  by  the  literature  review  and  in  particular  by  the 
comparison  of  different  complex  systems  methods  and  modelling  techniques  (see 
Chapter 2 and Annex I).  

Testing  and  validation  of  i-FRAME  1.5  resulted  thus  in  the  suggestion  that  these  two 
methods  should  be  combined  to  produce  what  we  have  called:  the  'Dynamic 
Simulation - Hybrid Model' (DS-HM).  

DS-HM  emerged  as  a  potentially  powerful  methodology  which  could  address  all  the 
specificities and complexity of evaluating the impact of social policy innovation initiatives 
promoting social investment in PSSGI delivery. The ABMS component of the DS-HM can 
model  the  dynamic  characteristics  of  each  individual  in  the  target  population  and 
simulate their behaviour during their life courses. The SD component can represent the 
complexity and the dynamic of the context in which an individual operates. It can show 
how this context evolves over time due to the  interaction among different layers  of the 
system.  These  interactions  are  represented  by  causal  relationships  and  feedback  loops 
that can interact with the behaviour of the individuals. 

Recognised  experts  in  the  field  (e.g.  Borshev,  2013)  have  argued  that  the  hybrid 
approach  to  modelling  and  simulation  is  preferable  to  other  methods  for  a  number  of 
reasons.  

Firstly,  the  SD-based  structure  represents  a  well-defined  paradigm  that  includes  high-
level perspectives and gives a high-level abstraction of the problem. In the language of 
stocks  and  flows,  the  general  logic  of  the  system  is  immediately  clear.  It  is  easily 
understood and depicts possible counter-intuitive situations. 

Secondly,  a  pure  SD  approach  does  not  immediately  show  the  behaviour  of  single 
classes  of  objects  (people,  means,  sub-structures)  involved  in  the  problem.  In  other 
words,  it  does  not  show  the  effort  needed  to  take  into  account  possible  behaviours  of 
single  classes  of  objects  that  are  part  of  the  problem.  This  requires  the  introduction  of 
other structures (arrays of stocks and flows) that tend to complicate the overall model. 

Finally,  a  pure  ABMS  approach  is  sometimes  difficult  to  apply  to  general  macro 
behaviours.  Because  the  ABMS  paradigm  is  based  on  a  bottom-up  approach,  it  focuses 
on the details of each single class of agents. The macro properties and dynamics of the 
whole system which depict the problem are often lost at the first glance at the structure 
of the model. These may become evident only ex-post, as a result of the simulation. 

22   A complex system is adaptive to changes in its  local environment and in general it is  composed of other 
complex systems, behaves in a nonlinear fashion, and exhibits emergent behaviour (Padula et al. 2014). 

33 

 
 
                                           
Therefore  a  Hybrid  Modelling  approach,  which  combines  SD  structures  with  ABMS 
features,  was  considered  as  a  feasible  development  of  the  i-FRAME  proposal.  This 
approach has the following advantages:  

• 

• 

• 

It captures both high-level structures and single agent behaviour;  

It depends on both (agents') behaviour and structures;  

It uses both stocks and events as sources of the dynamics of the problem;  

•  Finally,  since  the  macro  properties  of  the  system  emerge  from  the  interaction  of 
agents, the use of 'heroic assumptions' is reduced as it is limited and confined to the 
general structure of the problem (the related system). 

However, as the specific review of the state of the art in modelling and simulation clearly 
showed,  a  general  model  which includes  all  the  elements  of  any  single  component  of  a 
complex system does not exist. Thus, any attempt to build a general simulation model to 
study, assess and evaluate the effects (i.e. the outcomes and impacts at different levels 
of  scale:  micro,  meso,  macro)  of  a  specific  policy  initiative  would  not  produce  any 
valuable results. 

As  the  literature  underlines,  a  simulation  model  should  start  from  a  'well-defined  and 
specific  problem'  that  in  turn  has  its  own  specific  characteristics  and  properties  (for  all 
see:  Sterman,  2000).  These  characteristics  must  stem  from  both  the  context  of  the 
problem identified and the experience and knowledge of the problem's stakeholders.  

In other words, we must discard the idea that we can model a general system applicable 
to any identified subject. We can only model a specific problem and,  starting from that 
problem, we can model the system to which the problem belongs. In so doing, we must 
take into account the fact that the higher the number of appropriate contributions made 
by proactive stakeholders, the more effective the model will be.  

A  simulation  models  is  driven  by  a  specific  problem  and  should  be  conceived 
and  developed  to  explain  this  problem.  In  this  perspective,  the  system  represents 
the  set  of  components  surrounding  the  problem  to  be  modelled  in  order  to  understand 
the  dynamics  of  the  surrounding  environment  where  the  problem  is  formed,  fed  and 
generates other effects.  

The validation activities carried out demonstrate that the i-FRAME methodology is able to 
support the development of simulation models for a problem which is both well-defined 
and contains the main key points to be used in any practical simulation applications. 

The  methodological  approach  proposed  was  further  developed  and  tested  with  the 
support of dynamic simulation modelling techniques, which were applied to case studies 
and scenarios of use and led to the i-FRAME 1.5.  

The  key  activity  was  first  to  break  down  complex  problems  into  domains  and  then  into 
domain-related  sub-models.  These  are  simpler  and  easier  to  understand  for  domain 
experts, stakeholders and policymakers (Misuraca and Kucsera, 2016).  

In  this  connection,  the  review  of  literature  on  dynamic  simulation  applications  in  the 
social policy areas addressed, led to the identification of the following main domains: 

•  Relevant population: this sub system depicts the behaviour of relevant populations 
(e.g.  needy  families  living  on  subsidies;  people  with  chronic  diseases;  unemployed 
vocational college students; etc.). 

•  Investment  and  financing:  this  sub  system  includes  aspects  and  indicators  for 
evaluating  the  financial  and  investment  aspects  of  a  problem  (e.g.  use  of  social 
impact bonds; establishing private public partnerships; etc.). 

34 

 
 
•  Impacted domains: the sets of aspects that drive and change the domains that are 
impacted  by  the  social  policy  innovation  initiatives  (e.g.  labour  market;  economic 
development; industrial sector; occupational life course; etc.). 

•  Service  delivery  system:  the  sets  of  aspects  that  drive  the  related  sub-system 
providing  services  to  the  relevant  population  (e.g.  social  security;  care;  education; 
etc.). 

The  general  structure  of  the  domain  'relevant  population'  can  be  represented  using  a 
System Dynamics model, see Figure 9. 

Figure 9: Example of a System Dynamics model of 'relevant population' 

Attribute - structural 

Attribute - rate 

Source: Fair Dynamics elaboration for JRC on a J. Sterman model 

Relationship - logics 

Each  domain  contains  the  outcome/impact  indicators  and  can  be  defined  as  impacted 
domain, as it receives the effects from the other more 'scope-specific' domains that can 
be  called  impacting  domains.  These  impacting  domains  contain  the  resources  that 
deliver the services that affect the 'impacted' domains.  

For simulation modelling purposes, the main focus has to be on capturing and depicting 
the structure of the impacted domains. Usually, the impacting domains are represented 
by  collections  of  variables  acting  on  the  elements  of  the  impacted  domain.  In  practice, 
this  means  that  each  domain-related  sub  model  is  based  on  its  own  internal  structure, 
which  according  to  Sterman  (2007)  can  be  represented  through  a  set  of  attributes  and 
methods, where the attributes represent the variables and can be appropriately divided 
in Structural, Logical and Key Performance Indicators (KPIs). 

Depending  on  the  simulation  methods  used,  these  attributes  and  methods  may  be 
further  subdivided  into  technical  sub  categories  and  cross  comparisons,  for  example 
variables  and  indicators.  The  final  model  will  result  from  the  aggregation  of  different 
domain-related sub models, depending on the specific context, as depicted in Figure 10. 

35 

 
 
 
 
 
 
 
Figure 10: Example of 'domain-related sub-models of a System Dynamics model 

Source: Fair Dynamics elaboration for JRC 

Building  on  the  assumptions  and  key  aspects  introduced  above,  the  practical  steps  for 
implementing  the  proposed  methodology  for  modelling  and  simulation  of  complex 
systems can be summarised by the i-FRAME 1.5 Decalogue presented below. 

1.  Start from a definition of a case/problem/need, and reconstruct the logic model 
which  defines  how  the  case/problem/need  is  addressed  by  the  social  policy 
innovation initiative (see e.g. Epstein and Yuthas, 2014). 

2.  Define the levers for output, outcome and impact assessment according to the 
logic model identified in Step 1, and identify the indicators for impact, outcome and 
output assessment according to the levers defined. 

3.  Identify the impacted and impacting domains of the case/problem/need and 
how  they  are  addressed  by  the  social  policy  innovation  initiative.  This  could 
be  done  by  developing Causal  Loop  Diagrams  (CLD)  that  help  us  to  understand  the 
main  cause-effect  relationships  of  the  problem  under  examination  (Sterman,  2000; 
Forrester, 1994). 

4.  Check  whether  any  similar  dynamic  simulation  models  have  already  been 
developed (cases available in the literature and scenarios of use to be developed as 
part of the i-FRAME Web Platform). 

5.  Look for and check the Attributes and Methods for each domain-related sub-
model  of  the  existing  dynamic  simulation  model,  and  adapt  them  according  to  the 
case/problem/need addressed by the initiative. 

6.  Improve  the  dynamic  simulation  model  adding  the  domain-related  sub-model 
not  already  included  in  the  existing  dynamic  simulation  model  selected  from  the 
existing  ones,  and  complete  the  logical  representation  of  the  case/problem/need 
addressed  by  the  initiative.  To  this  end,  develop  a  methodological  pathway  in 
dynamic  model  development  that  combines  qualitative  (Causal  Loop  Diagram)  and 
quantitative (stocks and flows/agent based models). 

7.  Adapt  and  improve  each  domain-related  quantitative  sub-model  (Stocks  & 
Flows Diagram and/or state charts with analytical description of the state transitions) 
also through Group Model Building (Vennix, 1999; Zeigler  et al., 2000; Vanden belt, 
2004),  and  combine  the  sub-models  in  the  final  dynamic  simulation  model 
representing  the  case/problem/need  addressed  by  the  social  policy  innovation 
initiative.  

36 

POPULATIONFINANCIALSEMPLOYMENTHEALTH…SERVICE PROCESS SERVICE RESOURCES KPIINVESTMENT EFFETCS 
 
 
8.  Define  the  conditions  for  each  scenario  to  be  studied.  To  this  end  aggregate 
approaches  (i.e.  hybrid  models)  can  be  used  to  father  data  and  information  on  a 
specific policy problem, and to build consensus around challenging issues.  

9.  Analyse  the  scenario  through  different  experiments  (by  changing  the  internal 
levers  of  the  model).  This  would  facilitate  presentation  of  results  from  alternative 
scenarios, engaging policymakers' and stakeholders' to concentrate on feedbacks and 
develop an endogenous perspective of the policy interventions and their impacts. 

10. Compare the scenarios and make policy recommendations. 

Of course the above steps should not be considered as the 'perfect scheme' for achieving 
the rather ambitious objective of assessing the impact and making recommendations for 
designing  and  evaluating  social  policy  innovations.  However,  they  can  provide  valuable 
support  to  stakeholders  and  policymakers,  which  together  with  assistance  from  domain 
experts, can help them to achieve realistic impact assessment for a given policy relevant 
initiative. They can also provide general hints on how to identify all the elements needed 
to describe the problem in its full complexity. 

In this respect, it must be noted that this methodological approach should be considered 
circular  and  reiterative  rather  than  sequential.  It  requires  a  greater  involvement  of 
domain experts, stakeholders and policymakers. It is therefore important to foresee the 
implementation  of  a  structured  Group  Model  Building  (GMB)  approach,  together 
with the application of the proposed i-FRAME methodology. The main objective of GMB is 
to enhance knowledge within the team and bring out and share mental models across all 
the  stages  of  problem  modelling  and  simulation  (Tako  et  al.,  2010).  Different  methods 
can be used to engage the various actors in the modelling and simulation process, such 
as a structured series of workshops or policy lab sessions, also depending on the level of 
interaction of the two typologies of actors (Stakeholders and Domain Experts), following 
the various steps of the proposed i-FRAME methodology. 

In order to give a more practical idea of how the i-FRAME methodology could be applied, 
we  describe  below  an  SD  model  developed  by  Sutrisno  and  Handel,  (2011)  to 
understand  the  socio-economic  impact  of  demographic  changes  in  the  German 
population. In this example, the methodological steps of the i-FRAME have been applied 
as follows:23 

  Describe  the  problem/case/need:  the  impact  of  increasing  numbers  of  ICT-

enabled services on the ageing population. 

  Define  the  levers  for  output,  outcome  and  impact  assessment  according  to 
the  logic  model:  resources  allocated  to  ICT-enabled  initiatives  which  promote 
social services for ageing – functioning of social services provided to older people  – 
cost and quality of services provided  – benefits for elderly population (e.g. increase 
of healthy years) – impacts on welfare system (e.g. reduction of care cost and social 
security savings). 

 

 

 

Identify  the  main  domains:  Relevant  Population,  Employment  (Jobs),  Financials, 
etc. 

4&5.  
and Handel, (2011). 

Check the literature for existing Models, Attributes and Methods: Sutrisno 

6,7&8.  Adapt  and  improve  each  domain/sub-model;  combine  the  sub-models  and 
define the levels for output, outcome and impact assessment. The application leads 
to the illustrations reported in  Figures 11 and 12, where are reported as a Causal 
Loop  Diagram,  while  Stock  and  Flow  Diagrams  and  state  charts  -  step  6  of  the  i-
FRAME 1.5 methodological approach - have not been developed for this example. 

23   Steps 9 and 10 of the i-FRAME methodology have not been applied, as they would require the availability 

of a computer-based simulation model. 

37 

 
 
                                           
Figure 11: Example of a dynamic simulation model (in a CLD) assessing the impact of 
ICT investments on ageing 

Source: Fair Dynamics for JRC, adapted from Sutrisno and Handel, (2011) 

Figure  12:  Domain–related  sub-models  of  a  dynamic  simulation  model  (in  CLD) 
assessing the impact of ICT investments on ageing 

EMPL. DOMAIN 

POPULATION 

FINANCIAL DOMAIN 

Source: Fair Dynamics for JRC, adapted from Sutrisno and Handel, (2011) 

38 

 
 
 
 
 
 
 
 
 
 
 
4.3  Insights from testing, validation and review of i-FRAME 1.5 

The  results  achieved  during  the  development  and  testing  phase  of  the  i-FRAME  1.5  included 
the  findings  from  an  ad  hoc  analysis  of  the  state  of  the  art  and  review  of  literature,  which 
showed  that  the  application  of  dynamic  simulation  modelling,  especially  to  social  policies  and 
ICT-enabled  innovation  initiatives,  has  been  limited.  This  was  confirmed  by  the  results  of  the 
testing  and  validation  activities,  which  also  demonstrated  that  social  policy  actors  have  little 
experience  and  capacity  in  implementing  scientific  methodological  approaches  to  assess  the 
impacts of social policy interventions. 

The  proposed  i-FRAME  1.5  methodology  was  designed  as  a  structured  approach  which 
identifies  the  actions  that  should  be  followed  to  shape  a  dynamic  simulation  model  of  the 
impacts  of  social  policy  innovation  initiatives  which  promote  social  investment.  As 
demonstrated in the validation process of the i-FRAME methodology, the circular nature of the 
10 steps which compose the i-FRAME Decalogue facilitate consensus building on modelling the 
structure  of  social  policy  innovation  initiatives  and  their  expected  impacts  on  target 
beneficiaries in quantitative and less subjective ways. 

The  proposed  methodological  approach  was  tested  by  applying  it  qualitatively  to  different 
scenarios of use. Concrete case studies of ICT-enabled social innovation initiatives in different 
welfare  systems  across  the  EU,  each  addressing  different  combinations  of  Personal  Social 
Services of General Interest (PSSGI), were conducted.  

In  addition  to  this,  to  further  demonstrate  the  validity  of  the  methodological  approach 
underpinning  the  i-FRAME  1.5,  and  also  to  test  it  quantitatively,  the  simulation  modelling 
approach  proposed  was  applied  to  a  real-life  case.  We  chose  one  of  the  ICT-enabled  social 
innovation  initiatives  that  belong  to  the  IESI  mapping  sample:  Pathways  Accommodation 
and Support System  (PASS). Pass is a shared client support and bed management system 
for homeless services, and forms part of the priority actions in the National Homeless Strategy 
in Ireland.  

PASS improves the planning, delivery, monitoring, and coordination of services across various 
agencies  from  the  public  and  third  sectors.  The  data  collected  are  linked  to  profile  data, 
assessment  of  housing  and  support  needs  of  homeless  people,  ongoing  support  planning; 
engagement with accommodation, outreach and day services, and reasons for departure.  

The  system  was  launched  in  the  Dublin  region  in  2011  to  prevent  homelessness,  reduce  the 
duration  of  homelessness  to  less  than  six  months  and  ensure  the  delivery  of  services  for 
homeless  people  that  meet  their  needs.  All  funded  services  addressing  homelessness  are 
required to use PASS under Service Level Agreements (SLA). 

This case is very complex, as it implies numerous situations which interact with one another. 
To test the i-FRAME methodology, a simplified model was prepared to capture the main effects 
of the PASS system on a homeless person's transition from initial 'emergency accommodation' 
to more  stable accommodation.24 The model developed is coherent and  reproduces  over time 
data officially published by the Dublin Region Homeless Executive (DRHE).  

Figure 13 displays a Causal Loop Diagram featuring three main stocks in the model: homeless 
people,  people  in  emergency  accommodation,  and  people  in  other  accommodation.  The  flow 
rate is based on the support capacity of the system variable – i.e. transitional support capacity 
-  which  influences  the  rate  at  which  people  go  from  emergency  accommodation  to  other 
accommodation.  

The  whole  dynamic  mechanism  of  the  model  is  based  on  the  variation  in  the  probability  of 
success in transferring people from emergency to other accommodation. This may depend on 
the load of support that, in turn, depends on the PASS's transitional support capacity. In other 
words,  the  PASS  system  influences  transitional  capacity,  i.e.  the  number  of  'figurative' 
operators providing the service and the 'Load' of the Support offered. 

24   The  model  includes  impacted  domain-specific  sub-models  (i.e.  relevant  population,  social  housing  delivery,  care 

service delivery, financing of the intervention and labour market). 

39 

 
                                           
The  model  captures  the  dynamic  relationship  between  the  support  capacity  of  the  services 
offered  through  the  PASS  system  (transitional)  and  the  rate  of  homeless  people  going  into 
each type of service. The logic of the reinforcing loop is:  

People  in  emergency  accommodation  →  Load  of  transitional  support  (affected  by  the  PASS 
system through the change of the support capacity) → Probability of success → In Transitional 
rate → People in other accommodation. The support capacity of the service underlies the loop. 

Figure 13: CLD model of simplified PASS-like case 

Source: Fair Dynamics elaboration for JRC 

To  validate  the  model  quantitatively,  the  simulation  modelling  of  the  PASS-like  model  used 
data from the Homeless Agency Partnership (2008). A Group Model Building Approach was also 
used to present the dynamic simulation model of the  PASS case during an Experts  Workshop 
at JRC-Seville in July 2015.  

Figure  14  shows  the  stock  and  flow  diagram  used  for  the  PASS-like  simulation  modelling 
application. The first stock represents homeless people in the Dublin area.  

Thanks  to  PASS,  the  end  beneficiaries  of  the  service  can  move  to  the  stock  of  people  in 
emergency accommodation, the first type of accommodation provided. The transitional service 
capacity,  fed  by  the  PASS  system,  influences  the  number  of  homeless  people  who  can  enter 
the 'other accommodation' status.  

Accumulated costs represent the budget allocated to provide the services.  

The flows represent the rates of people going from one stock to another.  

Homeless people-in rates indicate the share of the population that becomes homeless.  

The emergency rate regulates the flow of people going into emergency accommodation.  

The  other  accommodation  rate  regulates  the  flow  of  people  going  from  emergency  to  other 
accommodation.  

In  this  case,  ICTs  act  as  enablers  that  increase  the  number  of  figurative  operators  providing 
and  coordinating  services,  which,  in  turn,  increases  the  probability  of  success:  more  people 
moving to housing from emergency accommodations. 

40 

 
 
 
 
 
Figure 14: Stock and Flow diagram of PASS-like model 

Source: Fair Dynamics elaboration for JRC 

The levers that regulate the flows used to calibrate the model are: 

• 

(Q1-2014) New homeless people in emergency accommodation at baseline (437). 

•  Existing homeless people in emergency accommodation at baseline (1,869). 

•  Transitional support capacity, measured by the number of figurative operators affected by 

the PASS system (15). 

•  Homeless  rate,  measured  by  the  number  of  homeless  people  per  week  (from  the  case 

documentation). 

•  Unit  cost,  measured  by  the  cost  per  homeless  person  per  week  (25,000€)  adjusted  to  a 

coherent value for the delivery of the service following discussion with experts. 

•  The time period used to run the model is 78 weeks from Q1-2014 to Q3-2015. 

By  using  the  levers  with  the  above  values,  in  this  timespan,  the  number  of  homeless  people 
amounted  to  3,383,  of  which  3,371  were  in  emergency  accommodation  and  1,449  were  in 
other accommodation (assuming 0 at baseline, Q1-2014).  

41 

 
 
 
 
Figure 15 shows the output of the simulation modelling of PASS. 

Figure 15: Output of simulation modelling of PASS 

Source: Fair Dynamics elaboration for JRC 

The quantitative validation of the PASS scenario indicates that this approach could be used for 
simulating the impact and the functioning of a given policy action involving ICTs as enablers in 
services  planning,  design  or  provision.  Furthermore,  the  proposed  approach  is  also  useful  for 
simulating the impacts of different policy options. For instance, experimenting with the levers 
used for the simulation model shows that, even after changing the number of operators in the 
system,  the  probability  of  success  reaches  an  upper  limit  over  time  and  the  number  of 
beneficiaries no longer increases. 

In this simplified simulation, the effect of ICTs  is represented by an increase in the figurative 
number  of  operators,  which  leads  to  an  increase  in  both  the  efficiency  and  effectiveness 
(probability  of  success)  of  the  initiative.  By  decreasing  the  'load'  or  increasing  the  number  of 
figurative  operators,  the  'probability  of  success'  increases  and  the  'time  to  move  to  housing' 
decreases, making the social intervention more efficient and effective. 

Applying  the  i-FRAME  approach  to  PASS  shows  the  effectiveness  of  this  intervention  for  the 
planning  and  execution  of  the  services.  The  real-time  tracking  of  homeless  people  and  the 
availability of beds made it possible to increase bed occupancy to 99%,  thanks to the sharing 
of  information  between  all  agencies,  other  stakeholders,  and  volunteer  organisations.  This 
ensured efficient use of available resources and reduces duplication of efforts.  

The results from the simulation confirm the estimates of the impact assessment conducted by 
The Dublin Region Homeless Executive (DRHE) using data provided by the PASS system.  

The  introduction  of  PASS  has  allowed  the  local  government  to  make  timely  use  of  the 
information generated to manage, monitor and evaluate the services efficiently. PASS provides 
statistical information on the homeless people's profiles and their use of services that is useful 
for  assessing  the  effectiveness  of  the  strategy,  identifying  emerging  trends  and  monitoring 
service delivery. Every record is unique and tracks a homeless person's progress, and assesses 
his/her income, employment, training, education and health needs.  

42 

 
 
 
Service delivery is improved through the  shared information system that allows agencies and 
other  stakeholders  involved  in  service  provision  to  track  and  share  tasks  and  provide  a  care 
continuum  and integrated  service  delivery.  The  statistics  provided  can  be  used  to  plan  future 
service  developments  and  monitor  the  quality  of  the  services  delivered.  They  also  provide 
insights  into  the  challenges  of  homelessness,  by  profiling  the  characteristics  of  homeless 
people  using  the  services,  and  contributing  to  a  reconfiguration  of  service  provision  to  fulfil 
long-term strategic policy objectives. 

The  initiative  thus  helps  to  optimise  the  use  of  financial  and  human  resources  so  that  an 
essential required service to any household experiencing homelessness can be delivered, while 
at the same time overall costs of homelessness for society are reduced.  

Statistics  show  a  significant  improvement  in  the  number  of  individuals  who  moved  into 
independent  living  accommodation  and  became  integrated  into  society  with  full-time 
employment  and  improved  health.  This  initiative  was  thus  highly  preferable  compared  to 
earlier approaches which aimed to get people 'housing ready'. 

The  results  of  the  this  phase  of  the  research  showed  that  dynamic  simulation  modelling 
can  make  a  considerable  contribution  to  our  understanding  of  complex  social 
systems. More specifically, the literature review and the analysis of the state of the art of the 
application  of  such  approaches  in  the  field  under  investigation,  demonstrated  that  exploiting 
the capabilities of Systems Dynamics (SD) and Agent-Based Modelling Simulation (ABMS) in a 
Hybrid Approach seems to be the most promising option for modelling and simulating complex 
systems such as ICT-enabled social innovation which promote social investment. 

The  analysis  also  demonstrated  that,  without  a  structured  methodological  framework 
like  the  i-FRAME,  the  dynamic  simulation  approach  cannot  be  easily  implemented, 
because it would require a high level of expertise, and also because it would be expensive. 

The proposed methodology suggests that a structured approach to Dynamic Simulation would 
instead  significantly  reduce  the  cost.  At  the  same  time,  through  a  sort  of  'standardisation',  it 
would  also  help  to  engage  policymakers,  relevant  stakeholders  and  domain  experts  in  the 
dynamic  modelling  and  simulation  process  of  a  specific  problem.  Though  each  problem  that 
needs to be modelled and simulated presents its own specificities and peculiarities that cannot 
be  generalised,  the  proposed  methodology  clearly  shows  how  to  reuse  already  developed 
models, with significant savings in time and costs. 

The  i-FRAME  methodology  allows  us  to  represent  problem-solving  as  a  circular  process  that 
first  defines  the  problem  and  then  simulates  the  impacts  of  given  policy  measures.  All  the 
policy  actors  and  domain  experts  participate  actively  in  both  shaping  the  problem  into  a 
dynamic  simulation  model  and  defining  the  most  suitable  policy  instruments.  This  shall 
contribute  to  maximising  positive  impacts  of  the  planned  policy  actions  on  the  target 
beneficiaries.  

The  encouraging  results  of  the  qualitative  and  quantitative  validation  process  and  also  the 
interest stakeholders have expressed in using the i-FRAME methodology clearly open the door 
for  more  extensive  and  systematic  implementation  of  the  proposed  methodological 
approach.  This  would  support  policy  actors  in  simulating  ex-ante,  in-itinere  and  ex-post 
impacts  of  initiatives  where  ICT-enabled  social  innovation  plays  an  important  role  in  the 
governance and delivery of social services. 

However, it would be naïve not to also consider the main problems that modelling and 
simulation  approaches  have.  Several  approaches  based  on  modelling  and  simulation  and 
how  they  are  used  provide  evidence  of  social-economic  impacts  of  policy  initiatives  are 
described in the literature. However, it is well known that solutions generated by a model are 
not the solutions to the problem that could be generated in the real world environment, were 
we  able  to  control  all  the  independent  variables  (Rainey  2004,  Ackoff  1999,  McHaney  1991, 
Neelamkavil 1987).  

How good the solution(s) generated by a model is/are will mainly depend on whether we:  

1) Can understand the problem;  

2) Have the expertise to make the right choice of modelling technique;  

43 

 
3) Know how to use these techniques properly;  

4) Can set up a proper validation process; and  

5) Have the considerable resources and time needed to create a model of complex system. 

Moreover the characteristics and features of complex systems are not 'fixed' or static, and can 
be  grouped  into  two  categories:  those  that  are  evident  at  the  elemental  (micro)  level  of  the 
system, and those that are observed at the macro-level.  

Complex  systems  normally  have  a  large  number  of  elements,  a  certain  number  of  relations 
(interdependence), and a set of shared rules by which they operate. All these features lead to 
a larger set of properties at the micro-level of the system.  

At  macro-level,  there  are  two  sets  of  characteristics.  First,  there  are  the  observable 
phenomena,  which  are  related  to  the  emergent  capability  of  complex  systems,  in  which 
patterns at the macro level that cannot be inferred by simply  analysing the parts in isolation. 
Second,  complex  systems  have  properties,  such  as  resilience,  robustness,  non-linearity, 
flexibility, and fitness (Couture, 2007). 

Reviews of the approach proposed suggest that, in light of the results achieved, the 
i-FRAME initiative  has the  potential to significantly  improve the quality of modelling 
for social policy  innovation initiatives and beyond. It can add real value to national and 
local initiatives  and  thus,  indirectly improve  the  lives  of  EU  citizens.  Although  this  framework 
was intended specifically for the evaluation of ICT-Enabled Social Innovations, its importance is 
much wider: a framework of this kind could be adopted to improve and complement the use of 
computer modelling for policy purposes throughout the EU. 

The  i-FRAME  approach  has  been  developed  in  parallel  with  standards  elaborated  by  national 
governments for policy modelling and analysis (e.g. the UK and the Netherlands). The relevant 
lessons from these different attempts  should be included in the further development of the i-
FRAME.  If  the  standards  and  frameworks  adopted  have  greater  consistency  with  each  other, 
they will probably exert a greater influence on how policy modelling is done across the EU. 

The  i-FRAME  proposal  includes  the  establishment  of  a  library  of  models  that  can  be 
adapted and reused. However, in order for models to be effectively reusable, high standards 
of development and documentation are needed. In the past, models have been very difficult to 
reuse  partly  because  their  development  was  rushed.  However,  persuading  policy  clients  to 
spend  the  necessary  time  and  resources  to  allow  good  model  structuring  and  documentation 
has also been difficult. This conflict needs to be addressed and the further development of the 
i-FRAME could make a contribution in this direction. The i-FRAME identified the difficulty policy 
actors  and  modellers  have  in  working  together,  and  proposed  a  sensible  10-stage  framework 
to  help  structure  this.  In  addition,  experts  involved  in  the  validation  of  the  i-FRAME  believe 
that this framework is timely and necessary and should be further developed. This could save 
time and money, and the emergence of reference models may encourage comparable project 
evaluation across the EU. 

The  i-FRAME  also  envisages  the  development  of  an  interface  to  guide  policy  actors 
through the process of specifying an evaluation model. An interface that fully covers this 
process will be difficult to achieve due to the large variety of needs, contexts, techniques and 
availability  of  data.  However,  an  interface  that  aids  this  process,  by  providing  prompts  and 
guidelines,  is  highly  desirable.  This  interface  should  be  clearly  distinguished  from  other 
evaluation models that may be developed.  

The  studies  carried  out  to  develop  the  i-FRAME  1.5  also  rightly  pointed  out  that  a  variety  of 
modelling techniques may be appropriate in different  circumstances, and proposed hybrids  of 
System Dynamics and Agent-Based Models. They also suggested that statistical models should 
be included. Adherence to traditional schools of modelling should be abandoned.  

Instead, an open and pluralistic approach is recommended whereby each aspect of a 
model is chosen and justified according to the modelling needs and situation. 

44 

 
 
Though the i-FRAME accepts the importance  of being able to monetise the impact of projects 
to  support  the  case  for  their  funding,  it  also  recommends  a  multi-dimensional  assessment 
using a variety of techniques. Guidance as to how these various techniques and elements can 
or should be combined could be further substantiated in the next versions of the i-FRAME. 

More importantly, initially the i-FRAME assumed there was a particular purpose for modelling, 
namely: to assess the impact (direct or otherwise) of initiatives or policies. This is what policy 
actors often say they want before going through the modelling process.  

However, there is a wider range of purposes for modelling which often turn out to be the 
real  goal  in  these  situations.  Much  confusion,  wasted  effort  and  the  failure  of  modelling 
projects  have  resulted  from  the  precise  purpose  for  modelling  not  being  established.  It  was 
therefore strongly recommended that these different purposes are recognised and included in 
the next versions of the i-FRAME. They include: 

•  Predicting the impact of variations of policy or intervention; 

•  Understanding the complex interactions that might occur; 

•  Explaining the observed outcomes of an intervention; 

•  Illustrating the impact of an intervention for the purposes of presenting a project; 

•  Performing a risk analysis of the ways in which projects may deviate from their intended 

outcomes, in order to inform monitoring and management; 

•  Providing a policy 'flight simulator' to train policy actors and sharpen their intuitions; 

•  Eliciting  evidence,  as  part  of  a  cycle  of  extracting  knowledge  from  experts  and 

stakeholders. 

For these reasons it was proposed that in the further development of the i-FRAME, guidelines 
which relate these different purposes to the different techniques available should be 
drawn  up.  The  modelling  technique  chosen  and  the  amount  of  effort  made  will  strongly 
depend on the model's  purpose. This is especially true when policy actors are included in the 
design  and  need  to  learn  how  to  use  models  and  when  modellers  need  to  be  involved  in  the 
policy cycle.  

Moreover, the importance of independent validation of models must be underlined. It 
is  very  easy  to  fool  oneself  with  a  plausible  model  which  seems  to  behave  in  the  right  way. 
However,  one  can  miss  factors  that  turn  out  to  be  crucial  or  one  can  make  an  unreliable 
assumption. If model results are to be used to make costly decisions, then good specifications 
and  verification  methods  are  not  enough  to  ensure  that  the  model  is  sufficiently  robust  and 
reliable.  In  particular,  whilst  stakeholder  validation  is  sufficient  for  some  purposes  for  many 
others  it  is  not.  Guidance  on  the  range  of  validation  techniques  and  when  they  are  needed 
should be included in future versions of the i-FRAME. 

45 

 
 
 
5  i-FRAME 2.0: pluralistic evidence-informed policy-making 

Summary of content of Chapter 5 

This  chapter  presents  the  final  proposal  for  developing  a  methodological 
framework  to  assess  the  social  and  economic  impact  of  social  policy  innovations 
which promote social investment - i-FRAME (V2.0). It is organised as follows: 

§5.1 discusses how the findings of the supplementary systematic literature review 
conducted  in  the  last  phase  of  the  research  have  improved  the  theoretical 
orientations which underpin the i-FRAME.  

§5.2  presents  the  revised  methodological  approach  and  operational  components 
proposed  to  develop  a  comprehensive  framework  for  evidence-informed  social 
policy innovation – i-FRAME 2.0; 

§5.3  provides  an  overview  of  the  research  underway  to  build  the  prototype  of 
some  of  the  operational  components  proposed  for  developing  a  computer-based 
simulation model and web platform for social impact evaluation 

5.1  Conceptualising and contextualising social policy innovation 

As anticipated in  §2.2, the IESI research progressed from its exploratory phase and widened 
the  scope  of  the  analysis  in  order  to  consider  the  broader  concept  of  social  policy 
innovations  which  promote  social  investment,  of  which  ICTs  form  an  important 
component.  This  was  prompted  not  only  by  the  findings  from  previous  reviews  and 
consultation  with  experts  and  representatives  of  stakeholders,  but  also  by  the  need  to 
investigate the possibility of applying the i-FRAME to other policy fields.  

Within  this  context,  social  policy  innovation  refers  to  'social  investment  approaches  that 
provide social and economic returns. It is linked to the process of modernising social protection 
systems  and  redesigning  social  service  delivery  through  innovative  systemic  reforms,  where 
ICTs generally play a key role' (Misuraca et al. 2017c). 

The  main  focus  of  the  analysis  is  on  social  policy  and  services  innovation  that  can  also  be 
defined as 'the design, production, and provision of PSSGI addressing the needs of individuals 
throughout  their  lives,  through  the  reconfiguration  or  recombination  of  practices  across  the 
value  chain  (upstream,  midstream,  and  downstream)'.  This  builds  on  a  previous  definition  of 
ICT-enabled social innovation proposed by the JRC (see Misuraca et al., 2015). 

A typology has been derived from this definition, using the three dimensions described below: 

•  Upstream: the extent to which the  underlying policy is reconfigured. For instance, 
conceptual  innovation  and  new  sources  of  evidence  may  define  new  needs  and/or  new 
target  users  and  redesign  services.  In  addition,  changes  in  policy  orientation  and 
policymakers'  objectives  may  introduce  new  financial  instruments  and  even  redefine  the 
regulatory  framework.  This  dimension  can  take  two  values  (low  or  high)  depending  on 
whether  changes  are  only  at  the  conceptual  and  design  level  or  whether  they  go  all  the 
way to new objectives, funding, and rules. 

•  Midstream:  service  production  reconfiguration  –  or  the  extent  to  which  service 
production  entails  the  integration  and  coordination  of  actors  across  traditional  functional 
units  in  the  public  sector,  and  also  across  other  non-public  sector  providers.  The  aim  of 
integration  is  to  put  the  final  users/beneficiaries  (including  service  intermediaries)  at  the 
centre  and  treat  their  needs  holistically.  This  can  be  simply  sectoral  (i.e.  within  one 
functional unit) or cross-sectoral (across functional units and across public and non-public 
sector actors).  

•  Downstream: service offering reconfiguration - or to the extent to which new services 
are  added  and  new  users  reached  (via  different  channels,  especially  digital)  or  existing 
services are simply rationalised and improved. 

46 

 
 
 
Table 3: Typology of social policy and service innovation 

(A) UPSTREAM: Social Policy Reconfiguration 

(B) MIDSTREAM 
Service 
reconfiguration 

Production 

Low 

Sectoral 

(C) 
DOWNSTREAM 
Services 
offering 
reconfiguration 

High 

Sectoral 
Expansionary 
innovation 

Low 

Sectoral 
Incremental 
innovation 

Source: own elaboration 

High 

Sectoral 

Cross-
sectoral 

Total sectoral 
innovation 

Total 
systemic 
innovation 

Sectoral 
development
al innovation 

Cross-
sectoral 
development
al innovation 

Cross-
sectoral 

Cross-
sectoral 
Expansionary 
innovation: 
i.e. 
Multipurpose 
horizontal 
one-stop-
shops 

Cross-
sectoral 
Incremental 
innovation: 
vertical one-
stop-shops 

An  effort  to  contextualise  is  then  also  required  to  better  understand  the  key  findings  of  the 
literature  review  and  the  revised  proposal  of  methodological  framework  and  operational 
components advanced as part of the i-FRAME 2.0. 

We  can  anticipate  that  the  evaluation  and  measurement  identified  and  analysed  by  the 
scientific and grey literature provide only a few tangential insights into the various typologies 
of  innovation.  Obviously,  this  can  be  seen  as  a  result  of  the  time  lag  between  an  innovation 
being introduced  and  then  analysed,  measured,  and  evaluated.  However,  it  can  also  be  seen 
as  a  lack  of  capacity  as  shown  by  the  complementary  components  of  the  IESI  research, 
namely  the  consolidated  mapping  and  analysis  of  ICT-enabled  social  innovation  initiatives 
across the EU (see Misuraca et al., 2017a). 

One of the key dimensions of innovation –and a main focus of the IESI research - is related to 
the attempt, through service integration and one-stop shops, to provide a single point of 
access  and  delivery  of  services,  which  integrates  their  production  and  administration.  This 
approach  has  been  emphasised  in  the  above  typology,  even  though  the  concept  of  social 
service integration and one stop shops have been around since the turn of the century (Aksim, 
et  al.,  2011;  Champion  and  Bonoli,  2011;  Ditch  and  Roberts,  2002;  McQuaid,  2010;  Minas, 
2014; OECD, 2003; Wiggan, 2007).  

Several  potential  benefits  have  been  identified,  such  as:  enhanced  government  efficiency 
through  the  reduction  of  transaction  costs,  more  responsive  services,  information  and 
knowledge  sharing,  etc.  (Budapest  Institute,  2014a;  Minas,  2014).  Yet,  to  date  there  is  no 
evidence  of  impacts,  and  the  evaluation  studies  on  activation  intervention  'normally  ignore 
various ways of administrating activation programmes' (Minas, 2014, p. S51). It has also been 
stressed  that  there  are  several  challenges  to  achieving  these  benefits  (McQuaid,  2010)  and 
that there is no one-size-fits-all solution to integrated service delivery (Munday, 2007).  

We  will  come  back  to  these  challenges,  as  they  contribute  to  the  discussion  on  the  limits  of 
strictly positivist evidence-based policy prescriptions, and to the need to deal with complexity.  

47 

 
 
 
 
 
It  is  also  worth  recalling  two  hypotheses  formulated  in  the  previous  phase  of  the  IESI 
research. The first one is that countries in which there is a greater level of local autonomy and 
decentralisation  of  funding  for  the  provision  of  social  services  are  more  likely  to  provide  a 
fertile  ground  for  ICT-enabled  social  innovation  with  a  'bottom-up'  approach.  The  second 
hypothesis  is  that  countries  with  a  long  history  of  ICT  use  in  public  administration,  a  public 
sector workforce familiar with ICTs, and a reasonable level of digital competence among their 
citizens,  are  likely  to  adopt  ICT-enabled  social  innovation  faster  than  others,  moving  from 
experimentation  to  deployment  in  different  areas  of  social  services  provision'  (Misuraca  et  al. 
2015, p. 36). However, these hypotheses will require separate ad hoc research. 

In  view  of  the  main  findings  of  the  scientific  literature  (which  mostly  concentrates  on  Active 
Labour  Market  Policies),  it  is  worth  underlining  how  more  recently  integration  and  one-
stop-shops have focused mostly on labour market policies and their integration with 
other social services.  

A review of the literature conducted for DG Employment, for instance, suggests that increased 
unemployment  and  changes  in  labour  market  risks  are  forcing  European  countries  to  change 
their approach. Instead of providing unemployment insurance for the temporarily unemployed 
male  breadwinner  and  social  assistance  to  those  unable  to  work,  they  are  adopting  a  more 
integrated and wider approach.  

It  has  been  observed  that  the  two-tier  system  that  discriminates  between  those  with  stable 
employment  and  those  with  more  unstable  and  precarious  employment  is  no  longer  fit  for 
purpose  (Minas,  2014).  The  former  group  can  rely  on  insurance  and  first-tier  activation 
programmes, whereas the latter has access to second tier benefits such as social assistance at 
the  discretion  of  local institutions.  However,  members  of  the  latter  group  often  have  multiple 
problems that entail contacts with several different public agencies.  

Hence,  there  is  a  move  toward  centring  service  integration  around  employment  and 
the concept of activation, defined by the OECD (2017) 'as a combination of policy tools that 
support and incentivise participation in the labour market and in society, to which other forms 
of social services and benefits are tied'.  

This  trend  has  been  conceptualised  as  'triple  integration'  in  which  three  processes  overlap 
(Clasen  and  Clegg,  2011):  1)  Unemployment  benefits  are  homogenised;  2)  risk  re-
categorisation that weakens the boundaries between in-work and out-of-work benefits (i.e. job 
search  obligations  might  be  extended  to  others  besides  the  unemployed,  implying  that 
boundaries  between  previously  separate  contribution  arrangements  are  erased  and  broader 
categories of risk are formed); 3) Closer links between benefits and activation with increasing 
application  of  job  search  conditionality.  So,  on  the  one  hand  there  is  a  more  integrated 
provision of services but on the other, a wide application of the 'job first' principle.  

One  recent  example  at  European  level  of  this  approach  is  presented  in  the  Council 
recommendations  on  long-term  unemployed  integration  measures  (European  Council, 
2016).  The  recommendations  have  three  objectives:  (1)  increase  coverage  with  higher 
registration  and  active  support  for  the  long-term  unemployed,  (2)  ensure  continuity  and 
coordination  between  relevant  services,  and  (3)  increase  the  effectiveness  of  interventions 
aimed  at  both  the long-term  unemployed  and  employers.  The  core  instruments  proposed  are 
Job  Integration  Agreements  (JIAs)  that  would  link  together  providers,  beneficiaries,  and 
employers  to  produce  a  seamless,  integrated,  and  personalised  stream  of  services  for  the 
Long-Term  Unemployed  (LTUs).  So,  it  seems  that  the  Council  envisions  employment  policies 
(through Job contracts) as the key to social services integration.  

This digression will help us to explain the main findings of the review of the scientific literature. 
However,  this  does  not  mean  that  other  PSSGI  that  cannot  be  tied  to  employment  and  job 
search conditionality are unimportant.  

The supplementary systematic review of literature conducted in the last phase of the research 
aimed to strengthen the theoretical orientations which underpin the proposed approach and to 
provide  further  inputs  into  refining  and  improving  the  methodological  framework  and  the 
operational components proposed to build the i-FRAME.  

48 

 
Integration  of  the  main  findings  of  the  review,  summarised  here  below,  with  a  critical 
analysis  of  orthodox  evidence-based  policy  approaches  presented  later,  will  justify  and 
anticipate the changes in the revised meta-framework and its operational components. 

The literature review identified three separate streams of evidence:  

i)  Experimental and quasi-experimental studies in the scientific literature;  

ii)  Macro-level  indexes  and  indicators  of  broadly-defined  social  progress  (i.e.  beyond  GDPs) 

indicators and approaches;  

iii)  Micro-level impact measurement tools for social innovation and social inclusion.  

There is no overlap between these three streams. For example,  the review found no scientific 
articles dealing with macro and micro-level measurement tools, neither did it find any insights 
from  the  scientific  literature  embedded  in  these  tools,  which  tend  to  be  purely  practical.  The 
scientific literature produces the most important inputs, outputs, and outcomes, but these are 
not used in the other two streams of evidence.  

The  search  across  the  scientific  literature  focused  on  the  following  six  broad  PSSGIs  :  1. 
Employment; 2. Social housing; 3. Child care; 4. Long-term care; 5. Social assistance; and 6. 
Social  inclusion.  The  findings  show  that  in  the  scientific  literature,  most  of  the  relevant 
interventions.  As 
contributions  focus  on  employment  policies,  services,  and 
anticipated above, this can be contextualised and makes sense with respect to the fact that in 
most  countries  the  efforts  are  on  employment,  activation,  and  on  linking  the  two  with  other 
forms of welfare.  

On  the  whole,  the  literature  on  what  works  in  term  of  employment  interventions  is 
inconclusive.  However,  there  are  indications  that  job  search  assistance  is  the  most  cost-
effective  intervention,  whereas  integrated  Active  Labout  Market  Policy  approaches  have 
different effects according to context and target.  

In  addition,  public  work  programmes  are  seen  as  ineffective,  and  the  evidence  on  enterprise 
zones  and  wage  tax  breaks  is  mixed.  The  evidence  on  targeted  interventions  (i.e.  youth, 
women) is limited, but it seems to suggest that these are not very effective,  especially those 
that target the youth. Unemployment Insurance has been shown to be effective in contrasting 
poverty. The psychological and behavioural implications of different approaches to intervention 
implementation  have  been  shown  to  matter.  Finally,  one  of  the  main  shortcomings  of  this 
literature  is  that  the  studies  tend  to  focus  on  the  type  of  intervention  and  the  outcomes 
produced, without considering the different organisational and governance forms under which 
the interventions are implemented. 

At  macro  level,  there  is  a  plethora  of  'Beyond  GDP'  types  of  indexes  and  indicators 
which  aim  to  measure  different  dimensions  of  societal  progress,  but  are  difficult  to 
link.  It  is  worth  noting  that  all  of  these  indexes  and  indicators  include  environmental 
sustainability (costs of economic growth in terms of resource depletion and negative impact of 
pollution on health and overall welfare) as a key strategic dimension. On the other hand, the 
possible link between these indexes/indicators and the actual objects of social policy and social 
service innovation is very weak and distant. The social dimensions considered are very macro 
and  building  a  logical  path  from  micro  interventions  to  these  macro  indicators  would  be  very 
complex  and  long.  Finally,  we  found  no  modelling  simulation  of  the  macro  impact  of  social 
policies and services similar to the one we are proposing for employment policies (see §5.3). 

Most micro-level impact measurements focus on a definition of social innovation that does not 
coincide  with  the  focus  of  i-FRAME  on  social  policy  innovation.  All  these  tools  start  from  the 
perspectives of social enterprises, companies doing CSR, or NGOs. The indicators they use are 
no  longer  relevant  measures  of  social  policy  innovation.  Instead,  these  indicators  will  require 
further development by building on results from case studies.  

An  important  outcome  of  the  review  concerns  the  distinction  between  micro-meso-macro 
levels of analysis. This distinction is made in very different and inconsistent ways in different 
contributions.  Moreover,  it  depends  on  one's  perspective,  and  could  become  a  sort  of  a 
logically articulated Matrioska.  

49 

 
 
For instance, interventions in kindergartens and primary schools for child-care, schooling, and 
welfare  in  a  given  community  can  address:  children;  households;  teachers;  schools;  and  the 
community as a whole at system level. What is micro, meso, or macro in this context?  

While  the  micro/macro  distinction  can  be  difficult,  the  meso  dimension  is  particularly 
problematic.  When  you  move  from  the  community  level  to  the  administrative  unit  (province) 
the community belongs to, how does this distinction change? What happens when you go one 
tier  up  (region)?  When  you  aggregate  at  the  level  of  the  national  system,  is  this  again  meso 
with respect to economy and society as whole, or is this macro with respect to the child-care 
national system?  

In  typical  economic  terms  one  only  has  micro  (individuals,  firms,  organisations,  etc.)  and 
macro  (economy  as  a  whole).  However,  in  addition  to  traditional  micro-  and  macro-economic 
effects,  which  focus  on  measurable  ways  of  describing  social  behaviour,  intermediate  (meso) 
effects  are  considered  in  the  literature  on  evolutionary  economics.25  This  approach  builds  on 
the  argument  that  the  intermediate  (meso)  scale  creates  effects  which  need  to  be  described 
using  different  measurements,  mathematical  formalisms  and  ideas.  Thus,  adapting  the 
thinking of Dopfer (2006) and Dopfer et al., (2004; 2008), it appears that there are important 
structures  which  are  not  reflected  in  traditional  micro  and  macro  indicators  (i.e.  price  signals 
and  supply  and demand  curves,  or  the  large  economic  measures  of  inflation,  Gross  Domestic 
Product, the unemployment rate, and other measures of aggregate demand and savings) that 
need to be considered.  

Therefore, in our revised meta-framework, we keep the meso-level perspective to support the 
understanding  of  what  we  call  'Social  Policy  Innovation  Ecosystems'  (which  are  adapted  from 
what  has  been  conceptualised  as  ICT-enabled  social  innovation  ecosystem  in  §3.2).  The 
analysis  of  the  meso  level  would,  in  fact,  require  the  development  of  specific  metrics  and 
indicators  for  impact  assessment  that  could  be  more  or  less  formalised  depending  on  the 
methodology  that  will  be  adopted  and  the  specific  context  and  policy  domain  under 
investigation.  

Finally,  as  mentioned  above,  the  supplementary  review  of  the  literature  sheds  more  light  on 
one-stop-shops  for  employment  and  social  welfare  services  and  highlights  the  variety  and 
complexity of these services. It also shows that, to date, there is no robust evaluation on these 
forms of institutional innovation. Standard experimental and quasi-experimental studies seem 
to  have  been  unable  to  capture,  within  the  limits  of  their  methodological  requirements,  this 
complexity. In this regard, the contextualisation of the findings from literature review brings to 
the fore the need to establish the epistemological and methodological underpinning of 
the improved meta-framework and the proposed evolution of i-FRAME. We will discuss this 
specifically  in  §5.2,  both  in  terms  of  the  methodological  approach  and  of  the  operational 
components proposed for implementing it.  

To  this  end,  we  need  to  digress  briefly,  in  order  to  debate  the  limits  of  Randomised 
Controlled Trials (RCTs) beyond Evidence-Based-Policy (EBP) positivist prescriptions.  

Evidence-Based  Policy  (EBP)  is  an  approach  that  emerged  in  the  1990s  and  was  especially 
pushed in the UK by the Labour Party under Tony Blair (Cabinet Office, 1999). It applied to all 
sorts  of  different  policy  fields  the  same  principles  that  in  the  1980s  inspired  the  Evidence-
Based Medicine (EBM) movement26.  

Obviously,  there  is  nothing  new  in  policymakers  using  evidence  from  research.  What 
distinguished  EBP  from  previous  use  of  evidence,  however,  was  that  it  tried  to  eliminate  any 
ideological  elements  and  judgments  from  the  formulation  of  policies,  and  to  limit  arbitrary 
decisions  of  professionals  (i.e.  teachers,  field  workers,  policy  officers,  etc.)  due  to  rising 
scepticism  about  their  expertise.  It  sought  a  rational  decision-making  process  which  kept 
politics and deliberation out of policy-making and rested on a positivist epistemology.  

25   See  Dopfer,  Kurt,  (2006),  Dopfer,  Kurt;  Foster,  John  and  Potts,  Jason  (2004  and  2008)  and  for  a  more  recent 

analysis focusing on evaluation of ICT-enabled services, see: Misuraca G., Codagnone, C., and Rossel, P., (2013). 

26  As  defined  by  one  of  the  key  proponents  EBM  is:  ‘the  conscientious,  explicit,  and  judicious  use  of  current  best 
evidence in making decisions about the care of individual patients. The practice of evidence-based medicine means 
integrating  individual  clinical  experience  with  the  best  available  clinical  evidence  from  systematic  research’ 
(Sackett et al., 1996) 

50 

 
                                           
A number of initiatives, think tanks, and policy research centres, established what makes good 
evidence  defining  a  hierarchy  dominated  by  Randomised  Controlled  Trials  (RCTs),  meta-
analysis  of  coherent  sets  of  RCTs  to  deal  with  the  limited  statistical  base  of  each  trial  alone, 
and systematic reviews. All other sources of evidence are either not considered or rated as less 
reliable.  Examples  of  EBP  orthodoxy  are:  Campbell  Collaboration27;  Cochrane  Collaboration28; 
Coalition for Evidence-Based Policy29; Grading of Recommendations Assessment, Development 
and  Evaluation  (GRADE)30;  Scottish  Intercollegiate  Guidelines  Network  (SIGN)31;  What  Works 
Clearing House32; California Evidence-Based Clearing house for Child Welfare33.  

EBP  has  been  criticised  from  various  perspectives,  including  some  that  look  at  the  policy 
processes and at the possible agenda conflicts between  policymakers  and scientists (McMillin, 
2012; Sanderson, 2011; Strassheim & Kettunen, 2014; Torriti, 2010). However, we will focus 
below  on  three  more  'technical'  criticisms,  following  the  analysis  presented  in  brief  by  Munro 
(2014) and at length by Cartwright & Hardie (2012):  

•  Randomised  Controlled  Trials  do  not  provide  sufficient  evidence  to  indicate  that  a  policy, 

which is successful 'there', will work 'here';  

•  EBP orthodoxy implies an inadequate view of causality; and  

•  EPB  approaches  have  limited  objectivity  due  to  social  constructions  and  values  affecting 

conceptualisation and measurement.  

By  discussing  these  criticisms  and  limitations  of  orthodox  EBP,  we  continue  to  lay  the 
foundations of the epistemology of complexity already presented in the initial phases of the i-
FRAME  research  (see  Chapters  3  and  4).  This  will  allow  us  to  identify  the  rationale  for  the 
revised meta-framework and for some of its operational components. 

Even  when  they  are  implemented  according  to  the  best  standards  (e.g.  random  assignment, 
masking and double blindness, etc.), the external validity of RCTs is limited, as is the extent to 
which lessons from one of them can be generalised to other settings. This is particularly true in 
the social sciences, given the complexity that characterises socio-economic ecosystems.  

A  more  concrete  way  of  illustrating  this  is  presented  in  Cartwright  and  Hardie  (2012)  and 
taken  from  an  evaluation  of  an  experiment  carried  out  in  California  to  improve  students' 
reading  scores  (Bohrnstedt  and  Stecher,  2002).  In  the  1990s,  the  Californian  authorities 
wanted to improve students' reading scores by replicating a successful experiment carried out 
in  Tennessee  in  1985  based  simply  on  reducing  the  number  of  students  per  class.  However, 
the Californian experiment yielded no positive results. Why?  

Cartwright  and  Hardie  (2012)  explain  that  this  failure  was  due  to  the  lack  of  what  they  call 
'support  factors'.  The  settings  in  California  were  different  from  those  of  the  original 
experiment; in particular, there was not enough space, nor were there enough good teachers 
for an increased number of smaller classes in the Californian school system.  

This explanation requires a brief digression into the notion of causality and the distinction 
between  a causal description and  a causal explanation.  To move from  'it worked there' 
to  'it  will  work  here',  we  need  to  know  facts  about  causal  principles  and  contextual  factors 
(support factors) both there and here. The recommendation one usually finds that the settings 
should  be  sufficiently  similar  is  too  vague.  In  some  sciences,  there  are  universal  causal 
principles  (i.e.  the  law  of  gravitation).  This  is  not  the  case,  however,  in  the  social  sciences 
where causal principles work ceteris paribus, and may shift, or turn out to be very local. 

27   https://www.campbellcollaboration.org 
28   http://www.cochrane.org 
29   http://coalition4evidence.org 
30   http://www.gradeworkinggroup.org 
31   http://www.sign.ac.uk 
32   https://ies.ed.gov/ncee/WWC 
33   http://www.cebc4cw.org 

51 

 
 
 
                                           
The  attractiveness  of  RCTs  is  that  they  are  self-validating  if  implemented  correctly  (i.e.  they 
ensure  internal  validity)  and  they  do  not  require  that  we  understand  why  a  certain  cause 
produces a certain effect. The How and Why can apparently be ignored. Policymakers 'can take 
a  free  ride  and  have  confidence  that  this  caused  that,  without  wondering  how  it  happened, 
because somebody knows' (Cartwright and Hardie, 2012, p. 124).  

In  practice,  however,  all  the  times  what  worked  there  turns  out  not  to  work  here  shows  that 
things  are  more  complex  and  we  cannot  get  away  from  theory,  and  eventually  deliberation. 
Methodologically,  one  should  make  a  clear  distinction  between  a  causal  description 
and  a causal explanation. In the former, we make a general statement relating two items. 
The classic example is the relationship between flicking a light switch and turning on the light. 
Instead, we make a causal explanation when we can account for the fact that the relationship 
between the light switch and the light may fail if the bulb burns out (Bogliacino et al., 2015). A 
single  RCT  will  provide  only  a  causal  description.  If  we  want  to  design  a  policy  that  will  work 
here,  however,  we  need  to  provide  a  causal  explanation.  Clearly  we  need  a  good  theory  for 
this  (non-overlapping  categories  with  high  explanatory  power)  and  good  operationalisation  of 
measurement.  It  is  also  clear  that  an  experiment  cannot  free  the  social  scientist  (or  the 
policymakers informed by it) from dealing with theory. 

Each single cause, including a policy intervention, is an INUS condition for its effect, meaning: 
'an Insufficient but Necessary part of an Unnecessary but Sufficient  condition for producing a 
contribution  to  the  effect'.  Put  it  another  way,  we  need  to  search  for  consequences  of  non-
redundant  pieces  of  non-necessary  but  sufficient  contextual  causes  (Bogliacino  et  al.,  2015). 
In simpler terms, a policy intervention can rarely cause an effect by itself but does so 
in  combination  with  other  contextual  support  factors.  Moreover,  there  are  many  other 
alternative combinations of factors that can produce the same effect.  

This  means  that  an  intervention  may  have  iatrogenic  (unintended)  impact  on  other  causal 
combinations  that  may  have  caused  the  sought-after  effect  anyway.  The  generalization  of  a 
tested policy is conditional on the equivalence between the policy to be implemented and the 
already  tested  one  and  on  assumptions  over  how  agents  in  different  context  respond  to  the 
tested  intervention.  As  a  result  we  always  have  problems  when  we  extrapolate  units, 
descriptively  different  treatments,  and  observations  to  alternative  settings  (Cronbach,  1982). 
Rigid  counterfactual  protocols  are  instrumental  to  this  process,  but  are  not  a  magic  bullet. 
Experiments provide highly localised evidence, while the intervention itself and the behavioural 
assumptions aim to be very general. Moreover, experiments deal with means, not ends. They 
provide insights into how to accomplish a target but cannot determine which aim to prefer.  

Policy  remains  a  domain  of  competing  interests  where  deliberation  cannot  be  ruled 
out.  Hence,  before  simply  replicating  an  intervention  here  because  an  RCT  showed  that  it 
worked there, policymakers and their advisers must always ask how and why questions, rather 
than  following  the  deterministic  prescriptions  of  orthodox  EBP.  Cartwright  &  Hardie  (2012) 
recommend  that  a  horizontal  search  of  support  factors  and  a  vertical  search  of  causal 
principles at the correct level of abstraction should always be carried out. 

As  explained  by  Munro  (2014),  orthodox  EBP  recipes:  'take  too  simplistic  a  view  of  causal 
processes in the social world, assuming a degree of regularity that allows for generalizing from 
observed causal connections. The complexity of the social world is presented as an obstacle to 
this  assumption.  Instead  of  conceiving  of  causality  as  simple  and  linear,  independent  of 
context  with  controllable  outcomes,  it  is  argued  that  the  social  world  is  an  open  system  in 
which causal processes are non-linear' (Munro, 2014, p. 63). 

As Pawson (2006, p. 18) put it, social sciences try to uncover what looks like a regular pattern 
to often discover that institutional and historical forces, which differ in different settings, shape 
these patterns and make them very local. For example, Munro evaluated a UK intervention for 
child  protection  (2011)  which  aimed  to  improve  the  welfare  of  children  and  young  people  by 
introducing stricter  rules on social workers. But the intervention failed because it had several 
unintended negative effects. Introducing rules which limit what social workers can do reduces 
their  sense  of  satisfaction  and  self-esteem.  The  first  negative  effect  is  an  increase  in  staff 
sickness  and  absence  rates.  The  second  is  an  increase  in  staff  turnover  rates.  In  turn,  these 
can increase the average social worker's caseload, which then leads to social workers spending 
less time with the children and young people and their families. So the policy may produce bad 

52 

 
unintended  consequences.  Worse,  these  negative  effects  can  become  amplified  via  the 
feedback loops. This is a clear example of a policy that was designed without considering the 
complex and non-linear causality that characterise social ecosystems. 

EBP's  claim  to  objectivity  raises  a  number  of  questions  about  the  fact  that  concepts  and 
measurement  in  the  social  sciences  can  be  socially  constructed  and  value-laden 
(Cartwright and Hardie, 2012; Cartwright and Runhardt, 2014; Munro, 2014). Some  of these 
challenges  are  more  politically  motivated.  Some  critiques  assert  that  the  concepts  and 
measurements  used  in  empirical  research  and  ranked  as  best  evidence  by  EBP  are  in  fact 
chosen  and  framed  by  particular  groups,  especially  by  those  in  power.  This  may  bias  the 
research and prevent us from considering the views of less powerful social groups.  

As regards more technical challenges, it is a matter  of fact that  concepts and measures in 
the social world must be socially constructed and entail subjective choices. This may 
limit the generalisation of findings at best, and can produce systematic measurement errors at 
worst.  In  the  social  sciences,  we  do  not  use  or  measure  concepts  like  those  of  the  natural 
sciences. Often social science concepts are unobservable, such as trust and satisfaction, or are 
fuzzy  umbrella  concepts  which  encompass  multiple  dimensions.  These  concepts  may  be 
difficult  to  operationalise  and  lead  us  to  use  proxies  that  sometimes  are  the  source  of 
measurement errors.  

Most importantly, as stated by Max Weber in his famous methodological essay  (1904), 'social 
scientists  must  study  concepts  that  they  and  the  society  in  which  they  work  care  about,  and 
that may be valued or disvalued'. But it is not always easy to use and measure concepts that 
validly reflect the underlying phenomenon and that enable explanation, prediction, and control. 
So, when moving to operationalisation, we may be forced to accept proxy measures that often 
do  not  really  reflect  the  underlying  phenomenon.  This  frequently  results  in  'fractional 
measurement' (Etzioni and Lehman, 1967), which means the lack of correspondence between 
the  explanation  of  a  concept/phenomenon  and  the  operational  definition  shaping  its 
measurement.  Researchers  may  therefore  be  tempted  to  solve  this  problem  by  defining  the 
concept  only  in  terms  of  what  is  operationally  measured,  which  Etzioni  and  Lehman  call 
'concept reduction'. This practice amounts to a sort of reification of measurements that causes 
systematic measurement errors, prevents robust evaluation, and sends the wrong messages to 
policymakers. For all these reasons, and especially because social sciences should study what 
we care about (as posited by Weber), we  cannot but agree with Cartwright's statement that: 
'social science measurement is value-laden through and through. In general, we can say that 
the aims we have with social science measurement decisively influence the results we will find' 
(2014, p. 284). 

5.2  Improved meta-framework and operational components 

5.2.1  The revised meta-framework 

As  previously  mentioned,  the  main  objective  of  this  report  and  of  this  concluding  part  of  the 
research  was  to  refine  and  revise  the  previous  versions  of  the  i-FRAME  meta-framework 
(Misuraca et al., 2015). In the methodology section, we introduced an important change with 
respect  to  the  overall  scope  of  the  focus  of  the  i-FRAME.  We  have  moved  from  a  focus  on 
strictly  defined  ICT-enabled  social  innovation  to  social  policy  innovation  promoting  social 
investment.  

This change underpins the revision of the meta-framework ,though the i-FRAME 2.0 continues 
to build on the key pillars of the original version – for example, the view of the '(digital) social 
policy innovation  ecosystem' (see Figure  5 in §3.2). Although the initial focus was on  ICT-
enabled social innovation, the conceptualisation made in the previous versions of the i-FRAME 
is still fully relevant for the broader definition of social policy innovation adopted in this report.  

This definition still considers ICTs to be necessary horizontal conditions, especially in the light 
of  current  and  emerging  trends  towards  the  digital  transformation  of  our  societies  and 
digitalisation  of  the  labour  markets.  Digitalisation  has  consequences  and  implications  for  the 
future  of  work  and  welfare  systems,  as  pointed  out  the  EU  Pillar  of  Social  Rights,  and  on  the 
White Paper on the Future of Europe on occasion of the 60th anniversary of the Rome Treaties.  

53 

 
The contextual ecosystem within which social policy innovation needs to be considered should 
be fleshed out in both the design and the evaluation phase of any policy, service, programme, 
or initiative.34 At the same time, the complexity of introducing innovations in practice remains 
a crucial pillar of the i-FRAME rationale and implementation. In this perspective recalling what 
has  been  discussed  at  length  in  the  i-FRAME  1.0  and  1.5,  Figure  16  provides  a  stylised 
representation of the elements of complexity in social policy innovation.  

Figure 16: Integrated social policy and service innovation in practice 

Source: Misuraca et al., 2015, p. 66 

For  instance,  this  stylised  representation  could  be  applied  to  disentangle  the  main  sources  of 
complexity  and  challenges  that  must  be  considered  when  launching  joined-up initiatives  such 
as one-stop shops (see §5.1); these aim to provide several PSGGIs with one point of contact 
and  require  governance  and  management  of  partnerships  and  coordination  (see  more  in 
Misuraca et al., 2015, pp. 66-67). 

The initial proposal for the meta-framework advanced in i-FRAME 1.0 was clearly a preliminary 
portrait  of  the  various  levels/dimensions  and  related  methods  and  approaches  that  could  be 
used to better understand and assess impacts of ICT-Enabled Social Innovations. The i-FRAME 
1.5  was  operationalised  in  a  dynamic  manner  with  respect  to  some  of  the  operational 
components (namely the dynamic simulation modelling).  

Figure 17 presents the refined version of the i-FRAME 2.0 as a generic meta-framework which 
can  be  applied  at  different  levels.  Here,  the  crucial  deliberation  and  design  step  has  been 
added.  In  the  previous  i-FRAME  1.0  version  it  was  hidden,  while  in  i-FRAME  1.5  it  was 
incorporated only at a technical level, under the Group-Modelling Building phase (see §4.2 and 
§4.3).  

In  this  improved  approach,  we  move  forward  from  the  idea  of  a  methodological  meta-
framework  towards  turning  the  i-FRAME  into  a  dynamic  'Knowledge  Base  on  Social  Policy 
Innovation' (see §5.3 and Chapter 6). 

34   For the discussion on the ICT-enabled social innovation ecosystem the reader is referred to the relevant report of 
the previous phases of i-FRAME development (Misuraca et al, 2015, pp. 42-46) as well as in brief to §3.2 of this 
report. 

54 

 
 
 
                                           
Figure 17: i-FRAME 2.0: Diamond for Evidence-Informed Social Policy Innovation 

Source: own elaboration 

The  revised  proposal  will  become  clearer  when  we  give  examples  of  the  operational 
components  in  §5.3.2.  As  groundwork  for  this  discussion,  we  briefly  illustrate  the  general 
principles and rationale for the improved proposal for the development of i-FRAME 2.0.  

First  of  all,  following  our  critique  of  an  orthodox  EBP  prescriptive  positivistic  approach,  if  we 
can never be sure that what worked there necessarily will work here, but we still want a policy 
measure to be successful, then we need policymakers to choose between competing proposals 
for  funding  allocation  on  the  basis  of  a  sound  assessment  of  the  support  factors  and  causal 
principles which provide the best possible guarantee that a given policy will work here.  

Obviously,  this  should  not  rely  on  politicians'  own  discretion  and  judgment,  but  should  take 
into account the suggestions from the more traditional and orthodox EBP approaches and also 
the alternatives. For this reason it is important to add the 'deliberation and design phase'. This 
step  will  help  shape  the  design  of  interventions  and  eventually  inform  the  three  phases  of 
evaluation  (ex-ante, 
in-itinere,  and  ex-post).  It  will  produce  gradual  but  constant 
improvements across the cycle depicted in what we have called the  Diamond for Evidence-
Informed Social Policy Innovation. 

In  our  approach,  we  are  not  offering  magic  bullet  prescriptions  as  alternatives  to  those  of 
orthodox EBP, but rather a set of options. These are inspired by methodological pluralism and 
by the recognition that, in social policy, we are not under conditions of paradigmatic normality. 
Instead, we are closer to those of post-normal science. As explained by (Cartwright & Hardie, 
2012, p. 93), it is not possible to provide unambiguous and prescriptive rules for social policy 
that  predict  the  results  of  interventions.  Some  discretion  will  be  needed,  albeit  informed  by 
pluralist, open, and transparent evidence. 

As  we  illustrated  earlier  (see  §5.1),  RCTs  as  such  only  provide  causal  descriptions  and  no 
automatic guarantees that what worked there will work here. In Annex IV, we further specify 
the issues of validity and the problem of generalisation from a single RCT, or from the meta-
analysis of many of them.  

55 

 
 
 
In order to check, in the deliberation phase, whether a successful policy can work in different 
settings,  we  must  look  for  causal  explanations  for  the  'how'  and  'why'  questions,  (i.e.  causal 
principles and support factors).  

This  can  be  done  in  two  ways.  One  is  structural  econometric  evaluation  (Heckman,  2008, 
2010),  which  is  not  considered  in  this  study  or  presented  with  the  operational  components. 
This  is  because  it  has  limited  results  in  the  field  under  investigation  as  discussed  above, 
though it is a consolidated technique.  

The  second  is  to  use  approaches  based  on  a  broadly  defined  theory  of  change  or  theory-
based  evaluation35.  In  theory-based  policy  evaluation  interventions  are  not  seen  as 
monoliths.  In  addition,  beneficiaries  and  stakeholders  are  not  simply  passive  recipients  and 
takers of the 'treatments': their views are crucial to the evaluation. These views are collected 
through  interviews  or  from  relevant  documents  (e.g.  programme  documents,  multi-annual 
plans,  research  agendas,  project  documents)  and  are  treated  as  'theories'  of  change  and 
action.  They  are  used  as  hypotheses 
tested  empirically.  Second,  unlike 
counterfactualism,  context  is  not  controlled  for  statistically.  Instead,  it  is  seen  as  the  key  to 
understanding  the  interplay  between  intervention  and  effects.  Contextual  variables  are 
'measured' from the perspective of involved players and through available external sources of 
evidence (i.e. statistics, review of relevant literature). 

to  be 

The  tools  for  the  proposed  deliberation  phase  are  inspired  by  this  latter  approach. 
Hypotheses  and  theories  can  be  derived  from  RCTs,  meta-analysis,  systematic  reviews,  and 
other mixed methods of evaluation. They should then be tested in the settings where a policy 
is  being  considered,  having  gathered  all  the  relevant  evidence  using  mixed  methods  and 
triangulating different sources of evidence36.  

The theory has to describe  what conditions are needed and  what causal principles may be at 
work for a policy to achieve the desired outcomes. Assumptions may be developed during the 
different phases and steps of an intervention and then checked with available evidence. In the 
deliberation  phase,  this  process  may  identify  where  conditions  may  be  lacking  or  where  the 
assumed causal principle may not work. This may lead to the planned policy being discarded or 
redesigned. The same logic can also be used ex-post to understand why a policy succeeded or 
failed.  

In the ex-ante evaluation phase, traditional Impact Assessment tools are obviously used in 
most  of  the  European  Commission's  evaluation  approaches.  Our  key  and  innovative 
contribution,  especially  with  respect  to  the  available  literature,  are  the  two  macro-level 
modelling simulations proposed (see §5.3). 

Finally, an operational tool that can be used across ex-ante, in itinere, and ex-post evaluations 
is  a  system  of  monitoring  indicators  of  inputs,  outputs,  and  outcomes.  Theory-based 
methods can also be used in itinere and ex post.  

Clearly,  for  the  ex-post  phase  there  are  experimental  and  quasi-experimental  methods, 
which are well known. However, the latter methods are not among the operational components 
proposed  because  they  are  already  recognised  and  methodological  guidance  on  their  use  is 
widely available in literature. 

The  operational components we present next  thus consider the deliberation and design tools, 
ex-ante  evaluation  methods,  measurement  indicators  and  the  experimental  application  and 
extension  of  a  simulation  approach  using  Agent-Based-Modelling.  This  last  component  are 
being  further  developed  and  tested  in  the  research  that  follows  the  research  conducted  to 
propose the i-FRAME 2.0 as part of the IESI project (see §5.3). 

35  The application of approaches of theory-based evaluation has increased in the last twenty years as documented in 
a recent systematic review (Coryn et al., 2011) and are considered as ‘coming of age’ (Mayne, 2012). In simple 
terms all these different authors use a variety of ways of developing a causal modal linking programme inputs and 
activities to a chain of intended or observed outcomes, and then using this model to guide the evaluation.  

36  Qualitative research has been the object of a long and controversial debate  for it is difficulty to establish its own 
equivalent to the three ‘‘truths’ of quantitative methods (validity, reliability, and generalisation). Several scholars 
hold  that  external  confirmation  of  qualitative  findings  through  well  designed  purposive  sampling  checking  and 
methods/sources  triangulation  can  enhance  the  credibility  and  validity  of  qualitative  research  (Bryman,  2006; 
Denzin & Lincoln, 2005; Miles & Huberman, 1994; Seale & Silverman, 1997).  

56 

 
                                           
5.2.2  Operational components of the i-FRAME 2.0 

5.2.2.1  Deliberation and design: from it worked there to it will work here 

Support factors. As illustrated in §5.1, the reduction of class size in California failed to raise 
reading  scores  as  had  happened  in  a  RCT  conducted  years  earlier  in  Tennessee.  Figure  18 
presents an evidence-informed view of the combination of support factors needed and explains 
why the policy failed in California. 

Figure 18: Support Factors - Reading Score Example 

Source: Adapted from Cooper et al., 2006 

The reduction of class size was just an INUS condition (an Insufficient but Necessary part of an 
Unnecessary  but  Sufficient  condition  for  producing  a  contribution  to  the  effect),  and  other 
conditions  were  lacking.  On  the  other  hand,  the  above  combination  of  conditions  is  sufficient 
but  unnecessary,  in  the  sense  that  there  are  other  possible  configurations  of  conditions  that 
can cause the effect sought (improved reading scores).  

Identifying the support factors and the corresponding causal principles at work in one setting is 
not  an  easy  task  and  there  are  no  clear-cut  recipes,  but  the  following  tools  can  help  the 
process of policy design and evaluation:  

(1) Problem tree; 

(2) Ex ante failure scenario with simplified causal model; 

(3) Step-by-step and backward theory-based evaluation thinking; 

(4) Quick exit tree. 

i.  Problem  tree.  As  explained  in  §5.1,  the  European  Council  (European  Council,  2016) 
released  recommendations  on how to better integrate support to the  Long-Term Unemployed 
(LTUs). One of the key instruments proposed included Job Integration Agreements (JIAs) that 
would link together providers, beneficiaries, and employers to produce a seamless, integrated, 
and personalised stream of services for LTUs. Figure 19 shows the problem tree developed to 
justify this recommendation. 

57 

 
 
 
 
 
 
Figure 19: LTU problem tree 

Source: Open Evidence and EY for JRC, 2017 

From the above problem tree, using the reviewed literature, as shown in Figure 20, can then 
be  developed  the  configuration  of  the  support  factors  needed  for  JIAs  to  produce  the  desired 
outcomes.  Decision  makers  should  take  these  factors  into  account  when  introducing  JIAs  in 
their settings. 

Figure 20: LTU intervention support factors 

Source: Open Evidence and EY for JRC, 2017 

58 

Low(cid:9)private(cid:9)investmentsInsufficient(cid:9)job(cid:9)creationBarriers(cid:9)to(cid:9)re-hiringWork(cid:9)does(cid:9)not(cid:9)payInadequate(cid:9)workforce(cid:9)skills(cid:9)Inefficient(cid:9)labour(cid:9)marketsBenefits(cid:9)trapLow(cid:9)skills(cid:9)&(cid:9)qualificationHigh(cid:9)unemployment(cid:9)and(cid:9)inflow(cid:9)from(cid:9)(cid:9)short(cid:9)term(cid:9)unemployment(cid:9)to(cid:9)LTUDRIVERSPROBLEMSCONSEQUENCESSocio-economic(cid:9)ecosystemALMPs(cid:9)not(cid:9)a(cid:9)priorityInsufficient(cid:9)coverage(cid:9)of(cid:9)ALMPsConditionality(cid:9)&(cid:9)behaviourSilos(cid:9)service(cid:9)deliveryLittle(cid:9)individualisedservicesLow(cid:9)transition(cid:9)to(cid:9)employmentPolicy(cid:9)ecosystemLow(cid:9)employers(cid:9)engagementLow(cid:9)employability(cid:9)relevanceDiscontinuity(cid:9)(cid:9)in(cid:9)support(cid:9)to(cid:9)LTUIneffective(cid:9)design(cid:9)(no(cid:9)personalisation&(cid:9)relevance(cid:9)of(cid:9)support)High(cid:9)LTU(cid:9)(cid:9)(cid:9)(cid:9)(cid:9)(cid:9)(cid:9)rates 
 
 
 
 
 
 
 
ii.  Ex-ante  failure  scenario.  An  alternative  instrument  is  to  develop  a  simplified  Causal 
Model  of  the  intervention  (here,  we  are  still  considering  the  JIAs  proposed  for  integrating 
services  for  LTUs)  see  Figure  21.  This  is  a  collaborative  approach  which involves  a  group  of 
policymakers,  stakeholders,  and  experts  in  a  collective  construction  of  an  ex-ante  failure 
scenario. They imagine the policy fails and then draw up a list of factors that are necessary for 
the policy to work. They must envision that the policy has been put in in place as planned but 
things go wrong: i.e. the question to ask is 'What could go wrong and why?'  

Figure 21: LTU intervention simplified causal depiction 

Source: Open Evidence & EY for JRC, 2017 

This approach is similar to the more technical approach of Group Modelling Building discussed 
in  Chapter  4,  which  is  a  crucial  element  of  the  methodology  proposed  in  i-FRAME  1.5.  It  is 
also  becoming  rather  common  in  Policy  Labs  and/or  Systems  Design  /  Design  Thinking 
methods, which are also connected to the methodology proposed for i-FRAME.  

The  structured  approach  followed  in  these  techniques  in  fact  facilitates  the  formalisation  of 
rules and relationships, and it also allows us to identify drivers and barriers, and indicators and 
variables. These can be used as inputs to develop simulation modelling approaches, based for 
instance  on  'Small-world'  system  dynamics  or  more  elaborated  simulation  modelling  methods 
supported  by  computer-based  simulation  tools  and  based  on  formal  structured  scenarios  of 
use.  

iii.  Step-by-step  and  backward  theory-based  evaluation  thinking.  This  approach,  also 
called process tracing aims to confirm the existence  of a causal connection between the start 
and the end of a process or policy by checking, one by one, a series of smaller causal steps in 
between against the available evidence. One good example, reported once again in Cartwright 
and  Hardie  (2012,  p.  102-107),  is  the  application  of  this  approach  to  the  World  Bank  social 
funds by Carvalho and White (2004), see Figure 22. 

59 

INTERVENTIONSOUTPUTSOUTCOMESAwareness(cid:9)&(cid:9)informationindividualisation(behaviourally(cid:9)informed)JIA(cid:9)(joined(cid:9)up(cid:9)delivery)Increased(cid:9)LTU(cid:9)registration(cid:9)&(cid:9)referral(cid:9)to(cid:9)ALMPsActivated(cid:9)LTU(cid:9)and(cid:9)employability(cid:9)relevant(cid:9)supportServices(cid:9)to(cid:9)more(cid:9)engaged(cid:9)private(cid:9)employersLTU(cid:9)in(cid:9)non-subsidized(cid:9)employmentLTU(cid:9)in(cid:9)secure(cid:9)and(cid:9)stable(cid:9)employmentReduced(cid:9)LTU(cid:9)ratesIMPACTLTU(cid:9)offered(cid:9)a(cid:9)JIA(cid:9)before(cid:9)18(cid:9)months 
 
 
 
 
 
Figure 22: Example of 'process tracing' approach on World Bank Social funds 

Source: Carvalho & White, 2004, p. 14S 

By  thinking  through  step  by  step  how  social  funds  may  lead  to  the  desired  outcome  one  can 
derive the support factors depicted in Figure 23. 

Figure 23: Support Factors for Social Funds 

Source: Cartwright & Hardie, 2012, p. 103 

60 

1.(cid:9)Social(cid:9)fund(cid:9)is(cid:9)created(cid:9)as(cid:9)semi-autonomous(cid:9)agency2.(cid:9)Social(cid:9)fund(cid:9)identifies(cid:9)target(cid:9)population(cid:9)and(cid:9)establishes(cid:9)eligibility(cid:9)criteria3.(cid:9)Social(cid:9)fund(cid:9)undertakes(cid:9)promotion(cid:9)(information(cid:9)campaign)4.(cid:9)Community(cid:9)prepares(cid:9)proposal(cid:9)&(cid:9)to(cid:9)applies(cid:9)to(cid:9)social(cid:9)fund5.(cid:9)Subproject(cid:9)is(cid:9)appraised(cid:9)and(cid:9)approved6.(cid:9)Subproject(cid:9)is(cid:9)implemented7.(cid:9)Maintenance(cid:9)committee(cid:9)is(cid:9)formed(cid:9)and(cid:9)receives(cid:9)training8.(cid:9)Subproject(cid:9)produces(cid:9)sustainable(cid:9)benefits(cid:9)for(cid:9)the(cid:9)poor6a.(cid:9)Central(cid:9)line(cid:9)ministry(cid:9)agrees(cid:9)to(cid:9)meet(cid:9)recurrent(cid:9)expenses4a.(cid:9)Community-based(cid:9)selection(cid:9)of(cid:9)subproject(cid:9)ensures(cid:9)community(cid:9)contribution(cid:9)to(cid:9)construction(cid:9)and(cid:9)maintenance6a-Construction(cid:9)quality(cid:9)is(cid:9)supervisedSame(cid:9)assumptions(cid:9)Autonomy(cid:9)of(cid:9)social(cid:9)fund(cid:9)preserves(cid:9)independence(cid:9)and(cid:9)promotes(cid:9)efficiency(cid:9)Anti-theoryGovernment(cid:9)is(cid:9)bypassed(cid:9)and(cid:9)so(cid:9)undeminedPoor(cid:9)communities(cid:9)can(cid:9)collectively(cid:9)decide(cid:9)on(cid:9)their(cid:9)common(cid:9)interest(cid:9)Information(cid:9)biases(cid:9)distort(cid:9)the(cid:9)allocation(cid:9)process(cid:9)against(cid:9)the(cid:9)poorCommunity(cid:9)capacity(cid:9)to(cid:9)implement(cid:9)and(cid:9)maintain(cid:9)investments(cid:9)exists,(cid:9)and(cid:9)is(cid:9)increased(cid:9)by(cid:9)project.(cid:9)Involvement(cid:9)of(cid:9)local(cid:9)government(cid:9)in(cid:9)subproject(cid:9)cycle(cid:9)enhances(cid:9)their(cid:9)capacity.Communities(cid:9)lack(cid:9)financial(cid:9)and(cid:9)institutional(cid:9)capacity(cid:9)to(cid:9)maintain(cid:9)investments.(cid:9)Government(cid:9)ministries(cid:9)are(cid:9)unable(cid:9)to(cid:9)meet(cid:9)recurrent(cid:9)obligations(cid:9)so(cid:9)that(cid:9)facilities(cid:9)are(cid:9)not(cid:9)sustainable.(cid:9)SubprojectProject 
 
 
 
 
iv.  Quick  Exit  Trees.  This  method  is  based  upon  simple  tools  which  aim  to  eliminate/select 
policy  options  by  answering  binary  Yes/No  questions  with  evidence.  The  approach  provides 
clear cut answers and may save efforts if the NO comes up at the very beginning of the policy 
design phase. As shown in Figure 24, the approach can be based on a simple 'pen and paper' 
exercise.  It  could  also  be  supported  by  computational  techniques,  which  would  formalise  the 
results  and  archive  previous  experiences  in  a  structured  manner  in  order  to  build  evidence 
over  time.  As  we  will  see  in  §5.2.3,  this  is  one  of  the  aims  of  the  i-FRAME  Web  Platform 
proposed to be developed. 

This method entails a series of questions on the presence or absence of a condition needed for 
a policy to work. If the first answer is YES, then one proceeds to the next question, but if the 
answer  is  NO  then  one  can  stop  and  discard  the  policy.  The  advantage  is  that  it  provides  an 
unequivocal answer as to whether a policy will work 'here'. This, however, requires that all the 
possible  conditions  are  laid  down  in  the  tree  and  that  evidence  is  gathered  to  answer  all  the 
questions.  

On  the  other  hand,  as  it  requires  a  dichotomous  answer  it  does  not  allow  for  more  nuanced 
answers or scores. The evidence available must be very robust to back the YES or NO answer. 
Moreover,  it  must  be  remembered  that  the  conditions  set  out  in  the  exit  trees  are  only  one 
possible configuration of support factors. If a NO answer occurs, before abandoning the policy 
a different tree with other possible support factors should be tried. 

Figure 24: Quick Exit Tree for Work Ready Policy for Problem Drug Users (PDU) 

Source: Cartwright & Hardie, 2012, p. 115 

Figure  24  shows  an  exit  tree  which  was  developed  by  the  United  Kingdom  Drug  Policy 
Commission (UKDPC) to evaluate a policy to help Problem Drug Users (PDUs) get back to work 
(UKDPC, 2008). This is a typical policy to take people off welfare and into work and was called 
Work  Ready).  Many  conditions  were  considered  and  analysed,  although  the  unrealistic 
character  of  this  policy  could  have  been  spotted  and  effort  saved  if  one  question  had  been 
asked  at  the  very  beginning.  Would  employers  be  willing  to  hire  PDUs?  A  later  piece  of 
research  showed  that  this  was  very  unlikely,  so  the  policy  was  put  on  hold  and  the  UKPDC 
recommended that further evidence be gathered before introducing this policy.  

The  same  kind  of  exit  tree  can  be  developed  for  the  JIAs  intervention  recommended  by  the 
Council and described in the example of LTU policy above (would employers buy-in?) and for 
most  'work  first'  policies  which  link  benefits  to  conditionality  and  to  employers  joining  in  the 
policy.  

61 

Are(cid:9)the(cid:9)PDUs(cid:9)motivated(cid:9)to(cid:9)work?NOExitYESAre(cid:9)they(cid:9)otherwise(cid:9)physically(cid:9)and(cid:9)mentally(cid:9)healthy?YESNOExitDo(cid:9)they(cid:9)have(cid:9)formal(cid:9)skills?YESNOExitWill(cid:9)they(cid:9)have(cid:9)in-work(cid:9)support?YESNOExitIs(cid:9)their(cid:9)drug(cid:9)use(cid:9)stabilized?YESNOExitDo(cid:9)they(cid:9)have(cid:9)soft(cid:9)skills?DO(cid:9)IT(cid:9)Exit 
 
 
5.2.2.2  Ex-ante evaluation 

Figure  25  represents  a  typical  instrument  in  the  European  Commission's  ex  ante  impact 
assessment toolkit, i.e. Intervention Logic Model (again, we use the example based on the 
Council's recommendation for JIAs to combat Long-Term Unemployment – LTU).  

Figure 25: LTU Intervention Logic Model 

Source: Open Evidence & EY for JRC, 2017 

Although  this  tool  entails  a  deterministic  approach  to  the  unfolding  of  an  intervention  and  to 
the  way  it  causes  the  desired  effects,  it  is  nonetheless  a  very  useful and  practical  support  to 
policy  design  and  evaluation.  However,  if  this  tool  is  not  backed  by  the  deliberation  tools 
described  earlier  (as  is  often  unfortunately  the  case,  especially  in  social  policy  innovation),  it 
only serves as a conceptual framework to define monitoring indicators.  

In our approach, we therefore propose two types of innovation: 

• 

the development of a computer-based measurement system with a pre-defined system 
of  indicators  for  monitoring  and  evaluation,  which  is  flexible  enough  for  customization  to 
specific interventions and different contexts, and also 

•  a  simple  toolkit  and  guidelines  for  ex-ante  and  in-itinere  impact  assessment,  to 
support,  the  structuring  and  formalisation  of  policy/programme/project  design  and 
evaluation.  Based  on  a  structured  repository  of  experiences  a  knowledge  base  with 
evidence of results can be built and support better informed decisions. The evidence-base 
gathered could be used to define variable and scenarios of use for modelling and simulation 
of impacts.  

62 

 
 
 
This approach, as we will see in §5.4.3 could be incorporated into the proposed i-FRAME Web 
Platform.  The  MIREIA  electronic  toolkit,  previously  developed  by  JRC  in  the  area  of 
employability and social inclusion, could be adapted and extended for this scope.37 

Another innovation refers to the two experiments with macro-level simulations of the impacts 
of  Active  Labour  Market  Policies  and  of  Mission-Oriented  Social  Innovation  Policy,  which  are 
based on an extended version of the K+S model. This will be briefly explained in §5.4 and in 
Annex II. 

5.2.2.3  Measurement indicators 

As  mentioned  above,  the  intervention  logic  which  underpins  traditional  policy  evaluation 
methodologies  includes  the  definition  and  quantification  of  indicators  for  monitoring.  These 
indicators make it possible to measure results during the implementation of a policy, and they 
also form the basis of evaluation of outputs/outcomes and impacts.  

Measurement  methodologies  are  based  on  accepted  approaches  and  standards.  As  we  have 
already  discussed  them  extensively  in  the  report  on  the  initial  version  of  i-FRAME  1.0,  38  and 
also  in  other  recent  studies  (see,  for  instance,  the  already  mentioned  JRC  MIREIA  Impact 
Assessment Framework, Misuraca et al., 2014), we do not enter into details here.  

Moreover,  as  mentioned  several  times,  the  specificities  and  complexity  of  social  policy 
innovation  limit  the  availability  of  evidence  that  could  be  used  to  operationalise  and 
differentiate measurement indicators for the various areas under investigation. In Figure 26, 
we  therefore  only  present  a  generic  example  of  a  system  of  measurement  indicators  which 
could be adapted to the various areas of analysis and structured as a formal electronic toolkit, 
see §5.4.3. 

Figure 26: Generic micro-level measurement tool 

Source: own elaboration 

37   See http://is.jrc.ec.europa.eu/pages/EAP/eInclusion/MIREIAeI2.html  
38   A set of indicators has been presented in version 1.0 of the i-FRAME (see Misuraca et al., 2015, pp. 59-60 Table 5 

and Table 6) These can be further developed in the i-FRAME system of indicators building on case study analysis. 

63 

 
 
 
                                           
In  Table  4,  we  present  a  set  of  indicators  to  monitor  the  JIAs  recommended  by  the  EU 
Council.  The  operationalised  and  differentiated  version  of  these  measurement  tools  could  be 
produced  as  part  of  the  i-FRAME  2.0  platform.  This  platform  will  also  benefit  from  in-depth 
case studies analysis, consultation, and validation and will be carried out during the course of 
2017. 

Table 4: LTU intervention monitoring indicators (exemplificative only) 

Objective 

Indicator 

Definition 

Increase 
registration rate 
among LTU 
Increase Referral 
to ALMP services 

Increase share of 
activated LTU 

Share of LTUs 
who are 
registered 
Number of 
referrals to 
ALMPs/ services 
included in JIAs 
Activation rates 
of LTU 

More 
personalised 
service provision 

Increase share of 
registered LTU 
who have been 
offered a JIA in 
last 18 months 
Increase share of 
JIA recipients 
who make the 
transition to non- 
subsidized 
employment 
Increase share of 
JIA recipients 
who make the 
transition to 
permanent full-
time contracts 

Increase services 
to private 
employers 

Share of LTUs 
reassessed 
before 
completing 18 
months of 
unemployment 
Share of LTUs 
with JIAs 

Successful 
transitions to 
non- subsidized 
employment 

Successful 
transitions to 
permanent/ 
full- time 
contracts 
following JIAs 

Share of ALMPs 
targeting 
private 
employers 
Share of JIAs 
accompanied 
by employer 
support 

Share of LTUs who self-
report in LFS that they 
are registered 
Average number of 
referrals to ALMPs., 
employment / social 
services in JIAs 
Number of LTUs 
participating in 
employment services and 
ALMPs/ total number of 
LTU 
Number of LTUs 
reassessed/ Total number 
of unemployed for 12- 18 
months 

Number of registered 
LTUs who sign JIAs/ total 
number of registered 
LTUs  

Number of JIA recipients 
in non-subsidized 
employment/ total 
number of JIA recipients 
(6 and 12 months after 
concluding JIAs) 
Number of JIA recipients 
in permanent (and/or) 
full-time employment / 
total number of JIA 
recipients who make the 
transition to employment 
(6 and 12 months of 
concluding JIAs) 
Ratio of expenditure on 
ALMPs / ratio expenditure 
on public works ALMP 
measure 
JIAs linked to an 
employer service/ total 
JIAs 

Source: Adapted from European Commission (2015b, p. 47) 

Unit of 
measurement 
% change over 
baseline 

Sources of 
data 
Labour Force 
Survey (LFS) 

Averages 
absolute values 

Reporting by 
implementing 
organisations 

% change over 
baseline 

% change over 
baseline 

Employment 
Committee 
(EMCO) Joint 
Indicators 
Framework  
Labour Market 
Policy (LMP) 
database 

% change over 
baseline 

Reporting by 
implementing 
organisations 

% change over 
baseline 

Possibly from 
PES 
benchlearning 

% change over 
baseline 

Possibly from 
PES 
benchlearning 

% change over 
baseline 

% change over 
baseline 

Labour Market 
Policy (LMP) 
database 

Reporting by 
implementing 
organisations 

64 

 
 
5.2.3  Prototyping a computer-based simulation model for i-FRAME 

The  JRC  is  developing  a  structured  web-based  environment  to  support  the  IESI  research 
activities  for  data  gathering,  management  and  analysis.  This  web-based  environment  could 
also  be  extended  to  integrate  a  computer-based  simulation  model  for  social  impact 
assessment.  It  would  be  based  on  the  i-FRAME  Web  Platform,  which  could  integrate  a 
number of different software application 'suites' for micro-data gathering and measurement of 
initiatives. These will be collected through the IESI relational database and web-tool developed 
by  JRC  to  support  mapping  activities  and  ad  hoc  in-depth  case  studies.  The  web-based 
environment could also include a more sophisticated platform to support the implementation of 
the operational components proposed as part of the i-FRAME methodological approach.  

As outlined during the  Final IESI Conference held in Brussels on 14-15 March 2017, the back 
end  and  front  end  of  i-FRAME  Web  Platform  could  be  structured  according  to  the  preliminary 
ideas  shown  in  Figures  27  and  28  below.  Figure  27  shows  the  IT  environment  already 
developed  and  functioning  at  the  JRC.  This  includes  the  web-based  relational  database  with 
interactive  dynamic  workflow  and  dashboard  for  data  management;  the  IESI  online  data 
gathering  portal  with  flexible  modular  approach  and  the  Online  Community  and  Research 
Collaboration  portal,  also  connected  to  the  JRC  Science  Hub  and  related  applications 
(accessible both internally and externally to the European Commission, and secured according 
to EC rules).  

Figure 27: IESI Online Knowledge Base and Portal 

Source: Presentation by G. Misuraca, at the Final IESI Conference, Brussels, 14-15 March 2017 

65 

IESI PortalIESI Online SurveyMulti-Modules: 1. Inventory2. Mapping3. Case Studies4. Scenarios of UseOnline Interactive Dynamic WorkflowRelational DatabaseJRC Science HubDashboard  forData management IESI Community & Collaboration Portal 
 
 
 
 
Figure 28 provides a preliminary snapshot of the prototype of the operational components of 
the  final  version  of  the  i-FRAME  currently  under  development.  This  prototype  of  computer-
based simulation model would encompass all the possible levels of analysis by using the same 
structural environment, and allow us to collect data at relevant levels of analysis through the 
definition of measurable and coherent indicators. 

To  this  end,  as  anticipated  in  §5.2.2,  a  number  of  relevant  tools  could  be  adapted  and/or 
extended  in  order  to  integrate  them  into  the  prototype  i-FRAME  2.0  platform,  as  part  of  an 
Interactive  and  dynamic  warehouse  for  Evidence-Informed  Social  Policy  Innovation 
(EISPI)  (see  §5.3).  These  tools  could  include  a  computer-based  problem  tree,  and  an 
adapted version of the JRC MIREIA electronic toolkit for impact measurement, to also support 
other methodologies and techniques, such as ex-ante failure scenarios with a simplified causal 
model;  step-by-step  and  backward  theory-based  evaluation  thinking;  or  the  quick  exit  tree 
mentioned in §5.2.2. 

Figure 28: Preliminary proposal of prototype of i-FRAME 2.0 Simulator Web-Platform 

Source: Presentation by G. Misuraca, at the Final IESI Conference, Brussels, 14-15 March 2017 

In  addition,  an  Interactive  Support  Tool  (IST)  to  funnel  users  to  the  'i-FRAME 
Simulator'  could  be  developed  as  part  of  the  i-FRAME  Web  Platform.  Direct  users  will 
be able to take a flexible approach, according to their degree of expertise and prior knowledge, 
to alternative or complementary simulation modelling (and related software packages already 
available when possible) that could be useful to address specific problems or policy issues.  

The  'i-FRAME  Simulator'  could  be  used  as  a  support  tool  for  policy  modelling  which 
would engage policymakers, representatives of stakeholders, domain experts and modellers, in 
more  or  less  formalised  and  structured  group-model  building  or  policy  lab/system  design 
sessions.  These  sessions  could  be  either  'real-time'  or  'on  demand',  by  setting  up  specific 
virtual policy lab sessions and/or workshops to be attended in person that could also serve as 
capacity building activities. 

66 

=(cid:9)meta-analysis,(cid:9)systematic(cid:9)reviews=ex(cid:9)ante(cid:9)deliberation(cid:9)&(cid:9)design(cid:9)tools=(cid:9)RCTs(cid:9)&(cid:9)quasi-experimental(cid:9)studies=(cid:9)micro(cid:9)level(cid:9)measurement(cid:9)indicators=Ex(cid:9)ante(cid:9)impact(cid:9)evaluation(cid:9)(including(cid:9)modelling(cid:9)simulations)=(cid:9)other(cid:9)ex(cid:9)post(cid:9)evaluationOutputs(cid:9)/outcomesemploymentchild(cid:9)care(cid:9)&(cid:9)welfare(cid:9)Integrated(cid:9)ALMPsTrainingJob(cid:9)searchChild(cid:9)levelHousehold(cid:9)levelCommunity(cid:9)levelParticipation(cid:9)of(cid:9)targeted(cid:9)groupEnrollmentSessions(cid:9)producedPersonalized(cid:9)services(cid:9)providedComplementary(cid:9)welfare(cid:9)benefits(cid:9)grantedServices(cid:9)to(cid:9)private(cid:9)employersComplementary(cid:9)welfare(cid:9)benefits(cid:9)grantedTarget(cid:9)group(cid:9)employment(cid:9)outcomesTarget(cid:9)group(cid:9)quality(cid:9)of(cid:9)employmentTarget(cid:9)group(cid:9)earning(cid:9)levelWelfare(cid:9)interventionsCash(cid:9)transfer(cid:9)interventionsCash(cid:9)transfer(cid:9)interventionsTeacher(cid:9)hired(cid:9)and(cid:9)trainedDrop(cid:9)(cid:9)out(cid:9)rateProgression(cid:9)and(cid:9)cognitive(cid:9)outcomes 
 
 
 
The  JRC  has  been  experimenting  with  specific  software  packages  already  on  the  market  for 
system dynamics and their possible hybridisation with Agent-based Modelling Simulation (e.g. 
Vensim, Anylogic) with a focus on ICT-enabled social innovation (as described in Chapter 4). 
In  addition,  the  JRC  is  currently  developing,  with  the  support  of  both  external  and  in-house 
resources, the prototype of some components for developing the i-FRAME Web Platform.  

In this context, a specific experiment  is being conducted on the use and extension of Macro-
economic  Agent-Based  Models  (MABMs)  from  the  'Keynes  plus  Schumpeter'  (K+S) 
family  (Dosi  et  al.,  2010;  Dosi  et  al.,  2014;  Fagiolo  and  Dosi,  2003;  Gerst,  et  al.,  2013; 
Lamperti, et al., 2016a; Roventini and Fagiolo, 2016).  

Based on the findings of the literature reviews conducted as part of the i-FRAME development, 
in  both  versions  1.5  and  2.0,  these  are  considered  to  be  more  flexible  tools  than  other 
traditional  models  (e.g.  QUEST,  RHOMOLO,  GEM-E3,  NEMESIS).  They  allow  a  fine-grained 
analysis  of  microeconomic  dynamics  leading  to  a  variety  of  aggregate  phenomena,  including 
but  not  limited  to,  self-sustained  growth,  employment  fluctuations  and  greenhouse  gas 
emissions.  More  details  on  the  model  proposed  and  the  experimental  simulations  being 
designed are reported in §5.3.  

Here below we briefly anticipate some of the technical features  that will be developed as part 
of  the  prototyping  of  an  interactive  support  tool  to  funnel  users  to  the  'i-FRAME 
Simulator', with specific regard to the extended K+S Modules developed ad hoc for i-
FRAME  (SIT-K+S).  This  tool  will  provide  a  user-friendly,  web-based  interface  to  the  K+S 
simulation  model  for  policy-making  and  general  usage  (K+S  Web  Interface  -  KSWI).  It  will 
offer a basic subset of the computer tools available for the configuration, execution, analysis of 
data  produced  by  existing  K+S  model  versions  and  the  export  of  this  data  to  other  software 
packages.  

The K+S Web Interface has a front- and a back-end, which communicate to allow the user to:  

(1) configure  existing  models  (i.e.  change  the  model's  parameters  and  initial 

conditions and define the simulation settings),  

(2) run the configured models (showing the execution status),  

(3) analyse  the  model  results  (presenting  the  produced  time  series  in  tabular  and 

graphical formats), and  

(4) export results data (as text files). 

The  front-  and  the  back-ends  will  communicate  over  secure  standard  HTTPS  protocols.  Other 
than  that,  KSWI  does  not  include  any  special  security  option.  User  sessions  are  open  (no 
id/password  required)  and  no  user  data  is  kept  on  the  server  side  (except  non-identified 
temporary configurations and simulation results data).  

KSWI  will  be  provided  with  non-exclusive  use  rights  on  the  corresponding  compiled  code, 
according to the  GNU  General Public License version 3 (GPLv3). KSWI, and the  prototypes  of 
the  other  front-end  components  of  the  i-FRAME  web  platform  will  be  developed  using 
standards-based  technologies  whenever  possible,  preferably  provided  under  the  open/public 
licensing models. 

The prototype of the software applications which compose the final version of the i-FRAME 2.0 
as  a  computer-based  simulation  model  could  be  finally  integrated  into  the  i-FRAME  Web-
Platform  and  relational  database,  developed  and  hosted  at  JRC-Server  and  compliant  with 
Commission IT governance rules and regulations. 

67 

 
 
 
 
5.3  Less is more: from a 10 to a 6-step approach beyond i-FRAME 2.0 

5.3.1  i-FRAME 2.0 approach for evidence informed social policy innovation 

In  this  final  section  on  the  revised  i-FRAME  2.0,  we  further  substantiate  how  the  framework 
could become an effective tool for evidence informed social policy innovation. To this end, the 
10-step  Decalogue  presented  in  version  1.5  of  the  i-FRAME  has  been  revised  to  include  not 
only  the  insights  from  the  final  phase  of  the  research  but  also  the  preliminary  results  of  the 
testing and validation phase of i-FRAME 2.0. Thus, by looking beyond i-FRAME 2.0 it is possible 
to envisage what could be developed from 2018 onwards.  

The main aims of i-FRAME 2.0 are to: 

1)  develop a dynamic 'Knowledge Base on Social Policy Innovation'; 

2)  design a fully-fledged i-FRAME simulator of social policy impact; 

3)  contribute  to  monitoring  the  implementation  of  the  EU  Pillar  of  Social  Rights  and  Member 

States' policies for modernising Social Protection Systems. 

As  regards  objective  3, it  must  be  said  that  a stable  system  of  national points  of  contact  will 
have to be established in order to monitor the implementation of the EU Pillar of Social Rights 
and Member States' policies for modernising social protection systems. Alternatively, a multi-
year benchmarking study could be set up. Further review of the literature and case studies to 
be conducted under the future i-FRAME research could provide exploratory evidence and help 
design  the  future  monitoring  systems.  These  could  then  support  the  establishment  of  a 
structured  approach  to  monitoring  social  policy  implementation  that  could  be  included  in  the 
European Semester.  

Once  set-up  and  fully  functioning,  the i-FRAME  Web  Platform  and  Simulator  would  be  able  to 
provide scientific advice to relevant actors, at Member State or Commission level, including the 
newly  established  Structural  Reform  Support  Service  of  the  European  Commission's  General 
Secretariat.  This  advice  could  support  evidence-informed  social  policy  innovation  and  help 
Member  States  to  design  more  effective  labour  market  policies  and  modernise  their  social 
protection systems.  

Yet, for our purpose, the important developments concern objectives 1 and 2, as they will be 
the  key  pillars  of  the  i-FRAME  Platform  and  the  operationalisation  of  the  methodological 
approach  proposed.  In  this  regard,  as  already  anticipated  in  §5.2.3  the  envisioned  structure 
and high-level functionalities of the i-FRAME Web Platform shall integrate a number of different 
software application 'suites'.  

These  should  be  linked  either  to  the  micro-data  gathering  and  measurement  activities  of 
initiatives  collected  through  the  IESI  relational  database  and  web-tool  developed  by  JRC  to 
support  the  mapping  activities  and  ad  hoc  in  depth  case  studies,  or  to  the  computer-based 
platform to support the simulation modelling proposed as part of the i-FRAME methodology.  

Here, we focus briefly on the methodological approach to implementing what we have labelled 
as 'Evidence-Informed Social Policy Innovation' warehouse. Whereas some components 
of the i-FRAME 1.5 Decalogue still underpin the final proposal for the development of i-FRAME 
2.0,  an  important  element  of  the  revised  methodology  is  that  it  is  not  tied  to  any  particular 
technique  and  is  more  generally  usable  regardless  of  the  specific  support  tools  one 
want/need/can use.39  

Furthermore,  the  improved  approach  for  i-FRAME  2.0  envisages  that  some  or  all  of  the  steps 
we  propose  could  be  conducted  as  policy  lab  sessions  where  group  work  would  enable  us  to 
extract  knowledge  from  experts  and  users  across  the  various  dimensions  of  the  'diamond' 
depicted  in  §5.1,  i.e.  Deliberation  &  design  followed  by  ex-ante,  in-itinere  and  ex-post 
evaluations.  

39   The i-FRAME 1.5 Decalogue, in fact, was instrumentally shaped around a specific tool and approach: Small System 

Dynamics based on a formalised Group Model Building (GMB) procedure. 

68 

 
 
                                           
This  interactive  approach  would  help  the  i-FRAME  to  become  a  reference  methodological 
framework  to:  a)  guide  policymakers  and  practitioners  on  what  evidence  and 
tools/methods  are  available  for  their  needs,  and  b)  help  them  run  simulations 
and/or measure and evaluate impacts. Whereas the former largely covers the deliberation 
and design phase, the latter covers all other dimensions of the diamond depicted above.  

The  proposed  approach  can  be  implemented  by  applying  the  following  steps  and  using  the 
operational components and tools envisaged to equip the i-FRAME Platform: 

1.  Problem  and  ecosystem  functioning.  Define  the  problem  an  intervention  aims  to 

address within the functioning of a given ecosystem; i.e., problem tree; 

2.  Interactive discussion on causal relationships. Engage stakeholders and experts in 

a discussion on possible causal logic; i.e., theory-based thinking, quick exit trees; 

3.  What  worked  elsewhere.  Engage  stakeholders  and  experts  in  a  discussion  of  the 
suitability for a given intervention of what worked elsewhere; i.e., interactive evidence-
informed social policy innovation warehouse (discussed in more detail in §5.3.2); 

4.  Decide  and  design  the  intervention.  The  final  deliberation  and  design  of  an 
intervention  must  follow  formal  and  prescribed  country-specific  rules.  Yet,  in  a  policy 
lab context, this step could be an exercise and should strategically embed the suitable 
measurement  and  evaluation  methods/tools  in  the  design  of  the intervention  from  the 
very beginning; 

5.  Identify key variables. This to a large extent corresponds to steps 2 and 3 of version 
1.5, namely definition of the input, output, outcomes, and impacts to be measured and 
evaluated.  It  can  be  implemented  using  'traditional'  tools  such  as  intervention  logic 
models, Causal Loop Diagrams (CLDs), and other support techniques, including Group-
Model Building, Focus Groups; Interviews; Experts Insights; etc. 

6.  Run  simulations  and/or  measure/evaluate  (ex-ante,  in-itinere,  ex-post). 
Depending  on  the  previous  steps  and  on  the  nature  of  the  intervention,  simulations, 
measurements,  or  evaluations  could  be  run  using  the  insights  from  the  evidence 
warehouse.  In  addition,  already  developed  simulations  and  models  could  be  used  (or 
re-used  where  possible)  according  to  hypothetical  scenarios  of  use  or  tested  against 
real-life case studies. In this case, there are several possible tools which could be used, 
depending on the resources and scope of the future research activities. Besides i-FRAME 
2.0, specific experiments could be conducted using either the methods proposed in the 
i-FRAME  1.5  (namely,  a  hybridisation  of  System  Dynamics  and  Agent-Based  Modelling 
Simulations)  or  a  further  development  of  the  experiments  and  modules  prototyped  ad 
hoc for ALMP using the K+S simulation model (see §5.2.3 and below). 

5.3.2  Evidence Informed Social Policy Innovation warehouse 

In view of the revised meta-framework and of the six steps described above, we propose that 
Objective 1  (i.e. the development  of a dynamic 'Knowledge Base  on  Social Policy Innovation) 
could  lead  to  the  production  of  a  pluralistic,  open,  interactive  and  ICT-enabled 
warehouse for evidence-informed social policy innovations.  

Figure  29  shows  what  this  warehouse  would  look  like  for  two  different  policies  (i.e. 
employment;  child  care  and  welfare.  This  is  just  a  diagram  which  has  been  filled  in 
hypothetically. It is limited not only with respect to the many other policy domains that could 
have  been  added,  but  also  in  interventions  (the  rows)  and  in  outputs  and  outcomes  (the 
columns) included for the two domains exemplified.  

Imagine that this diagram is placed into a web-based interface that is interactive and dynamic, 
and that it is accessed by a  policymaker or practitioner who is interested in, for example, the 
employment  outcomes  of  integrated  Active  Labour  Market  Policies  (ALMPs).  He  or  she  would 
be led to structured and organised information on the various tools and sources of evidence by 
clicking on different coloured circles. He/she could also be led to tutorials or simulations, and 
to the various tools illustrated as possible operational components in §5.2.  

69 

 
This interface would become an invaluable instrument for the community of practitioners and 
experts, especially in policy lab sessions or in more traditional capacity building activities.  

This vision embeds the lessons learned from the critiques we made of orthodox EBP and, thus 
the  warehouse  shall  gather  evidence  based  on  a  wide  range  of  methodologies,  and  shall 
substitute hierarchies of evidence with the contextualisation of what is useful for each purpose. 

Figure  29:  Interactive  and  dynamic  warehouse  for  evidence-informed  social  policy 
innovation 

Source: Open Evidence for JRC, 2017 

To build a warehouse of this kind, greater effort and time are needed. Possibly, at the end of 
this piloting phase, the results achieved will lead to a new research stream, along the lines of 
the suggestions advanced in Chapter 6 of this report.  

The experimental macro-level simulations, which are carried out using the K+S Macro Agent-
Based Model, represent a first step towards this broader goal. After that, further progress will 
require  to  develop  new  modules  of  the  K+S  model,  and  to  test  and  develop  alternative  tools 
(i.e. System Dynamics, Discrete Event Models, hybridisation of System Dynamics with Agent-
Based Modelling Simulations). 

A roadmap for building a fully-fledged i-FRAME simulator of social policy impact, both in terms 
of further K+S improvements and of using alternative instruments, will be produced at the end 
of the development phase and the pilot of the prototype i-FRAME 2.0 in 2018.  

5.3.3  i-FRAME 2.0 extended modules of the K+S Macro Agent-Based Model 

As an illustration of the i-FRAME 2.0 simulation modelling component proposed, we anticipate 
below  the  approach  being  piloted  and  tested  to  adapt  and  extend  the  family  of  'Keynes  plus 
Schumpeter'  Macroeconomic  Agent-Based  Models  (K+S  MABMs)  for  Active  Labour  Market 
Policies  (ALMP);  and  the  preliminary  conceptualisation  for  developing  a  scenario  of  use  on 
Mission-Oriented Social Innovation Policy (MOSIP). 

The  K+S  MABMs  has  been  developed  at  the  Scuola  Superiore  Sant'Anna  (Dosi  et  al.,  2010; 
Dosi  et  al.,  2014;  Dosi  et  al.,  2016a,  2016b;  Fagiolo  and  Dosi,  2003;  Fagiolo  and  Roventini, 
2016; Lamperti et al., 2016a; Lamperti et al., 2016b).  

70 

 
 
 
The  simulations  to  be  conducted  using  the  K+S  model  as  part  of  i-FRAME  2.0  must  be 
understood as pilot explorations. They represent one of the many possible  policy domains and 
modelling approaches that could have been chosen.  

Labour  market  policies  and  services  are  one  area  among  the  many  that  are  included  in  the 
concept of PSSGI. The special focus on the labour market was the choice of the research team, 
justified  on  two  grounds.  First,  it  is  uncontroversial  that  unemployment  and  long-term 
unemployment  are  among  the  most  pressing  social  problems  at  the  moment  and  are  social 
policy priorities for both the EU and the Member States. Second, it is in this domain  that the 
systematic review found the most solid empirical evidence, which is being used in developing 
the ad hoc module of the K+S model for i-FRAME.  

A  Macro  Economic  Agent-Based  Model  was  also  a  choice  made  from  the  growing  family  of 
modelling  tools  (which  have  been  reviewed  as  part  of  the  i-FRAME  research).  This  does  not 
mean that i-FRAME has abandoned the focus on social science approaches to embrace a more 
economic form of orthodoxy. First of all, the K+S model is very different from orthodox macro-
economic  models,  and  in  its  logic  there  is  space  for  the  role  of  institutions.  In  addition,  the 
assumptions about the behaviour of individuals in a K+S model are inspired by the concept of 
bounded rationality and the application of behavioural rules based on  empirically-documented 
heuristics.  

Needless  to  say,  we  pay  a  price  for  the  macro  perspective  in  terms  of  the  granularity  of  the 
transmission  mechanisms  of  the  various  possible  policy  innovations.  On  the  other  hand,  this 
perspective  provides  some  scenarios  of  potential  macro  impacts  that  could  encourage 
policymakers to act and researchers to develop more granular micro-simulations.  

The K+S family of models builds on evolutionary roots  (Nelson & Winter, 1982), but it is also 
in  tune  with  genuine  Keynesian  insights  (see  e.g.  Stiglitz,  1994a).  It  tries  to  explore  the 
feedback  between  factors  influencing  aggregate  demand  and  those  driving  technological 
change.  In  doing  so,  it  begins  to  offer  a  unified  framework,  which  accounts  for  long-term 
dynamics  and  short-  and  medium-term  frequency  fluctuations  at  the  same  time.  The  K+S 
model is structural, in the sense that it explicitly builds on a representation of what agents do, 
how they adjust, how they interact, and how they respond to policy changes.  

Recently,  building  on  Dosi  et  al.  (2010),  the  K+S  model  has  been  extended  to  account  for 
firm-worker interactions in a decentralised labour market (Dosi et al. 2016a, b). The model is 
portrayed in Figure 30 (see Annex II for a full description).  

Figure 30: The K+S model with the labour-market extension 

Source: Sant'Anna School of Advances Studies for JRC, 2017 

71 

 
 
As  part  of  the  final  phase  of  the  IESI  research  a  specific  module  of  the  K+S  for  i-FRAME  has 
been  developed  and  piloted  on  Active  Labour  Market  Policies  (ALPM).  In  particular,  the 
rationale  for  developing  the  K+S  i-FRAME  module  on  Active  Labour  Market  Policies 
(ALMP) and some preliminary results of the simulations conducted are presented here below. 

Since  2008,  the  world  economy  has  been  facing  a  severe  recession.  The  European  economy 
appears to be in a vicious circle of low growth, stagnating productivity and wages. Notions like 
jobless  recovery,  unemployment  hysteresis,  and  secular  stagnation  have  gained  momentum 
and are receiving a lot of attention in both academic and policy debates.  

Some  long-term  patterns,  particularly  apparent  in  the  US  economy  but  generally  occurring 
across developed countries, indicate the emergence of new and worrying trends both in labour 
markets and the whole macroeconomic machinery. These are:  

i) 

de-industrialisation; 

ii) 

stagnant wages and divergence between productivity growth and wage growth; 

iii)  declining labour share and related, massive surge in corporate profits, especially financial 

ones;  

iv)  declining labour force participation; 

v) 

declining business dynamism and net job creation; 

vi)  soaring inequality; 

vii)  skills polarisation; 

viii)  growing number of part-time jobs; 

ix) 

longer unemployment spells; and  

x) 

hysteresis40.  

Technology  is  probably  only  one  of  the  original  causes  of  these  long-term  patterns. 
Institutional  determinants  like  globalisation,  financialisation,  labour  market  liberalisation,  the 
declining  share  of  both  manufacturing  and  union  density  are  also  playing  a  major  role  in  the 
current  phase  of  capitalistic  re-organization,  triggering  potentially  prolonged  phases  of 
economic instability.  

Policy  institutions  need  to  understand  what  instruments  can  be  used  to  mitigate,  and 
potentially reverse these long-term trends. They also need to accurately assess the short- and 
long-term effects of different types of structural reforms on the labour markets. 

As part of the pilot and testing of  i-FRAME 2.0, we used the K+S model to study the relation 
between  the  institutional  conditions  of  the  labour  market,  declined  under  two  alternative 
'Fordist' and 'Competitive' scenarios, and the effects of labour market structural reforms.  

In particular, we studied the effects of structural reforms which aimed to:  

i) 

increase numerical flexibility,  

ii) 

lower the pass trough of productivity growth upon wage growth,  

iii) 

reduce unemployment benefits and minimum wages.  

To study these policies, we tested different types of firing schemes, which go from temporary 
to  permanent  type  of  contracts.  We  detected  that  structural  reforms  do  indeed  affect 
aggregate  macroeconomic  performance,  in  terms  of  productivity  and  GDP  growth  rate  and 
variance, unemployment, personal and functional income inequality.  

Simulation  results  show  that  structural  reforms  which  aim  to  increase  the  flexibility  of  the 
labour  market  worsen  the  performance  of  the  economy.  More  specifically,  they  increase  GDP 
volatility, the unemployment rate, and inequality (see e.g. Figures 31 and 32).  

40   Hysteresis refers  to  an even that  persists  in  a  system  even  when  the factors  that  caused  it  no  longer  occur. For 
instance, the unemployment rate after a recession may remain at a higher level than before the recession, even 
after GDP goes back to normal growth.  

72 

 
 
 
                                           
Figure 31: Labour market  structural  reforms:  impact on  unemployment  and  vacancy 
rates 

Source: Sant'Anna School of Advances Studies, 2017 

Figure 32: Labour market structural reforms: impact on inequality 

Source: Sant'Anna School of Advances Studies, 2017 

73 

 
 
 
 
 
In order to pursue the simulation objectives and extend the K+S model, it will be enriched  to 
account for: a) an endogenous process of skills accumulation by workers; b) a state-dependent 
process of entry; and c) a process of discouragement in the job-search activity.  

Detecting  the  sources  of  hysteresis  will  allow  policymakers  to  better  design  ALMP.  In  fact, 
hysteresis  is  both  a  micro  and  macroeconomic  phenomenon,  according  to  which  transitory 
shocks may last for long periods, preventing the system from returning to the pre-shock level. 
For example, a recession can persistently increase the structural unemployment rate.  

Recent evidence suggests that hysteresis is becoming ubiquitous, particularly in the aftermath 
of  the  Great  Recession.  There  are  many  potential  sources  of  hysteresis,  which  can  be 
particularly  pronounced  during  downturns,  for  example:  declining  innovation,  diffusion,  and 
investment rates due to the lack of demand opportunities; lower entry dynamics due in part to 
the  tightness  of  credit  markets;  skills  deterioration  processes  of  firm-based  capabilities  and 
labour force discouragement.  

The latter are some of the causes  of long-term unemployment and permanently lower output 
growth. The K+S model can account for the endogenous emergence of hysteresis in GDP and 
unemployment  and  detect  to  what  extent  structural  reforms  which  aim  to  make  the  labour 
market more flexible might favour the emergence of hysteresis.  

Preliminary simulation results show that labour-market structural reforms do indeed favour the 
emergence  of  hysteresis  and  worsen  both  the  short-  and  long-term  performance  of  the 
economy (see Figure 33). 

Figure 33: Labour market structural reforms and the emergence of hysteresis 

Source: Sant'Anna School of Advances Studies, 2017 

If growing numbers of  workers  experience long term unemployment spells (which is the case 
in many Mediterranean regions), the labour force participation rate may fall, as workers will be 
discouraged  from  searching  for  a  job.  After  modelling  the  pattern  of  increasing  labour  force 
discouragement  and  decreasing  participation  rates,  we  will  have  more  detailed  diagnostic 
instruments  to  examine  the  effects  of  a  set  of  ALMP.  If  there  were  no  diagnostic  phase  in 
which  the  model  can  be  enriched  and  tested  to  account  for  the  empirical  stylised  facts,  the 
policy exercises could not be undertaken.  

74 

 
 
 
After  the  empirical  validation  of  the  new  mechanisms  inbuilt  into  the  model  it  would  be 
possible  to  carry  out  policy  experiments  to  understand  how  hysteresis,  for  example,  can  be 
cured  by  active  intervention  on  the  part  of  institutions.  In  particular,  we  might  then  test  the 
effects of: a) Active labour market training policies directed at different cohorts of workers; b) 
Policy  aimed  at  public  skills  enhancement;  c)  Unemployment  insurance  vis-à-vis  permanent 
lay-off; and d) Labour hoarding and wage reduction vis-a-vis permanent lay-off. 

In  addition  to  developing  and  testing  the  K+S  i-FRAME  module  on  Active  Labour  Market 
Policies (ALMP), the JRC in collaboration with  external experts conducting a support study for 
operationalising  the  i-FRAME  2.0  is  also  defining  a  simulation  modelling  module  on  Mission-
Oriented  Social  Innovation  Policy,  borrowing  in  part  from  the  concept  of  Mission  Oriented 
Innovation à la Mazzucato (see Mazzucato, M., 2013).  

The  rationale  and  foundational  principle  for  the  K+S  i-FRAME  Mission-Oriented  Social 
Innovation Policy (MOSIP) are described below and are currently under further elaboration. 
In  the  current  research  this  is  limited  to  conceptual  and  theoretical  developments  for  future 
simulations of the impact of Mission Oriented Social Innovation Policies.  

This  proof-of-concept  will  not  entail  any  development  of  the  K+S  model  during  the  current 
research;  nevertheless,  a  stylised  and  abstract  development  of  the  K+S  model  in  order  to 
eventually  extend  and  simulate  the  Entrepreneurial  State  concept  developed  by  Mazzucato41 
from the domain of R&D and technology to that of social policy innovation, is being elaborated 
and will be proposed as a blueprint for future additional developments to the i-FRAME in 2018.  

In this respect, recalling the typologies of social policy and service innovation proposed in §5.1 
(Table 3), it is our contention that total systemic innovation could be seen as analogous to the 
Mazzucato's  concept  of  mission  oriented  policies  (2015a).  Thus,  Mission  Oriented  (Social) 
Innovation Policy à la Mazzucato fits the typology of total systemic innovation.  

To  show  this  we  can  apply  and  rephrase  the  following  excerpt  from  Mazzucato  'Rather  than 
focusing  on  particular  sectors  –  as  in  traditional  industrial  policy  –  mission-oriented  policy 
focuses on problem-specific societal challenges, which many different sectors interact to solve. 
The focus  on problems, and new types  of  collaborations between public and private  actors to 
solve  them,  creates  the  potential  for  greater  spillovers  than  a  sectoral  approach……  The  new 
framework  seeks  to  better  envision,  justify,  measure  and  assess  public  investments,  working 
within an eco-system of public, private and third sector actors across the innovation chain. It 
focuses on the  role of the state as shaping and creating markets, not only fixing them  – and 
enables  the  development  of  economic  policy  to  be  informed  by  a  broader  theoretical 
underpinning'. 

Today  the  way  in  which  the  role  of  the  State  is  conceived  is  crucial  for  future  recovery  and 
growth. This is because in most parts of the world we are witnessing a massive withdrawal of 
the  State,  one  that  has  been  justified  in  terms  of  debt  reduction  and  –  perhaps  more 
systematically – in terms of rendering the economy more dynamic, competitive and innovative 
(Mazzucato,  2015b).  Business  is  accepted  as  the  innovative  force,  while  the  State  is  cast  as 
inertial  –  necessary  for  the  basics,  but  too  large  and  heavy  to  be  the  dynamic  engine. 
Mazzucato  (2015a)  has  largely  dismantled  this  view  and  showed  that  major,  revolutionary 
technological changes had substantially benefited from an active role of the government (i) in 
directing and funding (on its own) the process of R&D and, on the other side, (ii) in taking the 
risk  that  private  business  alone  had  not  be  willing  to  sustain.  There  is  little  reason  today  to 
think that the role of the public government would be less important in solving contemporary 
major societal challenges and sustain value creation directly from the public sector.  

So, one could envisage a Socially Entrepreneurial State that would, at the same time, push 
and support the integrated production of services, new financial instruments and procurement 
rule  to  tackle,  for  instance,  the  goal  of  keeping  older  workers  on  the  market  with  skills,  job 
policies, and healthcare prevention policies.  

41   Mazzucato,  M.  (2015a).  The  Entrepreneurial  State:  Debunking  Public  vs.  Private  Sector  Myths.  London:  Anthem 

Press. 
Mazzucato, M. (2015b). The green entrepreneurial state: SPRU working paper series, 2015-28. 

75 

 
                                           
Moreover, a number of new and innovative financial instruments have been developed and 
or  adapted  in  the  recent  years  to  facilitate  access  to  funding  for  promoting  social  policy 
innovation initiatives and strengthening 'social infrastructures' across the EU.42 

Financial instruments are presented by the Commission as resource-efficient way of deploying 
cohesion  policy  resources.  Targeting  projects  with  potential  economic  viability,  financial 
instruments  provide  support  for  investments  by  way  of  loans,  guarantees,  equity  and  other 
risk-bearing mechanisms, possibly combined with technical support, interest rate subsidies or 
guarantee fee subsidies within the same operation.  

In  the  2014-2020  period,  the  Commission  is  encouraging  Member  States  to  double  their 
European Structural and Investment Funds (ESIF) used through such financial instruments.  

Among  such  financial  instruments  there  are,  for  instance:  1)  risk-sharing  loans,  based  on 
the  sharing  of  risks  between  public  and  private  resources;  2)  capped  guarantee 
instruments,  where  public  money  acts  as  guarantee  against  default  inside  a  bank's  loan 
portfolio.  Both  instruments  aim  to  provide  SMEs  with  better  access  to  finance;  and  3) 
renovation  loans,  for  energy  efficiency  and  renewable  energy  projects  in  the  residential 
building sector.  

The Commission is also launching two new instruments that are particularly interesting as they 
could  be  suitable  for  building  scenarios  of  use  for  simulation  modelling  under  the  i-FRAME 
approach, and to illustrate a possible re-thinking of the investment orientations in the field of 
social policy innovation (see focus box below). 

Selected new financial instruments to support Social Policy Innovation 

The  European  Commission  adopted  two  new  'off-the-shelf',  i.e.  ready-to-use  financial 
instruments for European Structural investment Funds (ESIF).  

Off-the-shelf financial instruments are designed to increase the take-up by Member States of 
revolving  financial  support  rather  than  traditional  grants,  and  to  combine  public  and  private 
resources. 

The two new instruments the Commission is launching are:  

A  co-investment  facility  to  provide  funding  to  start-ups  and  SMEs.  This  support  will 
enable  them  to  develop  their  business  models  and  attract  additional  funding  through  a 
collective investment scheme managed by one  main financial intermediary. Total investment 
combining public and private resources can amount to up to €15 million per SME.  

Urban  development  funds  to  support  sustainable  urban  projects,  in  public  transport, 
energy efficiency or the regeneration of urban areas, for example. Projects must be financially 
viable  and  part  of  an  Integrated  Sustainable  Urban  Development  strategy.  Total  investment 
combining  public  and  private  resources  can  amount  to  up  to  €20  million  per  project.  The 
support  will  take  the  form  of  a  loan  fund  managed  by  a  financial  intermediary,  with  ESIF 
resources and a contribution of at least 30% from private capital. 

Source: European Commission, 2017 

Despite the fact that the Commission has stressed the potential of the social service sector for 
creating jobs, austerity and other rules hamper the deployment of public funds to this purpose. 
This is part explained by the  controversy on  the extent to  which  EU rules on State aid, 
internal market and public procurement apply to social services. 

Whereas new financial instruments to support social policies have been introduced, the idea of 
the  Socially  Entrepreneurial  State  run  against  the  wall  of  the  debate  on  whether  or  not  EU 
rules on State aid, internal market and public procurement apply to social services.  

42   See  for  instance  the  FI-COMPASS  Handbooks  on  Financial  instruments  under  the  ESF  and  EFSI,  including 
microfinance  (2016);  and  on  Innovative  use  of  financial  instruments  within  the  ESF  (2017),  available  at: 
https://www.fi-compass.eu 

76 

 
 
 
                                           
This  debate  has  dominated  the  political  scene  for  at  least  the  last  decade.  Indeed,  in  recent 
years, several public authorities and civil society  organisations representing service users and 
providers  have  claimed  that  the  EU  rules  create  unnecessary  difficulties.  For  instance,  the 
Informal  Network  of  Social  Service  Providers  (INSSP)  published  a  report  of  a  seminar  about 
the  impact  of  EU  legislation  on  social  services  which  states  that  the  EU  rules  'in  many  cases 
impact  in  a  negative  manner  on  the  sector  of  social  NGOs,  part  of  the  social  economy,  and 
specifically  on  the  quality  of  social  services'.  INSSP  admits  however  that  the  negative  effects 
'are  not  only  due  to  the  EU  rules  but  also  linked  to  the  way  national  and  local  authorities 
implement these rules. Some authorities misunderstand the rules or implement them in a way 
that is more restrictive than necessary ...' 

The  Commission  has  argued  against  these  views,  explaining  that  EU  rules  already  take  into 
account  the  specific  characteristics  of  social  services  and  that,  if  the  public  authorities  apply 
them  correctly,  these  rules  can  help  them  organise  and  finance  high-quality  cost-effective 
social  services  in  a  transparent  manner.  Member  States  have  an  ample  choice  of  modes  of 
organisations of the services in line with their traditions and cultural backgrounds. 

Against  this  context  the  proposed  i-FRAME  K+S  MOSIP  simulation modelling  'scenario 
of  use'  envisages  integrated  interventions  that  are  not  merely  public  sector 
intervention,  rather  see  the  state  as  a  sort  of  entrepreneurial  incubator  enabling 
societal and private actors to leverage public and private funds. 

In the framework of piloting and testing the i-FRAME 2.0, by means of the K+S model, it would 
be  possible  to  provide  a  proof-of-concept  of  the  role  of  mission  oriented  social  innovation 
policy  interventions  for  triggering  the  emergence  of  a  new  paradigm  that  would  favour  the 
transition  towards  a  more  socially  and  environmentally  sustainable  growth  pattern.  This  will 
also investigate the role of green jobs in the context of employment dynamics (along the path 
suggested in Mazzucato, 2015a and 2015b).  

This includes building on the existing environmental module of K+S and links it to the following 
MOSIP scenario that should be further theorised and embedded into the model in the future: 

Social enterprises leveraging on financial resources for urban renewal engage either in helping 
people  restructuring  their  house  with  energy  efficient  solutions  or  build  smart  and  energy 
efficient homes and public infrastructures.  

This type of policy would aim at the same time at:  

(a) creating new green jobs in social enterprises;  

(b) provide support to disadvantaged social groups;  

(c) contribute to community building or re-building and urban renewal.  

By doing so, we could tackle social inclusion impacts, the issue of social impact investing (for 
energy  efficient  and  environmentally  friendly  urban  renewal)  and  the  issue  of  distributional 
inequality concerning both income and wealth.  

This  would  require  the  development  of  a  housing  market  and  a  real-estate  sector,  possibly 
endowed with a spatially explicit structure, as a new module of the model.  

Further, such a step will require enriching the financial sector, including additional actors other 
than  standard  commercial  banks  and,  as  a  consequence,  different  financial  instruments  than 
bonds  and  loans,  which  would  mimic  those  described  earlier  and  already  available  as  part  of 
the  portfolio  of  financial  instruments  that  can  be  used  to  implement  the  European  Structural 
and Investment Funds (ESIF) in the current and next programming period.  

In this perspective, the results of simulations that could be developed through operationalising 
this scenario of use may contribute to the current debate on the future of the European Social 
Fund (ESF) and its combination with the European Fund for Strategic Investment (EFSI). 

77 

 
 
 
 
6  Conclusions: paving the way to i-FRAME 3.0 and beyond 

Summary of content of Chapter 6 

This chapter draws conclusions on future research directions and start paving the 
way to i-FRAME 3.0 and beyond.  

It is organised as follows: 

§6.1 summarises the key results achieved by the research so far, elaborating on 
the improvements proposed for the methodological framework - i-FRAME 2.0 

§6.2  discusses  research  directions  that  could  be  followed  to  further  develop  a 
fully-fledged i-FRAME Simulator of social policy impact – i-FRAME3.0; 

§6.3 recommends further support for the i-FRAME, and discusses the contribution 
it could make to the current debate on strengthening the social dimension of the 
EMU and the future of welfare systems in the EU. 

6.1  Beyond reductionist and positivist evidence-based policy 

In order to appreciate fully the main results achieved during the designing and building of the 
i-FRAME,  it  is  important  to  briefly  recall  the  two  main  objectives  of  the  underlying  IESI 
research in general and of the i-FRAME in particular:  

 

 

to  help  policymakers  and  practitioners  better  understand  the  'Social  Policy  Innovation 
Ecosystems' where their interventions take place in the deliberation phase, and  

to  develop  micro-macro  operational  tools  for  the  design  and  evaluation  (ex-ante,  in-
itinere, and ex-post) of these interventions.  

To achieve these objectives a distinction has been made between the meta-framework and the 
specific micro and macro operational components that have been proposed, developed and/or 
piloted  during  the  research.  This  distinction  can  help  us  understand  the  methodological 
approach used in the development of the i-FRAME so far and the way forward to i-FRAME 3.0. 

The meta-framework encompasses both objectives, whereas the operational components 
focus  on  pragmatic  micro-level  measurement  tools,  computer-based  instruments  for  data 
gathering  and  analysis,  and  macro-level  simulation  modelling  approaches  rooted  in  complex 
systems theories. In both cases, a modular approach for development, testing, and validation 
was adopted.  

As  amply  demonstrated  by  the  findings  of  the  literature  reviews  conducted  during  this 
research,  it  was  necessary  to  deconstruct  and  interpret  social  policy  innovation  ecosystems 
using complexity thinking and tools in order to understand them. In some cases, this exercise 
could  be  conducted  by  using  more  formalised  techniques  such  as  modelling  simulations  that 
include Agent-Based Models (ABMs), System Dynamics (SD), AMB/SD hybridisation, and Social 
Network Analysis (SNA), all of which can be informed by behavioural insights. However, when 
this  is  not  possible,  ecosystems  could  be  reconstructed  through  in-depth  qualitative  case 
studies, which in turn fed into, and supported, the micro- and macro-level operational tools.  

In this respect, the final proposal for developing a methodological framework to assess 
social  policy  innovation  promoting  social  investment  -  i-FRAME  2.0  kept  both 
dimensions together, by developing and piloting the various operational components in diverse 
stages of development and application domains, as well as different timeframes.  

78 

 
 
 
 
 
 
This  approach  follows  almost  naturally  from  the  critique  of  orthodox  evidence-based  policy 
(EBP) we have advanced, based on a growing body of academic literature from which we have 
drawn  selectively.  The  i-FRAME  approach  is  to  move  from  reductionist  and  positivist 
evidence-based  policy  to  a  more  paradigmatically  and  methodologically  pluralistic 
'evidence-informed policy' (EIP).  

This critique of orthodox and deterministic approaches to evidence-based policy is fully in line 
with the current debate on the need for system thinking to tackle 'wicked' policy issues 
(OECD, 2017). The relationship between the cause and effect of wicked problems such as long-
term unemployment, income inequality, and poverty is unclear and there seem to be no easy 
solutions  (OECD,  2017,  p.  5).  These  contexts  require  not  only  integrated  inter-institutional 
policies,  but  also  new  analytical  tools  that  can  better  deal  with  uncertainty  and  complexity. 
Policy  analysis  should  accommodate  complexity  rather  than  filter  it  out,  using  outdated  and 
reductionist tools.  

As stated by the OECD (2017, pp. 12-13): 'as policy problems have changed toward systemic, 
interdependent  challenges,  their  understanding  and  analysis  needs  to  change.  In  highly 
complex  problems,  the  relationships  between  causes  and  effects  are  neither  linear  nor 
simplistic'. Though Randomised Control Trials (RCTs) and their meta-analysis remain important 
instruments,  they  are  not  the  only  ones  available.  They  cannot  be  used  mechanically  as 
prescribed  by  certain  EBP  recipes,  because:  a)  what  'worked  there'  will  not  necessarily  and 
automatically 'work here'; and b) some issues are too complex for classic experimental design.  

The  EBP  orthodoxy  prescribes  what  works  following  an  a-critical  positivist  approach.  It  treats 
social sciences and their policy application as though they were in a situation of paradigmatic 
normality in the sense defined by Kuhn  (1970). It is as if one consensual paradigm had been 
accepted in applied policy work by most social scientists and scholars, who had not challenged 
its main assumptions, models, hypotheses, and measures. The reality of social sciences, of 
applied  policy  research,  and  of  the  study  of  the  relation  between science  and  policy 
is much more pluralistic and has no dominant paradigm. 

However, in recent years, the  'Evidence-Based Policy' (EBP) agenda has been challenged and 
the  expression  turned  on  its  head  by  authors  who  talk  about  'Policy-Based  Evidence  Making' 
(Sanderson,  2011;  Strassheim  &  Kettunen,  2014;  Torriti,  2010).  There  are  increasing  doubts 
that science can inform policy as in the classical linear model of the relations between evidence 
and  policy  (Pielke,  2007,  pp.  12-14).  Accordingly,  the  idea  of  'speaking  truth  to  power' 
(Wildavsky, 1979) by way of scientific advice to politicians and policymakers is being seriously 
reconsidered.  There  have  been  frequent  attempts  to  turn  all  policy-making  into  technical 
exercises  that  obviate  the  need  for  political  debate  in  what  has  been  called  technocracy  or 
scientisation (Jasanoff, 1990; Ravetz, 1990; Sarewitz, 2000, 2004). However, in conditions of 
uncertainty, a detached 'pure science' approach is nothing more than 'stealth issue advocacy' 
(Pielke, 2007, chapter 5). 

In  many  policy  domains  (including  social  policy)  values  are  disputed,  facts  uncertain,  stakes 
high,  and  decisions  urgent.  These  are  the  conditions  in  which  'post-normal  science',  was 
introduced  and  further  developed  in  various  works  of  Funtowicz  and  Ravetz  (Funtowicz  and 
Ravetz, 1990, 1991, 1993, 2008; Ravetz, 1990). Under these conditions, scholars who advise 
policymakers should propose several options rather than prescribe hierarchies of what works.  

Inspired by the above critical considerations and by the analysis of the limits to orthodox EBP 
discussed more extensively in §5.1 the revised i-FRAME 2.0 meta-framework is proposed to be 
developed  as  an  open  and  epistemologically  pluralistic  framework  with  several  operational 
components  that  can  be  used  and  adapted  to  the  different  needs  of  policymakers  and 
practitioners  at  different  levels  (international,  national,  and  local).  It  should  employ  at  the 
same  time  formalised  quantitative  empirical  approaches,  theory-based  evaluation,  modelling 
simulations,  and  qualitative  methods.  This  ensemble  of  tools  can  be  used  both  in  interactive 
workshops within policy lab sessions and in more traditional capacity-building exercises. 

79 

 
 
6.2  Future research directions: towards i-FRAME 3.0 

In this report, we have presented the findings from the research conducted as part of the IESI 
study  between  2014  and  2016.  Based  on  these  findings,  we  proposed  a  methodological 
framework  to  assess  the  social  and  economic  impacts  of  social  policy  innovations  which 
promote social investment. During the course of the research, this quasi-concept turned out to 
be a better scope of analysis than focusing strictly on ICT-enabled social innovation potential. 
We carried out an iterative series of extensive and exhaustive reviews of sources and  set the 
basis  for  the  development  of  an  open,  interactive,  and  pluralistic  platform  for 
evidence-informed social policy innovation.  

In  this  connection  it  is  important  to  recall  the  findings  concerning  the  structuring  of  the 
operational  components  of  the  i-FRAME  2.0.  Though  these  are  still  depicted  in  a  high  level  / 
generic  fashion,  they  have  been  developed  and  tested  as  a  'prototype'  of  a  computer-
based simulation model which include:  

• 

tools to support the deliberation and design phase,  

•  an  electronic  micro-level  measurement  toolkit  that  serves  to  structure  the  interactive  and 

dynamic warehouse for evidence-informed social policy innovation (EISPI);  

•  a Support Interactive Tool (SIT) to funnel users. This tool is being piloted through a specific 
experiment  which  aims  to  apply  and  extend  the  Macro-economic  Agent-Based  Models 
(MABMs) from the 'Keynes plus Schumpeter' (K+S) family.  

In this respect  our proposal lays the foundations for both system-oriented formalised 
modelling  simulations  and  for  theory-based  approaches.  The  critique  of  orthodox  EBP 
enabled  us  to  refine  and  improve  the  earlier  version  of  i-FRAME  (V1.5)  especially  where  the 
design and deliberation phase was concerned, and to present a revised and integrated 6-step 
methodological approach, which replaced the previous i-FRAME Decalogue.  

We  have  also  started  setting  the  basis  for  developing  a  blueprint  for  conceptual 
modelling and have proposed and tested several operational components. Building on 
the results of piloting and validation activities, we designed the revised framework to be in line 
with  the  vision  for  i-FRAME  developments  beyond  2017.  To  do  so,  we  engaged  experts  from 
diverse research disciplines, representatives of key stakeholders and policymakers at different 
level. Finally, we paved the way to building the i-FRAME3.0 between 2018-2020.  

By  building  on  the  results  of  the  IESI  research  conducted  during  the  period  2014-2016,  we 
were  able  to  develop  and  validate  the  theoretical  and  methodological  approach  which 
underpins  the  i-FRAME.  We  proposed  a  refined  and  improved  design  of  the  operational 
components for the i-FRAME 2.0. The JRC is further developing and piloting, with the support 
of  external  experts  and  stakeholders,  a  prototype  of  a  computer-based  simulation  model  to 
implement the overall methodological framework. Once refined, this would enable researchers 
and policymakers to assess the impacts generated by social policy innovation initiatives which 
promote social investment in the EU. 

We  are  also  validating  the  conceptual  and  methodological  approach  of  the  i-FRAME  2.0  by 
applying it to selected case studies and scenarios of use which address social policy innovation 
promoting social investment. We will consider various domains, levels of governance of service 
integration implemented in diverse welfare systems and social service delivery models, drawn 
from different EU Member States. This work would set the ground for developing a fully-
fledged i-FRAME Simulator (V3.0) to be developed between 2018-2020. 

In  this  regard  it is  important  to  make  some  clarifications  with  respect  to  the  ambition  of  this 
proposal and its possible way forward.  

First of all, the i-FRAME in its current version 2.0 is still at the early stage of development of 
the  operational  components  that  are  envisaged  in  its  comprehensive  approach.  Some  of  the 
proposed tools may change in light of empirical work and testing in case studies and scenarios 
of use - for example, the 'diamond approach' should be further elaborated and refined. 

80 

 
 
Second,  only  when  the  web  platform  with  the  developed  instruments  should  be  finalised  it 
should  be  possible  to  start  build  capacity  by  further  developing  and  better  operationalising 
specific  aspects  underlying  the  i-FRAME  methodological  framework.  In  addition,  a  handbook 
and  manuals  to  navigate  through  the  i-FRAME  should  be  developed  to  facilitate  its 
understanding and use. This should benefit of guidance from experts and diverse communities 
collaborating  with  the  JRC  that  should  be  incorporated  into  the  further  development  of  the  i-
FRAME.  This  would  help  policymakers  also  to  look  at  what  others  have  done  and  select  the 
best approach and tools for their purpose. 

Third, the use of a Macro-Economic Agent-Based Model which focuses mostly on labour market 
policy in no way implies that this is the only approach where the i-FRAME shall focus on. It was 
a  choice  made  from  a  range  of  alternatives,  in  terms  of  both  modelling  tools  and  policy 
domains  that  was  shaped  by  the  constraints  –  in  terms  of  timeframe  and  resources  -  of  the 
project under which i-FRAME has been originated. As we have  seen, the philosophy  on which 
the  i-FRAME  is  based  adopts  a  non-paradigmatic  and  pluralistic  approach  to  methods  and 
sources to support policy making. In line with this approach, it is clear that there are different 
modelling  approaches  and  areas  of  policy  that  need  to  be  explored,  used  and  tested  in  the 
future. The use of K+S mostly for active labour market policies should be seen simply as a first 
exploration. 

Last  but  not least,  the  critical  stance  taken  by  i-FRAME  cannot  be  considered  in  anyway  as a 
'post-modernist' external critique or a 'constructivist critique'. Instead, it is an internal critique 
that  goes  back  to  the  principles  of  the  founding  fathers  of  the  1960s  experimentalism  and 
quasi-experimentalism  movements  in  support  of  policy,  (e.g.  Campbell,  Shasish,  Cook,  and 
Cronbach).  Standing  on  the  shoulders  of  these  giants,  we  simply  show  how  Evidence-Based 
Orthodoxy  has  betrayed  fundamental  epistemological  principles  and  all  the  work  about 
internal,  construct,  ecological,  and  external  validity  that  was  the  main  contribution  of  these 
scholars. Affirming that 'what worked there may not necessarily work here' arises from a truly 
scientific  epistemology  that  is  not  as  simplistic  and  ideological  as  the  one  proposed  by  the 
'Evidence-Based Policy mantra'. Furthermore, the relevant literature frequently documents how 
evidence-based  policy  has  been  often  turned  into  policy-based  evidence-making  simply  to 
support specific political agendas. 

In  this  vein,  it  is  our  opinion  that  the  proposed  i-FRAME  made  significant  progress 
towards  addressing  some  important  issues  in  the  area  of  supporting  the  evaluation 
of Social Policy Innovation promoting social investment.  

However  much  more  needs  to  be  done.  In  a  future  perspective,  the  i-FRAME  has  the 
potential  for  making  a  significant  contribution  especially  in  setting  standards  for  the  use  of 
models  for  the  evaluation  of  impacts  of  social  policy  initiatives.  Only  a  body  such  JRC  could 
avoid the danger of piecemeal approaches (national or regional), designing and implementing 
in a comprehensive manner a process of assessing and revising approaches and tools, able to 
boost the whole field (as with IEEE, W3C or ISO standards).  

For  this  purpose,  the  further  refinement  and  operationalisation  of  the  i-FRAME  shall  revisit 
some of the principles set out in its foundations, building on the rich knowledge repository for 
social  innovation  initiatives  that  have  been  developed  by  JRC  so  far.  A  number  of  proof-of-
concept  use  cases  shall  be  chosen  together  with  policymakers,  analysts  and  evaluators  and 
data  from  the  knowledge  repository  could  be  used  for  calibrating  simulation  experiments  for 
diverse  scenarios  of  use.  Results  of  experiments  and  test  validation  shall  be  then  discussed 
together  with  policymakers  to  assess  their  utility.  To  this  end,  connecting  to  other 
initiatives  and  activities  using  complex  systems  approach  to  support  policy  making 
and  evaluation  is  crucial  (e.g.  the  CECAN,  http://www.cecan.ac.uk  and  the  OECD  NAEC, 
http://www.oecd.org/naec)  just  to  mention  a  few).  Financial  resources  and  timeframe  should 
clearly  mirror  the  complexity  of  objectives  and  tasks  set  out  for  such  future  developments, 
which  shall  focus  even  more  on  the  needs  of  policy  actors,  aiming  also  at  enhancing  their 
capacity to embrace policy modelling. 

81 

 
 
6.3  Policy implications 

The  final  version  of  the  i-FRAME  set  the  foundations  for  developing  a  computer-based 
simulation  model  that  should  encompass  various  levels  of  analysis,  by  using  the  same 
structural  IT  environment  (i-FRAME  Web  Platform  and  Simulator).  This  shall  be 
complemented  by  concrete  technical  and  policy  recommendations  for  its  use  and 
how  to  scale  up  the  methodological  approach.  These  recommendations  would  help  the 
European Commission and relevant organisations in the EU Member States to conduct ex-ante 
analysis of potential impacts of innovative social policy initiatives. They could also help with in-
itinere and ex-post analysis of the impact of the investment implied by these interventions at 
the end of the policy cycle. In these analyses, possible counter-intuitive behaviours should be 
taken  into  account.  A  flexible  approach  should  be  applied  to  re-calibrating  the  various 
operational  components  developed  for  simulation  modelling,  as  a  consequence  of  ex-post 
analysis  or  changes  in  the  theoretical  assumptions/causal  relationships  and/or  dynamics 
hypothesis underlying the model and its components. 

To achieve these objectives and carry out this challenging, but very much needed and timely 
work,  an  ambitious  research  plan  should  be  envisaged,  both  to  reinforce  JRC  internal 
capacities and make available appropriate resources. This should be embedded into a specific 
high-level science for policy agenda. Experts and representatives of stakeholders should 
be closely involved on an ongoing basis: researchers from relevant scientific communities, 
and  practitioners  and  policymakers  should  be  called  on  directly  to  address  concrete  and 
specific complex policy challenges. 

instance,  when  considering  a  pressing  policy  challenge  such  as 

For 
long-term 
unemployment,  especially  among  the  younger  generation,  it is  evident  that  the  effects 
of the financial and economic crisis are not limited to (the lack of) economic growth. Tackling 
these  challenges  is  not  only  a  matter  of  'social  investment'  policies;  they  create  social 
problems  and  costs  that  must  be  addressed  by  social  insurance  and  welfare-enhancing 
policies. Long-term unemployment, for instance, reduces individual and household income, and 
increases the risk of poverty and exclusion (Addabbo et al., 2012).  

Both  long-term  unemployment  and  job  insecurity  and  precariousness  have  been  shown  to 
negatively impact on physical and mental health (Burchell, 2009; Herbig et al., 2013; Paul and 
Moser, 2009; Virtanen et al., 2005). Long-term unemployment can also reduce the individual's 
human and social capital, as well as trust in public policy and social cohesion (Easterly et al., 
2006; Martin, 2014). Precarious jobs become 'traps' as opposed to 'bridges' into secure work, 
as  they  reduce  social  mobility  (Cahuc  and  Kramarz,  2005;  Gash,  2008;  International  Labour 
Office, 1997).  

Trends  towards  more  flexible  work  have  also  been  associated  with  growing  inequality 
(Atkinson, 2015; Bernhardt, 2014; Kuttner, 2013, 2016; Standing, 2011; Summers and Balls, 
2015).  A  study  using  EU-SILC  data  for  24  European  countries  shows  that  the  temporarily 
employed  run  a  higher  risk  of  poverty  than  permanent  workers,  mainly  due  to  lower  wages 
(Van Lancker, 2012). The fiscal costs would in the end increase for governments dealing with 
the social costs of long-term unemployment and job insecurity (Adams and Deakin, 2014).  

For these reasons, among many others, social policies as productive factors must address both 
the economic and the social dimensions of the current crisis, including those instances where 
they cannot be justified purely in terms of investment, economic performance and efficiency.  

This  points  to  the  need  to  strengthen  the  social  dimension  of  the  European  Monetary  Union, 
not  only  in  Euro-Area  countries,  but  also  in  countries  about  to  join  the  Euro  (to  set  up  the 
economic and political pre-conditions for their access to Euro). Moreover, it calls for the reform 
of social policies and the modernisation of social protection systems (e.g. through innovations 
in  social  service  delivery,  deinstitutionalisation  and  other  innovative  regulatory  or  financial 
instruments, including social impact investment, for instance).  

This  is  clearly  an  area  where  'Social  Policy  Innovation'  could  contribute  to  finding  a  solution. 
The i-FRAME 2.0 offers a structured platform and approach which, if developed on a 
larger scale, could provide evidence and indicate what roads to follow.  

82 

 
In  the  same  vein,  both  the  social  investment  perspective  and  Active  Labour  Market  Policies 
(ALMPs)  to  tackle  LTU  conceive  social  policies  as  'investments'  rather  than  costs.  The 
Commission's  public  consultation  on  the  EU  Pillar  of  Social  Rights  (European  Commission, 
2016d)  has  reignited  the  debate  on  social  protection  and  social  policy  as  a  productive  factor. 
This is hardly new: the OECD produced a systematic review of evidence in this domain as early 
as  2002  (Arjona  et  al.,  2002).  In  2005,  the  International  Labour  Organisation  (ILO)  also 
released a brief entitled 'Social Protection as a Productive Factor' (ILO, 2005).  

This  approach  is  based  on  the  idea  that  well-designed  social  policies  can  constitute  a 
productive  factor  that  actually  underpins  the  performance  of  the economy.  However, 
the  investment  perspective  is  not  sufficient  and  must  be  integrated  with  the  idea  of  'Social 
insurance  in  response  to  corresponding  market  failures'  and  that  of  'Social  welfare-increasing 
policies'. In practice, social policies are considered to be possible stabilizers/correctors 
of  market  failures.  The  collective  cost  of  these  correctors  is  lower  than  the  cost  associated 
with  the  market  failures,  similar  to  insurance  against  important  risks.  It  is  also  possible  that 
policies  that  require  going  beyond  a  mere  investment  perspective  cannot  be  justified  only  in 
terms  of  their  direct  effects  on  economic  performance;  yet,  these  policies  can  be  seen  as 
establishing  framework  conditions  that  are  indirectly  conducive  to  improved  economic 
performance  through  the  generation  of  overall  social  welfare.  Distributional  and  equity 
producing  policies  can  reduce  social  costs  (i.e.  less  crime,  better  health  outcomes  as 
demonstrated  amply  by  Wilkinson  &  Pickett,  2009),  without  reducing  competitiveness  and 
economic performance. It has been recognised recently that a more equal society does not 
necessarily perform less well or less competitively (Lagarde, 2015; OECD, 2015a; World 
Economic Forum, 2015).  

Though  evidence  on  the  relationship  between  social  policy  and  economic  performance  is  not 
conclusive,  it  suggests  a  positive  rather  than  a  negative  correlation  (Arjona,  et  al.,  2002; 
Simões  et  al.,  2014).  At  a  descriptive  level,  it  is  uncontroversial  that  countries  with  a  high 
share  of  social  spending  in  GDP  often  have  high  economic  living-standards  as  measured  by 
GDP  per  capita.  However,  there  is  not  enough  evidence  yet  to  establish  unequivocally  the 
direction  of  causation.  Obviously,  not  all  social  policies  are  effective  and  efficiently 
administered.  Thus,  it  is  of  the  utmost  importance  that  they  are  assessed  with  evidence  in 
order  to  modernise  and  improve  them.  In  this  regard,  the  i-FRAME  methodological 
approach  could  provide  a  valuable  tool  to  help  policymakers  better  understand  the 
impact  of  diverse  policy  interventions,  and  possibly  simulate  new  options  in  different 
contexts  involving  various  stakeholders.  The  effective  engagement  of  policy  actors  and 
domains is crucial to bring out and share different 'mental models'. Thus, it will be easier reach 
consensus among diverse, at times alternative, and often conflicting policy options.  

The positive results of the preliminary application of the i-FRAME approach open the 
door  to  a  more  extensive  and  systematic 
implementation  of  the  proposed 
methodology  at  policy  level.  This  could  involve  building  a  knowledge  repository  of 
simulation models based on a portfolio of cases analysed as part of i-FRAME development and 
further  enriched  with  new  application  examples  and  scenarios  of  use  across  the  EU  and  in 
different policy domains. At the same time, it would require the development of a fully-fledged 
dynamic  electronic  toolkit  to  support  policymakers  in  modelling  and  simulating  in  real-time 
policies  and  programme  interventions  included  in  the  i-FRAME  knowledge  base.  To  this  end, 
large  scale  computational  modelling  and  systems  simulation  tools  could  be  exploited.  These 
are able to capture not only predictable human behaviour through linear top-down forecasting 
techniques,  but  also  unplanned  outcomes  of  complex  interactions,  taking  advantage  of  data 
analytics and computer-based policy modelling. This would lay the basis for what could be 
already  prospectively  called  Data-powered  i-FRAME  4.0,  which  would  include  real-time 
structured data inputting from initiatives gathered through the i-FRAME Web-Platform to the i-
FRAME  social  policy  innovation  simulator.  The  Data-powered  i-FRAME  4.0  could  help  the 
European  Commission  and  EU  Member  States  to  monitor  the  implementation  of  a  revamped 
'Social Union', and thus shape a better future for Europe. 

83 

 
Annexes 

Annex I - Modelling approaches 

A  model  is  a  simplified  representation  of  the  reality,  which  is  adopted  when  reality  is  too 
complex  to  be  analysed  in  any  form  of  granular  detail,  or  as  a  consequence  of  constraints 
(time, information, money, etc.) that make experimental methods unfeasible. Still when some 
data are available and the problem at hand not to complex one can try to analytically solve a 
model  with  data  to  estimate  parameters;  this  is  what  traditional  econometric  models  (not 
modelling  tools)  do.  When  also  this  approach  proves  too  complex,  then  researchers  resort  to 
computer based modelling simulations. In doing so, not all variables are of equal importance to 
the researcher and his/her choices reflect  'implicit' assumptions, rules, and strategies used to 
solve  problems  'explicit'  in  the  model.  In  view  of  this  premise,  the  table  below  provides  a 
synthetic comparison of different modelling approaches. 

Table 5: Comparison of models 

Source: own Elaboration 

Traditional  economic  modelling  tools  rest  on  unrealistic  characterisation  of  representative 
agents  and  the  unrealistic  assumption  of  general  equilibrium,  and  do  not  take  into  account 
heterogeneity of agents and activities.  

System  Dynamics  is  based  on  a  top  down  and  aggregate  view  with  a  strong  explanatory 
power that, however,  reflect the mental models of the modellers or the group model building 
exercises.  It  is  mostly applied in  managerial  studies  and  more  recently  also in  policy  studies. 
System  dynamics  is  a  perspective  and  set  of  conceptual  tools  that  enable  to  understand  the 
structure  and  dynamics  of  complex  systems.  SD  is  often  used  to  build  management  flight 
simulators-micro-worlds  where  space  and  time  can  be  compressed  and  slowed  so  one  can 
experience  the  long-term  side  effects  of  decisions,  speed  learning,  develop  an  understanding 
of complex systems, and design structures and strategies for greater success. State variables 
(stock  and  flows)  are  linked  by  algebraic  differential  equations.  They  are  system  centred  and 
do not account for heterogeneity of agents and activities. 

Discrete-event simulations are based on the concept of entities, resources and block charts 
describing  entity  flow  and  resource  sharing,  with  passive  entities.  The  focus  is  on  the 
'processes'  not  the  'agents'.  In  discrete-event  simulation,  the  operation  of  a  system  is 
represented as a chronological sequence of events. Each event occurs at an instant in time and 
marks a change of state in the system.  

Agent  based  Models  (ABM)  is  a  bottom  up  approach  and  populated  by  agents  that  are 
autonomous,  pro-active,  reactive,  spatial  aware,  able  to  learn  and  with  social  abilities,  that 
'live' in an environment and are driven by behaviour rules defined by 'state charts'. ABM is an 
essentially  decentralised,  individual-  centric  (as  opposed  to  system  level)  approach  to  model 
design.  Agent-based  models  (ABM)  as  they  have  been  used  in  social  simulation  (Balke  & 
Gilbert,  2014)  explicitly  describe  human  decision  making  and  range  on  a  continuum  from 

84 

(cid:9)Ontology(cid:9)Representative(cid:9)agents(cid:9)Process(cid:9)oriented(cid:9)top(cid:9)down(cid:9)Constructivist(cid:9)top(cid:9)down(cid:9)Bottom(cid:9)up(cid:9)Heterogeneous(cid:9)agents(cid:9)with(cid:9)empirically(cid:9)grounded(cid:9)Behavioural(cid:9)rules(cid:9)Epistemology(cid:9)Stochastic(cid:9)causation(cid:9)Passive(cid:9)entities(cid:9)mathematical(cid:9)and(cid:9)computer(cid:9)tools(cid:9)express(cid:9)expert(cid:9)knowledge(cid:9)and(cid:9)explicit(cid:9)the(cid:9)structures(cid:9)implicit(cid:9)in(cid:9)the(cid:9)social(cid:9)system(cid:9)modelled(cid:9)Emergent(cid:9)properties(cid:9)resulting(cid:9)from(cid:9)micro-meso(cid:9)links(cid:9)but(cid:9)not(cid:9)in(cid:9)isolation(cid:9)Methodology(cid:9)Equilibrium(cid:9)Prices(cid:9)Vector(cid:9)Chronological(cid:9)sequence(cid:9)of(cid:9)events(cid:9)SD(cid:9)logical(cid:9)truths(cid:9)and(cid:9)generic(cid:9)structures(cid:9)identify(cid:9)constants(cid:9)in(cid:9)our(cid:9)social(cid:9)practices(cid:9)Can(cid:9)work(cid:9)with(cid:9)disequilibrium(cid:9)Empirical(cid:9)validation(cid:9)(ex(cid:9)ante(cid:9)and(cid:9)ex(cid:9)post(cid:9)and(cid:9)using(cid:9)also(cid:9)experimental(cid:9)findings)(cid:9)Can(cid:9)work(cid:9)with(cid:9)disequilibrium(cid:9)Heterogeneous(cid:9)agents(cid:9)and(cid:9)activities(cid:9)No(cid:9)No(cid:9)No(cid:9)Yes(cid:9) 
 
rather  simple  interactions  among  agents  to  deliberative/cognitive  representations.  When 
designing an agent based model the modeller identifies the active entities, the agents (which 
can  be  people,  companies,  projects,  assets,  vehicles,  cities,  animals,  ships,  products,  etc.), 
defines  their  behaviour  (main  drivers,  reactions,  memory,  states,  ...),  puts  them  in  a  certain 
environment,  establishes  connections,  and  runs  the  simulation.  The  global  behaviour  then 
emerges  as  a  result  of  interactions  of  many  individual  behaviours.  It  combines  elements  of 
game  theory,  complex  systems,  emergence,  computational  sociology,  multi-agent  systems, 
and  evolutionary  programming.  Leaving  aside  the  traditional  economic  modelling  tools,  the 
other three families of modelling simulation discussed above can also be compared looking at 
the picture below.  

Figure 34: SD, Discrete Events, and ABM 

Source: Adapted from (Martin & Schlüter, 2015) 

The table below zooms more specifically into the comparison between SD and ABM, from which 
it clearly emerges in our view the superiority of the latter when dealing with Complex Adaptive 
Systems characterised by emerging effects and properties. 

Table 6: SD and ABM compared 

SD 

Perspective 

Top down 

Main building block 

Causal loop 

ABM 

Bottom up 

Agent entities 

Unit of analysis 

System structure 

Rule of agent behaviour 

Level of modelling 

Aggregate system behaviour 

Individual agent behaviour 

System structure 

Pre-determined 

Evolvable 

Time handling 

Continuous 

Continuous or discrete 

Source: Adapted from (Martin & Schlüter, 2015) 

ABMs are clearly preferable when one deals with situations characterised by the presence  of: 
a) identifiable and decentralised agents; b) when the agents are different or the environment 
is  heterogeneous;  c)  when  the  interaction  between  agents  is  local;  d)  when  agents  are 
adaptive  (and  adapt  their  adaptation  rules);  e)  when  individual  behaviours  (and  destiny) 
matters;  f)  when  agents  haves  spatial  presence.  Currently,  there  is  much  talk  about  the 
hybridisation between SD and ABM, although it something easier to say than to do.  

In the literature the two models are presented iconic model paradigms in the sense that they 
are  considered  as  exclusive  alternatives  to  analyse  complex  systems  either  from  a  top-
down/aggregate  or  bottom-  up/disaggregate  perspective  (Swinerd  &  McNaught,  2012; 
Vincenot  et  al.,  2011).  To  the  best  of  our  knowledge  such  hybridisation  has  been  used 
sequentially  only  when  interfacing  a  social  ecosystem  (rendered  as  ABM)  with  a  natural  one 
(i.e. the condition of a lake) modelled using SD (i.e., Martin & Schlüter, 2015). In a way if one 
starts  from  ABM  model,  including  an  SD  module  is  equivalent  to  'back-boxing'  part  of  the 
system;  vice  versa,  adding  an  ABM  module  to  a  SD  tools  means  opening  a  window  on  more 
realistic treatments of agent behaviour. The table below compare the three possibilities. 

85 

 
 
 
Table 7: SD, ABM, and their hybridisation compared 

ABM 

SD 

Hybrid 

Characteristic 
question 

Purposes 
In general for 
all: improve 
system 
understanding 
rather than 
prediction or 
forecasting 

How do emergent 
system-level patterns 
develop from micro-
level interaction (e.g., 
spatially, between 
individuals)? 

- To identify 
mechanisms (specific 
interactions) that are 
responsible for 
emerging system-level 
patterns 
(disaggregated). 

Focus 

- Generate hypotheses, 
exploration of micro-
level behaviour 
(Epstein, 2008) 

Tests for model 
calibration 

Suitable and 
traditional 
analysis tools, 
typical 
experiments 

Type of outcome 

Micro-level interactions 
between entities, 
network structure 
(heterogeneous 
characteristics of 
individuals/actors, 
temporal discrete 
behaviour), transient 
dynamics 
Statistical pattern 
matching—can the 
model grow patterns 
that are found in 
reality? 
Only through 
simulations, often with 
multiple repetitions 
because of stochastic 
elements: plotting 
group/system level 
characteristics over 
time (average), 
evaluating a limited 
parameter range, 
describing transient 
dynamics 

Source: Adapted from (Martin & Schlüter, 2015) 

- How do stocks change 
or stabilize? (given that 
rates are constant) 

How do changing 
process rates (impacted 
by decisions) affect 
dynamics? 

- Which 
process/feedback is 
dominating? 

- How do changing 
stocks affect agent 
states/the distribution 
of traits? 

Investigate system-
level dynamics 
(aggregated), stability 
properties of the 
system, loop 
dominance, explaining 
temporal dynamics, 
projection into the 
future 

Processes driving 
accumulation in stocks 
at (sub-)system level, 
stable-states, 
feedbacks (balancing, 
amplifying), non-
linearities 

Investigating different 
micro- or system level 
mechanisms that drive 
certain dynamics. 
Generate hypotheses of 
systems state-change 
(when does dominance 
of feedbacks change?) 
or structural 
development over time 
(when does an average 
trait of agents change?) 

Process of restructuring 
in a system which can 
focus either on a 
structure affecting the 
processes, or processes 
affecting the structure 

Stability analysis—
under which parameter 
setting can fixed 
points/equilibria occur? 
How stable are they? 

Separate sub-system 
tests (paradigm 
specific) and qualitative 
check for the coupled 
version 

Through simulations 
with a focus on either 

Simple models through 
analytical tools (basins 
of attraction, 
bifurcation analysis, 
overall stability), and 
more complex through 
simulations (state 
space plots from 
simulations, evaluating 
stable-states, 
equilibria) 

86 

 
 
 
Annex II – Overview of the K+S model 

The  economy  is  composed  of  three  populations  of  heterogeneous  agents,  i.e.  capital-good 
firms,  consumption-good  firms,  consumers/workers,  plus  a  bank  and  the  Government  (see 
Figure  below).  Capital-good  firms  invest  in  R&D  and  produce  heterogeneous  machine-tools, 
whose productivity evolves over time. Consumption-good firms combine machines bought from 
capital-good  firms  and  labour  in  order  to  produce  a  homogeneous  product  for  consumers. 
There is a minimal financial system represented by a single bank that provides finance to firms 
only,  so  they  can  borrow  to  produce  and  invest  (and  pay  interest  on  it).  Workers  submit  job 
applications  to  a  random  subset  of  firms,  which  hire  according  to  expectations  about  goods 
demand. Note that the  firm demand for labour is independent from the wage levels (alike an 
ensemble  of  fixed  coefficient  production  functions).  Consumption-good  firms  form  adaptive 
expectation  about  future  demand  for  product  and  decide  thereafter  their  desired  demand  for 
labour. The government levies taxes on firms and pays unemployment benefits. 

Figure 35: Schematic presentation of the K+S model 

Source: Sant'Anna School of Advanced Studies 

The capital- and consumption-good sectors. The capital-good industry is the locus where 
innovation  is  endogenously  generated  in  the  economy.  Capital-good  firms  develop  new 
machine-embodied techniques or imitate the ones of their competitors in order to produce and 
sell  more  productive  and  cheaper  machinery,  supplied  on  order  to  consumption-good  firms. 
The  capital-good  market  is  characterised  by  imperfect  information  and  Schumpeterian 
competition  driven  by  technological  innovation.  Machine-tool  firms  signal  the  price  and 
productivity of their machines to their existing customers as well to a subset of potential new 
ones and invest a fraction of past revenues in R&D in order to search for new machines or copy 
existing ones. On order, they produce machine-tools with labour only.  

Prices  are  set  using  a  fixed  mark-up  over  unit  costs  of  production.  Consumption-good  firms 
produce  a  homogeneous  good  employing  capital  (composed  by  different  ``vintages''  of 
machines) and labour under constant returns to scale.  

Desired production is determined according to adaptive demand expectations. Given the actual 
inventories, if the capital stock is not sufficient to produce the desired output, firms order new 
machines  in  order  to  expand  their  installed  capacity,  paying  in  advance  --  drawing  on  their 
cash flows or, up to a limit proportional to its size, on bank credit. Moreover, they replace old 
machines  according  to  a  payback-period  rule.  As  new  machines  embed  state-of-the-art 

87 

 
 
 
technologies, the labour productivity of consumption-good firms increases over time according 
to  the  mix  of  vintages  of  machines  present  in  their  capital  stocks.  Consumption-good  firms 
choose in every period their capital-good supplier comparing the price and the productivity of 
the machines they are aware of. Firms then fix their prices applying a variable mark-up rule on 
their production costs,  trying to balance higher profits and the growth of market  share. More 
specifically,  mark-up  dynamics  is  driven  by  the  evolution  of  the  latter:  firms  increase  their 
price whenever their market share is expanding and vice versa.  Imperfect information is also 
the normal state of the consumption-good market so consumers do not instantaneously switch 
to  the  most  competitive  producer.  Market  shares  evolve  according  to  a  (quasi)  replicator 
dynamics:  more  competitive  firms  expand  while  firms  with  relatively  lower  competitiveness 
levels shrink, or exit the market. 

The  labour  market.  We  study  two  labour  market  regimes,  which  we  call  Fordist  and 
Competitive.  The  two  regimes  capture  alternative  wage-labour  nexus.  Under  the  Fordist 
regime, wages are insensitive to the labour market conditions and indexed to the productivity 
gains  of  the  firms.  There  is  a  sort  of  covenant  between  firms  and  workers  concerning  long-
term  employment:  firms  fire  only  when  their  profits  get  negative,  while  workers  are  loyal  to 
employers and do not seek for alternative occupations. Labour market institutions contemplate 
a  minimum  wage  fully  indexed  to  aggregated  economy  productivity  and  unemployment 
benefits  financed  by  taxes  on  profits.  Conversely,  in  the  Competitive  regime,  flexible  wages 
respond to unemployment and market conditions, set by means  of  an asymmetric bargaining 
process  between  workers  and  firms,  which  have  the  last  say.  Employed  workers  search  for 
better  paid  jobs  with  some  positive  probability  and  firms  freely  adjust  (fire)  their  excess 
workforce according to their planned production. The competitive regime is also characterised 
by  different  labour  institutions:  minimum  wage  is  only  partially  indexed  to  productivity  and 
unemployment benefits -- and associated taxes on profits -- might or might not be there.  

(iv)  pro-cyclical  consumption, 

The labour-market K+S model has already proved to be able to generate jointly, as emergent 
properties,  a  wide  set  of  stylized  facts  regarding  both  micro/meso  phenomena  and  macro 
stylized  facts.  They  include  (i)  endogenous  growth;  (ii)  persistent  fluctuations;  (iii)  recurrent 
investment,  productivity, 
involuntary  unemployment; 
employment and changes in inventories; (v) fat-tailed distributions of aggregate growth rates; 
together  with  (persistent  asymmetries  in  productivity  across  firms;  (vi)  spiky  investment 
patterns; (vii) skewed firm size distributions; (viii) fat-tailed firm growth rates, and (ix) on the 
labour  market  side  Beveridge,  Wage  (or  Phillips),  and  Okun  curves.  To  repeat,  the  foregoing 
robust  statistical  regularities  and  relatively  stable  relations  amongst  aggregate  variables  do 
indeed  emerge  out  of  turbulent,  disequilibrium,  microeconomic  interactions.  As  such,  the 
model can be used also as sort of 'laboratory' for policy experiments. We have begun to do it, 
showing,  for  example,  the  complementarity  between  Keynesian  and  Schumpeterian  policies. 
Indeed,  the  policy  analysis  involves  a  fundamental  paradigmatic  change:  the  central  concern 
for policies ought to shift away from  efficient allocation issues, to the  (imperfect) governance 
of coordination, and genuinely uncertain processes of change.  

The  model  has  also  been  extended  beyond  the  structure  described  above  to  account  for  an 
energy  sector  that  provides  additional  inputs  to  labour  in  the  production  process.  Firms  are 
fueled by heterogeneous “dirty” and “green” energy production plants, constitute the principal 
CO2 emission contributors. In addition, we model the link between CO2 concentrations in the 
atmosphere  and  the  surface  temperature  through  non-linear  relationships  and,  as  the 
temperature  increases,  stochastic  damages  affects  both  labour  productivity  and  the  stock  of 
capital. In that, the model accounts both for frequent and mild environmental shocks and low-
probability but extreme climate events.  

Technical  change  occurs  both  in  the  manufacturing  and  energy  side  of  the  economy. 
Innovations  and  learning  contribute  to  determine  the  cost  of  energy  produced  by  differently 
carbon-intensive  technologies,  which,  in  turns,  affect  the  energy-technology  production  mix 
and the total amount of CO2 emissions per unit of time. As aggregate demand increases over 
time,  new  energy  plants  are  needed  to  fuel  the  production  system.  The  decision  about  what 
type  of  plants  to  invest  in  depends  upon  the  unitary  costs  of  production  stemming  from 
different  technologies.  In  general,  renewable  energy  plants  exhibit  higher  fixed  capital  and 
lower marginal costs with respect fossil fuel oriented plants. The process of technical change in 
the energy industry is also intimately linked to the dynamics of “green jobs”.  

88 

 
The  model  is  modular  and  flexible.  The  climate  box  can  be  optionally  activated  or  removed; 
relevantly,  it  allows  us  to  analyse  the  economy's  reaction  to  the  dynamics  of  workers' 
productivity and health conditions as well as to policies affecting them. The energy  sector and 
the  enriched  modelling  of  the  production  system  grant  evidence  on  the  short-  and  long-run 
sustainability  of  various  policy  combinations.  In  addition,  a  large  set  of  fiscal  and  industrial 
interventions can be introduced in the analysis.  

The model can reproduce a large set of empirical stylized facts. For instance, GDP and energy 
demand  grow  at  a  similar  rate.  Emissions  steadily  grow  as  well,  but  less  than  the  other  two 
quantities.  The  dynamics  of  emissions  resembles  another  feature  observed  in  the  data:  they 
stagnate  during  strong  recessions  and  rapidly  accelerate  immediately  after.  Models' 
simulations also mirror typical features of real data with respect to the volatility of output and 
its  components.  In  line  with  the  empirical literature,  investment  is  more  volatile  than  output, 
whereas consumption and another demand side component, energy demand, are less volatile. 
Unemployment rates seem to be quite stable across runs and average 12% in the Monte Carlo 
exercise,  a  result  in  relative  accordance  with  actual  data  for  different  countries.  Finally,  the 
share  of  renewable  energy  in  total  energy  production  exhibits  an  average  of  30%  over  the 
whole time span (2000-2010) and in only one period over three it finds above the threshold of 
20%, indicating the nearly absent room for green transitions in a business as usual scenario. 
Again,  the  foregoing  robust  statistical  regularities  and  relatively  stable  relations  amongst 
aggregate  variables  do  indeed  emerge  out  of  turbulent,  disequilibrium,  microeconomic 
interactions. The model can be conceived as a “laboratory” to test how policies affecting micro-
level  mechanisms  impact  the  aggregate  performances  across  various  time  scales  (short, 
medium, long run). 

Validation 

The  model  produces  time  series  data  for  a  number  of  relevant  variables,  including 
macroeconomic aggregates and micro-level characteristics of the agents. Here is a brief list of 
the main ones: 

Macro 

•  Output (GDP) 

•  Consumption 

• 

Investments  

•  Unemployment rate 

• 

Inflation rate 

•  Public Deficit 

•  Aggregate Debt 

•  Credit demand 

•  Average Productivity 

•  Average Energy Efficiency 

•  Wages 

Micro 

•  Firms' productivity 

•  Firms' competitiveness 

•  Firms' Sales (Turnover)  

•  Firms' Employees 

• 

Individual wages 

89 

 
 
 
The  simulation  protocol  is  simple  and it is  based  on  Monte  Carlo  exercises  where  we  account 
for the intrinsic stochasticity of the model. The size of the Monte Carlo exercises we perform is 
variable but, in the majority of cases, it is equal to 100 or 150. From these exercises a series 
of statistics can be computed and compared to the corresponding ones coming from empirical 
data and from the empirical literature.  

This  procedure  can  be  potentially  repeated  for  different  countries  of  groups  of  them.  The 
following figure briefly illustrates this approach. 

Figure 36: Analysis of simulation output 

Source: Sant'Anna School of Advanced Studies 

Sensitivity analysis. A number of sensitivity analyses are available for the model: 

•  Kriging  and  Global  Sensitivity  Analysis.  These  procedures  allow  us  to  infer  the 
sensibility  of  a  set  of  output  variables  to  variations  on  models'  parameters  and  initial 
conditions. 

•  Machine  Learning  Surrogate  Exploration.  This  procedure  allows  us  to  learn  an 
approximate and computationally efficient version of the true model, which is then used to 
explore its behaviour on possibly very large areas of the parameter space. 

•  Generalised Subtracted L-divergence. This procedure allows us to obtain a measure of 
the  distance  between  the  dynamical  behaviour  of  the  model  for  a  different  range  of 
parameters and the corresponding one observed in the empirical data.  

All the three procedures allow us to obtain Surface Plots which illustrate sensitivity of models' 
outcome to variations in parameters values for all possible scenarios. 

90 

 
 
 
 
 
 
 
Annex III – Details on method and sources for systematic review 

Scientific literature 

The  scope  of  the  systematic  review  was  defined  using  the  PICO  method:  Population, 
Intervention,  Comparison,  Outcomes.  The  population  of  the  study  is  the  EU.  Interventions 
have  been  defined  as  social  innovations  initiatives  in  one  of  the  six  fields  –  Employment  and 
training,  Child  care,  Long-term  care,  Social  Inclusion,  Social  Housing,  Social  Assistance  - 
whereas  Comparison  is  the  absence  of  treatment/intervention.  Outcomes  differ,  though  they 
all  relate  to  various  components  of  (social)  impact  and  its  assessment.  We  broadened  the 
scope  to  have  a  richer  insight  of  the  literature:  for  each  of  the  six  field,  we  have  carried  out 
two reviews - one targeting evaluations, the other targeting existing reviews or meta-analyses 
– in four electronic databases: IDEAS  – RePEc (Research Papers in Economics), SSRN (Social 
Science Research Network),  ISI  Web  of Science, and Scopus. The following table provides an 
overview of the data sources.  

Table 8: Database overview 

Source 

Scopus 

Description 

Scopus is the largest abstract and citation database of peer-reviewed 
literature: scientific journals, books and conference proceedings. It delivers 
a comprehensive overview of the world's research output in the fields of 
science, technology, medicine, social sciences, and arts and humanities.  

ISI Web of 
Science 

ISI provides access to the world's leading citation database, with 
multidisciplinary information from over 12,000 high impact journals and 
over 160,000 conference proceedings from around the world. 

IDEAS – RePEc  
(Research 
Papers in 
Economics) 

IDEAS is the largest bibliographic database dedicated to Economics and 
available freely on the Internet. It includes over 1,700,000 items of 
research. About 1500 institutions contribute their bibliographic data directly 
to this database.  

SSRN (Social 
Science 
Research 
Network) 

SSRN a website devoted to the rapid dissemination of scholarly research in 
the social sciences and humanities. It facilitates the free posting and sharing 
of research material (e.g., conference papers, preprints, non-peer-reviewed 
papers) areas.  

After several calibrations, 12 final search strings were developed using Boolean operators. Our 
overall framework for the search strings consists of three building blocks: Field, Domain, and 
Purpose.  Compared  to  the  provisional  strings  outlined  in  D1,  a  few  improvements  have  been 
added.  Under  the  previous  strings,  the  vast  majority  of  results  consisted  of  studies  on  the 
social dynamics of the six fields (e.g. effect of unemployment on mortality), whereas the focus 
of  the  project  lies  rather  in  studies  that  assess  initiatives  in  the  six  fields  (e.g.  impacts  of  a 
job-related  scheme  in  an  EU  region).  Therefore,  we  have  added  a  third  “Domain”  building 
block,  in  addition  to  the  "field-specific"  block  (employment  vs.  child  care  vs.  social 
assistance…) and to the "purpose" block (meta-analyses vs. individual evaluations). 

A second improvement relates to the Purpose building block. Unleashing the word “evaluation” 
in  the  search  string  ("evaluation"  |  "assessment"  |  "impact  assessment”),  as  it  was  the  case 
with  the  past  strings,  delivered  an  overwhelming  number  of  results  -  in  the  case  of 
employment  8,968  results  had  been  obtained  on  the  IDEAS  database  only,  even  by  looking 
only  at  abstract.  Instead,  hooking  evaluation  to  "impact"  (“impact  evaluation"  |  "impact 
assessment”)  gave  a  far  more  reasonable  number  –  in  the  case  of  employment,  169  results 
with  IDEAS,  and  140  including  the  Domain  block.  We  introduced  several  variations  in  the 
search  strings  to  check  how  this  modified  our  results,  and  to  better  adapt  to  the  nature  of 
individual databases. For example, SSRN does not allow for Boolean type of search, therefore 
we  had  to  reiterate  multiple  simple  strings  and  erase  duplicates  in  an  additional  step.  In 
IDEAS, it is not possible to search in titles and abstracts simultaneously. In WOS, we applied 
inclusion  criteria  to  some  of  the  12  strings,  considering  only  published  articles  written  in 
English and no older than 2010.  

91 

 
Table 9: Systematic review - Search strings 

#  Field 

IDEAS 

SCOPUS 

WOS 

SSRN 

1 

Employment 
(M) 

2 

Employment 
(E) 

(employment | 
unemployment | 
training) & ("meta 
analysis" | 
"systematic 
review") & (project 
| intervention | 
initiative | scheme | 
program | 
programme | policy 
| strategy) 

( TITLE-ABS-KEY ( employment  OR  
unemployment )  AND  TITLE-ABS-
KEY ( "meta analysis"  OR  
"systematic review" )  AND  TITLE-
ABS-KEY ( project  OR  intervention  
OR  initiative  OR  scheme  OR  
program  OR  programme  OR  
policy  OR  strategy ) )  AND  
SUBJAREA ( mult  OR  ceng  OR  
chem  OR  comp  OR  eart  OR  ener  
OR  engi  OR  envi  OR  mate  OR  
math  OR  phys  OR  mult  OR  arts  
OR  busi  OR  deci  OR  econ  OR  
psyc  OR  soci )  

(employment | 
unemployment | 
training) & ("impact 
evaluation" | 
"impact 
assessment") & 
(project | 
intervention | 
initiative | scheme | 
program | 
programme | policy 
| strategy) 

( TITLE-ABS-KEY ( employment OR 
unemployment) AND TITLE-ABS-
KEY ( "impact evaluation" OR 
"impact assessment" ) AND TITLE-
ABS-KEY ( project OR intervention 
OR initiative OR scheme OR 
program OR programme OR policy 
OR strategy ) ) AND SUBJAREA ( 
mult OR ceng OR chem OR comp OR 
eart OR ener OR engi OR envi OR 
mate OR math OR phys OR mult OR 
arts OR busi OR deci OR econ OR 
psyc OR soci ) 

((TS=(employment OR 
unemployment) AND 
TS=("meta analysis" OR 
"systematic review")) AND 
TS=(((((((project OR 
intervention) OR initiative) 
OR scheme) OR program) 
OR programmed) OR 
policy) OR strategy)) 

employment "meta analysis" ; 
unemployment "meta analysis"; 
training "meta analysis"; 
employment "systematic 
review"; unemployment 
"systematic review"; training 
"systematic review" 

((TS=(employment OR 
unemployment) AND 
TS=("impact evaluation" 
OR "impact assessment")) 
AND TS=(((((((project OR 
intervention) OR initiative) 
OR scheme) OR program) 
OR programmed) OR 
policy) OR strategy)) 

employment "impact 
evaluation"; unemployment 
"impact evaluation"; training 
"impact evaluation"; 
employment "impact 
assessment"; unemployment 
"impact assessment"; training 
"impact assessment" 

3 

Social 
Housing (M) 

(“social housing”) & 
(“systematic 
review” | “meta 
analysis”)     

( TITLE-ABS-KEY ( "social housing" 
)  AND  TITLE-ABS-KEY ( "meta 
analysis"  OR  "systematic review" ) 
) 

((TS=("social housing") 
AND TS=("meta analysis" 
OR "systematic review"))) 

"social housing" "meta 
analysis"; "social housing" 
"systematic review"; ; ; ;  

92 

 
4 

Social 
Housing (E) 

5 

Child 
Care(M) 

6 

Child Care 
(E) 

("social housing") & 
("impact 
evaluation" | 
"impact 
assessment") 

("childcare" | child | 
children | youth) & 
(health | care) & 
("meta analysis" | 
"systematic 
review") & (project 
| intervention | 
initiative | scheme | 
program | 
programme | policy 
| strategy) 

("childcare" | child | 
children | youth) & 
(health | care) & 
("impact 
evaluation" | 
"impact 
assessment") & 
(project | 
intervention | 
initiative | scheme | 
program | 
programme | policy 
| strategy) 

( TITLE-ABS-KEY ( "social housing" 
)  AND  TITLE-ABS-KEY ( "impact 
evaluation"  OR  "impact 
assessment" ) ) 

((TS=("social housing") 
AND TS=("impact 
evaluation" OR "impact 
assessment"))) 

"social housing" "impact 
evaluation"; "social housing 
"impact assessment"; ; ; ;  

( TITLE-ABS-KEY ( "childcare"  OR  
child  OR  children  )  AND  TITLE-
ABS-KEY ( health  OR  care )  AND  
TITLE ( "meta analysis"  OR  
"systematic review" )  AND  TITLE-
ABS-KEY ( project  OR  intervention  
OR  initiative  OR  scheme  OR  
program  OR  programme  OR  
policy  OR  strategy ) )  AND  
SUBJAREA ( mult  OR  arts  OR  busi  
OR  deci  OR  econ  OR  psyc  OR  
soci )  AND  PUBYEAR  >  2009 

((((TS=("childcare" OR 
child OR children) AND 
TS=(health OR care) AND 
TS=("meta analysis" OR 
"systematic review")) AND 
TS=(((((((project OR 
intervention) OR initiative) 
OR scheme) OR program) 
OR programmed) OR 
policy) OR strategy))))  

( TITLE-ABS-KEY ( "childcare"  OR  
child  OR  children  OR  youth )  
AND  TITLE-ABS-KEY ( health  OR  
care )  AND  TITLE-ABS-KEY ( 
"impact evaluation" OR "impact 
assessment" )  AND  TITLE-ABS-
KEY ( project  OR  intervention  OR  
initiative  OR  scheme  OR  program  
OR  programme  OR  policy  OR  
strategy ) )  AND  SUBJAREA ( mult  
OR  arts  OR  busi  OR  deci  OR  
econ  OR  psyc  OR  soci ) 

(((TS=("childcare" OR 
child OR children) AND 
TS=(health OR care) AND 
TS=("impact evaluation" 
OR "impact assessment")) 
AND TS=(((((((project OR 
intervention) OR initiative) 
OR scheme) OR program) 
OR programmed) OR 
policy) OR strategy))) 

childcare "meta analysis"; child 
"meta analysis"; children "meta 
analysis"; youth "meta 
analysis"; childcare "systematic 
review"; child "systematic 
review"; children "systematic 
review"; youth "systematic 
review" 

childcare "impact evaluation"; 
child "impact evaluation"; 
children "impact evaluation"; 
youth "impact evaluation"; 
childcare "impact assessment"; 
child "impact assessment"; 
children "impact assessment"; 
youth "impact assessment" 

93 

 
 
 
( TITLE-ABS-KEY ( "long term care"  
OR  "long term services" )  AND  
TITLE-ABS-KEY ( "meta analysis"  
OR  "systematic review" ) )  ( 
TITLE-ABS-KEY ( "long term care" )  
AND  TITLE ( "meta analysis"  OR  
"systematic review" )  AND  TITLE-
ABS-KEY ( project  OR  intervention  
OR  initiative  OR  scheme  OR  
program  OR  programme  OR  
policy  OR  strategy ) )  AND  
SUBJAREA ( mult  OR  arts  OR  busi  
OR  deci  OR  econ  OR  psyc  OR  
soci )  

((((TS=("long term care") 
AND TS=("meta analysis" 
OR "systematic review"))) 
AND TS=(((((((project OR 
intervention) OR initiative) 
OR scheme) OR program) 
OR programmed) OR 
policy) OR strategy))) 

"long term care" "meta 
analysis"; "long term care" 
"systematic review" 

( TITLE-ABS-KEY ( "long term care"  
OR  "long term services" )  AND  
TITLE-ABS-KEY ( "impact 
evaluation"  OR  "impact 
assessment" ) )   

((TS=("long term care") 
AND TS=("impact 
evaluation" OR "impact 
assessment"))) 

"long term care" "impact 
evaluation"; "long term care 
"impact assessment" 

( TITLE-ABS-KEY ( "social 
assistance" )  AND  TITLE-ABS-KEY 
( "meta analysis"  OR  "systematic 
review" )  AND  TITLE-ABS-KEY ( 
project  OR  intervention  OR  
initiative  OR  scheme  OR  program  
OR  programme  OR  policy  OR  
strategy ) )  

((TS=("social assistance") 
AND TS=("meta analysis" 
OR "systematic review")) 
AND TS=(((((((project OR 
intervention) OR initiative) 
OR scheme) OR program) 
OR programmed) OR 
policy) OR strategy)) 

"social assistance" "systematic 
review"; "social assistance 
"meta analysis" 

7 

Long Term 
Care (M) 

("long term care" | 
"long term 
services") & ("meta 
analysis" | 
"systematic 
review") 

8 

Long Term 
Care (E) 

9 

Social 
Assistance 
(M) 

("long term care" | 
"long term 
services") & 
("impact 
evaluation" | 
"impact 
assessment") 

("social assistance") 
& ("impact 
evaluation" | 
"impact 
assessment")  & 
(project | 
intervention | 
initiative | scheme | 
program | 
programme | policy 
| strategy) 

94 

 
 
 
10 

Social 
Assistance 
(E) 

("social assistance") 
& ("meta analysis" | 
"systematic 
review")  & (project 
| intervention | 
initiative | scheme | 
program | 
programme | policy 
| strategy) 

11 

Social 
Inclusion 
(M) 

(“social inclusion”) 
& (“systematic 
review” | “meta 
analysis”)     

12 

Social 
Inclusion 
(E) 

(“social inclusion”) 
& (“impact 
evaluation” | 
“impact 
assessment”)     

( TITLE-ABS-KEY ( "social 
assistance" )  AND  TITLE-ABS-KEY 
( "impact evaluation"  OR  "impact 
assessment" )  AND  TITLE-ABS-
KEY ( project  OR  intervention  OR  
initiative  OR  scheme  OR  program  
OR  programme  OR  policy  OR  
strategy ) )  AND  SUBJAREA ( mult  
OR  ceng  OR  chem  OR  comp  OR  
eart  OR  ener  OR  engi  OR  envi  
OR  mate  OR  math  OR  phys  OR  
mult  OR  arts  OR  busi  OR  deci  
OR  econ  OR  psyc  OR  soci ) 

( TITLE-ABS-KEY ( "social inclusion" 
)  AND  TITLE-ABS-KEY ( "meta 
analysis"  OR  "systematic review" ) 
) AND  SUBJAREA ( mult  OR  ceng  
OR  chem  OR  comp  OR  eart  OR  
ener  OR  engi  OR  envi  OR  mate  
OR  math  OR  phys  OR  mult  OR  
arts  OR  busi  OR  deci  OR  econ  
OR  psyc  OR  soci ) 

( TITLE-ABS-KEY ( "social inclusion" 
)  AND  TITLE-ABS-KEY ( "impact 
evaluation"  OR  "impact 
assessment" ) )  AND  SUBJAREA ( 
mult  OR  ceng  OR  chem  OR  
comp  OR  eart  OR  ener  OR  engi  
OR  envi  OR  mate  OR  math  OR  
phys  OR  mult  OR  arts  OR  busi  
OR  deci  OR  econ  OR  psyc  OR  
soci ) 

Source: Authors' elaboration. Note: (M) = Meta-analysis; (E) = Evaluation. 

95 

((TS=("social assistance") 
AND TS=("impact 
assessment" OR "impact 
evaluation")) AND 
TS=(((((((project OR 
intervention) OR initiative) 
OR scheme) OR program) 
OR programmed) OR 
policy) OR strategy)) 

"social assistance" "impact 
evaluation"; "social assistance" 
"impact assessment" 

((TS=("social inclusion") 
AND TS=("meta analysis" 
OR "systematic review"))) 

"social inclusion" "meta 
analysis"; "social inclusion" 
"systematic review" 

((TS=("social inclusion") 
AND TS=("impact 
evaluation" OR "impact 
assessment"))) 

"social inclusion" "impact 
evaluation"; "social inclusion" 
"impact assessment" 

 
Using  these  criteria  and  the  search  strings,  we  conducted  the  search  in  November  2016  on 
each database and for each string. After the search, results were downloaded and aggregated 
in a consolidated database. The search produced a total of 2,813 results across the 12 strings. 
After  checking  for  within-database  duplicates,  the  total  number  decreased  to  2,737  results. 
After  checking  for  across-database  duplicates,  the  total  number  decreased  to  2,400  results. 
Then, an in-depth content screening was carried out. As a result, the final selection consists of 
a total of 219 results, of which 155 meta-analyses and 64 evaluations.  

Table 10: Systematic review – full results 

# 

Field 

IDEAS  SCOPUS  WOS 

SSRN 
(1) 

SSRN 
(2) 

Total 
(1) 

Total 
(2) 

Total 
(3) 

Final 

1 

2 

3 

4 

5 

6 

7 

8 

9 

10 

11 

12 

Employment 
(M) 

Employment 
(E) 

60 

179 

77 

96 

65 

412 

381 

315 

47 

140 

254 

77 

88 

71 

559 

542 

440 

21 

Social Housing 
(M) 

10 

5 

Social Housing 
(E) 

0 

6 

3 

4 

0 

6 

0 

6 

18 

18 

14 

2 

16 

16 

9 

2 

Child Care (M)  32 

345 

687 

37 

25 

1,101  1,089  1,026  86 

Child Care (E)  33 

142 

135 

76 

60 

386 

370 

323 

26 

Long Term 
Care (M) 

Long Term 
Care (E) 

Social 
Assistance (M) 

Social 
Assistance (E) 

Social Inclusion 
(M) 

Social Inclusion 
(E) 

Total 

3 

0 

6 

1 

39 

44 

0 

31 

11 

0 

6 

3 

3 

2 

1 

0 

21 

15 

20 

0 

89 

14 

12 

0 

0 

0 

1 

0 

0 

0 

86 

86 

79 

12 

42 

42 

28 

10 

16 

16 

14 

3 

6 

6 

4 

1 

56 

56 

51 

5 

115 

115 

97 

4 

2,813  2,737  2,400  219 

Source:  Authors'  elaboration.  Note:  (M)  =  Meta-analysis;  (E)  =  Evaluation.  “SSRN  (1)”  includes  within-database 
duplicates. “Total (1)” includes other within-database duplicates. “Total (2)” includes across-database duplicates. 

96 

 
 
 
 
 
 
 
 
 
Policy-based grey literature 

Social innovation is addressed more or less directly in many policy areas that form the Europe 
2020  strategy.  The  most  important  development  has  been  the  launch  in  2013  of  the  Social 
Investment  Package  (SIP)  (European  Commission,  2013).  The  SIP  contains  the  idea  and 
ambition  that  a  number  of  social  policies  can  have  an  investment  character  and  produce 
returns  in  terms  of  human  capital  development  and  of  promoting  higher  employment  and 
output  levels.  The  SIP  urges  Member  States  to  prioritise  social  investment  and  the 
modernisation  of  their  welfare  systems  in  order  to  address  unemployment,  poverty,  social 
exclusion,  and  sustainability  challenges  brought  about  by  the  economic  crisis  and  the  ageing 
population  trends.  It  stresses  that  national  welfare  systems  should  aim  at  stabilising  the 
economy,  and  at  promoting  social  protection  and  social  investment.  Social  investment  is 
human  capital  investment,  or  any  set  of  policies  that  strengthen  people's  current  and  future 
capacity to participate in the labour market and in the social and economic life. In this respect 
SIP  aimed  at  improving  protection  for  EU  Citizens  also  by  stimulating  re-calibration  and 
strategic institutional innovation.  

Welfare  recalibration  has  proceeded  slowly  in  recent  years,  with  emergency  measures 
responding to the crisis taking precedence over a systematic and strategic modernisation with 
long-term  perspective.  Member  States  are  encouraged  to  pursue  active  policies  to  prioritise 
social  investment  and  modernise  their  welfare  systems:  spending  more  effectively  and 
efficiently to  ensure  adequate and sustainable social protection; investing in people's  skills to 
improve  opportunities  to  integrate  society  and  the  labour  market;  ensuring  social  protection 
systems respond to people's needs at critical moments during their lives. It is worth stressing, 
especially  with  respect  to  the  objectives  of  the  IESI  project  in  general  and  of  iFrame  2.0 43  in 
particular,  that  the  SIP  underscores  the  added  value  in  focusing  on  innovative  social  policies 
and embedding innovation in evidence-based policy-making.  

Of  the  seven  Europe  2020  flagship  initiatives,  two  deal  with  social  innovation  directly  – 
although  all  seven  are  related  to  a  certain  extent  (BEPA,  2014).  These  are  Innovation  Union 
and  the  European  Platform  against  Poverty  and  Social  Exclusion.  The  Innovation  Union is  the 
EU  strategy  to  create  an  innovation-friendly  environment  in  Europe.  Built  around  34  specific 
commitments, it consists of a bold, integrated and strategic approach to innovation combining 
various  policies,  tool  and  levels  of  intervention  –  Member  States  and  EU  –  to  improve 
framework  conditions  and  access  to  finance  for  research  and  innovation,  so  that  innovative 
ideas  can  translate  into  products  and  services  and  create  growth  and  jobs.  The  Innovation 
Union  states  that  “Social  innovation  is  an  important  new  field  which  should  be  nurtured”.  It 
also  states  that  “As  well  as  meeting  social  needs  and  tackling  societal  challenges,  social 
innovations empower people and create new social relationships and models of collaboration”, 
and that “while there is no shortage of good ideas, social innovations are not yet producing the 
impact that they should”. 

The main objective of the European Platform against Poverty and Social Exclusion is to 
help  Member  States  to  ensure  cohesion  in  social,  economic  and  territorial  terms.  Social 
innovation  is  a  means  to  address  poverty  dynamics,  to  help  recognising  specific  expertise  of 
people  experiencing  poverty  and  to  facilitate  a  partnership  between  civil  society,  social 
partners,  and  Member  States.  The  initiative  consists  of  five  areas  for  action,  two  of  which 
relate  directly  to  social  innovation:  “promoting  evidence-based  policy  innovation”  and 
“promoting  a  partnership  approach  to  the  social  economy”.  With  regards  to  policy  aimed  at 
deepening  and  consolidating  the  Single  Market,  the  Single  Market  Act  I  defined  twelve 
priorities to re-launch Europe's growth and social progress. The eighth priority encourages new 
emerging  business  models  where  "social,  ethical  or  environmental  objectives"  are  pursued 
alongside  financial  profit.  The  Social  Business  Initiative  (SBI)  aims  to  create  a  favourable 
environment  for  the  development  of  social  business  in  Europe;  and  the  Single  Market  Act  II 
foresees a methodology to measure the socio-economic benefits created by social enterprises.  

In  the  pillars  of  the  Youth  on  the Move  initiative  -  reduce  early  school  leaving,  modernise 
higher education, encourage mobility and youth employment - social innovation is highlighted, 
firstly  by  committing  to  young  people's  participation  in  the  design  of  policy  measures,  and 

43   See IESI - ICT-Enabled Social Innovation to support the implementation of the Social Investment Package at for a 

detailed description of the previous work conducted by the JRC. https://ec.europa.eu/jrc/en/iesi  

97 

 
                                           
secondly  by  encouraging  universities  to  improve  the  quality  of  their  courses  by  making  them 
more  responsive  to  student's  needs.  Youth  unemployment  is  also  tackled  through  more 
learning  experiences  and  more  possibilities  for  self-employment.  With  regards  to  energy  and 
environmental law, regulations such as the Energy Efficiency Directive (art 7 par 7, art 12, 
art 5 par 7, art 9) and the Eco-design Directive (art 3) relate to social innovation, whereas 
the  Citizen  Energy  Forum's  Vulnerable  Consumers  Working  Group  includes  it  in  its 
recommendations. Finally, the effort44 to design proper indicators to monitor improvements in 
the use of resources represents another relevant uptake of social innovation. 

Programme and project based sources 

The  EASI  programme  (2014-2020)  builds  on  its  predecessor  PROGRESS  (2007-2013)  and 
includes a dedicated budget to test social policy innovation, including: design, evaluation and 
larger-scale  implementation  of  new  social  policy  initiatives  in  line  with  the  SIP  approach; 
exploring the role of PPPs in welfare reforms and investment in human capital; award schemes 
for  social  entrepreneurs;  social  policy  experimentation  in  support  of  SIP  actions,  such  as 
'housing first', 'one-stop-shops' and 'work in stations' projects.  

The  Horizon  2020  programme  links  research  with  innovation.  The  take-up  of  social 
innovation  here  is  across  all  themes.  Social  innovation  projects  in  FP7  include  CRESSI45 
(Creating  Economic  Space  for  Social  Innovation);  EFESEIIS46  (Enabling  the  flourishing  and 
evolution of social entrepreneurship for innovative and inclusive societies); ImPRovE47 (Poverty 
Reduction  in  Europe:  Social  policy  and  innovation);  ITSSOIN48  (Social  Innovation  and  Civic 
Engagement);  SEFORIS49  (Social  Enterprise  as  Force  for  more  Inclusive  and  Innovative 
Societies);  SI-DRIVE50  (Social  Innovation:  Driving  Force  of  Social  Change);  SIMPACT 51 
(Boosting  the  Impact  of  Social  Innovation  in  Europe  through  Economic  Underpinnings); 
SocIEtY52  (Social  Innovation  -  Empowering  the  Young  for  the  Common  Good);  TEPSIE53 
(Theoretical,  Empirical  and  Policy  Foundations  for  Social  Innovation  in  Europe);  Third  Sector 
Impact54  (The  Contribution  of  the  Third  Sector  to  Europe's  Socio-economic  Development); 
TRANSIT55 (Transformative Social Innovation Theory).  

Under  the  EU  Cohesion  Policy,  the  European  Social  Fund  (ESF)  and  the  European 
Regional Development Fund (ERDF) support social innovation in a number of ways. Social 
innovation  is  a  tool  to  improve  the  employment,  social  inclusion,  education  and  institutional 
capacity-building policies supported by the  ESF. The  ERDF can contribute to the development 
of  social  enterprises  by  promoting  new  business  models  and  innovative  solutions  to  address 
societal  challenges.  The  ERDF  will  also  be  able  to  support  ICT  development  by  supporting 
product  development  and  by  strengthening  the  use  of  existing  applications  in  eHealth, 
eLearning and eInclusion. Other programmes where social innovation could be funded include 
COSME,  Erasmus  for  Young  Entrepreneurs,  Erasmus+,  Creative  Europe,  and  the  Eco-
Innovation Action Plan. 

With  regards  to  one-off  EU  initiatives  and  projects,  a  distinction  can  help  between 
interventions  directed  to  EU  policymakers,  and  to  social  innovators.  The  former  includes: 
events,  such  as  workshops,  conventions,  and  open  days,  to  facilitate  coordination  among 
actors; guidance, provided by studies and analyses such as the Guide to Social Innovation, to 
help officers to programme social innovations in ESIF; or the Social Experimentation guide, to 
help testing interventions on a small population so as to evaluate its efficiency before deciding 
to  scale  it  up;  new  policy  instruments,  such  as  expert  groups  and  EIPs;  and  regulatory 

44   http://ec.europa.eu/eurostat/web/environmental-data-centre-on-natural-resources/resource-efficiency-indicators  
45   CRESSI.  http://www.sbs.ox.ac.uk/faculty-research/research-projects/creating-economic-space-social-innovation-

cressi  

46   EFESEIIS. Available at http://www.fp7-efeseiis.eu/ (retrieved 31 January 2017). 
47   IMPROVE. Available at http://improve-research.eu/ (retrieved 31 January 2017). 
48   ITSSOIN. Available at http://itssoin.eu/ (retrieved 31 January 2017). 
49   SEFORIS. Available at http://www.seforis.eu/ (retrieved 31 January 2017). 
50   SI-DRIVE. Available at https://www.si-drive.eu/ (retrieved 31 January 2017). 
51   SIMPACT. Available at http://www.simpact-project.eu/ (retrieved 31 January 2017). 
52   SociETY. Available at http://www.society-youth.eu/ (retrieved 31 January 2017). 
53   TEPSIE. Available at http://www.tepsie.eu/ (retrieved 31 January 2017). 
54   Third Sector Impact. Available at http://thirdsectorimpact.eu/ (retrieved 31 January 2017). 
55   TRANSIT. Available at http://www.transitsocialinnovation.eu/ (retrieved 31 January 2017). 

98 

 
                                           
initiatives.  With  regard  to  the  second  category,  the  starting  point  is  the  large  2014  EU-wide 
study “A mapping of the social enterprises and their ecosystems in Europe”56.  

Other  initiatives  for  social  innovators  include:  platforms  and  communities,  such  as  Social 
Innovation  Community57;  Digital  Social  Innovation58;  Collective  Awareness  Platforms  for 
Sustainability  and  Social  Innovation  (CAPS)59;  European  Workplace  Innovation  Network; 
Sharing  Experience  Europe  –  Policy  Innovation  Design60;  networks  for  incubators,  such  as 
BENISI61,  and  TRANSITION62;  and  awards,  such  as  the  European  Social  Innovation 
Competition63  by  DG  GROW,  The  Social  Innovation  Tournament64  by  the  EIB,  and  RegioStars 
Awards65 by DG REGIO.  

56  Social enterprises_ report, DG EMPL, http://ec.europa.eu/social/main.jsp?langId=it&catId=89&newsId=2149. 
57  Social Innovation Community. Available at: https://www.siceurope.eu/about-sic (retrieved 31 January 2017). 
58  Digital Social Innovation. Available at https://digitalsocial.eu/ (retrieved 31 January 2017). 
59  Collective  Awareness  Platforms 

for  Sustainability  and  Social 

Innovation 

(CAPS).  Available  at 

https://ec.europa.eu/digital-single-market/en/collective-awareness (retrieved 31 January 2017). 

60  Sharing Experience Europe, Policy Innovation Design. http://www.seeplatform.eu/ (retrieved 31 January 2017). 
61  BENISI. Available at http://www.benisi.eu/ (retrieved 31 January 2017). 
62  TRANSITION. Available at http://transitionproject.eu/ (retrieved 31 January 2017). 
63  European Social Innovation Competition. Availble at 

https://ec.europa.eu/growth/industry/innovation/policy/social/competition_en (retrieved 31 January 2017). 

64  The  Social  Innovation  Tournament.  Available  at  http://institute.eib.org/whatwedo/social/social-innovation-

tournament/ (retrieved 31 January 2017). 

65  RegioStars  Awards.  Availble  at  http://ec.europa.eu/regional_policy/en/regio-stars-awards/  (retrieved  31  January 

2017).  

99 

 
 
 
                                           
Annex IV – On experiments and their validity 

Random Assignment 

The  use  of  randomised  experiments  is  widely  considered  as  the  gold  standard  for  recovering 
unbiased  estimates  of  causal  parameters66.  Below  we  illustrate  both  intuitively  and  in 
formalised  fashion  how  the  counterfactual  logic  of  causation  works.  As  we  discuss  in  further 
details  in  in  next  paragraph,  experiments  overall  validity  depends  on  several  dimensions 
(internal  validity,  ecological  validity,  external  validity,  construct  validity),  it  varies  depending 
on  how  the  experiment  is  realised  (laboratory/online  vs.  field  experiments),  as  well  as  on 
specific design choices (i.e., between-subjects vs. within-subjects; main effect, fractional, and 
full factorial design). For the moment let us start from a hypothetical between-subject design 
without further specification of the context (lab vs. field) and of other design aspects; and let 
us  consider  only  internal  validity  defined  as  the  extent  to  which  the  experiment  warrants  a 
causal conclusion regarding the effect of D (treatment) on Y (outcome). The key challenges for 
internal validity that may undermine the capacity to discern causation from correlation are the 
'selection  bias'  and  'unobservable'  variables  (from  which  springs  the  endogeneity  problem). 
Imagine the classical Mr Smith has a headache and takes the aspirin (see figure below).  

Figure 37: Counterfactual causation 

Source: Open Evidence 

Once  he  does  take  the  aspirin  we  can  no  longer  observe  what  would  have  happened  had  he 
not  taken  it.  Lacking  this  information,  we  cannot  solve  the  simple  operation  Y1-Y0  (Smith's 
health taking and not taking the aspirin); the observed difference in the health status of Smith 
does not give us the causal effect.  

Let us now reason in terms of a research question that may be typical of social policy context. 
For instance, let us consider a treatment D in the form of a training programme for Long Term 
Unemployed (LTU) whose desirable outcome is that the treated find a job.  

Let  us  add  that,  we  are  not  considering  an  experiment  but  rather  an  hypothetical  real  world 
situation  where  the  chances  that  different  units  in  the  population  of  LTU  are  exposed  to  the 
information that this training for finding employment is available and that being aware decide 
to  participate  depend  on  their  individual  level  characteristics  (i.e.,  educational  level,  age, 
digital skills, etc.).  

We can proceed to discuss this situation and the logic underlying randomisation in formalised 
fashion.  The  logic  underlying  Randomised  experiments  is  often  explicated  in  terms  of  a 
notational system that has its origins in Neyman (1923) and Rubin (1974) and that we present 
below in summarised and simplified version. In a way what follows is what in econometrics is 
called the problem of identification67. 

66  Although  this  is  changing  and  some  have  argued  it  is  a  ‘bronze  standard’  unless  they  are  carried  out  strictly 

according to all the precepts, including double blinded randomisation (Berk, 2005). 

67  As Manski explains (1995, 2003), econometricians distinguish between the identification and estimation of causal 
relationships.  Estimation problems have to do with statistical issues of whether, given the data set analysed and 
the assumptions made about the relationship between the data set and the population, the parameters of interest 
are  efficiently  and  consistently  estimated,  or  statistical  validity.  Manski  remarks  (2003,  p.  12):  "Statistical 
inference seeks to characterize how sampling variability affects the conclusions that can be drawn from samples of 

100 

 
 
                                           
The formalised notation is as follows: 

•  We have observation on N units indexed by i= 1,…, N; 

•  We use the binary indicator Di to indicate whether the unit i was treated or not treated is a 
dummy  variable  with:  Di=0  if  unit  i  is  not  'treated'  (did  not  attend  training  course),  and 
Di=1 if unit i is treated (attended the training course); 

•  Yi is the observed outcome (found or not a job sometime after the  training is finished) for 

each population unit i; 

•  Yi= Y1i if Di=1 (unit i is treated); 

•  Yi= Y0i if Di= 0 (unit i is not treated); 

•  Let (Y1, Y0 )i be the two outcomes corresponding to the i-th population unit being treated 

or not treated, respectively 

•  Y1i- Y0i= unit level causal effect; 

•  Y0i= is the counterfactual for an individual who took the course; 

•  Y1i= is the counterfactual for an individual who did not take the course. 

Factual Outcome 

Counterfactual Outcome 

Treated (D=1) 

Non-Treated (D=0) 

Y1 

Y0 

Y0 

Y1 

If  a  specific  member  of  the  population  is  exposed  to  the  intervention  then  Y1  is  observable, 
while Y0 is irreversibly unobservable on that specific member. Il follows that Y1 and Y0 are the 
factual and observable  outcomes for treated and non-treated respectively. The counterfactual 
outcome  for  a  member  of  the  population  who  was  treated  is  Y0  and  for  one  who  was  not 
treated is Y1. Henceforth we simplify the notation abandoning the index i and simply using the 
suffix  1  or  0  but  with  the  important  notation  that  blue  refers  to  factual  outcomes  and  red  to 
counterfactual ones as per the summary table above. 

Now  let  us  proceed  illustrating  the  standard  logic  identity  focusing  on  the  average  treatment 
effect on the treated68. 

Average Treatment on the Treated (ATT) = E {Y1 – Y0 | D= 1} = E {Y1 | D= 1} – E {Y0 | D= 
1}; 

Evidently  the  second  term  of  the  identity  is  unknown  and  unobservable  as  it  represents  the 
counterfactual  for  the  treated;  it  means  whether  a  hypothetical  subject  who  attended  the 
course  would  have  found  a  job  or  not,  had  he/she  not  attended  the  course;  even  more 
concretely  if  we  have  Marc  attending  the  course  and  finding  a  job,  then  the  second  term 
means would have Marc found a job had he not taken the course?  

Let us now consider the term E {(Y1 | D= 1) – E {Y0 | D= 0}, which represents the factually 
observed  difference  in  means  between  treated  and  non-treated  that  is  observed  and  known; 
then we add and subtract to it the term E {Y0 | D= 1}, that is the counterfactual for the non-
treated (Laura did not attend the course, what would her employment status has she taken the 
course? Again this is unknown and unobservable).  

limited  size."  In  contrast,  an  identification  problem  exists  when  it  is  problematic  to  establish  causal  inferences 
even  if  the  researcher  has  an  unlimited  sample  from  the  population.  In  the  logical  notation  of  identification  one 
considers the possibility of recovering a causal parameter in abstract but under the hypothesis of no limitation in 
data.  In  this  respect  the  econometric  concept  of  identification  is  analogous  to  one  of  the  three  dimensions  into 
which the internal validity of experimental research can be broken down that is , as we shall see in § 2.2, ‘causal 
validity’.  

68  This  is  the  causal  effect  of  major  interest  from  a  policy  perspective;  the  other  important  effect  is  the  Average 
Treatment Effect (ATE) defined as E {Y1 – Y0} or the average effect that would result from having all population 
members  (or  none  at  all)  take  part  in  the  programme.  ATE  perhaps  is  not  a  crucial  parameter  of  interest  for 
impact evaluation. It is the most relevant parameter when the programme under consideration is universal, in the 
sense that it would expose all units of the target-population to treatment. 

101 

 
 
                                                                                                                                            
We, thus, get: E {(Y1 | D= 1) – E {Y0 | D= 0} = [E {Y1 I D = 1} – E {Y0 I D = 1}] + [{E {Y0 
I D = 1} – E {Y0 | D= 0}]; in word (following the notation of using bold, underline, and italic) 
this  means  that  Observed  Means  Differences=  Average  Treatment  on  the  Treated  (ATT)+ 
Selection  Bias.  So,  the  ATT  and  the  selection  bias  cannot  be  disentangled  and  simply 
calculating the difference in observed means will not suffice to identify the causal effect of the 
treatment on the treated. The selection bias term of the identity, [{E {Y0 I D = 1}  – E {Y0 | 
D= 0}}, can be easily interpreted as the difference in the outcome Y that would be observed 
between  treated  and  non-treated  if  there  was  no  treatment,  and  depends  on  pre-existing 
differences between the two groups. That is, this term captures outcome differences between 
treated  and  non-treated  that  cannot  be  attributed  to  the  treatment.  The  first  term  of  the 
selection  bias  is  not  observable.  So,  unless  the  selection  bias  is  equal  to  zero  or  we  can  in 
some  way  control  it  (i.e.,  measure  it),  the  full  identity  above  cannot  be  given  a  causal 
interpretation. The selection bias is a function of X characteristics (possibly unobserved) of the 
treated and non-treated subjects. In other words, we are saying that a vector of Xs variables 
shapes the values of the variable D, or how the groups D=0 and D=1 are formed. With some 
additional equations we spare the readers, we can show that X is correlated to Y and hence the 
selection  bias  does  not  vanish,  since  the  two  groups  (treated  and  non-treated)  are  not 
equivalent with regard to Xs. In other words, there is a selection bias when: a) being treated 
or not depend on individual characteristics X known to affect also the outcome variable; and b) 
such characteristics are unequally distributed between treated and non-treated.  

Under  the  condition  of  a  randomised  experiment  the  selection  bias  vanish  and  then  we  can 
basically recover the causal effect as a  simple difference in means (see infra for a  regression 
based illustration of this). Under randomisation the dummy variable D is a toss of a coin and is 
not systematically related to either Y or Xs in anyway, hence the selection bias of the previous 
identity entirely vanishes (equal to zero). In concrete, if we randomise subjects to either take 
or  not  the  course,  their  unobserved  characteristics  would  no  longer  be  related  to  their 
employment  status  sometime  after  the  course  is  ended.  Well-performed  randomisation 
ensures  that  the  treated  and  non-treated  groups  are  balanced  with  respect  to  these 
unobserved characteristics. 

Under  the  condition  of  randomization,  thus,  the  treatment  effect  is  estimated  as  a  difference 
between  two  means.  Econometrically,  such  difference  in  mean  can  be  obtained  (if  it  exists) 
running a regression analysis where dependent variable is our outcome Y and the independent 
variable is a dummy equal to one if the subject took the course and zero otherwise. Let “y” be, 
for  instance,  “employment  status  after  a  time  lag  T”  measured  with  respect  to  whether 
subjects took (D=1) or not (D=0) the course. Hence, the general formula for the y is: 

y=a + b D1 + ε 

The  term  ε  includes  all  the  observable  and  unobservable  variables  that  impact  on  Y.  What  is 
important  is  that  under  randomization  the  expected  value  of  ε  under  treatment  (in  technical 
notation  E[ε|  D1])  is  equal  to  zero  (orthogonality  condition).  A  simple  Ordinary  Least  Square 
(OLS)  regression  is  capable  of  identifying  “b”,  since  the  exogeneity  condition  is  met.  The 
coefficient  “b”  is  precisely  the  treatment  effect.  In  fact,  for  those  that  did  not  receive  the 
treatment the expected value for y is: 

E[y | D1=0] = y0 = a 

While for the treated group we have: 

E[y | D1=1]= y1 = a + b 

So, by definition: 

b = y1 – y0 

This corresponds to the difference in the mean outcome between treated and untreated group, 
i.e.  the  treatment  effect.  The  coefficient  “b”  is  also  what  we  can  consistently  estimate  with 
Ordinary Least Square (OLS). 

102 

 
 
 
Validity 

Some  level  of  ambiguity  and  confusion  surrounds  the  validity  of  experimental  research  since 
different dimensions (internal, external, ecological) and, especially, different definitions of such 
dimensions are used. This springs particularly by the different norms and practices in different 
fields  and  especially in the  distinct  traditions  of  psychology  and  economics.  In  this  paragraph 
we  mostly  use  classic  sources  to  establish  the  baseline  that  will  inspire  our  approach  to  the 
design  of  behavioural  experiments.  To  start  with  we  can  take  the  essence  of  the  validity  of 
empirical  research  from  a  question  formulated  in  the  classic  work  of  Shadish,  Cook,  and 
Campbell  (2002)  and  the  answer  they  provide;  the  question  is  'what  one  can  believe  about 
what one learns from the data? The answer to this question is given by the overall validity of 
the  experiment  generating  such  data  that  can  be  defined  as  the  'approximate  truth'  of  the 
inference  or  knowledge  claim  made  on  the  basis  of  such  data.  This  is  an  intuitive  question 
given the discussion about how randomisation enables to recover unbiased estimates of causal 
relationships.  Random  assignment  enables  the  researcher  to  formulate  the  appropriate 
comparisons,  but  random  assignment  alone  does  not  ensure  that  the  comparison  will  speak 
convincingly to the original causal question.  

A  randomised  experiment  obviously  presents  an  ideal  opportunity  for  having:  a)  the  Data 
Generating Process (DGP) of the variable of intervention subject to exogenous variation; b) the 
participants to the study representative of the population object of the intervention; and if our 
in  a  laboratory  context  c)  the  behaviour  measured  in  the  lab  (on  average)  identical  to  the 
variable  of  interests  that  we  want  to  study  with  respect  to  our  theory  and/or  the  underlying 
real  environment  phenomenon  of  interest.  The  main  strategic  challenge  is  that  experiments 
may  remain  a  very  'local'  type  of  evidence  and  this  is  problematic  if  our  goal  is  to  support 
policymakers  who  are  interest  in  general  results.  As  matter  of  facts  all  of  the  literature  on 
experimental  validity  and  on  internal  and  external  validity  originated  in  the  theory  Campbell 
and  his  associates  developed  on  how  to  move  from  'local'  evidence  to  'generalisation'.  The 
problem  was  framed  as  one  of  extension  to  other  populations  and  settings  (external  validity) 
and of representation and measurement (construct validity). As noted  (McGraw and Hoekstra, 
1994;  Shadish,  et  al.,  2002),  in  this  effort  Campbell  early  on  abandoned  the  single  binary 
distinction between internal and external validity; first Cook and Campbell extended the notion 
of  internal  validity  (1979)  and  later  with  Shadish,  Cook,  and  Campbell  (2002),  incorporating 
clarifications advanced by Cronbach with his UTOS (Units, Treatments, Observation operations, 
Settings) framework (1982), validity was divided into four concepts: causal validity, construct 
validity, statistical validity, and external validity. Ecological validity, as we shall see it partially 
overlap  with  construct  validity  and  with  external  validity,  although  it  has  its  own  distinct 
tradition  in  psychology  starting  with  Brunswik's  suggestion  that  we  should  study  the 
participant's  setting  (1943),  and  then  being  further  developed  particularly  from  an  ecological 
perspective.  

If validity in general is the 'approximate truth the inference or knowledge claim', then internal 
validity  is  the  approximate  truth  the  inference  or  knowledge  claim  made  with  respect  to  a 
target population studied. Following the same general definition then external validity would be 
the approximate truth of the inference or knowledge claim for observations beyond the target 
population studied. 

Campbell  (1957)  considered  an  experiment  internally  valid  if  the  experimenter  finds  a 
significant difference between the treatment and control conditions. These differences are then 
assumed  to  provide  a  meaningful  reflection  of  the  causal  processes  at  play.  As  long  as  no 
reason  exists  to  assume  that  some  extraneous  mediating  factor  systematically  influenced 
subjects'  responses,  observers  can  attribute  changes  in  the  dependent  variable  to  systematic 
manipulations  across  the  independent  variables.  From  this  perspective,  internal  validity  is 
enhanced  by  experiments  that  are  well  designed,  carefully  controlled,  and  meticulously 
measured  so  that  alternative  explanations  for  the  phenomena  under  consideration  can  be 
excluded. Shadish, Cook, and Campbell (2002) are careful to note that internally valid findings 
remain discrete to the specific experimental context in which they are explored. As anticipated, 
Campbell and associates divided internal validity into three separate concepts: causal validity, 
construct validity, statistical validity. 

103 

 
Causal  validity.  Causal  validity  is  the  determination  of  whether  the  relationships  the 
researcher  finds  within  the  target  population  analysed  are  causal;  to  some  extent  it 
corresponds to the identification problem in econometrics that we mentioned earlier when we 
illustrated  the  formal  logic  behind  randomisation.    Establishing  causal  validity  for  the 
relationship between X and Y would mean establishing that changes in X causes changes in Y.  
Very  often  this  is  what  analysts  have  in  mind  when  they  consider  internal  validity,  whereas 
establishing a causal relation does not exhaust the criteria for assessing the internal validity of 
an  experiment,  if  we  recall  that  the  goals  are  those  of  extension  from  local  to  general 
knowledge  and  of  representation.  We  can  possibly  demonstrate  that  X  causes  variations  in  Y 
and that there are no other factor influencing them and yet score poorly in internal validity if 
the inferences we draw are flawed, for instance, by the fact that the variables we measured do 
not represent well the variable in the theory  or in the real world, which smoothly leads us to 
consider the next dimension of construct validity. 

Construct  validity.  First  and  foremost  construct  validity  has  to  do  with  how  valid  the 
inferences  of  the  data  are  for  the  theory  (or  constructs)  the  researcher  is  evaluating.  In 
experimental  research  the  question  is  whether  the  design  of  the  experiment  is  such  that  the 
variables investigated are closely equivalent to the variables the theory is concerned with. Are 
those  things  that  the  theory  holds  constant  held  constant in  the  experiment?  Are  the  choices 
before the subjects the same as the choices assumed in the theory? In other words, is there a 
close  match  between  what  the  theory  (the  real  world) is  about  and  what is  happening in  the 
manipulated  Data  Generating  Process?  As  the  theory  can  be  assumed  to  deal  with  what 
happens  in  the  real  world,  then  the  above  sentences  can  be  to  some  extent  be  rephrased  in 
terms of the adherence between the experiment and what happen in practice, for instance: are 
the choice before the subjects the same (or similar enough) as the choice they face in the real 
world? In this respect construct validity and ecological validity overlaps and the former is not 
immune  from  consideration  on  the  trade-off  between  'mundane'  and  'experimental'  realism. 
So,  although  construct  validity  is  a  dimension  of  internal  validity,  construct  validity  is  also 
about generalization and reflect the attempt to move from the local to the general by way of 
ensuring  valid  representation.  Results  from  experiments  with  high  construct  validity  can  help 
us answer more general questions than those without construct validity.  

Statistical  validity.  In  simple  terms  statistical  validity  depends  on  the  significance  of  the 
statistical relation estimated for the variables of interest and on the size of such relation. Given 
the assumption made and other aspects of the design and of the Data Generating Process, one 
must ask whether estimates are  efficient, accurate, significant, and sizable, and also whether 
the  data  set  is  representative  of  the  target  population.  Usually  other  aspects  of  validity  are 
considered more challenging and given more attention than statistical validity, as the latter is 
so  to  speak  'commoditized'  given  the  increasing  capacities  of  statistical  software  packages. 
Yet, there are questions that statistical packages do not solve and on which the research must 
reflect. What are in general the implications of different level of statistical significance? What if 
we find a relationship that is just on the edge of the 5% level? Also important is the extent to 
which  the  assumptions  about  variables  distribution  are  supported  and  whether  or  not  errors 
are correctly estimated. 

Mundane  versus  experimental  realism.  As  argued  (Aronson  et  al.,  1990),  despite  efforts 
internal  validity  remains  intrinsically  tied  to  experimental,  as  opposed  to  mundane,  realism. 
Those  who  are  sceptical  about  the  experimental  method  often  point  to  the  artificial  nature  of 
laboratory settings as a main weakness and obstacle to generalisation. The critique holds that 
the  tasks  presented  to  subjects  offer  a  poor  analogue  to  the  real-world  experiences  that 
individuals  confront.    Strong  proponents  of  the  experimental  method,  especially  in 
experimental  economics  game-theoretic  experiment  but  also  to  some  extent  in  psychology 
argue  on  the  contrary  that  stylized  and  stripped  down  settings  are  needed  to  carefully 
operationalise  and  measure  the  variables  of  interest  and  then,  through  multiple  tests  on 
numerous populations, to begin to define the conditions under which generality might obtain. 
The  reason  it  becomes  so  critical  to  uncover  these  mechanisms  is  because  unless  an 
investigator knows the underlying principles operating in a given dynamic, it will prove simply 
impossible to ascertain which aspect of behaviour is causing which effect within the context of 
real-world settings where many other variables and interactions occur simultaneously.  

104 

 
In  this  tradition  it  does  not  matter  whether  the  experimental  environment  does  not  overtly 
mimic  the  real-world  setting  as  long  as  the  subject  experiences  the  relevant  forces  that  the 
investigator seeks to elicit. A more balanced view is to recognise that if experimental subjects 
become  psychologically  disengaged  in  the  process  they  confront  and  do  not  pay  attention  in 
performing the tasks this clearly undermines internal validity and the possibility to extent the 
findings.  If  subjects  approach  a  task  with  scepticism  or  detachment,  then  genuine  responses 
fade  and  strategic  incentives  come  to  the  fore.  This  raises  the  possibility  that  measures 
obtained  do  not  accurately  reflect  the  process  being  manipulated,  but  rather  manifest  a 
different  underlying  construct  altogether.    This  is  a  clear  threat  to  construct  validity.  Hence, 
the motivation of the subjects and the internal experience of the experiment for them should 
ensure  engagement  and  for  this  to  occur  the  settings  need  not  necessarily  reflect  outside 
appearances.  

The  success  of  the  experiment  depends  on  the  subject  taking  the  task  seriously,  and 
experimenters can foster such engagement to the degree that they can create and establish a 
situation  that  forces  psychological  investment  on  the  part  of  subjects.  The  critical  operative 
feature in such experimental designs revolves around the ability of the experimenter to create 
a  psychological  situation  that  realistically  elicits  the  dynamics  under  consideration.  So, 
experimental realism remains more important than mundane realism in  maximising prospects 
for  internal  validity  because  it is  more  likely  to  elicit  the  critical dynamic  under investigation; 
but more highly stylized or abstract experimental protocols can risk both internal and external 
validity by failing to engage subjects' attention or interest.   

External validity refers to the generalizability of findings from a study, or the extent to which 
conclusions  can  be  applied  across  different  populations  or  situations;  too  often,  however, 
external  validity  is  simply  intended  as  a  matter  of  representativeness  or  of  the  size  of  the 
sample.  Privileging  of  external  validity  often  results  from  a  misunderstanding  that 
generalizability can result from single study as long as it is large enough or broad enough; this 
is  never  true.    External  validity  results  primarily  and  eventually  from  replication  of  particular 
experiments across diverse populations and different settings, using a variety of methods and 
measures.  

As put it by Aronson et al. 'No matter how similar or dissimilar the experimental context is to a 
real-life situation, it is still only one context: we cannot know how far the results will generalise 
to  other  contexts  unless  we  carry  on  an  integrated  programme  of  systematic  replication' 
(1990,  p.  77).  External  validity  is  the  extent  to  which  the  'causal  relationship  holds  over 
variations  in  persons,  settings,  treatments,  and  outcomes'  (Shadish,  Cook,  and  Campbell, 
2002, p. 83).  

Hence, it covers four aspects of experimental design:  

1)  whether  the  participants  resemble  the  actors  who  are  ordinarily  confronted  with  these 
stimuli;  

2) whether the context (including the time) within which actors operate resembles the context 
(and time) of interest;  

3) whether the stimulus used in the study resembles the stimulus of interest in the world; and  

4)  whether  the  outcome  measures  resemble  the  actual  outcomes  of  theoretical  or  practical 
interest.  

So,  the  challenge  for  external  validity  is  to  extend  to  other  subjects  (i.e.  representativeness 
strictu  sensu),  other  observations,  other  treatments,  and  other  settings  as  in  the  Units 
Treatments, Observing operations, Settings (UTOS) framework articulated by Cronbach (1982) 
for establishing valid causal inference. 

Finally, it is  worth  pointing  out  that  often  when  critics  say  that  an  experiment  does  not  have 
external  validity  they  just  have  in  mind  that  the  sample  used  (in laboratory)  is  not  randomly 
drawn;  yet,  this  has  to  do  with  internal  statistical  validity  and  not  with  external  validity. 
Random  sampling  from  a  target  population  does  not  necessarily  mean  that  a  result  is 
externally valid. External validity has to do with generalizing to populations beyond the target 
population,  so  whether  one  has  a  random  sample  from  the  target  population  tells  nothing 
about the external validity for other populations for which one has not taken a random sample.  

105 

 
As a starting point ecological validity can be seen as referring to the relation between real-
world phenomena and the investigation of these phenomena in experimental contexts. Yet, as 
for  other  dimension  of  validity  there  are  many  different  definitions  (Scheidt,  1981,  pp.  225-
226).  The  focus  on  ecological  validity  started  with  Brunswik's  (1943)  suggestion  to  study  the 
participant's  setting.  It  then  continued  with  Bronfenbrenner's  (1977)  echoing  of  the  same 
emphasis  and  with  Neisser  (1976)  arguing  that  the  stimulus  materials  should  be  ecologically 
valid;  this  was  also  the  concern  of  Gibson's  ecological  psychology  and  in  particular  on  the 
perceptual dimension of stimuli (1979).  

In brief, ecological validity concerns focussed on the realism of three aspects:  

a) the settings;  

b) the stimuli; and  

c) the responses.  

Ecological  validity  involves  maintaining  the  integrity  of  the  real-life  situation  in  the 
experimental  context  while  remaining  faithful  to  the  larger  social  and  cultural  context.  More 
intuitively  ecological  validity  is  similar  to  what  Harrison  and  List  (2004)  refer  to  as  the 
'fieldness' of an experiment.  

Ecological validity is a dimension that could be considered as a subset of both external validity 
and  of  construct  validity.  If  we  consider  it  in  terms  of  settings  (the  S  in  Cronbach's  UTOS 
framework,  see  infra  in  this  glossary)  then  obviously  ecological  validity  is  sub-dimension  of 
external validity.  

On the other hand, if we look at ecological validity as concerning the problems that may arise 
from the interaction between the aseptic (artificial) context typical of a laboratory experiment 
and  the  behaviour  of  subjects,  which  means  the  interaction  between  the  design  and  the 
measurement of the variable of interest, then lack of ecological validity can actually be seen as 
a  problem  of  construct  validity  inasmuch  as  the  measurement  generated  may  not  validly 
measure  the  underlying  construct  (phenomenon)  due  for  instance  to  subjects  disengagement 
with the experimental task.  It is about the similarity between the environment constructed in 
the research and a target environment. Some experimentalists call this mundane experimental 
realism  or  contextual  congruence.  The  experimental  environment  is  considered  ecologically 
valid  if  the  methods,  materials,  and  settings  of  the  research  are  similar  to  the  target 
environment.  

It  must  be  clarified,  however,  that  maximising  ecological  validity  may  or  may  not  enhance 
external  validity  of  the  results  because  the  target  environment  may  not  generalise.  External 
validity  can  be  conjectured  or  hypothesized  based  on  similar  studies  or  assumptions  about 
population  similarities  about  any  study,  experimental  or  non-experimental,  but  the  proof  of 
external validity is always empirical. 

In conclusion, generalising from RCTs, being field experiments or laboratory experiments, is 
a very difficult undertaking that requires theory accumulation and experiments replicability.  

As put it by Aronson  et al.: “Bringing the  research  out of the laboratory does not necessarily 
make  it  more  generalizable  or  “true”;  it  simply  makes  it  different.  The  question  of  which 
method – “artificial” laboratory experiments versus experiments conducted in the real world  – 
will provide the more generalizable results is simply the wrong question. The generalizability of 
any research finding is limited.  

This  limitation  can  be  explicated  only  by  systematically  testing  the  robustness  of  research 
results across different empirical realizations of both the independent and dependent variables 
via systematic replication to test the extent to which different translations of abstract concepts 
into concrete realizations yield similar results” (1990, 82). 

106 

 
 
 
 
Annex V - Synoptic summary of scientific literature review  

Table 11: Systematic review – Employment 

Active labour market programs (ALMP) in general, vocational training programmes, 
job search intervention 

Authors 

Geographical 
coverage 

Summary 

Greenberg, D. H., 
Michalopoulos, C., 
& Robins, P. K. 
(2003) 

U.S. 

Card, D., Kluve, J., 
& Weber, A. (2010) 

OECD 
countries 

Greenberg, D., & 
Cebulla, A. (2008).  

U.S. 

Kluve, J. (2010).  

OECD 
countries 

Card, D., Kluve, J., 
& Weber, A. 
(2015).  

Whole world 

The earnings effects of the programmes evaluated in the 
article seem to have been largest for women, quite 
modest for men, and negligible for youths. For men and 
women, the earnings effects of training appear to have 
persisted for at least several years after the training was 
complete.  

Job search assistance programmes yield relatively 
favourable programme impacts, whereas public sector 
employment programmes are less effective. Training 
programmes are associated with positive medium-term 
impacts, although in the short term they often appear 
ineffective. 

Costs of a typical welfare-to-work programme probably 
exceeded its benefits from the perspective of the 
government, but those assigned to the programme, and 
possibly society as a whole, may reap small positive net 
benefits.  

The article states that, rather than contextual factors 
such as labour market institutions or the business cycle, 
it is almost exclusively the programme type that seems 
to matter for programme effectiveness. While direct 
employment programmes in the public sector frequently 
appear detrimental, wage subsidies and "Services and 
Sanctions" can be effective in increasing participants' 
employment probability. 

1) Average impacts of ALMP are close to zero in the 
short run, but become more positive 2-3 years after 
completion of the programme; (2) the time profile of 
impacts varies by type of programme, with larger gains 
for programmes that emphasize human capital 
accumulation; (3) there is systematic heterogeneity 
across participant groups, with larger impacts for 
females and participants who enter from long term 
unemployment; (4) active labour market programmes 
are more likely to show positive impacts in a recession.  

107 

 
 
 
 
 
 
 
 
 
 
 
Liu, S., Huang, J. 
L., & Wang, M. 
(2014).  

Not specified 

Dar, A., & 
Tzannatos, Z. 
(1999) 

OECD 
countries 

Job search interventions that contain certain 
components, including teaching job search skills, 
improving selfpresentation, boosting self-efficacy, 
encouraging proactivity, promoting goal setting, and 
enlisting social support, are more effective than 
interventions that did not include such components. 
More important, job search interventions effectively 
promoted employment only when both skill development 
and motivation enhancement are included.  

ALMPS 1) help disadvantaged groups providing 
poverty/safety nets, but ineffective instruments in the 
pursue of permanent employment 2) can have positive 
impacts an be cost-effective but in general does not 
seem to improve quality of employment type and 
duration of contract, nor wages 3) Helpful under an 
improving economy 4) more expensive and ineffective 
than job search assistance 5) no positive impact on 
employment prospects. they cannot be a substitute to 
education systems failures 6) displacement effects, 
unlikely to have positive impacts. 

Nightingale, D. S., 
& Holcomb, P. A. 
(1997).  

U.S. 

ALMPS increase employment and earnings in some 
cases. Alsp reduction of welfare costs but in general only 
small gains in earnings that do not move most 
participants out of poverty. 

Walk, M., 
Greenspan, I., 
Crossley, H., & 
Handy, F. (2015).  

Canada 

Eppel, R., Horvath, 
T., & Mahringer, H. 
(2014).  

Austria 

Ragazzi, E., & 
Sella, L. (2013) 

Italy piedmont 

Clients evaluation of the impact of job-training and 
skills-building programmes. Findings illuminating the 
gap between expectations and actual experiences. 
Necessity of calibrating expectations when entering the 
programme and preparing to possible difficulties in the 
transitional period when facing with actual search of job. 
Importance of drawing participant's attention on by 
products of the programme. Self-improvement, nets of 
friendships and supports. 

Transitional employment in socio-economic enterprises, 
non-profit employment projects, training and socio-
pedagogical support to unemployed job-seekers are 
examined. Significant improvements of labour force 
participation and of individuals' employment prospects 
(women and older workers in particular) are found. 

The article examines employment rates of participants 
to vocational training courses targeting adults and 
young people. Employment rate of participants between 
38,0% and 42,5% with the odds of 95% , programme 
completion rate between 44,4% and 49,0%. Best 
outcomes concern adults with low educational level 
notably for programme preparing for medical care and 
social and long term care sector.  

108 

 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
Age stereotypes and employment 

Authors 

Dordoni, P., & 
Argentero, P. 
(2015).  

Geographical 
coverage 

Summary 

 Europe and U.S. 

The main findings of selected empirical studies 
indicated that: a) there is the need to focus on 
management policies for older workers; b) age 
stereotypes toward older workers are 
multidimensional and there is the simultaneous 
existence of both positive and negative stereotypical 
beliefs; c) managers' age stereotypes can affect 
several organizational outcomes. Self-categorization 
theory and intergroup bias are recommended as 
promising theoretical approaches for studying age 
stereotypes toward older workers and their cognitive 
identification with the age group. 

Graduated unemployment 

Authors 

Geographical 
coverage 

Summary 

Scurry, T., & 
Blenkinsopp, J. 
(2011) 

Area a not 
specified 

The review highlights significant issues around the 
conceptualisation and measurement of graduate 
under-employment. It argues that individual volition 
and meaning making are important issues that to 
date remain under-researched in relation to graduate 
under-employment.  

Unemployment insurance (UI) 

Authors 

Geographical 
coverage 

Summary 

EU countries 

O'Campo, P., 
Molnar, A., Ng, E., 
Renahy, E., 
Mitchell, C., 
Shankardass, K., & 
Muntaner, C. 
(2015).  

Poverty and psychological distress, among 
unemployed and even the employed, are impacted by 
generosity of UI in terms of eligibility, duration and 
wage replacement levels. Though unemployment 
benefits are not intended to compensate fully for a 
loss of earnings, generous UI programmes can 
moderate harmful consequences of unemployment. 

109 

 
 
 
 
 
 
 
 
 
Wage subsidies/Wage tax exemption for firms hiring part of their workforce locally 

Authors 

Geographical 
coverage 

Summary 

Gobillon, L., 
Magnac, T., & 
Selod, H. (2012).  

France 

Eppel, R. & 
Mahringer, H. 
(2012). 

Austria 

The article focuses on French enterprise zone 
programme: wage-tax exemption (about one third of 
total labour costs) to firms hiring at least 20% of their 
labour force locally. Effects on unemployment 
duration, distinguishing between short- and medium-
term effects are discussed. Results: small effect on 
the rate at which unemployed workers find a job 
(increased by a modest 3%). effect localised and 
significant only in the short run ( 3 years following 
the start of the policy). Cost-ineffective 

Wage subsidies schemes granted temporarily to 
employers who decide to recruit from particular 
groups of unemployed individuals aged 25-54 are 
examined. Results: Significant increase in 
employment for supported job seekers compared to 
similar non participants in the seven years from 
programme start. Considerably less time spent in 
unemployment and out of the labour force. 
Programme effectivity increases with age and pre-
treatment employment duration. Remarkable for 
older worker and long term unemployment. 
Participation has no significant impact on average 
wage level but there is a rise in cumulated earnings 
due to increase in employment. Wage subsidy seems 
therefore particularly effective in helping 
disadvantaged individuals back into employment. 

Youth training programmes 

Authors 

Geographical 
coverage 

Summary 

Employment impact of youth interventions tends to 
be more favorable in developing and transition 
countries than in industrialized countries. The meta-
analysis confirmed that the difference in programme 
impact by level of development remained even after 
the quality of the evaluation evidence was taken into 
account.  

Betcherman, G., 
Godfrey, M., 
Puerto, S., Rother, 
F., & Stavreska, A. 
(2007) 

42 from OECD 
countries; 24 
per cent Latin 
America, 14 per 
cent in Eastern 
Europe and 
Central Asia, 10 
per cent in 
SubSaharan 
Africa, 7 per 
cent in South 
and East Asia 
and the Pacific, 
3 per cent in the 
Middle East and 
North Africa. 

110 

 
 
 
 
 
 
 
Kluve, J., Puerto, 
S., Robalino, D., 
Romero, J. M., 
Rother, F., 
Stöterau, J., 
Weidenkaff, F., & 
Witte, M. (2016).  

Ibarraran, P., 
Kluve, J., Ripani, 
L., & Rosas, D. 
(2015).  

OECD  52%, 
Sub-Sahara 
Africa14%, 
Europe and 
Central Asia 4% 
Latin America 
and Caribbean 
21%, Middle 
East and North 
Africa 6%, South 
Asia  4% 

Santo Domingo 

Díaz, J. J., & Rosas 
Shady, D. (2016).  

Perú 

Just more than one-third of evaluation results from 
youth employment programmes implemented 
worldwide show a significant positive impact on 
labour market outcomes – either employment rates 
or earnings. In general, programmes have been more 
successful in middle- and low-income countries; this 
may be because these programmes' investments are 
especially helpful for the most vulnerable population 
groups – low-skilled, low-income – that they target. 

The long-term analysis shows that impacts of the 
programme examined are sustained and growing over 
time. There are no impacts on average employment, 
which is consistent with the low unemployment in 
countries with high informality and no unemployment 
insurance. Looking at the local labour market context, 
the analysis suggests that skills training programmes 
work particularly well in more dynamic local contexts, 
where there is actual demand for the skills provided. 

This paper evaluates the impact of The Peruvian Job 
Youth Training Programme (Projoven). The evaluation 
finds a high long term positive impact of Projoven on 
formal employment. It also finds certain 
heterogeneity of programme impacts across 
subpopulations. Impacts on formal employment vary 
depending on the beneficiaries' gender and age, with 
different patterns of statistical significance depending 
on the data source used to measure employment 
formality. Finally, it does not find significant impacts 
on socio-emotional skills. 

Women's employability 

Todd, P. E. (2012) 

Non OECD 
countries, 
developing and 
transition 
economies Latin 
america Asia 
Africa 

The article evaluate various labour market 
programmes, education and training programmes, 
programmes that facilitate work (childcare subsidies, 
parental leave programmes and land titling 
programmes), microfinance programmes, 
entrepreneurship and leadership programmes. 
Results: many programmes effective in increasing 
women's quantity of work as measured by increased 
rates of labour market participation and number of 
hours worked. In some cases, the programmes also 
increased women's quality of work, for example, by 
increasing the capacity for women to work in the 
formal rather than the informal sector. 

111 

 
 
 
 
 
 
 
 
 
 
 
 
Table 12: Systematic review – Social inclusion 

Authors 

Geographical 
coverage 

Summary 

Seyfang, G. 
(2003). 

England 

'Time banks' are a new form of community currency in the UK 
which are receiving government support. Time credits are 
earned for each hour of voluntary service given, and can be 
used to purchase services from other members in return. This 
article discusses new findings from the first national study of 
time banks to assess their impacts and potential. Time banks 
are found to be successful at engaging socially excluded and 
vulnerable groups of people in community activities - many for 
the first time - boosting their confidence, social networks, skills 
and well-being. 

Misuraca, G., 
Centeno, C., 
& Torrecillas, 
C. (2014). 

Das, Maitreyi 
B. (2016). 

EU 

The text presents both a conceptual model and an operational 
framework with guidelines for self-evaluation of practices, with 
specific regard to interventions addressed to the use of ICTs to 
enhance employability of groups at risk of exclusion. 

Not specified 

This study used systematic review methods to synthesise 
evidence in three main areas: (i) how people with LTC or 
disabilities and their family carers find and access information 
about the quality of services; (ii) how quality information is 
used in decision-making; and (iii) what type of quality 
information is most useful. 

112 

 
 
 
 
 
 
Table 13: Systematic review – Long-term care 

Authors 

Geographical 
coverage 

Summary 

Makai, P., 
Brouwer, W. B. 
F., 
Koopmanschap, 
M. A., Stolk, E. 
A., & Nieboer, A. 
P. (2014) 

Da Roit, B., & Le 
Bihan, B. (2010) 

Not specified 

The review identifies four well wellbeing instruments for 
inclusion in economic evaluation and recommend the use of 
ICECAP-O and ASCOT  

Austria, 
France, 
Germany, 
Italy, the 
Netherlands, 
and Sweden 

 A new typology of long-term care configurations is 
proposed based on the inclusiveness of the system, the role 
of cash-for-care schemes and their specific regulations, as 
well as the views of informal care and the care work that 
they require. 

Turnpenny, A., & 
Beadle-Brown, J. 
(2014).  

UK 

Scott, J. C., & 
Elstein, P. (2004). 

U. S. 

North West of 
England 

Vassilev, I., 
Rogers, A., 
Blickem, C., 
Brooks, H., 
Kapadia, D., 
Kennedy, A., & 
Reeves, D. 
(2013).  

The analysis highlights the use of multiple sources of 
information in decision-making about services and in 
particular the importance of informal sources and extended 
social networks in accessing information. . Trust emerged a 
key issue in relation to quality information. Experiential and 
subjective information is highly valued and trusted. 

This article presents a descriptive, conceptual discussion of 
strategic issues and a summary of research that would be 
valuable in informing critical areas of decision-making in 
quality and outcomes research in long term care over the 
next decade. The issues, dilemmas, and challenges 
discussed are grouped into 5 categories: measurement 
methods and tools, uses of quality data, organizational and 
cultural factors, information and informatics, and impact 
evaluation and research. 

The article analysis the use of networks for illness 
management. Survey conducted on 300 people from 
deprived areas and with chronic illnesses. A network 
perspective offers an opportunity to redress the balance of 
an exclusively individual focus on self-management 
because it addresses the broader set of contributions and 
resources available to people in need of chronic illness 
management and support. 

113 

Annex VI – Review of relevant statistics and datasets 

The following table lists the statistical databases reviewed during the i-FRAME 2.0 revision. 

Table 14: Relevant statistics and datasets 

Source  Theme 

Protection and 
Well-being 

OECD 

Science, 
Technology 
and Patents 

ICT 

Labour 

Digital 
economy and 
society 
(t_isoc) 

Science and 
technology 
(t_scitech)  

Eurostat 

Quality of Life 
(QOL) 

Social 
protection 

Database 
Social Protection (Social Expenditure Aggregated data; Detailed 
data;  Reference Series; Pensions at a Glance; ELS Pensions; 
Income distribution and poverty) 
Wealth 
Benefits, Taxes and Wages (Key Indicators; TAXBEN Data 
Visualisation; Benefits, Taxes and Wages – Net Incomes) 
Gender (Gender, Institutions and Development Database; 
Education; Entrepreneurship; Employment) 
Family 
Child Well-Being 
OECD Science Technology and Industry Outlook 
Patents Statistics 
Research and Development Statistics 
Science and Technology Indicators 
ICT Access and Usage by Households and Individuals 
ICT Access and Use by Businesses 
Earnings 
Employment Protection 
Labour Force Statistics 
Labour Market Programmes 
Trade Union 
World Indicators of Skills for Employment 
ILOSTAT Database 
Job quality 
ICT usage in households and by individuals (t_isoc_i) 
ICT usage in enterprises (t_isoc_e)  
Digital skills (t_isoc_sk) 
ICT sector (t_isoc_se) 
Research and development (t_research) 
Community innovation survey (t_inn) 
High-tech industry and knowledge-intensive services (t_htec) 
Human Resources in Science & Technology (t_hrst) Intellectual 
property rights (t_ipr) 
Material living conditions 
Productive or main activity 
Health 
Education 
Leisure and social interactions 
Economic and physical safety 
Governance and basic rights 
Natural and living environment 
Overall experience of life 
Classes of indicators 
Social protection (t_spr) 
Expenditure on social protection (tps00098)  
Total expenditure on social protection per head of population. 
ECU/EUR (tps00099)   
Expenditure on social protection per inhabitant (tps00100) 
Total expenditure on social protection by type (tps00101) 
Total expenditure on social benefits (tps00102) 
Total expenditure on administration costs (tps00104) 
Other expenditure on social protection (tps00105) 
Social benefits by function - % of total benefits (tps00106) 

114 

 
 
 
 
 
 
 
 
 
  
  
  
Social benefits by function - million EUR (tps00082) 
Net social benefits by function (tps00083)   
Social benefits per head of population by function (tps00107)  
Expenditure on pensions (tps00103) 
Expenditure on care for elderly (tsdde530)   
Social protection receipts by type (tps00108)  
Income and living conditions (t_ilc)  
People at risk of poverty or social exclusion (Europe 2020 
strategy) (t_ilc_pe) 
Income distribution and monetary poverty (t_ilc_ip) 
Living conditions (t_ilc_lv) 
Material deprivation (t_ilc_md) 
Labour market policy (source: DG EMPL) (t_lmp)   
Public expenditure on labour market policies, by type of action 
(source: DG EMPL) (tps00076) 
Public expenditure on labour market policy measures, by type of 
action (source: DG EMPL) (tps00077) 
Public expenditure on labour market policy supports, by type of 
action (source: DG EMPL) (tps00078) 
Participants in labour market policy measures, by type of action 
(source: DG EMPL) (tps00079) 
Beneficiaries of labour market policy supports, by type of action 
(source: DG EMPL) (tps00080) 
Persons registered with Public Employment Services (source: DG 
EMPL) (tps00081) 
Population 
Health 
Housing 
Education 
Work 

Income, social 
inclusion and 
living 
conditions 

Labour Market 
Policy 

United 
Nations 

Social 
Indicators 

115 

 
  
  
 
 
  
  
  
  
  
  
 
 
 
Bibliography 

Ackermann,  F.,  Andersen,  D.F.,  Eden,  C.,  Richardson,  G.P.  (2010).  Using  a  group  decision 
support system to add value to group model building. System Dynamics Review, 26, 4, 335–
346. 

Ackoff, R. L. (1999). Ackoff's best: his classic writings on management. John Wiley & Sons. 

Adams,  Z.,  &  Deakin,  S.  (2014).  Re-regulating  zero  hours  contracts.  In  S.  F.  Deakin,  i.  b. 
Institute of Employment Rights & i. b. Centre for Business Research (Eds.). Liverpool: Institute 
of Employment Rights. 

Addabbo, T., García-Fernández, R., Llorca-Rodríguez, C., & Maccagnan, A. (2012). Poverty and 
Unemployment: The Cases of Italy and Spain. In G. Parodi & D. Sciulli (Eds.), Social Exclusion: 
Short  and  Long  Term  Causes  and  Consequences  (pp.  199-219).  Heidelberg:  Physica-Verlag 
HD. 

Akkermans,  H.  (2001).  Emergent  supply  networks:  System  dynamics  simulation  of  adaptive 
supply  agents.  System  Sciences.  Proceedings  of  the  34th  Annual  Hawaii  International 
Conference, IEEE. 

Ahrweiler,  P.  (2017).  Agent-based  simulation  for  science,  technology,  and  innovation  policy, 
Scientometrics, 110; 391-415  

Ahrweiler,  P.,  Schilperoord,  M.,  Pyka,  A.  &  Gilbert,  N.  (2015).  Modelling  research  policy:  Ex-
ante  evaluation  of  complex  policy  instruments,  Journal  of  Artificial  Societies  and  Social 
Simulation 18;5 

Alborg, M. (2009). Study to support the interim evaluation of  ICT-P  under the CIP: Technical 
Report-T-Seniority, 140-150. http://goo.gl/Vn2Ajq 

Alkire,  S.,  Jindra,  C.,  Robles,  G.  &  Vaz,  A.  (2016).  Multidimensional  Poverty  Index-Summer 
2016: Brief Methodological Note and Results. University of Oxford, June. 

Allen, G. (2011). Early Intervention: The Next Steps. An Independent Report to Her Majesty's 
Government. London: Her Majesty's Government. 

Angers,  K.  (2011).  The  Governance  of  Activation  in  Europe:  What  Can  We  Learn  from  a 
Literature Review: Canada-Europe Transatlantic Dialogue. 

Angrist,  J.,  &  Pischke,  J.  (2009).  Mostly  harmless  econometrics:  An  empiricist's  companion. 
Princeton, NJ: Princeton University Press. 

Anheier, H. K.; Krlev,  G.; Preuss, S.; Mildenberger, G.; Bekkers, R.; Mensink, W.; Bauer, A.; 
Knapp,  M.;  Wistow,  G.;  Hernandez,  A,  &  Adelaja,  B.,  (2014).  Social  Innovation  as  Impact  of 
the  Third  Sector.  Deliverable  1.1  of  the  project:  Impact  of  the  Third  Sector  as  Social 
Innovation  (ITSSOIN),  European  Commission  –  7th  Framework  Programme,  Brussels: 
European Commission, DG Research. 

Annoni, P., Dijkstra, L. (2016). THE EU REGIONAL SPI: A MEASURE OF SOCIAL PROGRESS IN 
THE 
at 
http://ec.europa.eu/regional_policy/sources/information/maps/methodological_note_eu_spi_2
016.pdf (retrieved 24 January 2017).  

Commission, 

REGIONS. 

European 

Available 

REGIO. 

DG 

EU 

Arjona,  R.,  Ladaique,  M.,  &  Pearson,  M.  (2002).  Social  Protection  and  Growth.  Paris:  OECD 
Economic Studies No. 35, 2002/2. 

Aronson,  E.,  Phoebe,  C.,  Ellsworth,  J.,  Carlsmith,  M.,  et  al.  (1990).  Methods  of  Research  in 
Social Psychology. 2nd ed. New York: McGraw-Hill. 

Arthur,  W.  (2013).  Complexity  Economics:  A  Different  Framework  for  Economic  Thought. 
Santa Fe, Ca.: Santa Fe Institute Working Paper No. 2013-04-012. 

Arundel, A., & Hollanders, H. (2011). A taxonomy of innovation: How do public sector agencies 
innovate? - Results of the 2010 European Innobarometer survey of public agencies”. Brussels: 
European Commission, DG Enterprise. 

116 

 
Arvidson, M., Lyon, F., McKay, S., and Moro, D. (2010). The ambitions and challenges of SROI. 
Third Sector Research Centre. Working Paper 49. 

Askim,  J.,  Fimreite,  A.  L.,  Moseley,  A.,  &  Pedersen,  L.  H.  (2011).  One-Stop  Shops  For  Social 
Welfare: The Adaptation Of An Organizational Form In Three Countries.  Public Administration, 
89(4), 1451-1468. doi:10.1111/j.1467-9299.2011.01933.x 

Assenza,  T.,  &  Delli  Gatti,  D.  (2013).  E  Pluribus  Unum:  Macroeconomic  modelling  for  multi-
agent  economies.  Journal  of  Economic  Dynamics  and  Control,  37(8),  1659-1682.  doi: 
http://dx.doi.org/10.1016/j.jedc.2013.04.010 

Assenza,  T.,  Delli  Gatti,  D.,  &  Grazzini,  J.  (2015).  Emergent  dynamics  of  a  macroeconomic 
agent based model with capital and credit. Journal of Economic Dynamics and Control, 50, 5-
28. doi: http://dx.doi.org/10.1016/j.jedc.2014.07.001 

Atkinson, A., Piketty, T., & Saez, E. (2011). Top Incomes in the Long Run of History. Journal of 
Economic Literature, 49(1), 3-71.  

Atkinson,  T.  (2015).  Inequality:  What  Can  Be  Done?  Cambridge,  Mass.  /  London:  Harvard 
University Press. 

Autor,  D.  (2013).  The  "Task  Approach"  to  Labor  Markets  An  Overview.  Cambridge,  Mass: 
National Bureau of Economic Research (No. w18711). 

Autor, D. H., & Dorn, D. (2013). The Growth of Low-Skill Service Jobs and the Polarization of 
the 
1553-1597. 
doi:10.1257/aer.103.5.1553 

Labor  Market. 

Economic 

American 

Review, 

103(5), 

US 

Autor,  D.  H.,  Levy,  F.,  &  Murnane,  R.  J.  (2003).  The  Skill  Content  of  Recent  Technological 
Change: An Empirical Exploration. The Quarterly Journal of Economics, 118(4), 1279-1333.  

Axelrod,  R.  (1997).  The  Complexity  of  Cooperation:  Agent-Based  Models  of  Competition  and 
Collaboration, Princeton. 

Balke, T., & Gilbert, N. (2014). How Do Agents Make Decisions? A Survey. Journal of Artificial 
Societies and Social Simulation, 17(4), 13. doi:10.18564/jasss.2687 

Balshem, H., Helfand, M., Schünemann, H. J., Oxman, A. D., et al. (2011). GRADE guidelines: 
3.  Rating  the  quality  of  evidence.  Journal  of  Clinical  Epidemiology,  64(4),  401-406.  doi: 
http://dx.doi.org/10.1016/j.jclinepi.2010.07.015 

Bandt, J., Gadrey, J. (1994). Relations de service, marches de services, Paris: CNRS editions. 

Bayer, S.,  Barlow, J., Curry, C., (2007). Assessing the impact of a  care innovation: Telecare. 
System Dynamics Review, 23, 1, 61-80. 

BBI  (2011).  Study  on  Social  Services  of  General  Interest.  Final  Report  prepared  by  Bernard 
Brunhes  International  (F),  in  cooperation  with  London  School  of  Economics  (UK)  and  Work 
Research Center (IRL), for DG EMPLOY. 

Becker,  S.O.  (2012).  'EU  Structural  Funds:  Do  They  Generate  More  Growth?'  The  CAGE–
Chatham House Series, No 3, December 2012. 

Becker,  S.O.,  P.  Egger,  and  M.  von  Ehrlich  (2010).  'Going  NUTS:  The  Effect  of  EU  Structural 
Funds on Regional Performance', Journal 

Bellini, F. et al. (2016). Exploring Impacts of Collective Awareness Platforms for Sustainability 
and Social Innovation.  

BEPA  (2009).  Social  innovation  as  part  of  the  Europe  2020  strategy.  Brussels:  European 
Commission. 

BEPA  (2010).  Empowering  people,  driving  change:  social  innovation  in  the  European  Union. 
European Commission. 

Berk,  R.  (2005).  Randomised  experiments  as  the  bronze  standard.  Journal  of  Experimental 
Criminology, 1(4), 417-433. doi:10.1007/s11292-005-3538-2 

117 

 
Bernhardt, A. (2014). Labor Standards and the Reorganization of Work: Gaps in Data and 
Research. Berkeley: IRLE WORKING PAPER #100-14 
(http://www.irle.berkeley.edu/workingpapers/100-14.pdf). 

Bertrand,  M.,  Mullainathan,  S.,  &  Shafir,  E.  (2004).  A  Behavioral-Economics  View  of  Poverty. 
American Economic Review, 94(2), 419-423. doi:10.1257/0002828041302019 

Betcherman,  G.,  Godfrey,  M.,  Puerto,  S.,  Rother,  F.,  &  Stavreska,  A.  (2007).  A  review  of 
interventions  to  support  young  workers:  findings  of  the  youth  employment  inventory.  Social 
Protection discussion paper; no. SP 0715. Washington, DC: World Bank 

Biltgen,  P.T.  (May  2007).  A  Methodology  for  Capability-Based  Technology  Evaluation  for 
Systems-of-Systems. PhD thesis, Georgia Institute of Technology 

Blanchard,  O.  J.,  &  Summers,  L.  H.  (1986).  Hysteresis  and  the  European  Unemployment 
Problem. NBER Macroeconomics Annual, 1, 15-78. doi: 10.2307/3585159 

Blanchard, O., Cerutti, E., & Summers, L. (2015). Inflation and Activity – Two Explorations and 
their Monetary Policy Implications: NBER Working Paper No. 21726. 

Bloch,  C.,  &  Bugge,  M.  (2013).  Public  sector  innovation.  From  theory  to  measurement. 
Structural 
doi: 
http://dx.doi.org/10.1016/j.strueco.2013.06.008 

Dynamics, 

Economic 

133-145. 

Change 

27(0), 

and 

Bloom,  D.  E.,  Canning, D.,  &  Fink,  G.  (2010). Implications  of  population  ageing  for  economic 
growth. Oxford Review of Economic Policy, 26(4), 583-612.  

Bloom,  D.  E.,  Canning,  D.,  &  Sevilla,  J.  (2004).  The  Effect  of  Health  on  Economic  Growth:  A 
Production 
1-13. 
doi:http://dx.doi.org/10.1016/j.worlddev.2003.07.002 

Development, 

Approach. 

Function 

32(1), 

World 

Blume,  L.,  &  Durlauf,  S.  (2006).  Identifying  Social  Interactions:  A  Review.  In  Oakes  J  & 
Kaufman J (Eds.), Methods in Social Epidemiology. San Francisco: Jossey-Bass. 

Blundell,  R.,  &  Dias,  M.  C.  (2009).  Alternative  Approaches  to  Evaluation  in  Empirical 
Microeconomics. Journal of Human Resources, 44(3), 565-640.  

Boccara, N. (2004). Modelling Complex Systems. Springer. 

Bogliacino,  F.  (2014).  Inequality  and  Europe  2020.  Intereconomics,  49(5),  288-294. 
doi:10.1007/s10272-014-0511-1 

Bogliacino,  F.,  Codagnone,  C.,  &  Veltri,  G.  (2015).  The  Behavioural  Turn  in  Consumer  Policy: 
Perspectives  and  Clarifications.  Intereconomics,  50(2),  108-114.  doi:10.1007/s10272-015-
0532-4 

Bohrnstedt,  G.,  &  Stecher,  B.  (Eds.).  (2002).  What  We  have  Learned  about  Class  Size 
Reduction in California. Sacramento: California Department of Education. 

Börsch-Supan,  A.  (2013).  Myths,  scientific  evidence  and  economic  policy  in  an  aging  world. 
The Journal of the Economics of Ageing, 1–2, 3–15.  

Borshev, A. (2013). The Big Book of Simulation Modelling, edited by Anylogic NA. 

Borshev, A., Filippov, A. (2004). From system dynamics and discrete event to practical agent 
based  modelling:  reasons,  techniques,  tools.  In  Proceedings  of  the  22nd  international 
conference of the system dynamics society (Vol. 22). 

Bowles,  J.  (2014,  24  July).  The  computerisation  of  European  jobs.  Retrieved  from 
http://bruegel.org/2014/07/the-computerisation-of-european-jobs/ (28-12-2015) 

Brannen,  J.  (1992).  Combining  Qualitative  and  Quantitative  Approaches:  An  Overview.  In  J. 
Brannen  (Ed.),  Mixing  Methods:  Qualitative  and  Quantitative  Research  (pp.  3-37).  Avebury: 
Aldershot. 

Brittin, J., Araz, O. M., Nam, Y., & Huang, T. K. (2015). A system dynamics model to simulate 
sustainable  interventions  on  chronic  disease  outcomes  in  an  urban  community.  Journal  of 
Simulation, 9(2), 140-155. 

118 

 
Bronfenbrenner, U. (1977). Toward an experimental ecology of human development. American 
Psychologist, 32(513-531).  

Brunswik,  E.  (1943).  Organismic  achievement  and  environmental  probability.  The 
Psychological Review, 50, 255–272.  

Bryman,  A.  (2006).  Integrating  quantitative  and  qualitative  research:  how  is  it  done? 
Qualitative Research, 6(1), 97-113.  

Brynjolfsson, E., & McAfee, A. (2012). Race Against the Machine: How the Digital Revolution Is 
Acclerating Innovation, Driving Productivity and Irreversibly Transforming Employment and the 
Economy. Mexington, MA: Digital Frontier Press. 

Brynjolfsson,  E.,  &  McAfee,  A.  (2014).  The  second  machine  age:  work,  progress,  and 
prosperity  in  a  time  of  brilliant  technologies  (First  Edition.  ed.):  New  York  :  W.  W.  Norton  & 
Company. 

Budapest Institute. (2014). Literature review and identification of best practices on integrated 
social service delivery, Report for the European Commission, DG EMPL. 

Bugge, M., Mortensen, P., & Bloch, C. (2011). Measuring Public Innovation in Nordic Countries: 
Report on the Nordic Pilot Studies, Analyses of Methodology and Results. OSLO: MEPIN, NIFU. 

Bugg-Levine,  A.,  and  Emerson,  J.  (2011).  Impact  investing:  Transforming  how  we  make 
money while making a difference. Innovations, 6.3: 9-18. 

Burchell, B. (2009). Flexicurity as a moderator of the relationship between job insecurity and 
psychological well-being. Cambridge Journal of Regions, Economy and Society, 2(3), 365-378. 
doi:10.1093/cjres/rsp021 

CA/GIIN (2015). Introducing the Impact Investing Benchmark.  

Cabinet Office (1999). Modernising Government. London: Stationery Office. 

Cahuc,  P.,  &  Kramarz,  F.  (2005).  De  la  précarité  à  mobilité,  vers  une  sécurité  sociale 
professionnelle : rapport au ministre de l'économie, des finances et de l'industrie et au ministre 
de l'emploi, du travail et de la cohésion sociale. Paris: La Documentation française. 

Campbell, D., & Stanley, J. (1963). Experimental and Quasi-Experimental Evaluations in Social 
Research. Chicago: Rand McNally. 

Cantril,  H.  (1965).  The  pattern  of  human  concerns.  New  Brunswick,  NJ:  Rutgers  University 
Press. 

Card,  D.,  Kluve,  J.,  &  Weber,  A.  (2010).  Active  Labour  Market  Policy  Evaluations:  A  Meta-
Analysis. The Economic Journal, 120(548), F452–F477. 

Card, D., Kluve, J., & Weber, A. (2015). What Works? A Meta Analysis of Recent Active Labor 
Market Program Evaluations: IZA, Discussion Paper Series, IZA DP No. 9236. 

Carrozza,  C.  (2015).  Democratizing  Expertise  and  Environmental  Governance:  Different 
Approaches  to  the  Politics  of  Science  and  their  Relevance  for  Policy  Analysis.  Journal  of 
Environmental Policy & Planning, 17(1), 108-126. doi:10.1080/1523908X.2014.914894 

Cartwright, N. and Hardie, J. (2012) Evidence based policy: a practical guide to doing it better, 
Oxford: Oxford University Press. 

Cartwright, N., & Runhardt, R. (2014). Measurement. In N. Cartwright & E. Montuschi (Eds.), 
Philosophy  of  Social  Sciences:  A  New  Introduction  (pp.  265-287).  Oxford:  Oxford  University 
Press. 

Carvalho,  S.,  &  White,  H.  (2004).  Theory-Based  Evaluation:  The  Case  of  Social  Funds. 
American Journal of Evaluation, 25(2), 141-160. doi:10.1177/109821400402500202 

Champion,  C.,  Bonoli,  G.  (2011).  Institutional  fragmentation  and  coordination  initiatives  in 
western  European  welfare  states.  Journal  of  European  Social  Policy,  21(4),  323-334. 
doi:10.1177/0958928711412220 

119 

 
Charalabidis,  Y.,  Loukis,  E.,  &  Androutsopoulou,  A.  (2012).  A  System  Dynamics  Approach  for 
Complex  Government  Policies  Design.  Application  in  ICT  Diffusion.  In  Proceedings  of  the 
International Conference on Modelling, Simulation and Visualization Methods (MSV) (p. 1). The 
Steering  Committee  of  the  World  Congress  in  Computer  Science,  Computer  Engineering  and 
Applied Computing (WorldComp). 

Chen,  H.  (2005).  Practical  program  evaluation:  Assessing  and 
implementation, and effectiveness. Thousand Oaks, CA: Sage. 

improving  planning, 

Chen,  H.  (2006).  A  Theory-Driven  Evaluation  Perspective  on  Mixed  Methods  Research. 
Research in the Schools, 13(1), 75-83.  

Chen,  H.-T.,  Rossi,  P.  H.  (1980).  The  Multi-Goal,  Theory-Driven  Approach  to  Evaluation:  A 
Model  Linking  Basic  and  Applied  Social  Science.  Social  Forces,  59(1),  106-122. 
doi:10.2307/2577835 

Chen,  P.  C.  (2014).  Research  on  Dynamic  Effects  of  Employability  of  Vocational  College 
Students  in  Taiwan.  International  Journal  of  Computer  Science  &  Information  Technology, 
6(1), 161. 

Christensen,  T.,  Lægreid,  P.  (2007).  The  Whole-of-Government  Approach  to  Public  Sector 
Reform. 
doi:10.1111/j.1540-
6210.2007.00797.x 

Administration 

1059-1066. 

Review, 

67(6), 

Public 

Cingano,  F.  (2014).  Trends  in  Income  Inequality  and  its  Impact  on  Economic  Growth.  Paris: 
OECD Social, Employment and Migration Working Paper No. 163. 

Clasen,  J.,  Clegg,  D.  (Eds.).  (2011).  Regulating  the  risk  of  unemployment.  National 
adaptations to post-industrial labour markets in Europe. Oxford: Oxford University Press. 

Clasen,  J.,  Duncan,  G.,  Eardley,  T.,  Evans,  M.,  et  al.  (2001).  Towards  “single  gateways”?  A 
cross-  national  review  of  the  changing  roles  of  employment  offices  in  seven  countries. 
Zeitschrift für ausländisches und internationales Sozialrecht, 15(1), 43-63.  

Codagnone, C. (2006). Compendium to eGEP Measurement Framework. Brussels: Delivered for 
the European Commission, DG Information Society. 

Codagnone,  C.  (2007).  Measuring  eGovernment:  Reflections  from  eGEP  Measurement 
Framework Experience. European Review of Political Technologies, 4, 89-106.  

Coleman, J. S., C. (1990). Foundations of Social Theory. Cambridge Mass.: The Belknap Press 
of the Harvard University Press. 

Collen, B. E. N., Loh, J., Whitmee, S., McRAE, L. O. U. I. S. E., Amin, R., & Baillie, J. E. (2009). 
Monitoring  change  in  vertebrate  abundance:  the  Living  Planet  Index.  Conservation  Biology, 
23(2), 317-327. 

Cook,  T.,  &  Campbell,  D.  (1979).  Quasi-Experimentation:  Design  and  Analysis  for  Field 
Settings. Chicago: Rand McNally. 

Cooper,  H.,  Robinson,  J.  C.,  &  Patall,  E.  A.  (2006).  Does  Homework  Improve  Academic 
Achievement? A Synthesis of Research, 1987–2003. Review of Educational Research, 76(1), 1-
62. doi:10.3102/00346543076001001 

Coote,  A.  (2012).  The  Wisdom  of  Prevention:  Long-term  planning,  upstream  investment  and 
early 
Foundation, 
www.neweconomics.org/publications/entry/the-wisdom-of-prevention  

Economics 

London: 

prevent 

action 

harm. 

New 

to 

Corning, P.A., (1998). Complexity is just a word! Technological Forecasting and Social Change, 
vol. 59, 197-200. 

Coryn, C. L. S., Noakes, L. A., Westine, C. D., & Schröter, D. C. (2011). A systematic review of 
theory-driven  evaluation  practice  from  1990  to  2009.  American  Journal  of  Evaluation,  32(2), 
199-226. doi:10.1177/1098214010389321 

Council  of  the  European  Union.  (2015a).  Council  Conclusions  on  the  2015  Annual  Growth 
Survey and Joint Employment Report  — Political guidance on employment and social policies, 
Brussels 6147/15. 

120 

 
Council  of  the  European  Union.  (2015b).  Council  Recommendations  on  the  National  Reform 
Programmes. OJ 215/C 272 of 18.08.2015. 

Council of the European Union. (2016). Council Recommendation of 15 February 2016 on the 
integration of the long-term unemployed into the labour market. Brussels (2016/C 67/01), OJ 
C67/1-C67-4. 

Couture,  M.,  (1997).  Complexity  and  chaos  -  state-of-the-art;  overview  of  theoretical 
concepts. Technical Department of Defense, DoD Modelling and Simulation (M&S) Glossary. 

Crafts, N., (2004). Steam as a general purpose technology: A growth accounting perspective. 
The Economic Journal, 114(495), 338-351. 

Crepaldi, C., De Rosa, E., Pesce, F., (2012).  Literature review on innovation in social services 
in Europe – Sectors of health, education and welfare services. INNOSERV. 

Creswell,  J.  (2003).  Research  Design:  Qualitative,  Quantitative,  and  Mixed  Methods 
Approaches (2nd edition). Thousand Oaks, CA: Sage. 

Cronbach, L. (1982). Designing Evaluations of Educational and Social Programs. San Francisco: 
Jossey-Bass. 

Da Roit, B., & Le Bihan, B. (2010). Similar and Yet So Different: Cash-for-Care in Six European 
Countries' Long-Term Care Policies. Milbank Quarterly, 88(3), 286–309. 

Davies, H., Nutley, S., & Smith, P. (2000). What Works? Evidence-Based Policy and Practice in 
Public Services. Bristol: Policy Press in Evidence-based policy making: UNICEF. 

Davis  Jr,  F.  D.  (1986).  A  technology  acceptance  model  for  empirically  testing  new  end-user 
information systems: Theory and results. http://hdl.handle.net/1721.1/15192 

Dawid,  A.  (2007).  Counterfactuals,  hypotheticals  and  potential  responses:  a  philosophical 
examination of statistical causality. In Russo F & Williamson J (Eds.), Causality and Probability 
in  the  Sciences,  (pp.  503-532).  London:  College  Publications,  Texts  In  Philosophy  Series  Vol. 
5. 

Dawid,  H.  (2006).  Agent-based  Models  of  Innovation  and  Technological  Change.  In  L. 
Tesfatsion  &  L.  Kenneth  (Eds.),  Handbook  of  Computational  Economics,  Volume  2,  Agent-
based  Computational  Economics  (pp.  1235–1272).  Amsterdam  and  Boston:  Elsevier,  North-
Holland. 

Dawid,  H.,  &  Harting,  P.  (2012).  Capturing  Firm  Behavior  in  Agent-based  Models  of  Industry 
Evolution  and  Macroeconomic  Dynamics.  In  G.  Bünsdorf  (Ed.),  Evolution,  Organization  and 
Economic Behavior (pp. 103–130). Cheltenham and Northampton: Edward Elgar Publishing. 

Delli  Gatti,  D.,  Desiderio,  S.,  Gaffeo,  E.,  Cirillo,  P.,  et  al.  (2011).  Macroeconomics  from  the 
Bottom-up. Berlin: Springer Verlag. 

Delli Gatti, D., Gaffeo, E., Gallegati, M., Giulioni, G., et al. (2008). Emergent Macroeconomics. 
Berlin: Springer Verlag. 

Denzin, N., & Lincoln, Y. (Eds.). (2005). The Sage Handbook of Qualitative Research.  3rd Ed. 
Thousand Oaks, CA: Sage. 

Department  for  Education,  UK  Government,  (2011).  Munro  Review  of  Child  Protection,  Final 
Report,  available  at:  https://www.gov.uk/government/publications/munro-review-of-child-
protection-finalreport-a-child- centred-system. 

Department  of  Defense,  USA  Government,  (1998).  Modeling  and  Simulation  (M&S)  Glossary, 
DoD 5000.59-M, Department of Defense 

Department  of  Defense,  USA  Government,  (2010).  Update  of  the  DoD  M&S  Glossary. 
Proceedings of Simulation Interoperability Workshop, Florida, Orlando, USA 

Di Prete, T. A., Goux, D., Maurin, E., & Quesnel-Vallee, A. (2006). Work and pay in flexible and 
regulated  labor  markets:  A  generalised  perspective  on  institutional  evolution  and  inequality 
trends  in  Europe  and  the  U.S.  Research  in  Social  Stratification  and  Mobility,  24(3),  311-332. 
doi:10.1016/j.rssm.2006.04.001 

121 

 
Ditch,  J.,  Roberts,  E.  (2002).  Integrated  Approaches  to  Active  Welfare  and  Employment 
Policies. Dublin: European Foundation for the Improvement of Living and Working Conditions. 

Djellal,  F.,  and  Gallouj,  F.  (2001).  Patterns  of  innovation  organisation  in  service  firms:  postal 
survey 
theoretical  models.  Science  and  Public  Policy  28  57–67. 
doi:10.1093/spp/28.1.57. 

results  and 

Djellal, F., Gallouj, F.,  & Miles,  I. (2013). Two decades of  research  on innovation in services: 
Which  place  for  public  services?  Structural  Change  and  Economic  Dynamics,  27(0),  98-117. 
doi: http://dx.doi.org/10.1016/j.strueco.2013.06.005 

Donaldson, S. (2007). Program theory-driven evaluation science. New York: Lawrence. 

Dopfer,  K.  (2006).  The  Origins  of  Meso  Economics  Schumpeter's  Legacy.  The  Papers  on 
Economics and Evolution. Jena, Germany: Evolutionary Economics Group. 

Dopfer,  K.,  Foster,  J.  and  Potts,  J.(2004).  Micro-Meso-Macro.  Journal  of  Evolutionary 
Economics 14, 263-279. 

Dopfer,  K.,  Potts,  J.  (2008).  The  General  Theory  of  Economic  Evolution.  London;  New  York: 
Routledge. 

Dosi,  G.  (1988).  Sources,  Procedures,  and  Microeconomic  Effects  of  Innovation.  Journal  of 
Economic Literature, 26(3), 1120-1171.  

Dosi,  G.  (2013).  Economic  Organization,  Industrial  Dynamics  and  Development,  Selected 
Essays. Cheltenham and Northampton: Edward Elgar Publishing. 

Dosi,  G.,  Fagiolo,  G.,  &  Roventini,  A.  (2010).  Schumpeter  meeting  Keynes:  A  policy-friendly 
model of endogenous growth and business cycles. Journal of Economic Dynamics and Control, 
34(9), 1748-1767. doi: http://dx.doi.org/10.1016/j.jedc.2010.06.018 

Dosi,  G.,  Fagiolo,  G.,  Napoletano,  M.,  &  Roventini,  A.  (2013).  Income  distribution,  credit  and 
fiscal policies in an agent-based Keynesian model. Journal of Economic Dynamics and Control, 
37(8), 1598-1625. doi: http://dx.doi.org/10.1016/j.jedc.2012.11.008 

Dosi, G., Fagiolo, G., Napoletano, M., Roventini, A., et al. (2015). Fiscal and monetary policies 
in complex evolving economies. Journal of Economic Dynamics and Control, 52, 166-189. doi: 
http://dx.doi.org/10.1016/j.jedc.2014.11.014 

Dosi,  G.,  Napoletano,  M.,  Roventini,  A.,  &  Treibich,  T.  (2014).  Micro  and  Macro  Policies  in 
Keynes+Schumpeter  Evolutionary  Models.  Pisa,  Italy:  Working  Paper  Series  2014/21, 
Laboratory of Economics and Management (LEM), Scuola Superiore Sant'Anna. Retrieved from: 
http://papers.ssrn.com/sol3/papers.cfm?abstract_id=2523999SSRN: (09-06-2016). 

Dosi, G., Pereira, M., Roventini, A., & Virgillito, M. E. (2016a).  Labour market flexibility: more 
a source of macroeconomic fragility than a recipe for growth. Pisa, Italy: Working Paper Series 
Forthcoming, Laboratory of Economics and Management (LEM), Scuola Superiore Sant'Anna. 

Dosi,  G.,  Pereira,  M.,  Roventini,  A.,  &  Virgillito,  M.  E.  (2016b).  When  more  flexibility  yields 
more  fragility:  the  microfoundations  of  keynesian  aggregate  unemployment.  Pisa,  Italy: 
Working  Paper  Series  Forthcoming,  Laboratory  of  Economics  and  Management  (LEM),  Scuola 
Superiore Sant'Anna. 

Draghi,  M.  (2015).  Structural  reforms,  inflation  and  monetary  policy.  Paper  presented  at  the 
ECB Forum on Central Banking, Sintra.  

Dubiel  B.,  Tsimhoni,  O.  (2005).  Integrating  agent-based  modelling  into  a  discrete  event 
simulation. Proceedings of the 2005 Winter Simulation Conference, 1029-1037. 

Duggan, J., (2008). A simulator for continuous agent-based Modelling, in: Proceedings of the 
26th International Conference of the System Dynamics Society. 

Dunleavy,  P.,  &  Hood,  C.  (1994).  From  old  public  administration  to  new  public  management. 
Public Money & Management 3(14), 9-17.  

Dutta,  A.,  Roy,  R.,  &  Seetharaman,  P.  (2012).  System  Dynamics  Modelling  of  ICT  Diffusion. 
System, 7, 15. 

122 

 
Dutta, S., & Bilbao-Osorio, B. (Eds.). (2012). The Global Information Technology Report 2012 
Living in a Hyperconnected World. Geneva: World Economic Forum. 

Easterly,  W.,  Ritzen,  J.,  &  Woolcock,  M.  (2006).  Social  Cohesion,  Institutions,  and  Growth. 
Economics & Politics, 18(2), 103-120.  

Edmonds, B., (2000). The Use of Models – Making MABS More Informative, in: Moss, Scott and 
Paul Davidsson (Eds.) (2000): Multi-Agent-Based Simulation, Berlin, 15–32. 

Edmonds, B. and  Meyer, R. (forthcoming, 2017) Simulating Social Complexity  – a Handbook. 
2nd Edition. Springer. 

EFF  (2005).  Street  (UK):  Learning  From  Community  Finance.  London:  Esmée  Fairbairn 
Foundation (EFF). 

Eichhorst,  W.  (2013).  The  Unequal  Distribution  of  Labor  Market  Risks:  Permanent  vs. 
Temporary Employment. IZA VEF. Bonn.  

Eichhorst,  W.,  Braga,  M.,  Famira-Mühlberger,  U.,  Gerard,  M.,  et  al.  (2013).  Social  protection 
rights of economically dependent self-employed workers. Brussels: European Parliament. 

Eppel,  R.  &  Mahringer,  H.  (2012).  Do  wage  subsidies  work  in  boosting  economic  inclusion? 
Evidence on e. EcoMod 2012(4065). 

Eppel, R., Horvath, T., & Mahringer, H. (2014). The Effects of Subsidised Employment in Socio-
Economic  Enterprises  and  Non-Profit  Employment  Projects  on  the  Subsequent  Labour  Market 
Integration  of 
the  Unemployed.  Results  of  a  Microeconometric  Evaluation.  WIFO 
Monatsberichte (monthly reports), WIFO, vol. 87(11), pp. 783-794 

Epstein,  M.  J.,  Yuthas,  K.,  (2014).  Measuring  and  improving  social  impacts:  A  Guide  for 
nonprofits, companies, and impact investors. Berrett-Koehler Publishers. 

Eskinasi, M., Rouwette, E. A. J. A., & Vennix, J., (2011). Houdini: a system dynamics model for 
housing market reforms. In 29th International System Dynamics Conference. 

Esping-Andersen, G. (1990) The Three  Worlds of  Welfare Capitalism. Cambridge: Polity Press 
& Princeton: Princeton University Press, 1990. 

ESSC  (2011).  Sponsorship  Group  on  Measuring  Progress,  Well-being  and  Sustainable 
Development. Final Report adopted by the European Statistical System Committee. 

Etzioni,  A.,  &  Lehman,  E.  (1967).  Some  Dangers  in  'Valid'  Social  Measurements:  Preliminary 
Notes. The Annals of the American Academy of Political and Social Science, 373(1), 1-15.  

Eurofound (2015a, 5 May ). Latest Findings, European Jobs Monitor.  

Eurofound  (2015b).  New  forms  of  employment.  Luxembourg:  Publications  Office  of  the 
European Union. 

European Commission (2011a). Innobarometer 2010: Analytical Report on Innovation in Public 
Administration (Flash Eurobarometer 305). Brussels: European Commission, DG Enterprise. 

European  Commission  (2011b).  A  Quality  Framework  for  Services  of  General  Interest  in 
Europe. Brussels, COM(2011) 900 final. 

European  Commission  (2012).  Innobarometer  2011:  Innovation  in  the  public  sector,  its 
perception  in  and  impact  on  business  (Flash  Eurobarometer  343).  Brussels:  European 
Commission, DG Enterprise. 

European Commission (2013a). 'Social innovation research in the European Union Approaches, 
findings  and  future  directions:  Policy  Review'.  Brussels:  Directorate-General  for  Research  and 
Innovation Socio-economic Sciences and Humanities. 

European  Commission  (2013b).  Towards  Social  Investment  for  Growth  and  Cohesion  – 
including implementing the European Social Fund 2014-2020. Brussels, COM(2013) 83 final. 

European Commission (2014a). Annual Growth Survey 2015. Brussels, COM (2014) 902 final. . 

European  Commission  (2014b).  Commission  Work  Programme  2015.  A  New  Start  Brussels, 
COM(2014) 910 final. 

123 

 
European  Commission  (2014c).  Employment  and  Social  Developments  in  Europe  2013. 
Luxembourg: Publications Office of the European Union. 

European  Commission  (2014d).  European  Vacancy  and  Recruitment  Report  2014.  Brussels: 
European Commission. 

European Commission (2014e). A New Start for Europe: My Agenda for Jobs, Growth, Fairness 
and  Democratic  Change.  Political  Guidelines  for  the  next  European  Commission,  Opening 
Statement in the European Parliament Plenary Session by Jean Claude Juncker. 

European Commission (2015a). The 2015 Ageing Report: Economic and Budgetary Projections 
for  the  28  EU  Member  States  (2013-2060).  Brussels:  European  Commission,  Directorate 
General Economic and Financial Affairs. 

European  Commission  (2015b).  2015  European  Semester:  Assessment  of  growth  challenges, 
prevention and correction of macroeconomic imbalances, and results of in-depth reviews under 
Regulation (EU) No 1176/2011 Brussels, COM(2015) 85 final. 

European  Commission  (2015c).  Commission  Staff  Working  Document.  Analytical  Supporting 
Document  Accompanying  the  document  Proposal  for  a  Council  Recommendation  on  the 
integration  of  the  long-term  unemployed  into  the  labour  market.  Brussels,  SWD(2015)  176 
final / COM(2015) 462 final. 2015/0219 (NLE). 

European  Commission  (2015d).  Employment  and  Social  Developments  in  Europe  2014. 
Luxembourg: Publications Office of the European Union. 

European  Commission  (2015e).  Proposal  for  a  Council  Decision  on  guidelines  for  the 
employment policies of the Member States. Brussels, COM(2015) 98 final. 2015/0051 (NLE). 

European Commission (2015f). Proposal for a COUNCIL RECOMMENDATION On the integration 
of  the  long-term  unemployed  into  the  labour  market.  Brussels,  COM(2015)  462  final. 
2015/0219 (NLE). 

European  Commission  (2016a).  2016  European  Semester:  Assessment  of  progress  on 
structural reforms, prevention and correction of macroeconomic imbalances, and results of in-
depth reviews under Regulation (EU) No 1176/2011. Brussels, COM(2016) 95 final/2. 

European Commission (2016b). 2016 European Semester: Country-specific recommendations. 
Brussels, COM(2016) 321 final/2. 

European  Commission  (2016c).  Employment  and  Social  Developments  in  Europe  2015. 
Luxembourg: Publications Office of the European Union. 

European  Commission  (2016d).  Social  Policies  as  a  Productive  Factor.  Brussels:  European 
Commission, DG Employment, Social Affairs and Inclusion. Ares(2016)5531280 - 23/09/2016. 

European  Commission.  (2009). 
Commission 

Impact  Assessment  Guidelines.  Brussels:  European 

European  Council  (2014).  Employment,  Social  Policy,  Health  and  Consumer  Affairs  Council, 
07/12/2015. 

European Council (2016). Council Recommendation of 15 February 2016 on the integration of 
the long-term unemployed into the labour market. Brussels (2016/C 67/01), OJ C67/1-C67-4. 

European Parliament (2015). Report  on European Semester for  economic policy coordination: 
Employment and Social Aspects in the Annual Growth Survey 2015. Brussels (A8-0043/2015). 

European Trade Union Institute (2012). Benchmarking Working Europe 2012 Brussels: ETUI. 

European Trade Union Institute (2015). Benchmarking Working Europe 2015 Brussels: ETUI. 

Euroqol Copyright Group (1990). Euroqol: a new facility for the measurement of health-related 
quality of life. Health Policy, 16, 199-208. 

Eurostat  (2007).  Labour  Market  Policy,  2007  Edition.  Luxembourg:  Office  for  the  Official 
Publications of the European Communities. 

124 

 
Eurostat  (2012).  Labour  market  policy  –  expenditure  and  participants.  Data  2010. 
Luxembourg: Publications Office of the European Union. 

Eurostat (2013). Labour Market Policy Statistics, Methodology 2013. Luxembourg: Publications 
Office of the European Union. 

Eurostat (2016a). Euro area unemployment at 10.1%. Luxembourg: Eurostat, Newsrelease 
euroindicators, 163/2016. Retrieved from: 
http://ec.europa.eu/eurostat/documents/2995521/7601593/3-31082016-AP-EN.pdf/c416f4ad-
a1b4-4d29-b0f4-2adc41d4c951 (02-11-2016). 

Eurostat  (2016b).  Income  distribution  statistics.  Luxembourg:  Eurostat.  Retrieved  from: 
http://ec.europa.eu/eurostat/statistics-explained/index.php/Income_distribution_statistics 
(02-11-2016). 

from: 
Eurostat  (2016c).  Unemployment  Statistics.  Luxembourg:  Eurostat.  Retrieved 
http://ec.europa.eu/eurostat/statistics-explained/index.php/Unemployment_statistics  (02-11-
2016). 

Eurostat.  (2006).  Labour  Market  Policy  Database  Methodology.  Luxembourg:  Office  for  the 
Official Publications of the European Communities. 

EVPA  (2013).  A  Practical  Guide  to  Measuring  and  Managing  Impact.  European  Venture 
Philanthropy Association. 

EVPA,  (2013),  A  Practical  Guide  to  Measuring  and  Managing  Impact,  retrieved  from  EVPA 
website. 

Fagiolo,  G.,  &  Dosi,  G.  (2003).  Exploitation,  exploration  and  innovation  in  a  model  of 
endogenous growth with locally interacting agents. Structural Change and Economic Dynamics, 
14(3), 237-273. doi: http://dx.doi.org/10.1016/S0954-349X(03)00022-5 

Fagiolo,  G.,  &  Dosi,  G.  (2003).  Exploitation,  exploration  and  innovation  in  a  model  of 
endogenous growth with locally interacting agents. Structural Change and Economic Dynamics, 
14(3), 237-273. doi: http://dx.doi.org/10.1016/S0954-349X(03)00022-5 

Ferreira,  S.  and  Vincent,  J.  (2005).  Genuine  Savings:  Leading  Indicator  of  Sustainable 
Development? Economic Development and Cultural Change 53(3):737- 

Ferreira, S., Hamilton, K. Vincent, J. (2008). Comprehensive Wealth and Future Consumption: 
Accounting for Population Growth. World Bank Economic Review.  

FI-COMPASS  (2016).  Guide  on  Financial  instruments  under  the  ESF  and  EFSI,  including 
Investment  Bank,  available  at: 
microfinance,  European  Commission  and  European 
https://www.fi-compass.eu 

FI-COMPASS  (2017)  Innovative  use  of  financial  instruments  within  the  ESF,  European 
Commission and European Investment Bank available at: https://www.fi-compass.eu 

Folstein, M.F., Folstein, S.E., & McHugh, P. R. (1975). “Mini-Mental state”. A practical method 
for  grading  the  cognitive  state  of  patients  for  the  clinician.  Journal  of  Psychiatric  reports.  12, 
189-198. 

Foose, L., Folan, A. (2016). What Impact Investors Can Learn from the Microfinance Industry.  

Forrester,  J.W.,  (1994).  System  dynamics,  systems  thinking,  and  soft  OR.  System  Dynamics 
Review, 10(2‐3), 245-256. 

Freeman  A.M.  (1982).  Air  and  Water  Pollution  Control  –  A  Benefit−Cost  Assessment,  John 
Wiley & Sons, New York. 

Frey,  K.,  &  Osborne,  M.  (2013).  The  Future  of  Employment:  How  Susceptible  Are  Jobs  to 
Computerisation?  Oxford  Martin  School  Working  Paper.  Oxford.  Retrieved 
from 
http://www.oxfordmartin.ox.ac.uk/downloads/academic/The_Future_of_Employment.pdf  (28-
01-2016) 

Friends of Europe (2015). Unequal Europe. Recommendations for a more caring Europe. Final 
Report of the High-Level Group on 'Social Union'. 

125 

 
Funtowicz,  S.,  &  Ravetz,  J.  (1990).  Uncertainty  and  Quality  in  Science  for  Policy.  Dordrecht: 
Kluwer  

Funtowicz,  S.,  &  Ravetz,  J.  (1991).  A  new  scientific  methodology  for  global  environmental 
issues.  In  R.  Costanza  (Ed.),  Ecological  economics:  The  science  and  management  of 
sustainability (pp. 137-152). New York: Columbia University Press. 

Funtowicz, S., & Ravetz, J. (1993). Science for the post-normal age. Futures, 25(7), 739-755. 
doi:10.1016/0016-3287(93)90022-L 

Funtowicz, S., & Ravetz, J. (2008). Values and uncertainty. In G. Hirsch Hadorn, H. Hoffmann- 
Riem,  S.  Biber-Klemm,  W.  Grossenbacher-Mansuy,  D.  Joye,  C.  Pohl,  &  E.  Zemp  (Eds.), 
Handbook of transdisciplinary research (pp. 361–368). Heidelberg: Springer. 

Gadrey, J. (2003). Socio-économie des services, La Découverte, pp.128, Repères. 

Gambetta,  D.  (1988).  Trust:  Making  and  Breaking  Cooperative  Relations.  Oxford:  Basic 
Blackwell. 

Gash, V. (2008). Bridge or Trap? Temporary Workers' Transitions to Unemployment and to the 
Standard  Employment  Contract.  European  Sociological  Review,  24(5),  651-668. 
doi:10.1093/esr/jcn027 

Gaube  V.  et  al.,  (2009).  Combining  agent-based  and  stock-flow  Modelling  approaches  in  a 
participative analysis of the integrated land system in Reichraming, Austria. Landscape Ecology 
24 1149-1165. 

Gaved, M., and Anderson, B., (2006), The impact of local  ICT initiatives on social capital and 
quality of life, Chimera working paper, 2006. 

GECES  Sub-group  on  Impact  Measurement  (2014).  Proposed  Approaches  to  Social  Impact 
Measurement in the European Commission legislation and practice relating to: EuSEFs and the 
EaSI. GECES Group. 

Ghaffarzadegan,  N.,  Lyneis,  J.,  &  Richardson,  G.  P.,  (2011).  How  small  system  dynamics 
models can help the public policy process. System Dynamics Review, 27(1), 22-44. 

Gibson, J. (1979). The ecological approach to visual perception. Boston, MA Houghton Mifflin 

Giovannini,  E.,  &  Rondinella,  T.  (2012).  Measuring  equitable  and  sustainable  well-being  in 
Italy. In Quality of life in Italy (pp. 9-25). Springer Netherlands. 

GLA  (2010).  Standards  of  Evidence  the  Greater  London  Authority.  Oracle  Project.  London: 
Greater London Authority (GLA). 

Glaeser,  R.  (2014).  Secular  joblessness.  In  C.  Teulings  &  R.  Baldwin  (Eds.),  Secular 
Stagnation:  Facts,  Causes  and  Cures  (pp.  69-80).  London:  Centre  for  Economic  Policy 
Research (CEPR). 

Glaser,  B.,  &  Strauss,  A.  (1967).  The  discovery  of  grounded  theory:  strategies  for  qualitative 
research. Chicago: Aldine. 

Glass,  T.  A.,  &  McAtee,  M.  J.,  (2006).  Behavioral  science  at  the  crossroads  in  public  health: 
extending horizons, envisioning the future. Social science & medicine. 62(7), 1650-1671. 

Gobillon, L., Magnac, T., & Selod, H. (2012).  Do unemployed workers benefit from enterprise 
zones? The French experience. Journal of Public Economics, 96(9-10), 881–892. 

Gordon,  G.  (1966).  Simulation  languages  for  discrete  systems.  In Proceedings  IBM  Scientific 
Computing Symposium on Simulation Models and Gaming, White Plains, New York (101-118). 

Gordon, G., (1969). System Simulation. Prentice Hall. 

Gordon,  R.  (2014).  The  turtle's  progress:  Secular  stagnation  meets  the  headwinds.  In  C. 
Teulings  &  R.  Baldwin  (Eds.),  Secular  Stagnation:  Facts,  Causes  and  Cures  (pp.  47-59). 
London: Centre for Economic Policy Research (CEPR). 

Granovetter, M. (1973). The Strength of Weak Ties. American Journal of Sociology, 78, 1360-
1380.  

126 

 
Grant,  M.  J.,  &  Booth,  A.  (2009).  A  typology  of  reviews:  An  analysis  of  14  review  types  and 
associated methodologies. Health Information and Libraries Journal, 26(2), 91–108. 

Greenberg,  D.,  &  Cebulla,  A.  (2008).  The  Cost-Effectiveness  of  Welfare-to-Work  Programs:  A 
Meta-Analysis. Public Budgeting & Finance, 28(2), 112–145. 

Greenhalgh,  T.,  Robert,  G.,  Macfarlane,  F.,  Bate,  P.,  et  al.  (2004).  Diffusion  of  innovations in 
service  organizations:  systematic  review  and  recommendations.  Milbank  Q,  82(4),  581-629. 
doi:10.1111/j.0887-378X.2004.00325.x 

Greenhalgh,  T.,  Robert,  G.,  Macfarlane,  F.,  Bate,  P.,  et  al.  (2005).  Diffusion  of  innovations in 
service  organizations.  A  systematic  review.  (2004/12/15  ed.).  Oxford:  Blackwell  Publishing 
(BMJ Books). 

Guba, Y., & Lincoln, E. (1989). Fourth Generation Eualuation. London: Sage. 

Guyatt,  G.,  Berman,  L.,  Townsend,  M.,  Pugsley,  S,.  &  Chambers  L.  (1987).  A  measure  of 
quality of life for clinical trials in chronic lung disease. Thorax, 42,773-778. 

Hall, C., Galvez, M., & Isaac, M. (2014). Assumptions About Behavior and Choice in Response 
to  Public  Assistance.  Policy  Insights  from  the  Behavioral  and  Brain  Sciences,  1(1),  137-143. 
doi:10.1177/2372732214550833 

Hallqvist,  J.,  Lynch,  J.,  Bartley,  M.,  Lang,  T.,  &  Blane,  D.,  (2004).  Can  we  disentangle  life 
course  processes  of  accumulation,  critical  period  and  social  mobility?  An  analysis  of 
disadvantaged  socio-economic  positions  and  myocardial  infarction  in  the  Stockholm  Heart 
Epidemiology Program. Social Science & Medicine, 58(8), 1555-1562. 

Haynes, L., Service, O., Goldacre, B., & Torgerson, D. (2012). Test, Learn, Adapt: Developing 
Public  Policy  with  Randomised  Controlled  Trials.  London:  Cabinet  Office,  Behavioural  Insights 
Team. 

Heckman,  J.  J.  (2001).  Micro  data,  heterogeneity,  and  the  evaluation  of  public  policy:  Nobel 
lecture. Journal of Political Economy, 109(4), 673-748.  

Heckman,  J.  J.  (2005).  The  scientific  model  of  causality.  Sociological  Methodology  2005,  Vol 
35, 35, 1-97.  

Heckman,  J.  J.  (2008).  Econometric  causality.  International  Statistical  Review,  76(1),  1-27. 
doi:10.1111/j.1751-5823.2007.00024.x 

Heckman,  J.  J.  (2010).  Building  Bridges  between  Structural  and  Program  Evaluation 
Approaches 
to  Evaluating  Policy.  Journal  of  Economic  Literature,  48(2),  356-398. 
doi:10.1257/jel.48.2.356 

Hedstròm,  P.  (1994).  Contagious  Collectivities:  On  the  Spatial  Diffusion  of  Swedish  Trade 
Unions, 1890-1940. American Journal of Sociology, 99(5), 1157-1179.  

Hemerijck, A., Brian Burgoon, Alessandra Di Pietro, and Simon Vydra (2017). Assessing Social 
Investment Synergies (ASIS). European Commission: DG EMPL. 

Henry,  G.  T.  (2001).  How  Modern  Democracies  Are  Shaping  Evaluation  and  the  Emerging 
Challenges 
419-429. 
doi:10.1177/109821400102200320 

Evaluation.  American 

Evaluation, 

Journal 

22(3), 

for 

of 

Herbig,  B.,  Dragano,  N.,  &  Angerer,  P.  (2013).  Health  in  the  long-term  unemployed.  Dtsch 
Arztebl Int 110(23-24), 413-419.  

Hernandez,  M.  &  Hodges,  S.  (2003).  Crafting  Logic  Models  for  Systems  of  Care:  Ideas  into 
Action. 

Herzberg,  J.T.  (2014).  Foundations  in  human  services  practice:  A  generalist  perspective  on 
individual, agency, and community. Upper Saddle River: Pearson Education. 

Hessels,  L.  K.,  &  van  Lente,  H.  (2008).  Re-thinking  new  knowledge  production:  A  literature 
review 
740-760. 
research 
doi:10.1016/j.respol.2008.01.008 

Research 

agenda. 

Policy, 

37(4), 

and 

a 

127 

 
Higgins,  J.  PT.,  &  Green,  S.,  (2011).  Cochrane  handbook  for  Systematic  Reviews  of 
interventions, Cochrane Book series, Ed. A John Wiley & Sons, Ltd., Publication.  

Hirsch,  G.,  Homer, J.,  Milstein, B., Scherrer, L.,  Ingersoll, C., Landy,  L.,  & Fisher,  E., (2012). 
ReThink  Health  Dynamics:  Understanding  and  influencing  local  health  system  change. 
30th International Conference of the System Dynamics Society, St. Gallen, Switzerland. 

Holland,  P.  (1986b).  Statistics  and  Causal  Inference  -  Rejoinder.  Journal  of  the  American 
Statistical Association, 81(396), 968-970.  

Hollanders,  H.,  Arundel,  A.,  Buligescu,  B.,  Peter,  V.,  et  al.  (2013).  European  Public  Sector 
Innovation Scoreboard 2013. A pilot exercise. Brussels: European Commission, DG Enterprise. 

Homeless  Agency,  (2008).  Counted  in,  2008  –  A  report  on  the  extent  of  homelessness  in 
Dublin. Homeless Agency, SPSS. 

Homer  J  (2012).  Models  That  Matter:  Selected  Writings  on  System  Dynamics,  Grapeseed, 
Press, New York. 

Homer  J.B.,  Milstein  B,  Hirsch  G  (2007).  Chronic  illness  in  a  complex  health  economy:  the 
perils, and promises of downstream and upstream reforms, System Dynamics Review, 23, 2–
3, 313-343.  

Homer, J.B., (1999). Macro- and micro-Modelling of field service dynamics. System Dynamics 
Review 15, pages 139–162. 

Homer, J.B., Milstein B, Wile K, et al., (2010). Simulating and evaluating local interventions to 
improve cardiovascular health. Preventing Chronic Disease; 7(1). 

House,  J.  S.,  Lepkowski,  J.  M.,  Kinney,  A.  M.,  Mero,  R.  P.,  Kessler,  R.  C.,  &  Herzog,  A.  R., 
(1994).  The  social  stratification  of  aging  and  health.  Journal  of  Health  and  Social  Behavior, 
35(3), 213-234. 

Howick, S., & Whalley, J., (2008). Understanding the drivers of broadband adoption: the case 
of rural and remote Scotland. Journal of the Operational Research Society, 59(10), 1299-1311. 

Huber,  M.,  Maucher,  M.,  &  Sak,  B.,  (2006).  Study  on  Social  and  Health  Services  of  General 
Interest  in  the  European  Union.  Report  prepared  for  the  European  Commission  DG 
EMPLOYMENT. 

Hughes, W.,  ed., (1997). Military Modelling for Decision Making.  Military  Operations Research 
Society, 3rd ed. 

Hume, D. (1739). A Treatise of Human Nature. London: John Noon. 

Huy,  C.,  &  Schnieder,  S.  (2008).  Instrument  for  the  assessment  of  middle-aged  and  older 
adults'  physical  activity:  design,  eliability  and  application  of  the  German-PAQ-50+. 
Gerontology Geriatrics, 41, 208-216. 

Ibáñez,  F.,  Playfoot,  J.,  Fabregat,  M.  E.,  Costa,  M.,  Torres,  S.,  &  Cretu,  C.  (2010).  REPLAY 
project  gaming  technology  platform  for  social  reintegration  of  marginalised  youth.  In  CSEDU 
2010  -  2nd  International  Conference  on  Computer  Supported  Education,  Proceedings  (Vol.  2, 
489–495). 

Ilachinski,  A.,  (1996).  Land  warfare  and  complexity,  part  I:  Mathematical  background  and 
technical sourcebook. Memorandum CIM 461, Center for Naval Analyses. 

ILO (1997). Part-time work: Solution or trap? International Labour Review, 136(4), 557-579. 

ILO (2005). Social protection as a productive factor. Geneva: International Labour Office. 

ILO (2015). Non-standard forms of employment. Geneva: International Labour Office. 

Imbens,  G.  W.,  &  Wooldridge,  J.  M.  (2009).  Recent  Developments  in  the  Econometrics  of 
Program Evaluation. Journal of Economic Literature, 47(1), 5-86. doi:10.1257/jel.47.1.5 

Jaarsma, T., Strömberg, A., Mårtensson, J.,  &  Dracup K. (2003).  Development and testing of 
the European Heart Failure Self-Care Behaviour Scale. The European Journal of Heart Failure, 
5,363-370. 

128 

Jackson,  T.  (2004).  Chasing  progress:  Beyond  measuring  economic  growth.  New  Economics 
Foundation. 

Jacobs, M., Mazzucato, M. (2016). Rethinking Capitalism: Economics and Policy for Sustainable 
and Inclusive Growth, Wiley-Blackwell. 

Jankuj, M., & Voracek, J., (2015). Dynamic Modelling of national healthcare system. Measuring 
Business Excellence, 19(3), 76-89. 

Jäntti, M., Kanbur, R., & Pirttilä, J. (2014).  Poverty, Development, and Behavioral Economics. 
Review of Income and Wealth, 60(1), 1-6. doi: 10.1111/roiw.12085 

Jasanoff,  S.  (1990).  The  fifth  branch:  science  advisers  as  policymakers.  Cambridge,  MA: 
Harvard University Press. 

Jenkinson, C., Layte, R., & Jenkinson D. (1997). A shorter form health survey: Can the SF-12 
replicate results from the SF-36 in longitudinal studies? Journal of Public Health Medicine, 19, 
179-186. 

Jenson, J. (2012). Redesigning citizenship regimes after neoliberalism: moving towards social 
investment.  In  B.  Morel,  N.  Palme,  J.  Palier,  ed.  Towards  a  Social  Investment  Welfare  State? 
Ideas, Policies and Challenges. Bristol: Policy Press, pp. 61–87. 

Jones,  C.,  William  S.  Hesterly,  and  Stephen  P.  Borgatti.  (1997).  A  general  theory  of  network 
governance:  Exchange  conditions  and  social  mechanisms.  Academy  of  Management  Review 
22: 911–945. doi:10.5465/amr.1997.9711022109. 

JRC (2015). The challenge of resilience in a globalised world. 

Kahn,  L.  (2012).  Temporary  jobs  and  job  search  effort  in  Europe.  Labour  Economics,  19(1), 
113.  

Kapsalis,  A.  P.,  Lamprinakos,  G.,  Papadopoulos,  K.  A.,  Kaklamani,  D.  I.,  &  Venieris,  I.  S. 
(2012).  The  inCASA  project:  Improving  the  quality  of  life  and  social  care  for  the  ageing 
population. International Journal of Integrated Care, 12. doi: http://www.ijic.org. 

Kautto, M., (2002). Investing in services in West European welfare states. Journal of European 
Social Policy. 12, 1 pp. 53-65 

Kesselring,  A.,  Smith,  S.,  Dobner,  S.,  &  Schrammel,  M.  (2014).  Social  Innovation  for  Active 
and Healthy Ageing: A Case Study Collection. King Baudouin Foundation. 

Kieckhäfer, K. G. Walther, J. Axmann, T. Spengler, (2009). Integrating agent-based simulation 
and  system  dynamics  to  support  product  strategy  decisions  in  the  automotive  industry,  in: 
Proceedings of the 2009 Winter Simulation Conference, pp. 1433–1443. 

Kirman,  A.  (2010).  Complex  Economics:  Individual  and  Collective  Rationality.  London: 
Routledge. 

Kodner,  D.L.  (2009)  All  together  now:  a  conceptual  exploration  of  integrated  care.  Health 
Quarterly, 2009 Oct; 13 Spec No: 6-15. 

KPMG  International  and  MOWAT  Centre,  (2013),  'The  Integration  Imperative:  reshaping  the 
delivery  of  human  and  social  services',  Authors:  Gold,  J.,  and  Dragicevic,  N.,  retrieved  from 
http://mowatcentre.ca/the-integration-imperative 

KPMG International, (2014a), 'A New Vision of Value: Connecting corporate and societal value', 
retrieved 
http://www.kpmg.com/global/en/topics/climate-change-sustainability-
services/pages/a-new-vision-connecting-corporate.aspx  

from 

KPMG  International,  (2014b)  Unlocking  the  Value  of  Social  Investment',  retrieved  from 
http://www.kpmg.com/global/en/issuesandinsights/articlespublications/sustainable-
insight/pages/unlocking-value-social-investment.aspx  

Krlev, G., Münscher, R., Mülbert K. (2012) Social Return on Investment (SROI): State-of-the-
Art and Perspectives A Meta-Analysis of practice in Social Return on Investment (SROI) studies 
published 2002-2012, Centre for Social Investment, Heidelberg University 

129 

 
Kroenke,  K.,  Spitzer,  R.L.,  &  Williams,  J.B.  (2001).The  PHQ-9:  validity  of  a  brief  depression 
severity measure. Journal of General Internal Medicine, 16(9), 606-13. 

Kuhn,  T.  (1970).  The  Structure  of  Scientific  Revolutions,  2nd  edn.  Chicago:  University  of 
Chicago Press. 

Kuttner, R. (2013). TheTask Rabbit Economy. The American Prospect, Sep-Oct 2013, 46-55.  

Kuttner, R. (2016). The New Inequality Debate. American Prospect, Winter 2016. 

Kvist,  J.,  A  framework  for  social  investment  strategies:  Integrating  generational,  life  course 
and gender perspectives in the EU social investment strategy, Comparative European Politics, 
Vol. 13, No. 1.  

Kydland,  F.  E.,  and  E.  C  Prescott  (1982).  Time  to  build  and  aggregate  fluctuations. 
Econometrica 50:1345–70. 

Lagarde, C. (2015). Lifting the Small Boats. Speech by Christine Lagarde, Managing Director, 
IMF, at Grandes Conferences Catholiques, Brussels, June 17, 2015. 

Lalonde,  R.  (1986).  Evaluating  the  Econometric  Evaluations  of  Training  Programs  with 
Experimental Data. American Economic Review, 74(6), 604-620.  

Lalonde,  R.  (2013).  Employment  and  Training  Programs.  In  R.  Moffit  (Ed.),  Means  Tested 
Transfer Programs in the United States. Chicago University of Chicago Press  

Lamperti,  F.,  Dosi,  G.,  Napoletano,  M.,  Roventini,  A.,  et  al.  (2016a).  Faraway,  so  Close:  An 
Agent-Based Model for Climate, Energy and Macroeconomic Policies. Pisa, Italy: Working Paper 
Series  Forthcoming,  Laboratory  of  Economics  and  Management  (LEM),  Scuola  Superiore 
Sant'Anna. 

Lamperti,  F.,  Napoletano,  M.,  &  Roventini,  A.  (2016b).  Preventing  Environmental  Disasters: 
Market-Based  vs.  Command-and-Control  Policies.  Pisa,  Italy:  Working  Paper  Series  2015/34, 
Laboratory of Economics and Management (LEM), Scuola Superiore Sant'Anna. Retrieved from 
SSRN: http://papers.ssrn.com/sol3/papers.cfm?abstract_id=2709577  (09-06-2016). 

Lane  DC,  Monefeldt  C,  Rosenhead  JV  (2007)  Looking  in  the  wrong  place  for  health 
improvements:  A  system  dynamics  study  of  an  accident  and  emergency  department,  Journal 
of the Operational Research Society, 51, 518–531. 

Laville, J. L. (2011),  'Innovation sociale, economie sociale et solidaire, entrepreneuriat social. 
Elements pour une mise en perspective historique.'  

Lawton,  M.P.,  &  Brody,  E.M.  (1969).  Assessment  of  older  people:  Self-maintaining  and 
instrumental activities of daily living. Gerontologist, 9, 179-186. 

Le Gallo, J., S. Dall'Erba, R. Guillain (2011). 'The Local Versus Global Dilemma of the Effects of 
Structural Funds', Growth and Change, Vol. 42, No 4, December 2011. 

Lee,  Sang  M.,  G.  Kim,  and  J.  Kim.,  (2009).  Comparative  feasibility  analysis  of  Wi-Fi  in 
metropolitan  and  small  municipalities:  a  system  dynamics  approach.  International  Journal  of 
Mobile Communications 7.4, 395-414. 

Lesage,  J.P.  and  M.  Fischer  (2008).  'Spatial  Growth  Regressions:  Model  Specification, 
Estimation and Interpretation', Spatial Economic Analysis, Vol. 3, No 3, November 2008. 

Leviner, N., Crutchfield, L., Wells, D. (2007) Understanding the Impact of Social Entrepreneurs 
-  Ashoka's  answer  to  the  Challenge  of  Measuring  Effectiveness,  in  Research  on  Social 
Entrepreneurship:  Understanding  and  Contributing  to  an  Emerging  Field,  Mosher-Williams,  R. 
(editor) 

Lewe,  J.-H.,  (April  2005).  An  Integrated  Decision-Making  Framework  for  Transportation 
Architectures:  Application  to  Aviation  Systems  Design.  PhD  thesis,  Georgia  Institute  of 
Technology. 

Lewis, K. E., Annandale, J. A., Warm, D. L., Hurlin, C., Lewis, M. J., & Lewis, L. (2013). Home 
telemonitoring  and  quality  of  life  in  stable,  optimised  chronic  obstructive  pulmonary  disease. 
Journal of Telemedicine and Telecare, 16, 253-259. 

130 

 
Leys,  M.  (2009)  Technology  and  innovation  in  the  elderly  care.  Care  symposium.  Aalst,  10 
September 2009. 

Lindenberg,  S.  (1992).  The  Method  of  Decreasing  Abstraction.  In  J.  S.  Coleman  &  T.  Fararo 
(Eds.), Rational Choice Theory: Advocacy and Critique (pp. 3-20). Newbury Park: Sage. 

Lindsay, C., McQuaid, R., & Dutton, M. (2008). Inter-Agency Cooperation and New Approaches 
to Employability. Social Policy & Administration 42(7), 715-732.  

Liu,  S.,  Huang,  J.  L.,  &  Wang,  M.  (2014).  Effectiveness  of  job  search  interventions:  A  meta-
analytic review. Psychological Bulletin, 140(4), 1009–1041. 

Lucas, R. E., (1975). An equilibrium model of the business cycle. Journal of Political Economy 
83:1113– 44. 

Luebker,  M.  (2014).  Income  Inequality,  Redistribution,  and  Poverty:  Contrasting  Rational 
Choice  and  Behavioral  Perspectives.  Review  of  Income  and  Wealth,  60(1),  133-154.  doi: 
10.1111/roiw.12100 

Lyons,  L.C.  (2003).  Meta  -  Analysis:  Methods  of  Accumulating  Results  Across  Research 
Domains. George Washington University Medical Center. Ross 312, 2300 I St. NW Washington, 
DC 20037. 

Lytinen, S. L., & Railsback, S. F., (2009). The evaluation of agent-based simulation platforms: 
A  review  of  NetLogo  5.0  and  ReLogo.  Paper  presented  at  the  European  Meetings  on 
Cybernetics and Systems Research. 

Maas,  K.,  Liket,  k.  (2011).  Talk  the  Walk:  Measuring  the  Impact  of  Strategic  Philanthropy, 
Journal of Business Ethics, 100: 445-464 

Maase, S.J.F.M., & Dorst, K. H. (2007). Exploring the development process of grassroots social 
entrepreneurship.  Paper  presented  at  the  International  Social  Entrepreneurship  Research 
Conference, Copenhagen, Denmark. June. 

Macal,  C.  M.,  &  North,  M.,  (2008).  Agent-based  modelling  and  simulation:  ABMS  examples. 
Paper presented at the Winter Simulation Conference, Miami, FL. 

Mackie, J. (1965). Causes and Conditions. American Philosophical Quarterly, 2, 245–264.  

Maggio,  C.  A.,  Pi-Sunyer,  F.  X.,  (1997).  The  prevention  and  treatment  of  obesity.  Diabetes 
care, 20(11), 1744. 

Magnusson,  L.,  &  Hanson,  E.  (2005).  Supporting  frail  older  pople  and  their  family  carers  at 
home  using  information  and  communication  technology:  cost  analysis.  Journal  of  Advanced 
Nursing, 51, 645-657  

Mani, A., Mullainathan, S., Shafir, E., & Zhao, J. (2013).  Poverty Impedes Cognitive Function. 
Science, 341(6149), 976-980.  

Manski,  C.  (1995).  Identification  Problems  in  the  Social  Sciences.  Cambridge:  Harvard 
University Press. 

Mantwill, S., et al. (2015). EMPOWER- support of patient empowerment by an intelligent self-
management  pathway  for  patients:  Study  protocol.  BMC  Medical  Informatics  and  Decision 
Making, 15: 18. doi: doi:10.1186/s12911-015-0142-x 

Marmot, M., Shipley, M., Brunner, E., & Hemingway, H. (2001). Relative contribution of early 
life  and  adult  socioeconomic  factors  to  adult  morbidity  in  the  Whitehall  II  study.  Journal  of 
Epidemiology and Community Health, 55(5), 301–307. 

Marshall, D. A., Burgos-Liz, L., IJzerman, M. J., Osgood, N. D., Padula, W. V., Higashi, M. K., 
...  &  Crown,  W.,  (2015).  Applying  dynamic  simulation  Modelling  methods  in  health  care 
delivery  research—the  SIMULATE  Checklist:  report  of  the  ISPOR  Simulation  Modelling 
Emerging Good Practices Task Force. Value in health, 18(1), 5-16. 

Marteau T, & Bekker H. (1992). The development of a six-item short-form of the state scale of 
the  Spielberger  State-Trait  Anxiety  Inventory  (STAI).  British  Journal  of  Clinical  Psychology, 
31:301-306. 

131 

 
Martin,  J.  (2014).  Activation  and  Active  Labour  Market  Policies  in  OECD  Countries:  Stylized 
Facts and Evidence on their Effectiveness: IZA, Policy Paper Series, IZA Policy Paper No. 84. 

Martin,  R.,  &  Schlüter,  M.  (2015).  Combining  system  dynamics  and  agent-based  modeling  to 
analyze social-ecological interactions—an example from modeling restoration of a shallow lake. 
[10.3389/fenvs.2015.00066]. Frontiers in Environmental Science, 3, 66.  

Martinez-Moyano  I.J.,  D.L.  Sallach,  M.J.  Bragen,  P.R.  Thimmapuram,  (2007).  Design  for  a 
multilayer model of financial stability: exploring the integration of system dynamics and agent-
based  models,  in:  Proceedings  of  the  25th  International  Conference  of  the  System  Dynamics 
Society. 

MASON Web site., (2012), Retrieved August 15, 2013, from 
http://cs.gmu.edu/~eclab/projects/mason/. 

Massimo,  C.,  &  Bottone,  N.  (2014).  Final  Report:  inCASA.  Integrated  network  or  completely 
assisted senior citizen's autonomy. http://goo.gl/r3PUkj 

Matlabi, H., Parker, S. G., & McKee, K. (2011). The contribution of home-based technology to 
older people's quality of life in extra care housing. BMC Geriatrics, 11, 68. doi:10.1186/1471-
2318-11-68. 

Mayne,  J.  (2011).  Addressing  Cause  and  Effect  in  Simple  and  Complex  Settings  through 
Contribution  Analysis.  In  R.  Schwartz,  K.  Forss  &  M.  Marra  (Eds.),  Evaluating  the  Complex: 
Transaction Publishers. 

Mayne,  J.  (2012).  Contribution  analysis:  Coming  of  age?  Evaluation,  18(3),  270-280. 
doi:10.1177/1356389012451663 

Mazzucato, M. (2015a). The Entrepreneurial State: Debunking Public vs. Private Sector Myths. 
London: Anthem Press. 

Mazzucato,  M.  (2015b).  The  green  entrepreneurial  state:  SPRU  Working  Paper  Series,  2015-
28. 

McCawley,  P.F.  (2001).  The  Logic  Model  for  Program  planning  and  Evaluation.  University  of 
Idaho Extension. Retrieved at http://www.uiweb.uidaho.edu/extension/LogicModel.pdf 

McGraw, K., & Hoekstra, V. (1994). Experimentation in Political Science: Historical Trends and 
Future Directions. In M. Delli Carpini, L. Huddy & R. Shapiro (Eds.), search in Micropolitics (pp. 
3-30). Greenwood: JAI Press. 

McHaney, R., (1991). Computer Simulation: A Practical Perspective. Academic Press. 

McKee,  K.J.,  Parker,  S,G,,  Elvish,  J.,  Clubb,  V.J,  Nahas,  M.E,  Kendray,  E.  et  al.  (2005).  The 
quality  of  life  of  older  and  younger  people  who  receive  renal  replacement  therapy.  Ageing  & 
Society, 25, 903–923. doi:10.1017/S0144686X05004046 

McKelvie,  D.,  (2012).  Envisaging  Social  Impact  Bonds  Modelling  'What  Might  Be'  A  Whole 
System Collaborative Approach, The Symmetric Partneship LLP. 

McKnight, A., Duque, M., & Rucci, M. (2016). Creating More Equal Societies–What Works. 

McLoughlin, J., Kaminski, J., Sodagar, B., Khan, S., Harris, R., Arnaudo, G., & Mc Brearty, S. 
(2009). A strategic approach to social impact measurement of social enterprises: The SIMPLE 
methodology. Social Enterprise Journal, 5(2), 154-178. 

McManus,  R.J.,  et  al.  (2009).  Protocol  for  a  randomised  controlled  trial  f  telemonitoring  and 
self-management in hypertension. BMC Cardiovascular Disorders 9:6. doi:10.1186/1471-2261-
9-6 

McMillin, S. E. (2012). Proposing a Test for Policy-Based Evidence Making: A Content Analysis 
of  the  Don't  Ask,  Don't  Tell  Review.  Journal  of  Policy  Practice,  11(4),  255-272. 
doi:10.1080/15588742.2012.713800 

McQuaid,  R.  (2010).  Theory  of  Organizational  Partnerships:  partnership  advantages, 
disadvantages  and  success  factors.  In  S.  Osborne  (Ed.),  The  New  Public  Governance? 

132 

 
Emerging  Perspectives  on  the  Theory  and  Practice  of  Public  Governance  (pp.  127-148). 
London: Routledge. 

McQuaid, R., Lindsay, C., Dutton, M.,  & McCracken, M. (2007). Best practice in inter- agency 
co-operation on employability. Edinburgh: Napier University, Employment Research Institute. 

Meadows D, Randers J, Meadows D (2004) Limits to Growth: the 30-year Update,  Earthscan, 
London & Washington DC. 

Meadows, D. H., Meadows, D. I., Randers, J., & Behrens, W. (1972). The Limits to Growth: A 
Report to the Club of Rome. 

Meadows,  K.,  Abrams,  C.,  &  Sandbaek  A.  (2000).  Adaptation  of  the  Diabetes  Health  Profile 
(DHP-1)  for  use  with  patients  with  Type  2  diabetes  mellitus:  psychometric  evaluation  and 
cross-cultural comparison. Diabetic Medicine, 17, 572-580. 

Merrell,  C.,  Tymms,  P.  (2011).  Changes  in  Children's  Cognitive  Development  at  the  Start  of 
School in England 2001 – 2008, Oxford Review of Education 37; 333-345. 

Mildenberger,  G.,  Münscher,  R.,  Schmitz,  B.  (2012).  Dimensionen  der  Bewertung 
gemeinnütziger  Organisationen,  In:  Anheier,  Helmut  K./Schröer,  Andreas/Then,  Volker 
(Hrsg.):  Soziale  Investionen.  Interdisziplinäre  Perspektiven,  Wiesbaden:  vs.  Verlag  für 
Sozialwissenschaften.  

Miles,  H.,  Huberman,  A.  (1994).  Qualitative  Data  Analysis.  An  Expanded  Sourcebook. 
Thousand Oaks, CA: Sage. 

Millar,  A.,  R.S.  Simeone,  and  J.T.  Carnevale.  (2001).  Logic  models:  a  systems  tool  for 
performance management. Evaluation and Program Planning. 24:73-81. 

Miller, Ronald E., Peter  D. Blair., (2009). Input-Output Analysis: Foundations and Extensions. 
2nd ed. New York: Cambridge University Press. 

Minas,  R.  (2014).  One-stop  shops:  Increasing  employability  and  overcoming  welfare  state 
fragmentation? International Journal of Social Welfare, 23, S40-S53. doi:10.1111/ijsw.12090 

Misuraca, G. (2012). Assessing ICT-enabled innovation for governance and policy making, PhD 
Thesis - College of Management and Technology, EPFL, Lausanne, Switzerland. 

Misuraca, G., Centeno, C., & Torrecillas, C. (2014). Measuring the Impact of eInclusion Actors. 
Impact  Assessment  Framework  Main  Report  and  Annex–MIREIA  eI2-IAF  Toolkit  (No. 
JRC89462). Institute for Prospective and Technological Studies, Joint Research Centre. 

Misuraca, G., Codagnone, C., Rossel, P. (2013). From Practice to Theory and back to Practice: 
Reflexivity in Measurement and Evaluation for Evidence-based Policy Making in the Information 
Society. Government Information Quarterly, 30: 68-82.  

Misuraca,  G.,  Colombo,  Kucsera,  C.,  Carretero,  S.,  Bacigalupo,  M.,  and  Radescu,  R.  (2015a). 
Mapping and analysis of ICT-enabled Social Innovation initiatives promoting social investment 
through integrated approaches to the provision of social services, European Commission’s Joint 
Research  Centre,  Institute  for  Prospective  Technological  Studies,  EUR  2745  EN,  JRC  Science 
and Policy Report Series.  

Misuraca,  G.,  Colombo,  C.,  Carretero,  S.,  Kucsera,  C.,  (2015b).  i-FRAME:  initial  proposal  of 
methodological  framework  to  assess  the  social  and  economic  impact  of  ICT-enabled  social 
innovation  initiatives  promoting  social  investment  in  integrated  approaches  to  social  services 
provision, including active and healthy ageing, European Commission’s Joint Research Centre, 
Institute for Prospective Technological Studies, JRC Technical Reports Series  

Misuraca, G., Colombo, C., Kucsera, C., Bacigalupo, M., Radescu, R.,  & Carretero, S. (2015c) 
Mapping and Analysis of ICT-enabled Social Innovation initiatives promoting social investment 
in Integrated approaches to the provision of Social Services. JRC Science and Policy Report 

Misuraca,  G.,  Kucsera,  C.,  (Eds.)  (2016).  Proposed  methodological  framework  to  assess  the 
social  and  economic  impact  of  ICT-enabled  social  innovation  initiatives  promoting  social 
investment  in  the  EU-  i-FRAME  V1.5  (D2.2)',  European  Commission's  Joint  Research  Centre, 
Institute for Prospective Technological Studies, JRC Technical Reports Series. 

133 

 
Misuraca,  G.,  Kucsera,  C.,  Lipparini,  F.,  Voigt,  C.,  Radescu,  R.  (2016a).  ICT-Enabled  Social 
Innovation  to  support  the  Implementation  of  the  Social  Investment  Package  -  Mapping  and 
Analysis of ICT-enabled Social Innovation initiatives promoting social investment in integrated 
approaches to the provision of social services: IESI Knowledge Map 2015 (EUR 27838 EN).  

Misuraca,  G.,  Kucsera  C.,  Pasi,  G.,  Gagliardi,  D.,  Abadie,  F.,  (2017a).  ICT-Enabled  Social 
Innovation  to  support  the  Implementation  of  the  Social  Investment  Package:  Mapping  and 
Analysis  of  ICT-enabled  Social  Innovation  initiatives  promoting  social  investment  across  the 
EU: IESI Knowledge Map 2016 (EUR 28444 EN).  

Misuraca, G., Pasi, G., Abadie, F., Kucsera, C., Virginillo, M. (2017b). Exploring the role of ICT-
Enabled  Social  Innovation  to  support  the  modernisation  of  EU  Social  Protection  Systems: 
findings and insights from analysis of case studies in fourteen Member States (EUR 28570 EN).  

Misuraca,  G.,  Pasi,  G.,  Abadie,  F.,  (2017c).  Innovating  EU  Social  Protection  Systems  through 
ICTs. Findings from analysis of case studies in fourteen Member States, JRC Insights on Social 
Policy Innovation, Issue #3, Seville.  

Misuraca,  G.,  Geppert,  L.,  Kucsera,  C.,  (2017d).  Deconstructing  social  policy  innovation 
through  the  use  of  complex  systems  theory:  A  methodology  for  modelling  and  simulation  of 
the impact of ICT-enabled social innovation. In Gil-Garcia, J. R., Pardo, T., and Luna-Reyes, L. 
F.:  Policy  Analyitics,  Modeling  and  Informatics:  Innovative  Tools  for  Solving  Complex  Social 
Problems. New York, NY: Springer.  

Misuraca,  G.,  Viscusi,  G.  (2014).  Digital  Governance  in  the  Public  Sector:  challenging  the 
Policy-Maker’s innovation dilemma. ICEGOV. 27-30 2014.  

Misuraca,  G.,  Viscusi,  G.,  (2015).  Shaping  public  sector  innovation  theory:  An  interpretative 
framework for assessing ICT-enabled innovation for governance and policy-making. Electronic 
Commerce  Research.  Special  Issue  on  Electronic  Government:  Investment  in  Communities, 
Firms, Technologies and Infrastructure.  

Misuraca,  G.,  Viscusi,  G.,  Pasi,  G.  (2016).  Digital  Governance  Challenges  for  ICT-Enabled 
Innovation of Social Protection Systems in the EU. In Joint Proceedings of IFIP EGOV and ePart 
2016, IOS Press: 172-179. 

Mookherjee,  D.  (2003).  Poverty  Persistence  and  Design  of  Anti-Poverty  Policies.  In  A. 
Banerjee,  R.  Bénabou  &  D.  Mookherjee  (Eds.),  Understanding  Poverty.  Oxford:  Oxford 
University Press. 

Moore,  G.,  F  (2015).  Process  evaluation  of  complex  interventions:  MRC  guidance.  BMJ,  350, 
h1258 

Morgan, S., & Winship, C. (2007). Counterfactuals and causal inference. New York: Cambridge 
University Press. 

Mosley, H. (2011). Decentralisation of Public Employment Services. Brussels: Analytical Paper. 
PES  to  PES  Dialogue,  The  European  Commission  Mutual  Learning  Programme  for  Public 
Employment Services. DG Employment, Social Affairs and Inclusion. 

Mulgan, G. et al. (2006). Mapping value in the built environment. Young Foundation 

Mulgan, G., Tucker, S., Ali, R., Sanders, B., Social innovation: What it is, why it matters and 
how it can be accelerated. London: The Young Foundation. (2007) 

Munday, B. (2007). Integrated Social Services in Europe: Report. Brussels: Council of Europe. 

Munday,  B.,  (2007).  European  Social  Services  a  Map  of  Characteristics  and  Trends.  Report 
prepared for the European Council. 

Munro, E. (2011). The Munro Review of Child Protection: Final Report. London: UK Department 
of Education. 

Munro, E. (2014). Evidence-Based Policy. In N. Cartwright & E. Montuschi (Eds.), Philosophy of 
Social Sciences: A New Introduction (pp. 48-67). Oxford: Oxford University Press. 

Murray, R., Caulier-Grice, J., Mulgan, G., 2010, The Open Book of Social Innovation 

134 

 
Mutti, A. (1998). Capitale sociale e sviluppo. Bologna: Il Mulino. 

Narayan,  D.,  Pritchett,  L.,  &  Kapoor,  S.  (2009).  Moving  out  of  poverty:  Success  from  the 
bottom up (Vol. 2). World Bank Publications 

Nardo,  M.,  Saisana,  M.,  Saltelli,  A.,  Tarantola,  S.,  Hoffman,  A.,  &  Giovannini,  E.  (2008). 
Handbook on constructing composite indicators. JRC/OECD. 

Neelamkavil, F., (January 1987). Computer Simulation and Modelling. Wiley. 

NEF (2016). Happy Planet Index 2016 Methods Paper. Available at 
https://static1.squarespace.com/static/5735c421e321402778ee0ce9/t/578dec7837c58157b92
9b3d6/1468918904805/Methods+paper_2016.pdf (retrieved 24 January 2017). 

Neisser, U. (1976). Cognition and reality. San Francisco Freeman. 

Nelson, R., & Winter, S. (1982). An Evolutionary Theory of Economic Change. Cambridge and 
London: The Belknap Press of Harvard University Press. 

Neyman, J. (1923). On the Application of Probability Theory to Agricultural Experiments. Essay 
on Principles. Statistical Science, 5(4), 465–480. doi:10.2307/2245382 

Ni Bhrolchain, M.,  & Dyson,  T. (2007).  On  causation in demography:  Issues and illustrations. 
Population and Development Review, 33(1), 1-36.  

Nieboer, A., et al. (2005). Dimensions of well-being and their measurement: The SPF-IL Scale. 
Social Indicators Research, 73, 313-353. 

Nightingale, D. S., & Holcomb, P. A. (1997). Alternative Strategies for Increasing Employment. 
The Future of Children, 7(1), 52-64. 

Nikolai, C., & Madey, G., (2009). Tools of the trade: A survey of various agent based modelling 
platforms. Journal of Artificial Societies and Social Simulation, 12(No. 22). 

Nonaka, I., Hirotaka, H., (1995). The knowledge creating company: how Japanese companies 
create the dynamics of innovation. New York Oxford University Press. 

Nordhaus, WD and Tobin, J (1972) Is Growth Obsolete? Economic Growth, National Bureau of 
Economic Research, no 96, New York. 

Novak,  M.,  &  Guest,  C.  (1989).  Application  of  a  multidimensional  caregiver  burden  survey. 
Gerontologist, 29, 798-803. 

Nunez,  I.,  &  Livanos,  I.  (2014).  Temps  “by  choice”?  An  Investigation  of  the  Reasons  Behind 
Temporary  Employment  Among  Young  Workers  in  Europe.  [Journal  article].  Journal  of  Labor 
Research, 36(1), 44-66. doi:10.1007/s12122-014-9195-3 

Ocepek, J., Roberts, A. E.K., & Vidmar, G. (2013). Evaluation of treatment in the smart home 
IRIS in terms of functional independence and occupational performance and satisfaction. 
Computational and Mathematical Methods in Medicine. 
http://dx.doi.org/10.1155/2013/926858  

OECD (2003). From Red Tape to Smart Tape, Administrative Simplification in OECD Countries. 
Paris: OECD. 

OECD  (2008).  Growing  Unequal?  Income  Distribution  and  Poverty  in  OECD  Countries.  Paris: 
OECD. 

OECD (2011). Divided we stand. Why inequality keeps rising. Paris: OECD. 

OECD (2011). Doing Better for Families, Paris: OECD Publishing. 

OECD (2014). OECD Employment Outlook 2014. Paris: OECD Publishing. 

OECD (2015a). How's Life? 2015: Measuring Well-being, OECD Publishing. 

OECD  (2015b).  Policy  Brief  on  Social  Impact  Measurement  for  Social  Enterprises  Policies  for 
Social Entrepreneurship. Luxembourg: Publications Office of the European Union. 

OECD  (2017).  Working  with  Change:  Systems  approaches  to  public  sector  challenges.  Paris: 
OECD. 

135 

 
O'Flynn,  J.  (2007).  From  New  Public  Management  to  Public  Value:  Paradigmatic  Change  and 
Managerial Implications. Australian Journal of Public Administration, 66(3), 353-366.  

Ógáin, E., Lumley, T. and Prichard, D., 2012. Making an impact: Impact measurement among 
charities and social enterprises in the UK. London: New Philanthropy Capital.  

Ógáin, E., Svistak, M. and de Las Casas, L., 2013. Blueprint for shared measurement. London: 
NPC. 

Olsen,  S.,  Galimidi,  B.  (2008).  'Catalog  of  Approaches  to  Impact  Measurement  -  Assessing 
social impact in private ventures.  

Osborne, D., & Gaebler, T. (1993). Reinventing Government.  How the  Entreprenurial Spirit  is 
Transforming the Public Sector. New York: Penguin Books. 

Osborne, S., Public. (2006). The New Public Governance? Management Review, 377-387(8). 

Øverbye,  E.,  Strohmeier  Navarro  Smith,  R.,  Karjalainen,  V.,  &  Stremlow,  J.  (2010).  The 
coordination  challenge.  In  Y.  Kazepov  (Ed.),  Rescaling  social  policies:  Towards  multilevel 
governance in Europe (pp. 389–414). Farnham: Ashgate. 

Padgett, J.F., Powell, W., W., (2012), The emergence of organizations and markets, Princetopn 
University Press.  

Padula  WV,  Duffy  MP,  Yilmaz  T,  Mishra  MK.  (2014).  Integrating  systems  Engineering  practice 
with health-care delivery. Health Systems, 3:159–64. 

Pareto V (1909) Manual of political economy. Kelley, New York, (1971). 

Patton M. (1982). Practical Evaluation. Beverly Hills, CA Sage. 

Patton,  M.  (1997).  Utilization-focused  evaluation:  The  new  century  text.  Thousand  Oaks,  CA: 
Sage Publications. 

Paul, K., & Moser, K. (2009). Unemployment impairs mental health: Meta-analyses. Journal of 
Vocational Behavior, 74, 264-282.  

Pawson, R. (2002). Evidence-based Policy: In Search of a Method. Evaluation, 8(2), 157-181. 
doi:10.1177/1358902002008002512 

Pawson, R. (2006). Evidence-Based Policy: A Realist Perspective. London: Sage. 

Perry,  G.,  J.  Whalley  and  G.  McMahon,  (2001).  Fiscal  Reform  and  Structural  Change  in 
Developing Countries, New-York: Palgrave-Macmillan. 

Petticrew, M., & Roberts, H. (2003). Evidence, hierarchies, and typologies: horses for courses. 
Journal of Epidemiology and Community Health, 57(7), 527-529.  

Phelan, Steven E., (2001). What is complexity science, really? Emergence, Vol. 3, No. 1, 120–
136. 

Philip, Deiglmeier & Miller, 2008 in The Open Book of Social Innovation, 2010. 

Pielke,  R.  (2007).  The  Honest  Broker:  Making  Sense  of  Science  in  Policy  and  Politics. 
Cambridge: Cambridge University Press. 

Pieńkowski,  J.  and  Berkowitz,  P.  (2015).  Econometric  assessments  of  Cohesion  Policy  growth 
effects:  How  to  make  them  more  relevant  for  policymakers?  DG  REGIO  Working  Paper 
02/2015. 

Pinkett,  R,  and  O'Bryant,  (2003),  Building  community,  empowerment  and  self-sufficiency, 
Information, Communication & Society 

Podsiadlo,  D.,  &  Richardson,  S.  (1991).  "The  timed  "Up  &  Go":  a  test  of  basic  functional 
mobility for frail elderly persons. Journal of the American Geriatric Society, 39, 142-148. 

Porter, M., Kramer, M. (2011). Creating Shared Value. HBR. January 2011. 

Prettner, K. (2013). Population aging and endogenous economic growth. Journal of Population 
Economics, 26(2), 811-834. doi:10.1007/s00148-012-0441-9 

136 

Prettner, K., & Prskawetz, A. (2010). Demographic change in models of endogenous economic 
growth.  A  survey.  Central  European  Journal  of  Operations  Research,  18(4),  593-608. 
doi:10.1007/s10100-010-0179-y 

Ragazzi, E., & Sella, L. (2013). Una valutazione di impatto delle politiche formative regionali: il 
caso  piemontese.  CERIS  Working  Paper  2013/15,  Institute  for  Economic  Research  on  Firms 
and Growth - Moncalieri (TO). 

Rahmandad H, Sterman JD., (2008). Heterogeneity and network structure in the dynamics of 
diffusion:  Comparing  agent-based  and  differential  equation  models.  Management  Science 
54(5): 998-1014. 

Railsback,  S.  F.,  Lytinen,  S.  L.,  &  Jackson,  S.  K.,  (2006).  Agent-based  simulation  platforms: 
Review and development recommendations. Simulation, 82(9), 609–623. 

Raine,  K.  D.,  (2010).  Addressing  poor  nutrition  to  promote  heart  health:  moving  upstream. 
Canadian Journal of Cardiology, 26, 21C-24C. 

Rainey, L. B., ed., (2004). Space Modelling and Simulation: Roles and Applications throughout 
the System Life Cycle. AIAA. 

Raitano, M. (2016).  Income  Inequality in Europe Since the Crisis.  Intereconomics, 51(2), 67-
72. doi:10.1007/s10272-016-0579-x 

Randle, A. and Kippin, H. (2014) “Managing Demand – Building Future Public Services, RSA  

Rauscher,  O., Schober, C., Millner, R., (2012), Social Impact Measurement und Social Return 
on Investment (SROI)-Analysis, New methods of economic evaluation? Working Paper, Vienna 
University of Economics and Business 

Ravetz,  J.  (1990).  The  merger  of  knowledge  with  power:  Essays  in  critical  science  (Vol. 
Mansell): London. 

Reeder,  N.,  Peron,  C.  and  Richter,  K.  (2015)  Deriving  a  framework  to  assess  the  impacts 
generated  by  ICT-enabled  social  innovation  initiatives  promoting  social  investment  for 
integrated approaches to the provision of personal social services, report by EngagedX for JRC-
IPTS 

Riain,  A.N.,  Shahahan,  E.,  &  Collins,  C.  (2014).  Evaluation  of  the  acceptability  and 
effectiveness  of  telemedicine  in  chronic  conditions  in  a  rural  general  practice  in  Ireland. 
European Journal of General Practice, 20, 1381 

Robinson,  B.  (1983).  Validation  of  a  caregiver  strain  index.  Journal  of  Gerontology,  38,  344-
348. 

Rodriguez-Pose,  A.  and  E.  Garcilazo  (2013).  'Quality  of  Government  and  the  Returns  of 
Investment:  Examining  the  Impact  of  Cohesion  Expenditure  in  European  Regions',  OECD 
Regional Development Working Papers, 12, OECD Publishing, 2013. 

Rogers, P. J. (2008). Using programme theory to evaluate complicated and complex aspects of 
interventions. Evaluation, 14(1), 29-48. doi:10.1177/1356389007084674 

Rossi, P.H., Lipsey, M.W., Freeman, H.E. (2004). Evaluation: A Systematic Approach. Sage. 

Roworth-Gaunt, C, Ridout, A., Wormald, G., Magee, G., Penkett, J., Richings, R., et al. (2009). 
Occupational  Therapy  Just  Checking  Telecare:  1  Year  Pilot  Report.  Leeds  NHS  Foundation 
Trust. http://goo.gl/eaUa0n  

Sackett,  D.  L.,  Rosenberg,  W.  M.  C.,  Gray,  J.  A.  M.,  Haynes,  R.  B.,  et  al.  (1996).  Evidence 
based medicine: what it is and what it isn't [10.1136/bmj.312.7023.71]. BMJ, 312(7023), 71.  

Salamon, L., & Anheier, H. (1996). The Emerging Nonprofit Sector: An Overview. Manchester: 
Manchester University Press. 

Salverda,  W.,  Nolan,  B.,  Checchi,  D.,  Marx,  I.,  et  al.  (Eds.).  (2014).  Changing  Inequalities  in 
Rich Countries. Analytical and Comparative Perspectives. Oxford: Oxford University Press. 

Sanderson, I. (2011). Evidence-based policy or policy-based evidence? Reflections on Scottish 
experience. Evidence & Policy, 7(1), 59-76.  

137 

 
Sarewitz,  D.  (2000).  Science  and  environmental  policy:  an  excess  of  objectivity.  In  R. 
Frodeman & V. Baker (Eds.), Earth Matters: The Earth Sciences, Philosophy, and the Claims of 
Community (pp. 79-98). Upper Saddle River, NJ: Prentice Hall. 

Sarewitz,  D.  (2004).  How  science  makes  environmental  controversies  worse.  Environmental 
Science & Policy, 7(5), 385-403. doi:http://dx.doi.org/10.1016/j.envsci.2004.06.001 

Scharle, A., Weber, T., & Puchwein Roberts, I. (2014). Approaches for Sustainable Integration 
of  Long-term  Unemployed  -  Toolkit.  Brussels:  European  Commission,  DG  Employment,  Social 
Affairs. 

Scheidt, R. (1981). Ecologically-valid inquiry: Fait accompli? . Human Development, 24, 225–
228.  

Scherr,  P.  A.,  &  Wallace,  R.  B.  (1994).  A  short  physical  performance  battery  assessing  lower 
extremity  function:  association  with  self-reported  disability  and  prediction  of  mortality  and 
nursing home admission. Journal of Gerontology, 49, M85-M94. 

Seale,  C.,  &  Silverman,  D.  (1997).  Ensuring  rigour  in  qualitative  research.  The  European 
Journal of Public Health, 7(4), 379-384.  

Sen, A. (2001). Development as freedom. Oxford Paperbacks. 

Serenko,  A.,  &  Detlor,  B.,  (2002).  Agent  toolkits:  A  general  overview  of  the  market  and  an 
assessment  of  instructor  satisfaction  with  utlizing  toolkits  in  the  classroom.  Research  and 
Working  Paper  Series  (Vol.  No.  455).  Ontario,  Canada:  Michael  G.  DeGroote  School  of 
Business, McMaster University. 

Sewell,  W.  H.  (1992).  A  Theory  of  Structure:  Duality,  Agency,  and  Transformation.  The 
American Journal of Sociology, 98, 1–29 

Seyfang,  G.  (2003).  Growing  cohesive  communities  one  favour  at  a  time:  social  exclusion, 
active  citizenship  and  time  banks.  International  Journal  of  Urban  and  Regional  Research, 
27(3), 699–706. 

Shanthikumar,  J.  G.,  &  Sargent,  R.  G.  (1983).  A  unifying  view  of  hybrid  simulation/analytic 
models and modelling. Operations research, 31(6), 1030-1052. 

Shaw I., Bell M., Sinclair I., Sloper P., Mitchell W., Dyson P., Clayden J., Rafferty J. (2009) An 
exemplary  scheme?  An  evaluation  of  the  integrated  children's  system.  Br  J  Soc  Work  (2009) 
39 (4): 613-626. 

Shirky,  C.,  (2009).  Here  Comes  Everybody:  How  Change  Happens  When  People  Come 
Together. New York: Penguin. 

Simões,  M.,  Duarte,  A.,  &  Andrade,  J.  (2014).  Assessing  the  Impact  of  the  Welfare  State  on 
Economic  Growth:  A  Survey  of  Recent  Developments.  Coimbra:  Faculdade  de  Economia, 
Universidade de Coimbra, GEMF Working Papers 2014/October. 

Simon,  H.  A.,  (1976).  How  complex  are  complex  systems?  in  Proceedings  of  the  Biennial 
Meeting of the Philosophy of Science Association, vol. 1976, pp. 507-522. 

Simonazzi,  A.,  (2008).  Care  regimes  and  national  employment  models.  Working  paper. 
Università de la Sapienza, Roma. 

Singh-Manoux,  A.,  Ferrie,  J.  E.,  Chandola,  T.,  &  Marmot,  M.,  (2004).  Socioeconomic 
trajectories  across  the  life  course  and  health  outcomes  in  midlife:  Evidence  for  the 
accumulation hypothesis? International Journal of Epidemiology, 33(5), 1072–1079. 

Single,  A.,  &  Donnelly,  G.  (2010).  'At  home,  not  alone'  COPD  Telehealth  project:  Final 
evaluation. NHS South East Essex. 

Soumitra,  D.,  Lanvin,  B.,  Wunsch-Vincent,  S.,  2014,  The  Global  Innovation  Index,  retrieved 
from https://www.globalinnovationindex.org/userfiles/file/reportpdf/GII-2014-v5.pdf 

Spreckley,  F.  (1981).  "Social  Audit  –  A  Management  Tool  for  Co-operative  Working".  Local 
Livelihoods. 

138 

 
Squires  and  Tappenden,  (2011),  Methods  Review  of  Mathematical  Modelling,  School  of  Social 
Care Research - SSCR, UK 

Standing, G. (2011). The Precariat: The Dangerous Class. London: Bloomsbury. 

Stanig,  P.,  Kayser,  M.,  &  Kelvin,  L.  (2013).  Governance  indicators:  some  proposals.  Oxford: 
Oxford University Press. 

Sterman  JD.,  (2000).  Business  Dynamics:  Systems  Thinking  and  Modelling  for  a  Complex 
World. McGraw: New York. 

Sterman, J. D. (2006). Learning from evidence in a complex world. Am J Public Health, 96(3), 
505-514. doi:AJPH.2005.066043 [pii]10.2105/AJPH.2005.066043 

Stiglitz, J., Sen, A., & Fitoussi, J.-P. (2009), Report of the Commission on the Measurement of 
Economic performance and Social Progress, 292 p. 

Stockmann, R., Meyer, W., Caspari, A. (2006). Evaluation: Eine Einführung. UTB. 

Stoddart,  A.,  et  al.  (2013).  Telemonitoring-based  service  redesign  for  the  management  of 
uncontrolled  hypertension  (HITS):  Cost  and  cost-effectiveness  analysis  of  a  randomised 
controlled trial, British Medical Journal 

Strassheim,  H.,  &  Kettunen,  P.  (2014).  When  does  evidence-based  policy  turn  into  policy-
based  evidence  configurations,  contexts  and  mechanisms.  Evidence  and  Policy,  10(2),  259-
277. doi:10.1332/174426514X13990433991320 

Strauss,  E.,  Sherman,  E.  M.  S.,  &  Sprenn,  O.  (2006).  A  compendium  of  neuropsychological 
tests: Administration, norms and commentary (3rd ed.). Oxford: Oxford University Press 

Struewer and Tews. (2015). Impact Investing: Financial Returns are Only Half the Story. 

Struyven,  L.  (2004).  Design  Choices  in  Market  Competition  for  Employment  Services  for  the 
Long-Term Unemployed. Paris: OECD Social, Employment and Migration Working Papers. 

Stufflebeam,  D.L.,  Shinkfield,  A.J.  (2007).  Evaluation  Theory,  Models,  and  Applications.  John 
Wiley & Sons.  

Summers, L. (2014a). Reflections on the 'New Secular Stagnation Hypothesis'. In C. Teulings & 
R.  Baldwin  (Eds.),  Secular  Stagnation:  Facts,  Causes  and  Cures  (pp.  27-38).  London:  Centre 
for Economic Policy Research (CEPR). 

Summers,  L.  (2014b).  US  Economic  Prospects:  Secular  Stagnation,  Hysteresis  and  the  Zero 
Lower  Bound.  speech  delivered  to  the  National  Association  for  Business  Economics'  Economic 
Policy Conference.  

Summers,  L.,  &  Balls,  E.  (2015).  Report  of  the  Commission  on  Inclusive  Prosperity. 
Washington:  Center 
(https://cdn.americanprogress.org/wp-
Progress 
content/uploads/2015/01/IPC-PDF-full.pdf). 

for  American 

Sutrisno,  A.,  Handel,  O.,  (2011).  Dynamic  Aging  Population  in  Germany:  A  case  study  about 
demographic change. Bergen University. 

Swinerd, C., & McNaught, K. R. (2012). Design classes for hybrid simulations involving agent-
based  and  system  dynamics  models.  Simulation  Modelling  Practice  and  Theory,  25,  118-133. 
doi: http://dx.doi.org/10.1016/j.simpat.2011.09.002 

Tako, A. A., Kotiadis, K., & Vasilakis, C., (March, 2010). A conceptual modelling framework for 
stakeholder  participation  in  simulation  studies.  In  Proceedings  of  the  2010  Operational 
Research Society Simulation Conference (SW10) (pp. 76-85). 

Tashakkori,  A.,  &  Teddlie,  C.  (2003).  Handbook  of  Mixed  Methods  in  Social  and  Behavioral 
Research. Thousand Oaks, CA: Sage. 

Teulings,  C.,  &  Baldwin,  R.  (Eds.).  (2014).  Secular  Stagnation:  Facts,  Causes  and  Cures. 
London: Centre for Economic Policy Research (CEPR). 

Thornton,  M.,  &  Travis,  S.  (2011).  Analysis  of  the  reliability  of  the  Modified  Caregiver  Strain 
Index. Journal of Gerontology ,58B,S127-S132 

139 

Tisue,  S.,  &  Wilensky,  U.,  (2004).  NetLogo:  A  simple  environment  for  modelling  complexity. 
Paper presented at the International Conference on Complex Systems, Boston, MA. 

Tomova, M. et al. (2013) 'EU governance and EU funds – testing the effectiveness of EU funds 
in a sound macroeconomic framework', European Commission, DG ECFIN, European Economy, 
Economic Papers No 510, 2013. 

Toobert, D., Hampson, S., & Glasgow R. (2000). The summary of diabetes self-care activities 
measure: results from seven studies and revised scale. Diabetes Care, 23, 943-950. 

Torp,  S.,  et  al.  (2008).  A  pilot  study  of  how  information  and  communication  technology  may 
contribute to health promotion among elderly carers in Norway. Health and Social Care in the 
Community, 16, 75-85. 

Torriti,  J.  (2010).  Impact  assessment  and  the  liberalisation  of  the  EU  energy  markets: 
Evidence-based  policy-making  or  policy-based  evidence-making?  Journal  of  Common  Market 
Studies, 48(4), 1065-1081. doi:10.1111/j.1468-5965.2010.02089.x 

Townsend P. (1979). Poverty in the United Kingdom. Harmondsworth: Pelican. 

Turnpenny, A., & Beadle-Brown, J. (2014). Use of quality information in decision-making about 
health and social care services - a systematic review. Health & Social Care in the Community, 
23(4), 349–361. 

U.S.  PREVENTIVE  SERVICES  TASK  FORCE  (2008).  Procedure  Manual.  Rockville,  MD:  Agency 
for Healthcare. 

UK Treasury (2003). The Green Book Appraisal and Evaluation in Central Government. London: 
TSO. 

UKDPC. (2008). Working towards Recovery: Getting Problem Drug Users into Jobs. London: UK 
Drug Policy Commission (UKDPC). 

UN  (2015).  Report  of  the  Inter-Agency  and  Expert  Group  on  Sustainable  Development  Goal 
Indicators 

UNICEF (2008). Bridging the gap: The role of monitoring and evaluation 

Valkila,  N.,  &  Saari,  A.  (2011).  The  productivity  impact  of  the  voice  link  between  elderly  and 
nurses:  An  assisted  living  facility  pilot.  Archives  of  Gerontology  and  Geriatrics,  52,  e44-e49. 
doi:10.1016/j.archger.2010.05.008 

Van Berkel, R., de Graaf, W., & Sirovátka, T. (2012).  Governance of the activation policies in 
Europe:  Introduction.  International  Journal  of  Sociology  and  Social  Policy,  32(5/6),  260-272. 
doi:10.1108/01443331211236943 

Van Hemelrjick (2010) Case presentation on measuring complex systemic changes, conference 
on Evaluation in Development, 20th to 21st May 2000 

Van Lancker, W. (2012). The European World of Temporary Employment: Gendered and poor? 
European Societies, 14(1), 83-111. doi: 10.1080/14616696.2011.638082 

Vanclay,  F.  (2003).  International  principles  for  social  impact  assessment.  Impact  assessment 
and project appraisal, 21(1), 5-12. 

Vanclay,  F.  (2013)  International  Principles  for  Social  Impact  Assessment,  Impact  Assessment 
and Project Appraisal 21(1) pp. 5–11 

Vennix,  J.  A.,  (1999).  Group  model-building:  tackling  messy  problems. System  Dynamics 
Review, 15(4), 379-401. 

Verburg,  P.V.,  K.P.  Overmars,  (2009).  Combining  top-down  and  bottom-up  dynamics  in  land 
use  Modelling:  exploring  the  future  of  abandoned  farmlands  in  Europe  with  the  Dyna-CLUE 
model, Landscape Ecology 24 1167–1181. 

Vincenot, C. E., Giannino, F., Rietkerk, M., Moriya, K., et al. (2011). Theoretical considerations 
on the combined use of System Dynamics and individual-based modeling in ecology. Ecological 
Modelling, 222(1), 210-218. doi: http://dx.doi.org/10.1016/j.ecolmodel.2010.09.029" 

140 

Virtanen,  M.,  Kivimäki,  M.,  Joensuu,  M.,  Virtanen,  P.,  et  al.  (2005).  Temporary  employment 
and health: a review. International Journal of Epidemiology, 34(3), 610-622.  

Visser,  R.  (2003).  Trends  in  program  evaluation  literature:  The  emergence  of  pragmatism: 
Texas Center for Adult Literacy, Occasional Research Paper No. 5, Retrieved August 29, 2011, 
from http://www-tcall.tamu.edu/orp/orp5.htm. 

Wackernagel, M., & Rees, W. (1998). Our ecological footprint: reducing human impact on the 
earth (No. 9). New Society Publishers. 

Wade R., Cartwright C., & Shaw K. (2012). Factors relating to home telehealth acceptance and 
usage compliance. Risk Management and. Healthcare Policy, 5, 25–33. 
doi:10.2147/RMHP.S30204. 

Waterworth,E.L.;  Waterworth,J.A.;  Peter,C.;  Ballesteros,S.  (2012)  Seniors  in  charge  of  ICT 
innovation. PETRA Conference 2012, June 6-8, 2012, Crete, Greece.  

Weber, M. (1904). The Methodology of the Social Sciences (1949 ed.). Glencoe: Free Press. 

Wechsler,  D.  (1997).  Wechsler  Adult  Intelligence  Scale—3rd  Edition  (WAIS-3®)  San  Antonio, 
TX: Harcourt Assessment 

Weinbren, D., & James, B., (2005). Getting a Grip: the Roles of Friendly Societies in Australia 
and Britain Reappraised. Labour History, 88. 

Weiss,  C.  (1976).  Using  research  in  the  policy  process:  potential  and  constraints.  Policy 
Studies Journal 4, 224-228.  

Weiss,  C.  (1995).  Nothing  as  practical  as  good  theory:  Exploring  theory-based  evaluation  for 
comprehensive  community  initiatives  for  children  and  families.  In  A.  Connell,  A.  Kubisch,  L. 
Schorr  &  C.  Weiss  (Eds.),  New  approaches  to  evaluating  community  initiatives:  Volume  1, 
concepts, methods, and contexts (pp. 65-92). New York: Aspen Institute. 

Wiggan,  J.  (2007).  Reforming  the  United  Kingdom's  public  employment  and  social  security 
409-424. 
agencies. 
doi:10.1177/0020852307081150 

Administrative 

International 

Sciences, 

Review 

73(3), 

of 

Wiggins,R.D.,  Higgs,  P.F.D.,  Hyde,  M.,  &  Blane,  D.B.  (2004).  Quality  of  life  in  the  third  age: 
key  predictors  of 
the  CASP-19  measure.  Ageing  &  Society,  24,  693–708.  doi: 
10.1017/S0144686X04002284. 

Wildavsky, A. (1979).  Speaking Truth to Power: The Art and Craft of Policy Analysis. Boston: 
Little, Brown. 

Wilkinson, R., & Pickett, K. (2009). The Spirit Level: Why More Equal Societies Almost Always 
Do Better. London: Allen Lane. 

Wilson,  J.  (1989),  Bureaucracy:  what  government  agencies  do  and  why  they  do  it,  Basic 
Books. 

Wilson,  K.  E.  (2014).  "New  Investment  Approaches  for  Addressing  Social  and  Economic 
Challenges", OECD Science, Technology and Industry Policy Papers, No. 15, OECD Publishing. 
DOI: 10.1787/5jz2bz8g00jj-en 

Windrum,  P.  (2008).  Innovation  and  entrepreneurship  in  public  services.  In  P.  Windrum  &  P. 
Koch (Eds.), Innovation in Public Sector Services. Cheltenham: Edward Elgar. 

Wood, C., Leighton, D. (2010). 'Measuring Social Value – the gap between Policy and Practice', 
DEMOS. 

Wooldridge, M., & Jennings, N. R. (1995). Intelligent agents: Theory and practice. Knowledge 
engineering review, 10(2), 115-152. 

World Economic Forum. (2015). The Inclusive Growth and Development Report 2015. Geneva: 
World Economic Forum. 

Yavas, M., & Bacaksizlar, N. G., (2012). Financial sustainability of social security institutions in 
the presence of aging populations. 

141 

Zarit,  S.  H.,  Reever,  K.  E.,  Back-Peterson,  J.  (1980).  Relatives  of  the  impaired  elderly: 
correlates of feelings of burden. The Gerontologist, 20, 649-655. 

Zeigler,  B.  P.,  Praehofer,  H.,  &  Kim,  T.  G.,  (2000).  Theory  of  Modelling  and  simulation: 
integrating discrete event and continuous complex dynamic systems. Academic press. 

Zigmond,  A.  S.,  &  Snaith,  R.P.  (1983).  The  Hospital  Anxiety  and  Depression  Scale.  Acta 
Psychiatrica Scandinavica, 67, 361-370.  

Zimmet,  G.,  et  al.  (1988).  The  multidimensional  scale  of  perceived  social  support.  Journal  of 
Personality Assessment, 52, 30-41. 

Zolotas, X. (1981). Economic Growth and Declining Social Welfare, New York. 

Zung, W.W.K. (1971). A rating instrument for anxiety disorders. Psychosomatics, 12, 371-379. 

142 

143 

Europe Direct is a service to help you find answers 
to your questions about the European Union. 

Freephone number (*): 

00 800 6 7 8 9 10 11 
(*) The information given is free, as are most calls (though some operators, phone boxes or hotels may 
charge you). 

More information on the European Union is available on the internet (http://europa.eu). 

HOW TO OBTAIN EU PUBLICATIONS 

Free publications: 

• one copy:

via EU Bookshop (http://bookshop.europa.eu);

• more than one copy or posters/maps:

from the European Union’s representations (http://ec.europa.eu/represent_en.htm);
from the delegations in non-EU countries (http://eeas.europa.eu/delegations/index_en.htm);
by contacting the Europe Direct service (http://europa.eu/europedirect/index_en.htm) or
calling 00 800 6 7 8 9 10 11 (freephone number from anywhere in the EU) (*).

(*) The information given is free, as are most calls (though some operators, phone boxes or hotels may charge you).

Priced publications: 

•

via EU Bookshop (http://bookshop.europa.eu).

K
J
-
N
A
-
2
8
8
2
4
-
E
N
-
N

doi:10.2760/83089 

ISBN 978-92-79-74271-2