# Demonstrator D4.2 Set of Methods to Assess SI Implications at Different Levels: Instructions for WPs 5 & 6

Laura Secco, Elena Pisani, Catie Burlando, Riccardo Da Re, Paola Gatto, Davide Pettenella, Achilleas Vassilopoulus, Ebun Akinsete, Phoebe Koundouri, Antonio Lopolito, et al.

<figure><img src="image_1.png" /></figure>

## To cite this version:

Laura Secco, Elena Pisani, Catie Burlando, Riccardo Da Re, Paola Gatto, et al. Demonstrator D4.2 Set of Methods to Assess SI Implications at Different Levels: Instructions for WPs 5 & 6. Department of Territorio e Sistemi Agro-Forestali (TESAF) of the University of Padova (Italy). 2017, http://www.simra-h2020.eu/wp-content/uploads/2017/10/SIMRA_D4.2_Set_of_Methods_to_Assess_SI_Implications_at_Different_Levels hal-04179020

HAL Id: hal-04179020

https://hal.science/hal-04179020

Submitted on 10 Aug 2023

HAL is a multi-disciplinary open access archive for the deposit and dissemination of scientific research documents, whether they are published or not. The documents may come from teaching and research institutions in France or abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est destinée au dépôt et à la diffusion de documents scientifiques de niveau recherche, publiés ou non, émanant des établissements d'enseignement et de recherche français ou étrangers, des laboratoires publics ou privés.

Public Domain
<figure><img src="image_2.png" /></figure>

<figure><img src="image_3.png" /></figure>

<figure><img src="image_4.png" /></figure>

**Social Innovation in Marginalised Rural Areas**

**Call: H2020-ISIB-2015-2**

**Innovative, Sustainable and Inclusive Bioeconomy**

**Work Programme:** Topic ISIB-03-2015. Unlocking the growth potential of rural areas through enhanced governance and social innovation

**Demonstrator D4.2**

**Set of Methods to Assess SI Implications at Different Levels: Instructions for WPs 5 & 6**

**Authors:** Laura Secco, Elena Pisani, Catie Burlando, Riccardo Da Re, Paola Gatto, Davide Pettenella, Achilleas Vassilopoulus, Ebun Akinsete, Phoebe Koundouri; Antonio Lopolito, Maurizio Prosperi; Diana Tuomasiukka, Micheal Den Herde, Marko Lovric, Nico Polman, Marijke Dijkshoorn, Katerine Soma, Alice Ludvig, Gerhard Weiss, Ivana Zivojinovic; Simo Sarkki, Elisa Ravazzoli, Cristina Dalla Torre, Thomas Streifeneder, Bill Slee; Maria Nijnik, David Miller, Carla Barlagne and Irina Prokofieva.

**Approved by Work Package Manager of WP4:** Laura Secco, University of Padova
**Date:** 30/09/2017

**Approved by Project Coordinator:** Maria Nijnik, James Hutton Institute
**Date:** 30/09/2017

This document was produced under the terms and conditions of Grant Agreement No. 677622 for the European Commission. It does not necessary reflect the view of the European Union and in no way anticipates the Commission's future policy in this area.
<figure><img src="image_5.png" /></figure>

This page is left intentionally blank
<figure><img src="image_6.png" /></figure>

<table><tr><td>6.2.1.</td><td>Elements of the evaluation approach</td><td>78</td></tr><tr><td>6.2.2.</td><td>Subject of the evaluation: the SI initiative in MRA and its impacts</td><td>80</td></tr><tr><td>6.2.3.</td><td>Subject of the evaluation: SI initiative at local level</td><td>83</td></tr><tr><td>6.2.4.</td><td>Timing of the evaluation: ex-post/phases</td><td>86</td></tr><tr><td>6.2.5.</td><td>Setting the scene: complementarity with existing evaluation frameworks/methods</td><td>87</td></tr><tr><td>6.2.6.</td><td>Setting the scene: policy analysis</td><td>90</td></tr><tr><td>6.2.7.</td><td>Setting the scene: Who?</td><td>91</td></tr><tr><td>6.2.8.</td><td>Setting the scene: domains of impact</td><td>92</td></tr><tr><td>6.2.9.</td><td>Evaluation questions</td><td>92</td></tr><tr><td>6.2.10.</td><td>Judgement references: indicators</td><td>94</td></tr><tr><td>6.2.11.</td><td>Methodological design: analysis of impacts</td><td>97</td></tr><tr><td>6.2.12.</td><td>SIMRA approach 1 to impact evaluation</td><td>98</td></tr><tr><td>6.2.13.</td><td>SIMRA approach 2 to impact evaluation (in the statistical meaning of the term)</td><td>99</td></tr><tr><td>6.2.14.</td><td>In summary: SIMRA approach to impact evaluation</td><td>102</td></tr><tr><td>6.2.15.</td><td>Methodological design: policy analysis</td><td>103</td></tr><tr><td colspan="2">PART IV – INSTRUCTIONS FOR ASSESSMENT AND EVALUATION FOR WP5 AND WP6</td><td>106</td></tr><tr><td>7.</td><td>Instructions for the Pilot Test of the Assessment and Evaluation Methods in Pioneer SIMRA Case Studies</td><td>107</td></tr><tr><td>7.1.</td><td>Steps of Evaluation Method</td><td>107</td></tr><tr><td>7.2.</td><td>Step 1: Consultation of Experts and Key Informants</td><td>109</td></tr><tr><td>7.3.</td><td>Step 2a: Survey Design for Analysis of Policy Processes (Work Package 6)</td><td>115</td></tr><tr><td>7.4.</td><td>Step 3: Data Processing and Analysis</td><td>129</td></tr><tr><td>7.5.</td><td>Step 4: Additional Instruments for Analysis</td><td>132</td></tr><tr><td>7.6.</td><td>Step 5: Common Final Reporting</td><td>135</td></tr><tr><td>8.</td><td>Conclusions</td><td>137</td></tr><tr><td>9.</td><td>Acknowledgments</td><td>139</td></tr><tr><td>10.</td><td>References</td><td>140</td></tr><tr><td colspan="2">Annex 1: Preliminary List of Questions for the Assessment and Evaluation of Social Innovation Initiatives in Pilot Case Studies in Marginalised Rural Areas</td><td>153</td></tr><tr><td colspan="2">Annex 2 – List of Key Terms to Facilitate the Pilot Test</td><td>192</td></tr></table>
<figure><img src="image_7.png" /></figure>

## List of Acronyms

<table><tr><td>BENISI</td><td>Scaling Social Innovation</td></tr><tr><td>CBA</td><td>Cost Benefit Analysis</td></tr><tr><td>CEA</td><td>Cost Effectiveness Analysis</td></tr><tr><td>CMEF</td><td>Common Monitoring and Evaluation Framework</td></tr><tr><td>CrESSI</td><td>Creating Economic Space for Social Innovation</td></tr><tr><td>CS</td><td>Case Studies</td></tr><tr><td>DD/DiD</td><td>Difference in Difference</td></tr><tr><td>EC</td><td>European Commission</td></tr><tr><td>EIA</td><td>Environmental Impact Assessment</td></tr><tr><td>EIP</td><td>European Innovation Partnerships</td></tr><tr><td>EU</td><td>European Union</td></tr><tr><td>GIS</td><td>Geographic Information Systems</td></tr><tr><td>LAU</td><td>Local Administrative Units</td></tr><tr><td>LEADER</td><td>Liaison Entre Actions de Développement de l'Économie Rurale</td></tr><tr><td>MIAA</td><td>Methodology for Impact Analysis and Assessment</td></tr><tr><td>MRAs</td><td>Marginalised Rural Areas</td></tr><tr><td>NGO</td><td>Non-Governmental Organisation</td></tr><tr><td>NUTS</td><td>Nomenclature des Unités Territoriales Statistiques</td></tr><tr><td>OECD</td><td>Organisation for Economic Co-operation and Development</td></tr><tr><td>POT</td><td>Perceived opportunities and threats</td></tr><tr><td>RACER</td><td>Relevant, Accepted, Credible, Easy and Robust</td></tr><tr><td>RBM</td><td>Results Based Management</td></tr><tr><td>RDP</td><td>Rural Development Programme</td></tr><tr><td>SAB</td><td>Scientific Advisory Board</td></tr><tr><td>SEA</td><td>Strategic Impact Assessment</td></tr><tr><td>SI</td><td>Social Innovation</td></tr><tr><td>SI-DRIVE</td><td>Social Innovation: Driving Force of Social Change</td></tr><tr><td>SIMPACT</td><td>Boosting the Impact of Social Innovation in Europe through Economic Underpinnings</td></tr><tr><td>SIMRA</td><td>Social Innovation in Marginalised Rural Areas</td></tr><tr><td>SITT</td><td>Social Innovation Think Thank</td></tr><tr><td>SMART</td><td>Specific, Measurable, Achievable, Relevant and Timebound</td></tr><tr><td>SNA</td><td>Social Network Analysis</td></tr><tr><td>SROI</td><td>Social Return on Investment</td></tr><tr><td>TEEB</td><td>The Economics of Ecosystems and Biodiversity</td></tr><tr><td>TEPSIE</td><td>Theoretical, Empirical and Policy Foundations for Social Innovation in Europe</td></tr><tr><td>ToC</td><td>Theory of Change</td></tr><tr><td>TSI</td><td>Third Sector Impact</td></tr><tr><td>WP</td><td>Work Package</td></tr></table>
<figure><img src="image_8.png" /></figure>

Executive Summary

This Deliverable provides a preliminary set of methods for the evaluation of the implications of SI at different levels, and provide instructions for their test in pioneer case studies in Work Packages 5 and the analysis of policy in Work Package 6.

The results from the pioneer case studies will be used to improve and refine the set of methods and tools, and in particular the starting point (i.e. the list of questions for data collection), the sampling strategy, and identification of impacts to create a consolidated version to be explained in training CS Teams, and used in the evaluation of case studies (Work Package 5) and in analysis of policy processes (Work Package 6). This will refine its applicability for use in the full set of case studies in 2018 (Work Package 5), and finalised in Deliverable 4.3 as a manual of methods to assess social innovation and its impacts.

Using an evaluation approach, Deliverable 4.2 refers to the set of options proposed to design, test, refine and finalise an innovative methodology for assessing and evaluating social innovation in marginalised rural areas at the local level. The foundation of any evaluation approach is its framework. An evaluation agenda is developed for the assessment and evaluation in which the key dimensions of SI are defined. It includes the key steps, interconnections and relationships to be considered in evaluating social innovation across all of its phases, and the key determinants, dimensions and consequences of SI (e.g. supporting conditions, blocking processes, participants, outputs and impacts, etc.).

The evaluation framework proposed draws on the theoretical underpinning of work in SIMRA on social innovation and marginalised rural areas, and the engagement with stakeholders in the Social Innovation think Tank (SITT), and the theory of change. These underpin the identification of a set of core dimensions for addressing SI process, SI project and related SI effects (Section 4), and a detailed analysis of the sub-dimensions, components and variables for assessing and evaluating SI (Section 5). One of the core dimensions of the evaluation framework is 'agency', in processes of reconfiguration of social practices. Agency refers to the initiator and developer of SI in a territory (not necessarily as a single individual but rather a network or organisation), as well as to one of the key respondents in the assessment process. It is the starting point for exploring why and how the SI initiative emerged in a certain marginalized rural area, for understanding the process of social innovation, and the role of small networks amongst actors as the initial aggregation (clique) that initiate the change.

A second core dimension is the reconfiguring of-reconfigured social practices. This dimension can be considered as the 'black box' of social innovation in marginalized rural areas. The evaluation framework seeks to deepen understanding of the components and variables that explain this process of changing relations, conditions and ultimately social practices. These key sub-dimensions are new governance arrangements, new networks and new attitudes. Finally, special attention is given to the effects of SI initiatives, included measurable immediate outputs, outcomes on direct beneficiaries of the SI initiative, and impacts on both direct and indirect beneficiaries in a given territory.

Social innovation initiatives may be established in a marginalised rural area within a EU or non-EU Mediterranean target country. In the approach developed, any SI initiative is considered to comprise of two parts requiring evaluation in conjunction with each other. The first part is the SI process, and the second is the SI project. The SI process occurs with the emergence of the SI idea through the reconfiguration of social practices. The SI process is based on the following dimensions: perceived context, agency (actors and preparatory action), from reconfiguring to reconfigured social practices (i.e. new governance arrangements, new networks, new attitudes). Once these dimensions have occurred, the SI project emerges based on inputs, activities and outputs. These two parts determine outcomes and impacts (identified as 'SI effects') in the economic, social, environmental and institutional domains.

In setting the evaluation methodology, the assumption is that the local level (in spatial, jurisdictional and social scales) is the most relevant for assessing and evaluating a SI initiative and its impacts in a marginalized rural
<figure><img src="image_9.png" /></figure>

area. SI often happens at lower spatial levels, and in particular, at the local level where individuals, single organisations and local networks matter in project implementation. The majority of the SIMRA database of examples of SI are developed at a local level, within relatively limited territorial areas and size. Even if SI emerges in a multi-level context and its dimensions, sub-dimensions and components refer to different spatial levels, their effects are typically local. For example, while triggers that lead to the emergence of a SI may occur at any spatial level (e.g. international, national, regional or local), they become salient for SI when their effects and reactions are at a local level.

The local level at which the SI occurs in a certain territory can have distinct boundaries (e.g. a group of landowners within the same municipality) or indistinct boundaries (e.g. a private-public hybrid network engaged in different nearby areas but not in the same municipality). Different solutions are proposed for the evaluation, depending on specific circumstances of the case study. A link to higher jurisdictional and spatial levels is provided by the policy analysis included as a key part of the evaluation. Local effects of regional or national policies are identified, and if incongruences or inconsistencies between policies (e.g. policy gaps) are identified, the matching between local and the higher levels is explored through interviews with policy experts.

The focus for data collection is at the micro-level, and the statistical units of analysis are individuals or single organisations or local networks that act in SI initiatives. The focus on SI impacts is at the meso-level of analysis, and the statistical units of analysis are the specific SI initiative (i.e. the case study) as well as the territory where the SI project is implemented. Primary data are collected at the local and or territorial level while secondary socio-economic, institutional and environmental data, often available at regional, national, European and other macro-regional spatial levels, can be found in various databases (but the data availability and level of aggregation have to be checked on a case-by-case basis). In terms of the *momentum* when the evaluation is carried out with respect to the SI initiative phase of development, in SIMRA only ex-post evaluations are possible. SI cannot be foreseen or planned, but it emerges spontaneously due to the convergence of factors (e.g. an innovator having an idea, supporting perceived context conditions, social dynamic processes that lead to develop the initial idea until its implementation in practice). Then, the proposed evaluation approach and set of methods can be integrated with existing frameworks, for example the EC Common Monitoring and Evaluation Framework. The proposed methods for data collection, processing and evaluation integrate both qualitative approaches and tools (e.g. focus groups, semi-structured interviews, document analysis), and quantitative approaches and tools (e.g. structured interviews based on closed and open coded questions, indicators and complex indices for measuring the SI initiative performance in terms of effectiveness, efficiency, relevance, sustainability, impact and other evaluation criteria). Final reporting is preliminary designed for triangulating information collected by means of qualitative-oriented tools (and described by narrative texts) with data collected by means of quantitative-oriented tools (and described by means of performance indicators, numbers and figures).

An impact evaluation will be applied in SI case studies by adopting two different approaches, the choice between which depends upon the case. An impact evaluation is an assessment of the causal effect of a project, a programme or a policy on beneficiaries (compared to non-beneficiaries). It uses a counterfactual to estimate what the state of the beneficiaries would have been in the absence of the programme (the control or comparison group), compared to the observed status of beneficiaries (treatment group), and to determine the intermediate or final outcomes attributable to the intervention (in this case, the SI initiative).

In SIMRA, an impact evaluation based on robust statistical techniques will be applied only in those specific SI case studies where such a type of impact evaluation is technically feasible (i.e. where the cause-effect relation between the SI and its impacts can be identified and measured separately with respect to the effects due to other factors). The identification of the cases to be evaluated using statistical techniques of impact evaluation will be performed by the CS Teams on a case by case basis in the initial steps of the evaluation process, in collaboration with Work Package 5. Those case studies that have the minimum basic characteristics for being evaluated using the proposed statistical techniques will be used as explicatory case studies to demonstrate SI impacts in MRAs, to check how an impact evaluation of this type should be implemented in practice. They will be used for cross-checking whether the analysis based on stakeholders' perceptions of changes (based on
<figure><img src="image_10.png" /></figure>

primary data and qualitative methods) is consistent/aligned with the findings based on the statistical techniques of impact evaluation (based on secondary data and quantitative methods). To focus the evaluation on key impacts (to reduce costs and capture the real meaning and consequences of SI), the method is based on the identification of 'significant aspects and impacts' on the economic, social, environmental and institutional domains. Specific criteria and methods for identifying 'significant impacts' are proposed.

Finally, the preliminary set of methods and tools developed and presented was co-created, taking into consideration the suggestions provided by members of the SIMRA SITT and the literature on SI evaluation and other fields of evaluation. This includes reports published by other EU-funded research projects on SI, i.e. it integrates the requests expressed by practitioners, evaluators, and experts in agriculture, forestry and rural development with the scientific knowledge on, and technical solutions for, evaluation.

Deliverable 4.2 is divided into four Parts. The review of existing methods is in Part I. Part II explains the relationship between the general theoretical background and the evaluation framework proposed (Section 4) and provides details of the dimensions and variables proposed for evaluation (Section 5). The architecture of the proposed SIMRA evaluation approach, process, framework and set of methods is in Part III (Section 6). The instructions for testing the preliminary set of assessment and evaluation methods is in Part IV (Section 7) are operational and include the preliminary questions in Annex 1, and a list of key terms to facilitate the comprehension by CS Teams in Annex 2. The questions by target population (i.e. the questionnaires to be submitted - by means of face-to-face interviews - to the different categories of actors in the case study) are provided in Deliverable 5.1. As a demonstrator, additional supporting resources for assessment and evaluation are accessible on the SIMRA Web Site (www.simra-h2020.eu/resources), including the methods identified and the database of the methods analysed.

Each part of this deliverable can be read standalone. To facilitate the reading, target readers and users are identified for each Part.

Part I - EXISTING KNOWLEDGE provides a review of the methods analysed in Section 2 and a detailed analysis of consultation with members of SIMRA SITT in Section 3. The list of identified and analysed frameworks, approaches, methods and tools (as a Word file) and the database with the methods analysed (as an Excel file) are available online in SIMRA Web Site. The main target readers and users of Part I are SIMRA partners and SITT members.

Part II - SETTING THE SCENE: WHAT WE NEED TO EVALUATE AND WHY contextualises the rationale for the development of the evaluation framework in Section 4 and describes the phases, dimensions, sub-dimensions, variables and related interconnections that likely characterise social innovation in marginalised rural areas (Section 5). These elements represent the core contents of the evaluation, responding to questions of what we need to evaluate and why. Section 5 describes in detail the evaluation framework, designed for the specific purpose of targeting methods and tools of evaluation that address the different elements of an SI initiative. The main target readers and users of Part II are leaders and project partners in Work Packages 5 and 6.

Part III - SETTING THE METHODS: HOW TO ASSESS AND EVALUATE SI IN MRAs provides the fundamentals for understanding the proposed evaluation strategy. Section 6 provides a detailed summary of the elements and sub-elements of the evaluation approach adopted in SIMRA and contextualises them within the overall architecture for assessment and evaluation that SIMRA will have by the end of the project. Based on the evaluation framework proposed in Section 5, it distinguishes the three phases of SI innovation (SI Process, SI Project and SI Effects), and identifies the spatial and temporal levels for evaluation, as well as a set of preliminary questions for assessing their relationship to standard evaluation criteria (relevance, efficiency, effectiveness, impact and sustainability). It provides a specific section on the conditions for carrying out impact evaluation, assuming that two approaches to impact evaluation are possible in SIMRA. The main target readers and users are leaders of Work Packages 5 and 6, and the CSs Teams.

Part IV - INSTRUCTIONS FOR PILOT ASSESSMENT AND EVALUATION OF SI IN MRAs FOR WORK PACKAGEs 5 AND 6 specifies the steps and operational instructions that SIMRA CS Teams will follow to test the preliminary set of methods and tools in pilot SI CS initiatives (Section 7). It also specifies the steps and operational
<figure><img src="image_11.png" /></figure>

<figure><img src="image_12.png" /></figure>

instructions that Work Package 6 will adopt for the analysis of policy processes, including questions on policy issues that will need to be used in a complementary way in the evaluation of CSs at a local level. Information relevant to policy analysis will be collected at local level by CS Teams using the guide for semi-structured interviews. This will use the questions in Annex 1 once be tested in Work Packages 5 and 6. A list of key terms to facilitate the interpretation and translation of questions and other elements of the method (e.g. identification of persons to be interviewed) into the local languages of case studies completes this part (Annex 2). The main target readers/users are WP5 and WP6 Leaders and pilot CSs Teams.

A final set of concluding remarks (Section 8), acknowledgements and references complete the document.
<figure><img src="image_13.png" /></figure>

# 1. Introduction

## 1.1. Objective and Purpose

The overall goal of SIMRA Work Package 4 (WP4) is "to build up an innovative, integrated set of methods to evaluate a range of aspects/impacts of social innovation (SI) activities/organisations/networks on, in conjunction or separately according to the needs, economic, social, ecological/environmental and institutional/governance components of territorial capital in marginalised rural areas (MRAs) across the target region" (SIMRA Grant Agreement 677622: 31). The first objective of WP4 is to develop and test an innovative, draft set of methods to measure SI in MRAs, its impacts and related policy implications. In order to reach this stated research objective, two series of research Tasks and related Deliverables were planned. The main output of Task 4.1 was Deliverable 4.1 (Secco et al., 2016). It provided guidelines for the identification and analysis of existing frameworks, approaches, methods and tools of evaluation for used or adaption for the evaluation of social innovation and its impacts in relation to the economic, social, environmental and institutional and governance domains, as well as to its policy implications. The main goal of D4.1 was to provide a standardised tool for exploring what other projects and publications had reported (i.e. how to explore and analyse the literature).

This deliverable aims to provide instructions on the evaluation practice and the data collection in the SI case studies in order to assess SI, its impacts and related policy processes in each case study (for use in Work Packages 5 and 6). More specifically, it provides guidelines for an initial application of the proposed method in a few pioneer<fnref n="1" /> case studies. The main aim of such pilot application is to test the applicability (the practical assessment) of the preliminary set of methods and related tools that are proposed for use in SIMRA. This deliverable describes the background on how the set of methods has been developed and how it should be applied in practice. To do this, it combines: a) the analysis of existing frameworks, approaches, methods and tools to evaluate SI and its impacts that have been identified in literature (Task 4.2); b) the analysis of the existing qualitative frameworks, approaches, methods and tools to evaluate policy implications of SI (Task 4.3); c) the opinions expressed by the members of the SITT at their workshop in Bratislava (October 2016), and in the second online consultation (July 2017); d) a step-wise process of development of an integrated and innovative set of methods and tools, that will be tested and refined during the next months of the project on the basis of the pioneer case studies and pilot tests results.

The purpose of Deliverable 4.2 is as a set of instructions to guide the design of assessment and evaluation and test the collection of first empirical results of SI initiatives and their impacts in pioneer SIMRA case studies (WP5), and in selected political processes (WP6). Feedback on its implementation will be used to refine the proposed set of methods and tools, and develop a more advanced version to be used in 2018 to collect empirical results across all SIMRA case studies and policy processes. In doing so, Deliverable 4.2 prepares the ground for the final manual (Deliverable 4.3, i.e. the main expected output of WP4 at the end of SIMRA) that will be used by evaluators to assess SI in marginalised rural areas (MRAs), its impacts and policy implications. The final, integrated set of methods for use in SIMRA, will be a timely and novel element for supporting decision-making, based on SI assessment and evaluation<fnref n="2" />.

<footnote n="1">¹ Pioneer case studies are SI case studies identified in the SIMRA online database of social innovation in marginalised rural areas (Bryce et al., 2017) which the preliminary set of methods will be on with, to check the assessment in practice. They are intended to be "pilot tests", to verify whether the proposed evaluation framework, methods and tools are complete, clear, functional to the evaluation purposes, and understandable by data collectors and analysts (CS Teams, WP5 and WP6). They will be piloted in the testing of the proposed methodology.</footnote>

<footnote n="2">² Despite these two terms being often used synonymously, they specify two different concepts: while assessment means "the action of assessing something", evaluation means "the making of a judgement about something" (Oxford Dictionary online, 2017). SI assessment includes all those activities that are necessary (e.g. identification of methods, data collection, etc.) for completing the evaluation (i.e. for having a final judgement) of a SI initiative. The term evaluation used in D4.2 includes also the term assessment (unless differently specified).</footnote>
<figure><img src="image_14.png" /></figure>

Deliverable 4.2 provides the rationale, and a first attempt at specifying the key dimensions on which to focus the evaluation of SI in marginalised rural areas (MRAs). It proposes a specific and innovative approach to the assessment and evaluation in a field as diverse and complex as that of social innovation in MRAs in relation to agriculture, forestry and rural development, to be tested in Work Packages 5 and 6.

## 1.2. An Overview of Work Package 4 Activities

An assessment and evaluation framework<fnref n="3" /> for social innovation, its impacts and related policy processes needs to be specific and strictly connected to the definition of SI, yet broad enough to include the variability and breadth of cases of social innovation in marginalised rural areas throughout Europe and non-EU Mediterranean areas. The various steps and paths towards developing the SIMRA evaluation methods and tools for evaluating SI and its impacts and policy implications, briefly recalled in the previous paragraph, are presented in Figure 1.

<figure><img src="image_15.png" /><figcaption>**Figure 1.** Paths to the development of evaluation methods and tools for use in SIMRA. (Source: SIMRA research team).</figcaption></figure>

Phase 1 consists of the development of the Guidelines for identifying and analysing, following a standard approach, existing methods to be used in or adapted to the evaluation of social innovation in marginalised rural areas (Secco et al., 2016). At the same time, members of the SIMRA Social Innovation Think Tank (SITT) members are consulted on methods for evaluation in a world café session held at the SITT workshop (Bratislava, 28th October 2017). The main outputs of Phase 1 are reported in Secco et al. (2016; D4.1)

<footnote n="3">³ From now onwards “assessment and evaluation framework” is “evaluation framework”.</footnote>
<figure><img src="image_16.png" /></figure>

Deliverable 4.1 and the summary report to stakeholders on the results of the consultation. These outputs
derive from completion of Tasks 4.1 and 2.1 (Polman et al., 2017).

Phase 2 comprises the compilation of an Excel file for analysing the identified existing methods, and a more
detailed analysis of the results that emerged from consultations with members of the SITT at the workshop
(Bratislava, 2016) and online (July 2017). The principal aim outputs of Phase 2 are internal to SIMRA, and
consist of: (i) a database of existing methods, analysed in detail (in an Excel structure, provided in Secco et al.,
2016; D4.1)<fnref n="4" />, and (ii) main observations and suggestions made by SITT members in the consultations. These
documents have driven the development of the evaluation framework and the related proposed methods and
tools.

Phase 3 comprises the development of the evaluation framework in its conceptual and operational
fundamentals. It includes two sub-phases:

(i) the compilation of a document collecting and organising potentially useful ideas/methods/variables, extracted from the analysis of existing methods and consultation with members of the SITT (Figure 1, Phase 3.a), and

(ii) the development of an evaluation framework and related methods and tools, specifically designed for targeting such methods and tools to the various dimensions and aspects of SIs in MRAs (Figure 1, Phase 3.b).

The main output of Phase 3 is this deliverable, which provides:

(i) a detailed explanation of the requirements for the assessments and evaluations in order to understand a SI initiative, its impacts and related policy implications;

(ii) an overview of the SIMRA approach and methods proposed to assess and evaluate a SI initiative and, finally, the guiding operational instructions to WP5 and WP6 for understanding and testing this preliminary set of methods and tools.

Phases 4 and 5 of the development process will be reported in Deliverable 4.3.

<footnote n="4">⁴ This database in available online in the SIMRA Web Site at www.simra-h2020.eu/resources.</footnote>
<figure><img src="image_17.png" /></figure>

<figure><img src="image_18.png" /></figure>

# PART I – EXISTING KNOWLEDGE
<figure><img src="image_19.png" /></figure>

**2. Literature Review: A Brief Overview of Existing Methods**

This Section provides an overview of the literature analysis<fnref n="5" /> on existing frameworks, approaches, methods and tools for evaluation that could be applied or adapted to the case of social innovation in marginalised rural cases. In the following we provide: i) an outline of the methodological steps used for the literature analysis, and ii) a synthesis of main results. The SIMRA database of frameworks, approaches and methods, with detail of analysis and specifications of evaluation tools, are accessible in the SIMRA Resources webpage (see www.simra-h2020.eu/resources/)

**2.1. Materials and Methods of Literature Analysis**

The selection of publications for the analysis was carried out separately for each domain (economic, social, environmental, and governance/institutional). However, the selection criteria used were common. An initial screening enabled the identification of a number of publications in each domain, presenting frameworks, approaches, methods and tools used in different fields for evaluation. An electronic search aimed at identifying the frameworks, approaches, methods and tools in Scopus, Google Scholar and Web of Science databases was carried out using text terms with appropriate truncation and relevant indexing terms<fnref n="6" />. Additional publications were identified from the reference lists of reviews retrieved within the electronic search.

The search was not limited to any language or country of publication. However, not all the identified publications were relevant to social innovation and/or social innovation in marginalised rural areas. To be included in the analysis, a framework, approach, method and related tool needed to respect the following criteria: i) to refer to the specific domain of the sub-task (economic, social, environmental or governance/institutional); ii) to encompass, directly or indirectly, social issues; iii) to be accessible, so that judgement of its relevance based on the other criteria was feasible. The full text the publications and studies fulfilling the inclusion criteria were collected and evaluated for relevance according to the aim of the review. Reviewing the scientific literature provided few insights, so the search was expanded to include white papers, grey literature and other reports. As a result, the above procedure was repeated on Google and Bing search engines rather than only in scientific databases<fnref n="7" />.

The main inclusion/exclusion criterion applied to all the domains was that of relevance to the specific domain<fnref n="8" />. However, overlaps between domains have been identified. For example, some frameworks, approaches and methods are related to social and environmental domains, but use economic (monetary) terms and/or borrow heavily from methods developed in the discipline of economics. Several tools have been excluded due to the fact that they are not free to use.

<footnote n="5">⁵ Secco et al. (2016) in Deliverable 4.1 provided guidelines for the identification of existing frameworks, approaches, methods and tools of evaluation and a standardized reporting form for their analysis (Annex - Excel file). The literature review was carried out by SIMRA sub-Tasks 4.2.1, 4.2.2, 4.2.3 and 4.2.4 on economic, social, environmental and governance/institutional aspects of SI, as well as by Task 4.3 on qualitative methods for policy analysis. Work Package 4 Leader focused on the evaluation frameworks developed by the EU, large research and evaluation consortia, and think tanks and studies more explicitly focused on social innovation.</footnote>

<footnote n="6">⁶ The search in the economic domain was in the form: [economic terms] and [evaluation terms] and [area/sector terms] and [social innovation terms]. The search terms used were: economic, cost, investment, benefit, turnover, value added, wealth, income, employment, wage, development, growth, capital, spending, efficiency, effectiveness, impact, evaluation, valuation, assessment, framework, approach, tool, toolkit, toolbox, method, rural, urban, peri-urban, local, regional, agriculture, forestry, social innovation, innovation, NGO and social enterprise.</footnote>

<footnote n="7">⁷ In the example of the economic domain, the search in the Tools and Resources for Assessing Social Impact database (http://mckinseyonsociety.com/social-impact-assessment/what-is-social-impact-assessment/) used the criteria: i) Approach: Tool, Method, Best Practice; ii) Purpose: Assessment; iii) Organisation: Social Enterprise, Social Investor; iv) Sector: General Applicability; v) Focus: Organisational Effectiveness, Impact, Outcomes, Outputs; vi) Geography: Developed Countries; vii) Techniques: Benchmarking, Cost Analysis, Descriptive Statistics, Expert Review, Mapping Methods, Multidimensional Indices, Regression Methods, Strategic Assessment.</footnote>

<footnote n="8">⁸ A situation often encountered by the economic subtask was that economic aspects were confused with individual user benefits or broader social impacts, such as the valuation of changes in amenity or quality of life factors (e.g. health, safety, recreation, air or noise quality). As a result, these kinds of frameworks, approaches and methods were excluded, except insofar as they examined an area's level of economic activity.</footnote>
<figure><img src="image_20.png" /></figure>

Where relevant, frameworks, approaches or methods have been included if designed for applications in the field and potentially applicable at a local scale (with minor or major changes and based on data availability). However, references to social innovation aspects as defined by SIMRA is not guaranteed.

For each framework, approach or method identified through the search, general information on the overall assessment method and related tools were provided. The level of specificity of the method in relation to social innovation was considered, its domains and territorial sector described, and the evaluation characteristics specified in detail. An overview of the frameworks, approaches and methods in the “hot-spot” have been provided<fnref n="9" />.

## 2.2. Main Findings

In total, 163 frameworks, approaches and methods were collected, but only the most relevant (n. 111) were analysed fully. Often, a single framework, approach and/or method comprises a number of assessment or evaluation tools. Consequently, within the 111 analysed sets, 214 tools were identified and described. Although different meanings may be used, 62.2% of the tools mentioned the word “framework”, “approach, or “tool” in the text, and 69.4% use the term “method”. The majority of the methods have a full reference (97.3%), and in 73.9% of the cases a website address is also provided. Less than half of the cases (45%) include an evaluation design. The main source of information is scientific articles (approximately half), followed by technical manuals, books and webpages. In the majority of cases, the intended context of application is not specified, and in only 30.9% of cases the focus is explicitly on rural areas (Figure 2). The scale of application is very often global (54.1%), or for Europe (32,4%), with few in Africa and Asia (Figure 2).

<figure><img src="image_21.png" /><figcaption>**Figure 2.** Applications of the frameworks, approaches and methods analysed, in terms of geographical region, and context (Source: SIMRA research team based upon the SIMRA database of frameworks and methods).</figcaption></figure>

<footnote n="9">⁹ As an example, in the economic domain the main observations about frameworks were that: i) IRIS has a very rich database with metrics; ii) SIMPACT (an FP7 EU-funded project) is very relevant to the SIMRA approach but, at the time of our literature survey, has not published its assessment framework; iii) TEEB has provided a standard for environmental valuation, and is applied in economics; iv) MIAA framework provides concrete and detailed explanations of how it should be applied (which is useful for the development of SIMRA methods); v) “Assessing social impact in private ventures” and “Double bottom line” frameworks are collections of tools; vi) Poverty and Social Impact Analysis is addressed for use at a higher scale, but some methods might be adjustable to local use; vii) Bill & Melinda Gates Foundation contains important methods and tools; viii) the UK HM treasury is the only one referring to stated preference methods, but relatively basic examples; ix) making the most of it, economic evaluation in the social welfare field has interesting case studies and narrative; x) some (e.g. the polyhedral model) are very new and it is not possible to obtain much information In terms of approaches and methods. Cost-Benefit Analysis (CBA), Cost Effectiveness Analysis (CEA) and Social Return on Investment (SROI) are the most popular methods. The Local Multiplier 3, Stated Preference Surveys and Revealed Preference Surveys are the most widely used.</footnote>
<figure><img src="image_22.png" /></figure>

Scientists are the main developers or proponents of individual frameworks, approaches or methods and related tools, sometimes in collaboration with other categories (e.g. institutional organisations)<fnref n="10" />. Target users are heterogeneous and principally comprise the scientific community (73%), policy programme and project evaluators (62%), consultants (47%) and community-level organisations (39%) (Figure 3).

<figure><img src="image_23.png" /><figcaption>**Figure 3.** Developers of the frameworks, approaches and methods by community of origin, and intended target users (Source: SIMRA research team based upon the SIMRA database of frameworks and methods)</figcaption></figure>

Over two-thirds of the analysed methods do not explicitly mention social innovation. In a few cases the topic is secondary or an indirect area of interest, while only in the remaining cases (21.6%) is it the main scope of application. Amongst the evaluation frameworks specifically focused on social innovation, there are several EU-funded projects, included: CrESSI (Creating Economic Space for Social Innovation), TEPSIE (The Theoretical, Empirical and Policy Foundations for Social Innovation in Europe), TSI (Third Sector Impact), SI-DRIVE (Social Innovation: Driving Force of Social Change), BENISI (Scaling Social Innovation), and SIMPACT (Boosting the Impact of Social Innovation in Europe trough Economic Underpinnings). However, they are typically oriented towards urban areas and other types of innovation. The innovation to which publications and reports refer in

<footnote n="10">¹⁰ Joint development of frameworks, approaches and methods by more than one community (e.g. scientists together with institutional organisations, or with NGOs) is the reason for the total percentage being greater than 100%.</footnote>
<figure><img src="image_24.png" /></figure>

the evaluation frameworks are mainly organisational (87.5% of cases), with a smaller percentage dealing with innovative processes (54.2%), products and markets.

Several frameworks, approaches and methods of evaluation do not directly relate to social innovation. However, they were included in the analysis because they deal with analysing or assessing issues related to one of the four domains of interest for SIMRA (i.e. economic, social, environmental, and institutional). The largest group of frameworks, approaches and methods (35 in total) deals with the social domain, which includes a broad literature on social capital and network theory. All those identified have also been fully analysed in the Excel file.

In the economic aspects domain, 18 frameworks, approaches and methods have been identified; all of them have been analysed in detail and provided in the Excel file. In the environmental aspects domain, there are 28; only a sub-set of them have been fully analysed in the Excel file, focusing on those with promise in relation to social innovation issues. Forty-seven frameworks, approaches and methods have been identified on governance and institutional aspects. However, only a few have been included and fully analysed in the Excel file, as in many cases they were considered inappropriate for the operationalization of assessment and evaluation in the field and at local level.

Governance/institutional aspects can be distinguished into two main categories: i) governing related; ii) institutional change/institutional environments. The former is observed in the forms of acting, interacting, steering, and the latter as existing frames, rules and regulations, norms, and administrative systems. These two categories interrelate.

The reasons for a noticeable lack of methods in the literature is related to there being few operative approaches to evaluate governance and institutions (especially at local level), and because the topic addresses many interrelated issues, several of which overlap with other domains. These include social network analysis, stakeholder analysis, dialogue instruments/conferences, participatory methods, GIS mapping, and cost-benefit analysis. Many of these approaches and methods are included in the social or economic domains, as well as in the cross-sectoral, interdisciplinary and general sets of frameworks, approaches and methods. The latter have been analysed as a separate category (39 in total). Finally, a set of 8 frameworks, approaches and methods have been identified and analysed as qualitative methods specifically oriented to policy analysis.

Some of the frameworks, approaches, methods or tools specifically apply to the primary sector: 33.3% refer to agriculture, 21.6% to livestock and 23.4% to forestry. Other sectors identified are food, energy, cultural heritage, business, defence, education, industry, fishery, environment, medicine, mining, policy, and water. Rural developed is referred to by 39.4% of methods, and 29.8% to land use management. These typically adopt a multi-sector and multi-functionality focus.

At least 54.6% of the evaluation frameworks, approaches and methods require an external evaluator, while 24% of them can be used for self-assessment. Counterfactual analysese are used by 27.9%. Almost a third of cases can be applied to any phase (30.8%). Of those proposed for use in a typical final policy cycle phase 41.3% are for evaluation. Of those applied to the initial or intermediate phases of the typical policy cycle, 10.6% were in conception, 8.7% in formulation and 8.7% in implementation. As for the *momentum* (phase) of evaluation when the framework, approach or method can be applied, in 31.1% of the cases it is not specified, while in 30.1% it is for an ex-post evaluation. Only 16.5% are applied for ex-ante evaluations (see Figure 4). A few methods are designed specifically for application at the local scale.
<figure><img src="image_25.png" /></figure>

<figure><img src="image_26.png" /><figcaption>**Figure 4.** Percentage of frameworks and methods intended for different phases of evaluation and different phases in the policy cycle (Source: SIMRA research team based upon the SIMRA database of frameworks and methods).</figcaption></figure>

Few methods adopt specific evaluation criteria, but the main ones are: impact (55%), effectiveness (45%), relevance (41.4%) and efficiency (33.3%). Other minor criteria are: equity, capacity and sustainability. Explicit use of indicators as main judgement references are used by 66.3%. Amongst them, the majority are focused on results, outcomes and impacts rather than process and inputs. The main source of data derives from primary data (76.8%), and indicators are based both on perceptions and objective facts. Only a few indicators respect the quality criteria for being considered SMART (Specific, Measurable, Achievable, Relevant and Timebound), or RACER (Relevant, Accepted, Credible, Easy and Robust) indicators.

Of the methods analysed, 42.3% propose a participatory approach assessment, involving multi-stakeholders as beneficiaries, policy makers, citizens, experts, community representatives, farmers, decision makers, NGOs, companies, suppliers, public operators, households. Specific software is required 33.7%, for example for modelling, Social Network Analysis and GIS mapping.

More than 200 specific tools for assessment and evaluation were collected and described. These tools are used both for mixed (quantitative and qualitative) analysis (27.1%); or focus on the use of quantitative (36.4%) rather than qualitative data (31.3%). The majority (54.2%) require primary data collection, and only 10.3% are based on secondary data (Figure 5).

<figure><img src="image_27.png" /><figcaption>**Figure 5.** Tools identified for qualitative and quantitative analysis from both primary and secondary data collection sources (Source: SIMRA research team based upon the SIMRA database of frameworks and methods).</figcaption></figure>

Each framework, approach, method and tool which was analysed was described in a qualitative way. The strengths and weaknesses were considered in detail, whether or not the frameworks, approaches or methods
<figure><img src="image_28.png" /></figure>

<figure><img src="image_29.png" /></figure>

refer explicitly to social innovation. Consideration was given to their applicability, replicability and adaptability to the assessment of SI in marginalised rural areas, within the SIMRA project. All these details can be found in the list of frameworks, approaches and methods, in the SIMRA Frameworks and Methods database, which includes the specifications of evaluation tools and their characteristics. Full references are also provided. This material is accessible in the SIMRA Resources webpage (www.simra-h2020.eu/resources/).
<figure><img src="image_30.png" /></figure>

# 3. Consultation with SIRA SITT

## 3.1. Consultation Process

In 2016, the SIMRA Consortium invited experts and stakeholders to join the **SIMRA Social Innovation Think Tank (SITT)**. Since its creation, SIMRA has carried out three consultations, two online (July 2016 and July 2017), and one face-to face workshop (Bratislava, Slovakia, October 26 to 28 2016). The approach adopted in WP4 was to use the consultations as a guide to the development of the evaluation methods through a participatory approach. A [summary report](#) was produced of the consultations of the SITT workshop in Bratislava.

This section provides insights and results obtained from the consultation with the SITT in the workshop and the online survey in July 2017 (Box 1). The summary of the first online consultation was provided in [SIMRA Deliverable 4.1](#) (Secco et al., 2016).

**Box 1. Consultation of SITT members on evaluation methods (WP4): meeting in Bratislava (October 2016) and second online consultation (July 2017)**

The objectives of the consultation on evaluation methods were to: 1) identify the most appropriate and useful approaches, frameworks, methods and tools that could be used for assessing SI, based on the experience and expertise of the members of the SITT; 2) understand the types of outputs stakeholders would have liked to obtain from an assessment of SI in marginalised rural areas, in terms of types of data and information, format and level; and 3) describe and classify the frameworks, methods and tools for assessing SI by economic, social, environmental, governance/institutional domains.

The consultation session on evaluation methods was divided into two parts. For the first part of the session, the world café participatory approach was adopted. Four topics were addressed in three rounds of discussions focusing on issues related to approaches, assessment frameworks, methods and tools. There were rich discussions in all four groups. The issues covered were: 1) outcome-oriented versus process-oriented evaluation methods; 2) participatory versus experts-based evaluation methods; 3) primary and secondary data; and 4) qualitative versus quantitative methods. The underlying guiding questions were: *what* information would you like to obtain from an evaluation of social innovation? If you were tasked, *how* would you evaluate social innovation? How would you measure SI? What would you expect from those methods? What are their pros and cons? During the second part of the session, participants were encouraged to brainstorm potential indicators to evaluate SI.

The results were based on the joint work of four facilitators and four rapporteurs. The results are derived from the posters (see Figure 7) and the notes taken by the group rapporteurs. Each round of discussions was recorded with permission of SITT members and the recording used to complete the rapporteur notes.

The second online consultation (July 2017) was led by WP2 by means of SurveyMonkey. For WP4, the SITT members were asked for their reflections on a fifth dichotomy: 5) the use of a descriptive versus a normative approach in assessing SI. The specific aim was to understand the opinions of members of the SITT, and suggestions on when it would be best and/or feasible to adopt either or both approaches. The questions comprised a set of three main and sub-questions. Replies were received from 18 members of the SITT.

The following sections contain the results of the consultation process. Details are provided because of the relevance of the results in guiding the development of the evaluation methods and their inclusion in the preliminary list of questions proposed in Annex 1. These results are presented by topic, as discussed in the consultation workshop held in Bratislava, and the online consultation in July 2017. These include tables in which are summarised the strengths and weakness as identified by SITT members, a preliminary list of suggested indicators, and a summary of how suggestions were used in development of the evaluation method.
<figure><img src="image_31.png" /></figure>

## 3.2. Topic A: Outcome-oriented Versus Process-oriented Evaluation

The discussion on process-oriented versus outcome-oriented evaluation<fnref n="11" /> revealed the opinions of stakeholder on the role of two different types of evaluation in relation to SI, outcome-oriented and process-oriented evaluations (Table 1, Figure 6). Some SITT members highlighted how the evaluation depends on whether SI is defined as a process or as a result, and on the length of the project or activity. Other members of the SITT suggested that both process and result requires to be evaluated because of the relevance of the importance of the relationship between process and outcome. In all the cases, a process-oriented evaluation was seen as a learning process.

Stakeholders highlighted the importance of measuring the tangible (hard) and intangible (soft) elements of SI, and identifying factors that contributed to success or failure in the SI process. Ex-ante evaluation was viewed as helping to reflect on strategies for selecting case studies for the evaluation of SI. Different starting points for SI (e.g. the degree of social involvement is different in different places), and the continuing development of the SI, was seen as a challenge to analysing results and outcomes. Different national contexts mean that what is SI in one country may be standard practice in another. Therefore, the social improvement to be measured as an outcome depended upon the context.

For some SITT members, there were different opinions on the measurement of social improvement, outcomes are the starting point for identifying the elements that led to failure or success; while the opinion of other members of the SITT was that evaluation should start with the situation (context analysis), then the assessment of process and outcomes. In both outcome and process-oriented evaluations, the use of common indicators was identified as important for comparability. In conclusion, it was agreed that a definition of SI was needed to enable the identification of what to measure and how that should be done (See Box 2 for the detailed results of the discussion).

<table><caption>**Table 1.** Summary of responses of SITT members on Topic A: outcome-oriented versus process-oriented data.</caption><thead><tr><td>Outcome-based evaluation</td><td>Process-based evaluation</td></tr></thead><tbody><tr><td><ul><li>Fulfills political expectations.</li><li>A limited set of indicators may be enough to measure the concrete results of a project, but may not be enough to identify it as a SI or not.</li></ul></td><td><ul><li>Process is an important element in SI</li><li>Process-oriented evaluation is a learning process</li><li>Evaluation needs to rely on participatory approaches</li><li>It needs to focus on the informality of process</li><li>It needs to address motivation of participants</li></ul></td></tr></tbody></table>

Source: SITT workshop, Bratislava, 28th October 2016 (Topic A).

<footnote n="11">¹¹ Process-oriented evaluation is defined by the OECD Glossary of Key Terms in Evaluation and Results based Management as: "Evaluation of the internal dynamics of implementing organisations, their policy instruments, their service delivery mechanisms, their management practices, and the linkages among these. Related term: formative evaluation" (OECD, 2010: 30) Outcome-oriented evaluation is defined as: "A study conducted at the end of an intervention (or a phase of that intervention) to determine the extent to which anticipated outcomes were produced. Summative evaluation is intended to provide information about the worth of the programme. Related term: impact evaluation or summative evaluation" (OECD, 2010: 35, see: www.oecd.org/dac/evaluation/2754804.pdf)</footnote>
<figure><img src="image_32.png" /></figure>

<figure><img src="image_33.png" /><figcaption>**Figure 6.** Results of the discussion on outcome-based versus process-based evaluation (SITT workshop, Bratislava, October 2016).</figcaption></figure>

## Box 2. The detailed contents of discussions on Topic A

During the 1st round of discussions, stakeholders expressed the need to clarify the meaning of evaluation, including whether what matters for evaluation is to understand how the process is developed, and how the nature of the outcome is related to SI. According to other members of the SITT, there are differences between results and outcomes. Consequently, for evaluation purposes everything depends on whether or not SI is defined as a process or as a result, and on which types of SI SIMRA is considering. More specifically there are two levels of results: (1) the SI itself, and (2) the impact of the SI on the elements addressed, such as the improvement of health in a community.

In relation to the last point, and following discussions in meetings of the SITT workshop with SIMRA Working Packages 2, 3 and 5, stakeholders highlighted the importance of measuring the tangible and the intangible elements of SI. Specifically, the recommendation was for the work on evaluation methods (WP4) to focus on different elements in relation to process versus outcome-oriented evaluations.

In relation to process-oriented evaluation, the objective is to improve processes. Therefore, the evaluation of SI in process-oriented evaluation requires focusing on the intangible features of development (e.g. SI processes comprising participation and interactions between different types of actors, with proactive or reactive attitudes), and on failures. In cases where results have not been achieved, reviewing the process may enable the identification of the factors that led to failure.

In relation to outcome oriented evaluation, it is important to have clarity from the outset about the results being sought. If the evaluation is of a specific project, then the results matter. If the results are good, then process matters. This statement was discussed at length reflecting the different visions of the stakeholders. This discussion highlighted the importance of process oriented approaches to the evaluation of successful projects.

During the 2nd round of discussions, a sharp difference in visions emerged between the process (the 'how') of the SI and the outcome (the 'how much') of the SI. The discussion about "how" highlighted that outcome-
<figure><img src="image_34.png" /></figure>

oriented evaluation was important, should be adopted first, and should include the evaluation of process: "first you look at outcomes and then, depending on the results (positive/negative) we look to the process". This view underlined the challenge for evaluators in changing the frames of evaluations from outcome-oriented to process-oriented, and raised questions such as which parameters are the easiest to obtain?

Discussion considered whether these indicators related to process or outcomes. Indicators relating to process are generally soft indicators, whilst indicators relating to outcomes are hard indicators. Hard outcomes are easier to evaluate and transform or translate into numerical estimates: "Political expectations are always outcome oriented so you have to get them done".

The second expressed how the "difference between SI and other innovation is related to the process". According to this view, in SI the important element is the process, so process-oriented evaluation is the most important, and should be able to address the 'informality of processes' in SI. To achieve this, it would be important to deal with unrealistic expectations, and to find ways to motivate actors to participate in evaluation processes. Stakeholders also highlighted that ways for dealing with a process-oriented evaluation could be different. They noted that with greater levels of involvement in evaluation, the more participatory the process could be, and the more opportunity of establishing a successful learning process through evaluation. A further item highlighted was the importance of the **relationship between process and outcome** and the need to evaluate both. For example, in the LEADER Programme the focus should be on both the result and whether it was more effective than a top-down approach.

During the 3rd round of discussions, it was stated that while policy makers like outcome-oriented, process-oriented are equally important, and more valuable to other actors. As noted in the 2nd round of discussions, an opinion expressed was that consideration should not be of process-oriented versus outcome-oriented, but both. For example, in the case of cooperatives, the results approach may indicate that they are unsuccessful because of networking failures, and yet a process-oriented evaluation captures their role.

Stakeholders highlighted that the choice of process versus outcome-oriented evaluation depends upon the objective of the evaluation. While the objective of outcome-oriented evaluation is clear, that of process-oriented may not be. The process may not be interesting in itself, but it is in relation to the results. Process ends with outputs, whilst the evaluation of outcomes draws on baseline indicators to evaluate impacts. However, since SI is not well defined, it is difficult to capture all the issues. Therefore, it was concluded that there are different levels of evaluation, which involve different stakeholders, with interests in different aspects of addressing the complexity of the process.

Finally, stakeholders highlighted how the focus is generally about socio-economic evaluation. However, it is also important to measure environmental impacts of SI if we understand SI as a sustainability issue, even if they are not defining SI. This led to a final observation about the added value of SI of measuring its environmental impacts.

Source: SITT workshop, Bratislava, 28th October 2016 (Topic A) (Facilitator: Elena Pisani, UNIPD; rapporteur: Diana Valero, Perth College).

### 3.3. Topic B: Participatory Versus experts-based Evaluation

The discussions in roundtable on participatory versus experts-based evaluations highlighted their strengths and weaknesses, and the complementarity of the approaches (Table 2, Figure 7). Expert-based approaches were considered to have more credibility associated with the results, while examples supported the use of participatory processes to legitimate, increase ownership and ultimately adoption and implementation. For example, participatory approaches were seen as crucial for assessing the 'feeling' or intangible aspects of those involved in SI (e.g. in the evaluation of LEADER) through indicators of trust, involvement of the community in innovative approaches, connection to other actors, and the level of acceptance and exchange of new practices. One of the perceived risks of participatory approaches was the loss of interest of participants (e.g. public).
<figure><img src="image_35.png" /></figure>

SITT members also highlighted the importance of **evaluating the impacts of social innovation through actual changes in policy**. The group concluded that it was important to understand "how you follow a story", and how you look at long-term impacts of SI (See Box 3 for the detailed contents of the discussion on Topic B).

<table><caption>**Table 2.** Summary of responses of SITT members on Topic B: participatory versus expert-based data.</caption><thead><tr><td>Expert-based</td><td>Participatory-based</td></tr></thead><tbody><tr><td><ul><li>Avoid bias, strive for objectivity</li><li>More experience in evaluation</li><li>Credibility of report results</li><li>Creates recommendations derived from participatory approaches</li><li>Effective for quantitative evaluations</li><li>Ensure validity of feedback to qualitative methods</li><li>Increase confidentiality of personal opinions</li><li>Possibility to generalise</li></ul></td><td><ul><li>More effective from the perspective of end-users</li><li>Can validate results</li><li>Can be designed by experts</li><li>Ensures legitimacy</li><li>Enables capturing of process</li><li>Ensures ownership of results</li><li>Enables bottom-up mobilisation/lobbying</li><li>Allows scaling-up of results</li></ul></td></tr></tbody></table>

Source: SITT workshop, Bratislava, 28th October 2016 (Topic B).

<figure><img src="image_36.png" /><figcaption>**Figure 7.** Findings of the expert-based versus participatory-based evaluation approaches (SITT workshop, Bratislava, October 2016).</figcaption></figure>
<figure><img src="image_37.png" /></figure>

Box 3. The detailed contents of discussions on Topic B

During the 1st round of discussion, the group considered the advantages and disadvantages of adopting expert and/or participatory based approaches for the evaluation of social innovation. On the one hand, expert-based evaluation was seen as avoiding bias, being more objective, lending credibility to the evaluation and to the recommendations that were proposed. It was also seen as increasing confidence in participatory processes, by setting up the methodology, analysis and reporting. On the other hand, participatory approaches were seen as a way to include different stakeholders ("*the vision of the end-user is very important*"), validate surveys (which could be expert-based), and provide legitimacy to the evaluation, although problems of bias were raised. It was argued that participation increases the ownership of the results obtained for the evaluation. For example, stakeholders suggested that the evaluation could be expert-based but the process participatory.

Stakeholders agreed that both systems could be integrated, without being mutually exclusive: "you can have *the best of both worlds, by asking the right questions*". For example, a climate change adaptation strategy that could have been developed over a short timescale by experts, instead was developed over a two-year period to ensure that stakeholders were involved at each stage of the process. This secured the implementation of the strategy. Better implementation (through the involvement of many stakeholders), as well as the division of responsibility, were identified as consistent with SI.

A second example discussed was that of food security. One stakeholder suggested that it was necessary to involve civil society in the evaluation of the flow of information required in securing access to food. In this example, the involvement of civil society was consistent with the definition of SI.

A third example was the provision of community health services: experts are needed to undertake a situation analysis, develop a strategy with goals and objectives, and to identify quantifiable indicators. However, the SI is about adoption: how you convince people to adopt this plan. A similar example was provided for agricultural programmes where farmers were encouraged to provide their opinions in a process leading to a plan they could adopt.

During the 2nd round of discussion, expert-based evaluation was identified as important in qualitative methods because it ensured a more objective perspective, and a better basis for generalising the results. Participatory interaction was seen as crucial for assessing the 'feeling' or intangible aspects of those who were involved in social innovation. The example of public-private partnerships in LEADER was identified as difficult to evaluate, and it was agreed that simple proportion (between public and private actors) was not enough to capture more intangible features such as power dynamics and quality of the relationships. One suggestion was to measure: 1) trust, by analysing before and after situations, 2) involvement of the community, for example in cooperatives where people with disabilities work, 3) connection to other actors, and 4) the level of acceptance and exchange of best practices. It was agreed that contracts were not enough to show the strength of a network. Instead, a plan for joint activities, both formal and informal, was needed. Finally, one of the limitations of the participatory approaches was the loss of participant interest: "*if you do not keep people involved during a participatory process, and follow up on their suggestions, then they will become less inclined to collaborate*".

During the 3rd round of discussion, the group focused on the importance of assessing SI through the process of social mobilisation, and to evaluate the impacts of social innovation through actual changes in policy. The first example referred to focused on measuring the process of developing a women's declaration on climate change within international negotiations. This was seen as an example of bringing people together, bottom-up. The second example concerned social farming, which illustrated the importance that experts (law-makers) can play in the process of achieving specific results by helping to tell the story of those working at the community level. In this example, the result was changes in legislation supporting public procurement of social farming products. The group concluded that it was important to understand "*how you follow a story*", and suggested reflecting on measuring social return on investment of these efforts, and long-term impacts of SI.

Source: SITT workshop, Bratislava, 28th October 2016 (Topic B) (Facilitator: Catie Burlando, UNIPD; rapporteur: Martin Špaček, IFE SAS).
<figure><img src="image_38.png" /></figure>

## 3.4. Topic C: Primary Versus Secondary Data

The discussions highlighted the lack of secondary data on social innovation, the importance of primary data to identify the specific context of social innovation, and the need to use them both complementarily (Table 3). Stakeholders suggested the following methods for data collection: 1) focus groups and participatory methods; 2) semi-structured interviews; 3) longitudinal surveys for studying pre- and post- conditions; and 4) stakeholder analysis, emphasising the importance of gathering soft data on interactions, feelings and activities. These types of data could be collected internally or externally, but should be seen as a two-way communication process. Key points raised were that: 1) data should be comparable; 2) triangulation should be used to verify the quality of data; and 3) data should be made publicly available.

Secondary data are considered relevant and useful for analysis at a large scale (e.g. national level), whereas primary data are considered necessary as social innovation is locally specific. Moreover, “soft data”, such as perception, vision, expectation, preferences of stakeholders and involved actors, are considered particularly relevant for social innovation evaluation. Data should be easy to use and interpret, available, recent and reliable. SITT members suggested the use of participatory approaches to gather soft data on interactions, feelings and activities but urged reflection to avoid the risks of subjective biases and negligent answers.

<table><caption>**Table 3.** Summary of responses of SITT members on Topic C: primary versus secondary data.</caption><thead><tr><td>Secondary data</td><td>Primary data</td></tr></thead><tbody><tr><td><ul><li>Data on economic and environmental impacts is difficult to access and interpret with respect to SI (e.g. green energy)</li><li>Data on enterprise innovation is available in some countries but without an evaluation of impacts (e.g. Czech Republic – Innovation Survey)</li><li>Cheaper and processed</li><li>Support analysis at large scale [i.e. local] (but need to define first at what scale the SI happens)</li><li>Not available for the evaluation of SI</li><li>No data on activities and social impacts of SI</li><li>Reflection: does secondary data actually help SI evaluation?</li></ul></td><td><ul><li>Primary data are necessary! SI is locally specific and needs primary data</li><li>Local data should be of broad use, for various purposes and relevant for policy and research</li><li>The focus should be on processes</li><li>“Soft data” is key: perceptions, visions, expectations, preferences, gaps</li><li>Analyse impact on different groups</li><li>Analyse intermediate impacts</li><li>Self-assessment</li><li>Data should be directly usable (easy to interpret), available, easy to use, recently collected, reliable</li><li>Enable a two-way communication process</li><li>Collected internally or externally</li><li>Requires time, money, and generally not supported by institutions</li><li>Careful with data transfer and generalisations</li><li>Potential for subjectivity in data collection</li></ul></td></tr></tbody></table>

Source: SITT workshop, Bratislava, 28th October 2016 (Topic C) (Facilitator: Riccardo Da Re, UNIPD; rapporteur: Andrej Udovč, IFE SAS).

## 3.5. Topic D: Qualitative Versus Quantitative Methods

Members of the SITT agreed that evaluation methods and tools should be tailored to: 1) the needs and purpose of the evaluation; 2) the type of project being evaluated; and 3) the object of measurement. Qualitative and quantitative approaches were identified as **complementary and suitable for use in the context of a specific evaluation**. Used together that can be used in triangulation, yielding different types of results which support each other by providing in-depth information on the process and results of a project. It was agreed that the objectives and framework for evaluation needs to guide the approaches and methods used, and the use of when to use internal and external evaluations. Quantitative methods are not always popular amongst practitioners but are considered useful (or fundamental) when lobbying policy makers and funders. Qualitative methods provide in-depth information, providing insight on what different groups of stakeholders
<figure><img src="image_39.png" /></figure>

obtain from a project or activity (e.g. SI), and their perceptions of the benefits. Qualitative approaches were considered necessary when describing results (see Table 4 and Table 5).

<table><caption>**Table 4.** Characteristics of qualitative and quantitative methods identified by SITT members.</caption><thead><tr><th></th><th>Qualitative</th><th>Quantitative</th></tr></thead><tbody><tr><th>Focus</th><td>Process<br>Soft skills<br>People's perception of the project<br>Quality of the project<br>Quality of the network: e.g. frequency of exchanges</td><td>Outcomes/Results</td></tr><tr><th>Stage of the project<sup>12</sup></th><td>Beginning<br>During the project</td><td>End of the project</td></tr><tr><th>Methods/tools</th><td>Semi-structured interviews<br>Surveys<br>Personal paths</td><td>Statistical models (e.g. Logit models)</td></tr></tbody></table>

Source: SITT workshop, Bratislava, 28th October 2016 (Topic D).

The discussion focused on the difference between quantitative methods and quantitative results (Figure 8). SITT members had different perspectives. Some SITT members reported that there are techniques for transforming data collected with qualitative-based methods (e.g. face-to-face interviews) into quantitative data (e.g. numbers or indexes). These techniques could help inform policy makers with synthetic reports based on figures, graphs and numbers that are easily understood and accessible for decision-making. The evaluation could be based on a multi-criteria analysis involving combinations of indicators. Such indicators should be as synthetic as possible. It was noted that for small or new projects, it might be difficult to measure long-term impacts.

Finally, the importance of both an internal and external evaluation was raised, including an evaluation carried out by local, internal evaluators and an evaluation carried out by national or international experts who do not know the local conditions and the needs or aspirations of the local people who initiated the social innovation. Perception of actors directly involved in the initiative might significantly differ from those of external evaluators. A suggestion made was to identify and evaluate the level of satisfaction of those involved in the initiative SI. Box 4 summarises the contents of the discussion on Topic D.

<table><caption>**Table 5.** Summary of responses of SITT members on Topic D: qualitative versus quantitative methods.</caption><thead><tr><th>Qualitative methods</th><th>Quantitative methods</th></tr></thead><tbody><tr><td><ul><li>More suited to measure SI because it explains how and why the project was successful</li><li>In-depth information about a case study and the process</li><li>Used to assess short-term impacts of projects</li><li>Used to collect information that can be transformed into quantitative data</li></ul><p>Drawbacks:</p><ul><li>Subjective</li><li>Training of evaluators can be lengthy</li><li>Requiring further resources</li></ul></td><td><ul><li>Easier to communicate to policy-makers</li><li>Easier to measure</li><li>Can be used to assess longer-term impacts of project</li><li>Quantitative data are important for policy makers</li></ul><p>Drawbacks:</p><ul><li>Data analysis can be subjective</li></ul></td></tr></tbody></table>

Source: SITT consultation, Topic D, Bratislava, 28th October 2016.

<footnote n="12">¹² This sequence was the most commonly cited during the three rounds but the opposite sequence was also mentioned as happening and relevant depending on the type of project.</footnote>
<figure><img src="image_40.png" /></figure>

<figure><img src="image_41.png" /><figcaption>**Figure 8.** Results of the discussion on qualitative versus quantitative methods.</figcaption></figure>

**Box 4. The detailed contents of discussions on Topic D**

During the 1st round of discussions, several issues were identified and are listed below.

1. "I don't like quantitative methods but they are important".

2. Stakeholders raised the importance of measuring the number of different types of networks and the diversity of the stakeholders who are involved in that network as well as the quality.

3. Benchmarking of the impacts of SI and how they influence other aspects of the project. For this, there is a need for synthetic indicators, and a combination of indicators.

4. Stakeholders expressed support for seeing complementarity between qualitative and quantitative methods rather than qualitative versus quantitative.

5. "Personal paths" or life stories could be used to follow an individual's personal history.

6. "Silence" can be an indicator of the level of a person's involvement or attitude towards a project (e.g. either a lack of engagement, or tacit approval).

7. A "Welcome your remarks" box may be a useful way to obtain feedback from beneficiaries.

8. For small or new projects, analysis could focus on the short-term impacts of the project. These may be qualitative, with a significant challenging to obtain quantitative results, and costly to evaluate.

Satisfaction with the project, self-confidence, self-esteem and capacity building are characteristics that can develop in people and are important to measure. For example, for people with a disability, regular meetings of the team over 2 years led to important improvements in their skills. This points to the importance of monitoring and an integrative assessment to analyse workers, funders, users, feedback from the local community, and supporting communications with the outside world.
<figure><img src="image_42.png" /></figure>

During the 2nd round of discussions, some issues were reiterated and new ones raised by SITT members. The suggestions are listed below.

1. Individual approaches are insufficient. A mix of approaches is needed.

2. Results depend upon the evaluation being undertaken: an evaluation framework and design for a system analysis is created initially, preferably in as part of a participatory process.

3. Discussion considered what constituted qualitative and quantitative evaluations. Qualitative data is about the description of reality in the participant's own words. It was described as similar to 'dummy variables in econometrics.' Quantitative evaluation was summarised as being about numerical information: "When I use those numbers in a system approach then it is quantitative." Examples of methods discussed at this stage included multi-criteria assessments, which can take both qualitative and quantitative methods into account, illustrating that qualitative and quantitative methods are not exclusive, they can complement each other, and can be used to triangulate results. Methods to transform qualitative information and/or conversation into numerical representation and scales are available. Qualitative results can be compared to quantitative ones. For example, in a project on new market opportunities for farmers, a quantitative measure might be the share of products sold through a shop, or the increase in their income. However, this does not explain anything about the underlying process. For that, a qualitative method may be a valuable accompaniment. To measure the degree of satisfaction on a scale is a qualitative process. Thus, qualitative aspects of the SI require using qualitative methods. SITT members noted that policy makers are not interested in qualitative methods, preferring quantitative information on subjects such as the amount spent (e.g. econometrics to measure marginal effects or a system approach). So, where possible, qualitative data are translated into quantitative data. Discussion considered other approaches, such as system analysis noting that for this it is necessary to have steps that can be measured, and inferential statistics. Methods exist to measure complex things, however as all SI cases are context specific, it is better to add a qualitative description to the results.

During the 3rd round of discussions new insights on qualitative and quantitative methods were provided by SITT members. Although qualitative approaches are about telling stories, it is possible to extract numerical data out of stories, and to tell stories out of numbers. However, the capability of researchers to do so may be limited by experiences by discipline or familiarity with certain methods.

Discussion identified the importance to funder of quantitative outcomes. Throughout a process or project, qualitative methods may be used, and then quantitative methods to measure results. The type of quantitative indicators or measures can be the "number of people reached by the programme". For example, in a screening programme for cancer research, the cure is in getting rid of the disease, while the palliative measure is relieving pain without dealing with the cause. The effectiveness of the palliative measure can be assessed on a qualitative scale. Discussion about the type or quality of quantitative data considered how meaningful it can be. i.e. the number of people who are involved does not tell you about the SI process, but rather whether or not and how much they are involved or active.

1. Qualitative approaches provide rich and deep information, but which have overheads of the close involvement of the evaluator, requiring time and training.

2. Consideration was also given to changes in the SI process, and the need for quantitative methods to provide some final evaluation.

3. Both types of methods have positive and negative aspects. Quantitative information helps to communicate, and can be objective. For example, the pulse of a patient may be 70 beats per minute (a quantitative measure). But to what extent can the pulse be considered to be weak? This can be very subjective.

The disadvantages of qualitative methods are their subjectivity. For example, the accuracy of impressions of what is happening rather what is really happening will largely depend upon the respondent. A key question for qualitative approaches is whether a story can provide the information required for an evaluation? An internal evaluation of the group may help to understand how people involved in the process feel. An external evaluation might come to a different conclusion. Obtaining results is important, but it is the quality of the
<figure><img src="image_43.png" /></figure>

process that enables good results to be obtained. It was reiterated that SI is about process and it would be wrong to focus on results alone. Social network analysis was identified as one approach to the measurement of networks and strength of links within it, both before and after the project. In an evaluation this would mean studying the situation before the programme started to create a baseline or situational analysis, against which the results could be compared. It was suggested that there may also be a need for a baseline for process-oriented evaluations. Learning from previous experience could also provide something of a baseline. Mixed methods approaches were discussed, stressing the benefits of complementary methods. However, it was recognised that one type of method might be more important or useful than another in any given situation. Sometimes, quantitative approaches might be easier and cheaper to conduct than qualitative ones.

Source: SITT workshop, Bratislava, 28th October 2016 (Topic D) (Facilitator: Laura Secco, UNIPD; rapporteur: Carla Barlagne, HUT).

### 3.6. Indicators Suggested by SITT Members

Members of the SITT were asked to suggest possible indicators during the workshop in Bratislava, as well as on the July 2017 online consultation. Overarching indicators suggested by members of the SITT included "quality of life", "happiness/satisfaction", "well-being", or the role of learning. Some indicators specifically related to networks in terms of the quality of relationships, regular activity, size, and/or structure. Consultation feedback included recommendations that focus attention on policy issues. Some of these indicators (general and specific) have been considered and included in the list of indicators developed for the set of methods. The full set of indicators proposed is listed in Table 6.

<table><caption>**Table 6.** Candidate indicators for the evaluation of SI and its impacts, suggested by SITT members.</caption><thead><tr><td>Indicators</td></tr></thead><tbody><tr><td>Happiness as a complex indicator reflecting all four dimensions</td></tr><tr><td>Life satisfaction/happiness</td></tr><tr><td>Impacts on the quality of life</td></tr><tr><td>Quality of products before and after</td></tr><tr><td>Indicator of well-being with respect to economy, goal achievement, goal met, and overall consensus on leadership</td></tr><tr><td>What has been learned?</td></tr><tr><td>Increase capacity</td></tr><tr><td>Number of network members and change over time to measure the effects of policy interventions</td></tr><tr><td>Quality of linkages between network members to measure the sustainability of the network</td></tr><tr><td>More active/regular networks</td></tr><tr><td>Measure the connectedness of the network through an index of quality and quantity of linkages between network members</td></tr><tr><td>Marginalised people connected with "others"</td></tr><tr><td>Number of informal networks over the long-term</td></tr><tr><td>Changes in policy</td></tr><tr><td>Policy responses</td></tr><tr><td>New measures and regulation of SI</td></tr><tr><td>Number of different actors involved</td></tr><tr><td>Number of sectors involved</td></tr><tr><td>Number of partners involved</td></tr><tr><td>Diversity of enterprises and number of networks</td></tr><tr><td>Number of self-employed enterprises</td></tr><tr><td>Implementing healthcare plan in rural area</td></tr><tr><td>Number of people (inhabitants) in rural areas</td></tr><tr><td>Social return on investment</td></tr><tr><td>Programmes and communication on TV channels on marginalised rural areas</td></tr></tbody></table>
<figure><img src="image_44.png" /></figure>

<table><tr><td>Increase animal production</td></tr><tr><td>Quality of a service or a project</td></tr><tr><td>Perceptions of project and benefits gained</td></tr><tr><td>Soft skills enhancement: self-confidence, self-esteem, capacity building</td></tr><tr><td>Intensity of the network: how often people interact and exchange within the network (e.g. number of ties in a network and strengths of those ties before and after a project; frequency of exchange)</td></tr><tr><td>People's attitude in meetings (e.g. "silence" of a person might indicate dissatisfaction)</td></tr><tr><td>Successes as well as failure and reasons that can explain it</td></tr><tr><td>Meaningfulness of a programme</td></tr><tr><td>People's involvement in a programme (e.g. number of people reached by the programme, level of involvement and activity)</td></tr><tr><td>Internal and external evaluation</td></tr><tr><td>Short-term impact of a project on new/innovative projects</td></tr><tr><td>Number of different types of networks and diversity of the stakeholders involved in the network (i.e. quantity and quality of networks)</td></tr><tr><td>Number of new jobs</td></tr><tr><td>Various impacts of SI</td></tr><tr><td>External communication (e.g., social media, press releases)</td></tr><tr><td>Income of the household</td></tr></table>

Source: SITT consultation, Bratislava, 28th October 2016 and online consultation in July 2017.

## 3.7. Descriptive Approach Versus Normative Approach for Assessing SI

The second online consultation with members of the SITT was carried out in July 2017 under the coordination of Work Package 2. An explanation of the two terms, descriptive and normative, was provided. Descriptive means "describing or classifying social innovation in an objective, non-judgemental way (e.g. "four people were the initial innovators", "the initiative was publicly announced on the website"); normative means establishing or deriving from a standard or a pre-defined "good" required for social innovation (e.g. "four people are the right number for starting social innovation because there is a need for a minimum number", "the initiative was transparent because the innovators published an announcement on the website").

The introduction explained that while it is accepted that we can describe social innovation (e.g. how it is structured, who are the actors involved, how it works, etc.), there is an on-going debate on adopting normative standards for evaluating whether a social innovation initiative is "good" or not, in its various phases. Responses to the questions are provided in Box 5, with the wording presented as submitted.
<figure><img src="image_45.png" /></figure>

**Box 5.** Summary of the second online stakeholder consultation process (July and August 2017)

In July and August 2017, a second consultation was carried out with members of the SITT. Eighteen members of the SITT responded. Four questions were included for the development of the evaluation method, the responses to which follow (wording unedited):

1. If you were tasked to evaluate social innovation, would you use a descriptive approach with respect to the process phase of social innovation (ideation and development of the initiative)?

<figure><img src="image_46.png" /></figure>

The descriptive indicators suggested were:

* Number of actors initiating the process

* Number of actors participating in the process, representatives of different groups (this was shared by most respondents)

* Motivations (i.e. expected benefits)

* Roles and contributions towards solving the societal issue at hand

* Resources (i.e. human, financial, technological) in support of the process development

* Existence of a process coordinating body

* Institutions (i.e. rules of the game) that support the process and its sustainability

* Number of social groups engaged

* Number of local community members (population) who benefit from SI

* Number of innovative products that the SI process brought

* Number of innovative solutions (examples of good practice) that can be used by other communities, municipalities, regions

* Number of new non-profit supporting SI established, or initiatives or networks created

* Number of new jobs for local residents created by SI

* Indicators about dissemination of information (e.g. information about SI, in the local media, seminars, workshops...)

* Reaching goals and targets

The sources of information suggested were: primary data from questionnaires and surveys, and regular monitoring of progress of the SI, local offices, public speeches, social media, institutional communication, statistical information and research studies. One stakeholder further summarized the **critical factors of success of social innovation** in the following list:

* Setting clear and ambitious targets early in the process

* Involving the widest possible range of stakeholders, including trained and professional experts

* Securing a division of tasks and responsibilities among all involved actors and ensuring mutual trust among them

* Setting realistic expectations and time frames, including benchmarks

* Linking the ongoing project, process or initiative to a broader framework
* Designing and developing a wide range of scenarios and proposals for alternative solutions
<figure><img src="image_47.png" /></figure>

* Presenting the project or process outputs in terms of benefits to the public

* Ensuring regular meetings and discussion forums for both addressing key tasks and resolving possible conflicts

* Using the available means to provide information to the local media

* Gaining support and interest in the proposed solutions from local self-government, state administration, opinion leaders, entrepreneurs, NGOs, and both the local community and the wider public

* Securing a partnership between the lead partner responsible for the project's implementation (activities) and project initiators

* Clearly formulating the requirements for securing the sustainability of the project's solution, including measurable parameters (indicators).

2. If you were tasked to evaluate social innovation, would you use the descriptive approach with respect to the results phase of social innovation (outcomes, impacts and scaling up)?

<figure><img src="image_48.png" /></figure>

In the second questions, stakeholders clearly expressed support for using a descriptive approach in relation to the results of the social innovation. The indicators proposed for this phase include:

* Positive outcomes that the SI is likely to result in

* Benefits for the participating agents and the community the SI targets

* Institutions and resources to facilitate the transfer of the outcomes across communities

* Employment created

* Number of visitors

* Reduction in costs

* Population change

* Number of individuals/groups profiting from SI (improved social status, life, well-being, and increased income/economical profit)

* Number of activities derived by the action people involved after the starting of the process and new actions for developing the results

* Number and type of stakeholders involved

* Investment made (expenditure)

* % of rural population covered by the social innovation

* % of rural population benefiting from the social innovation

* Consensus on the results

* Inclusiveness

One respondent noted that descriptive indicators can also be perceived as input and output indicators. However, it is difficult to provide examples of descriptive indicators as they depend on the type of project/social innovation (and its objectives) and context of its implementation. Another stakeholder suggested open discussion with the community.
<figure><img src="image_49.png" /></figure>

3. A descriptive evaluation is usually feasible. Is it also possible and advisable to evaluate social innovation using normative standards of "good" social innovation?

<figure><img src="image_50.png" /></figure>

4. If you were tasked to evaluate social innovation by using a normative approach, what standards, thresholds, requirements, etc. would you adopt for judging whether SI is "good" or not? Please provide us guidance and suggestions (e.g. links to documents, specific examples).

* I would assess a SI as "successful", "progressive" or "unsuccessful" with respect to its ultimate goal, which first needs to be quantified (by using indicators). I would not use the "good" or "not good" classification as it does not provide any direction for policy action.

* It depends on the nature of SI: if it is focused /has potential to involve/influences whole community or certain groups. Involvement/influence of 50% of potential participants seems to be a very good threshold, but I would prefer relative measures - comparisons in %.

* It is very difficult to exclude particular standards, requirements, because it depends on a case by case basis. For me it would be more important to use the group of standards (for example environmental standards, economic standards...). The same would hold with regards to the requirements.

* We may use criteria and indicators to assess the SI. Defining an initiative as good or bad is tricky and should be based among others on evidences from surveys/analysis of success and failures of SI initiatives. Score cards or tracking tools may be used in evaluation, especially at project level.

* The standards, thresholds, requirements will probably rely upon on the particular types of social innovation, territories, or target groups. However, there are standards as in any evaluation, e.g. acknowledge the context, provide useful information, efficient (relation between costs and utility), reliable, objective, need to involve all the stakeholders.

* Well defined social need; Innovative approach; Sustainability of the results.

* I would refuse to use them since I doubt their usefulness in valuing.

* Environmental sustainability, justice and equity.

## 3.8. How SITT Member Suggestions are Integrated into SIMRA Methods

Early engagement with SITT members provided suggestions that were included in the development of the evaluation approach, method, and the preliminary list of questions (Annex 1) for the analysis of pilot case studies (SIMRA WP5) and selected policy processes (SIMRA WP6). These include the following important guidelines.

The evaluation of a social innovation initiative must consider both process and outcomes, and be based on learning the dynamics that foster or hinder social innovation. The evaluation framework accounts for the dynamics that occur at different temporal phases of the social innovation (from trigger to outcomes), from different perspectives. The evaluation questionnaire reconstructs the storyline using a calendar of events, the dynamics of the process using through tools of social network analysis and qualitative assessment of
<figure><img src="image_51.png" /></figure>

motivation/competence/capacities/actions, and the impacts and effects of the social innovation using indicators of the social, environmental, economic and institutional domains. The actors are: (1) experts/key actors; (2) the clique; (3) the reconfiguring network; (4) project partners; and, (5) beneficiaries. The elements of SI captured in the questionnaire are both tangible (e.g. the number and role of participants in the clique and in the reconfiguring network; the number of beneficiaries reached), and intangible (e.g. motivation, attitude, enthusiasm, voice).

While participatory evaluation of social innovation can increase legitimacy, sense of ownership and adoption of the results, expert-based evaluation can increase the validity and credibility of results for policy-makers. The evaluation questionnaire proposed has been designed for combining self- or participatory-based evaluation with expert-based evaluations. By targeting different actors, the aim of the questionnaire seeks the inputs of different perspectives to unravel 'how you follow the story'.

The dearth of secondary data on social innovation processes calls for greater emphasis on the collection of primary data that is comparable, can be triangulated, is reliable and publicly available. Results of a survey using an evaluation questionnaire cannot always be made publically accessible due to the loss of anonymity in some aspects of methodologies. However, the structured questionnaire, with both open and closed questions, enables the evaluator to obtain and report on data that are comparable, reliable and that can be triangulated across the different target respondents.

Qualitative and quantitative methods are needed to reach policy-makers and funders, as well as support understanding of perception, motivation, quality of the network, and organisational capacity. The evaluation questionnaire combines qualitative and quantitative questions. The variables obtained can be converted into individual indicators, and aggregated into composite indicators and indexes. One stakeholder noted: "Qualitative is about telling stories, but you can take numbers out of stories and tell stories out of figures".

A descriptive approach can guide the evaluation not only of process, but also of results (outcomes and impacts), whereas the application of a normative approach would depend on the targets and objectives defined. The evaluation questionnaire provides space for a description of the evolution and organisation of the SI. In addition, the evaluation questions seek to verify whether SI has positive or negative, intended or unintended impacts and asks respondents to state their own judgement and perspectives of the intervention.

Indicators suggested by SITT stakeholders are relevant for evaluating some of the key dimensions of SI (e.g., networks, attitudes). The following indicators were suggested by the SITT members:

* Satisfaction of different categories of actors (perceptions on benefits gained)

* Motivations of different categories of actors (expected benefits)

* Impacts on the quality of life

* Lessons learned

* Change in capacity

* Quantitative and qualitative characteristics of networks and their dynamics

* Quantitative and qualitative characteristics of different types of networks (e.g., of the actors initiating the process, of those participating in the process)

* Level of civil society engagement: roles and contributions of actors towards solving the societal issue,

* Mechanisms of coordination

* Policy changes

* Demographic data

* Social return on investment and other socio-economic outcomes (e.g., employment, cost, inclusiveness)

* Changes in soft skills

* Stories and factors of successes and failures

* Resources in support of the process development
<figure><img src="image_52.png" /></figure>

* Institutions (i.e. 'rules of the game') to support SI and transfer it across communities

* Number and characteristics of beneficiaries

* Number and quality of innovative products and services that the SI initiative brought

* Best practices and innovative solutions used by other communities in other places

* Communication about the SI initiative

* Expected and real outcomes

* New activities promoted by the SI initiative

* Percentage of rural population interested (covered by the SI, benefiting from the SI)

* Social needs

* Consensus on the results of SI

* Sustainability of the SI results

Currently, questions in the structured interviews proposed for data collection of primary data on SI in the pioneer SI case studies (see Part IV and Annex 1) include the variables and aspects highlighted by the indicators that the SITT stakeholders suggested. However, specific indicators and analysis of qualitative data will be developed and carried out once the pilot testing of the set of methods and tools proposed in Part IV in selected pioneer SI case studies and policy processes is carried out. When the data for the proposed variables are available and functional to the SI evaluation, the variables will be combined to identify the most useful indicators.
<figure><img src="image_53.png" /></figure>

# PART II – SETTING THE SCENE: WHAT WE NEED TO EVALUATE AND WHY
<figure><img src="image_54.png" /></figure>

4. Towards a SIMRA Evaluation Framework

An evaluation agenda requires a clear framework for evaluation that defines the key dimensions of SI and visualises the steps, interconnections and relationships within an SI initiative (e.g. supporting conditions, limiting processes, participants, outputs and impacts). The SIMRA evaluation framework needs to be functional to guide an effective evaluation practice.

The first step in building the SIMRA evaluation method is based on the identification of the key dimensions and sub-dimensions, phases, changes and variables for measurement that are expected to be connected to SI initiatives and related processes and impacts. The evaluation framework aims to provide the basis for informing evaluation of SI, by describing the methods and tools that are most appropriate to use to evaluate the different components of a SI initiative.

The key dimensions and sub-dimensions, phases, changes and variables of SI and their interconnections, valid during the SI initiative and once it is completed, are explained in detail and used to guide the creation of the SIMRA evaluation method in Section 5. Specific assessment tools (e.g. questions for use in interviews with local actors) and evaluation approaches (e.g. integration of quantitative and qualitative methods) for each of these elements are outlined in Part IV and listed in Annex 1. These are to guide the preliminary testing of SI case-studies (WP5) and policy processes (WP6), in a few selected pilot/pioneer cases.

The SIMRA evaluation framework is grounded on theory (See Polman et al., 2017, D2.1; Kluvánková et al., 2017, D2.2), introducing innovative insights aimed at advancing existing knowledge for evaluation of SI. The concept of social innovation draws from a wealth of research and work in a variety of fields, including economics, sociology, ecology and political sciences and is defined in SIMRA as, *"the reconfiguring of social practices, in response to societal challenges, which seeks to enhance outcomes on societal well-being and necessarily includes the engagement of civil society actors"* (Polman et al., 2017; D2.1). Specifically, innovation theory, endogenous and neo-endogenous development, social capital, socio-ecological systems, regional development and social enterprises and entrepreneurship are considered prominent precursors to social innovation in marginalised rural areas (see Polman et al., 2017, D2.1; Kluvánková et al., 2017, D2.2; Slee et al., submitted). These have dynamics in marginalised rural areas related to the level of rurality, physical geography, limited access to infrastructure and often negative population trends (e.g. depopulation, ageing, loss of youth) (Price et al., 2017; D3.1). To operationalise an evaluation framework capable of addressing the specificities of social innovation, Deliverable 4.2 combines theoretical insights on social innovation in MRAs with an approach using theory of change (Morra Imas and Rist, 2009; Khandher et al., 2010; see Part III for more details and Secco et al., 2016; D4.1).

Figure 9 shows the connections between the interdisciplinary theoretical background adopted for the SIMRA definition of social innovation (Polman et al., 2017, D2.1; Kluvánková et al., 2017, D2.2)<fnref n="13" />, the characteristics of marginalised rural areas (Price et al., 2017; D3.1) and the evaluation framework proposed. It also shows how the evaluation framework expands on the abovementioned work to identify the core factors and variables of SI for the specific purposes of evaluation. To do this, it identifies SI *dimensions* (i.e. elements of assessment), *sub-dimensions* (i.e. sub-elements of assessment), components and possible interconnections.

The development of specific concepts leads to the identification, for each sub-dimension, of *components* of SI that can be assessed by means of measurable variables and/or their aggregations (see Pisani et al., in print). Finally, the evaluation framework proposed in SIMRA sets the scene for developing *indicators*, which are derived from both quantitative and qualitative data, and measured from the data collected at the case study level (see Part IV for instructions for data collection and evaluation). These indicators are proposed as principal instruments for the evaluation of SI in SIMRA (see Part III for more details on their development).

<footnote n="13">¹³ Details are provided in Polman *et al.* (2017; D2.1) and Kluvánková *et al.* (2017; D2.2).</footnote>
<figure><img src="image_55.png" /></figure>

<figure><img src="image_56.png" /><figcaption>**Figure 9.** Relationship between the general theoretical framework and the evaluation framework.</figcaption></figure>

Source: SIMRA research team.

The theory of change approach provides the basis for identifying key questions for collecting data on the different elements that constitute SI in MRAs. In the evaluation framework proposed, key questions are used to collect both quantitative and qualitative data on SI. First, this will be undertaken in the pioneer case studies, from which feedback will enable the revision of the questionnaires, which will then be used in the selected case study areas (WP5).

The collection of data at the local level is carried out using questionnaires that are submitted in structured interviews, while the analysis of policy processes, guided by the WP6 Leaders, is carried out using a guide for semi-structured interviews. The questions are tailored for use with each of the different types of actors. Specific details on the evaluation approach and questions are provided in Parts III and IV, and the list of questions for pilot testing is provided in Annex 1.
<figure><img src="image_57.png" /></figure>

# 5. Explaining the SIMRA Evaluation Framework

## 5.1. General Approach

The emergence of social innovation processes depends on factors that vary according to sector(s) and targets for implementation, as well as geographical, territorial, socio-economic and policy contexts (Polman et al., 2017, D2.1; Kluvánková et al., 2017, D2.2; and Ludvig et al., 2017, D6.1) and that have special characteristics in marginalised rural areas (Price et al., 2017; D3.1). In this section, the proposed SIMRA evaluation framework if described, with detailed analysis of the dimensions, and interrelationships amongst different dimensions, sub-dimensions and variables.

To analyse the emergence, development and impacts of social innovation, a structure agency framework based a multi-level, cross-scale approach is used (Sewell, 1992; Hays, 1994). The structure agency framework seeks to provide a unified understanding of collective action for social innovation, by bringing together the analysis of both process and outcomes (Cajaiba-Santana, 2014; Howaldt et al., 2015). According to Sewell (1992: 19), “structures are sets of mutually sustaining schemas and resources that empower and constrain social action and that tend to be reproduced by that social action”, a concept which is used and adapted as (perceived) context in the framework (Section 5.2.2). In other words, “[s]tructures emerge at the macro level as a result of the actions of the agents and their interactions with other agents” and change slowly (Janssen and Ostrom, 2006).

Agency “is the actor's capacity to reinterpret and mobilize an array of resources in terms of cultural schemas other than those that initially constituted the array” (Sewell 1992: 19; see also Jensen and Meckling, 1976). This perspective recognises agency as “embracing social choices that occur within structurally defined limits among structurally provided alternatives”, and where “human agency and social structure, then, have a simultaneously antagonistic and mutually dependent relationship” (Hays, 1994: 65). Rather than privileging an actor-centered perspective or a context-dependent perspective (e.g. Turker and Vural, 2017), the structure and agency framework draws on an understanding of SI as emerging from the actions of individuals embedded in networks, or a collective, who operate within the enabling and constraining conditions of their own social, economic, environmental and institutional environments to reconfigure practices that seek to improve wellbeing (Cajaiba-Santana, 2014; Howaldt et al., 2015; Haxeltine et al., 2016; Turker and Vural, 2017). The framework recognises a relational framing of the SI process, which sees agency (and cognition) as ‘distributed’ across a system (Chilvers and Longhurst, 2015: 3) rather than easily attributed to individual actors or groups, and reproduced by the practices and commitments of multiple actors (ibidem).

Cajaiba-Santana (2014) develops a conceptual framework that draws from the dialectic relationship between structure and agency to address how social innovation can lead to change and new social practices that seek to enhance outcomes on societal well-being. In this sense, social innovation should not be considered as a “neutral process”, i.e. just a change; rather, it should be considered for its capacity to lead to something new as compared with historical and/or recent trajectories in social action (Pike et al., 2010; Moulaert, 2013; Nicholls and Ziegler, 2015).

The focus on practices has enabled the academic debate to bring together concerns for the ways in which structures facilitate and constrain individual and collective actions, and the ways in which agency is indeed able to “make choices that have transformational consequences” (Hays, 1995: 62) “in creative and innovative ways” (Sewell, 1992: 4)<fnref n="14" />. As explained by Howaldt et al. (2015: 30), “Practice theories overcome the dichotomies between structure and action, subject and object, rule and application, society and the individual, that arbitrarily define micro and macro levels or sociological ‘reality rules’” (“Latour and Lépinay, 2010, p. 114, as cited in Howaldt et al., 2015; Giddens, 1979). In short, they allow us to see how new practices may

<footnote n="14">¹⁴ Referring to Giddens’ “duality of structure”, Sewell explains: “how, in a great variety of times and places, structures are in fact dual: how historical agents’ thoughts, motives, and intentions are constituted by the cultures and social institutions into which they are born, how these cultures and institutions are reproduced by the structurally shaped and constrained actions of those agents, but also how, in certain circumstances, the agents can (or are forced to) improvise or innovate in structurally shaped ways that significantly reconfigure the very structures that constituted them” (1992: 5).</footnote>
<figure><img src="image_58.png" /></figure>

reproduce existing structures, or conversely, lead to new social systems being created and institutionalised (or up-scaled), a key issue in the study of social innovation (Hays, 1994; Cajaiba-Santana, 2014; Haxeltine et al., 2016). From this perspective, SI can be seen as “a black box”, to be assessed and evaluated through the analysis of its components and their interactions.

Figure 10 represents the components and interactions of social innovation that may be suited to evaluation.

<figure><img src="image_59.png" /><figcaption>**Figure 10.** SIMRA evaluation framework proposed to evaluate SI and its impacts in MRAs. Source: SIMRA research team informed by the theoretical framework in Kluvánková *et al.* (2017; D2.2).</figcaption></figure>
<figure><img src="image_60.png" /></figure>

The following Sections describe the key phases and dimensions of social innovation identified in the framework. The explanation follows a linear, step-wise, and temporal process (i.e. from the initial drivers of SI and factors that determine the emergence of SI, to the intermediate stages of changing social practices, and the outputs/outcomes and impacts that derive by SI). However, this should not be considered as a strictly deterministic approach<sup>15</sup>. Rather, it should be reflected as an operational-functional approach required for an evaluation based on the theory of change and on a result model, i.e. based on identifying the stages, dimensions and variables of a SI (and the changes that a SI may induce) that are to be analysed and "measured".

## 5.2. Key Dimensions for the Evaluation of Social Innovation

The framework proposed for the evaluation of social innovation and its impacts includes the following key dimensions: "Individual and collective needs", "Perceived context", "Agency", "Reconfiguring and reconfigured social practices", "Activities", "Outputholzs", "Outcomes and impacts", "Learning processes" (see

Figure 10). These are explained in two sub-Sections: the first provides an overview of the main concepts, and the second provides "practical implications" for the SIMRA evaluation methods and tools. To aid introduction, some text used in the first Section has been restated in the second part.

### 5.2.1. Trigger, individual and collective needs

For the purposes of evaluation, two key elements are connected to the set of variables "Individual and collective needs" identified in Kluvánková et al. (2017; D2.2). These key elements are "trigger" and "needs".

#### 5.2.1.1. Trigger and needs: an overview

The evaluation framework proposed starts from the premises of a *trigger and response model*: unmet social needs, societal challenges and governance shifts<sup>16,17</sup> are framed as the 'needs' (or unwell-being) to which agency (comprising actors with their own ideas, visions and trust, willingness to act, reflexivity and capacity to change) responds following a 'trigger' at any level, i.e. local, national or international. Unmet social needs, societal challenges and governance shifts are the focal challenge to which a SI responds. In principle, a

<footnote n="15"><sup>15</sup> More in-depth considerations on the non-linearity of SI initiatives, are discussed from a theoretical point of view in Kluvánková *et al.* (2017; D2.2), and included in Part III.</footnote>

<footnote n="16"><sup>16</sup> We refer to unmet social needs, societal challenges and governance shifts as the focal challenge to which a SI responds following a trigger (BEPA, 2011). Unmet needs can be individual or collective. Mulgan *et al.* (2007) identify the areas with a "deficit" of social innovation, and thus a potential need, as: rising life expectancy, climate change, growing diversity of countries and cities, stark inequality (leading to rising violence and mental illnesses), rising incidence of health-related long-term conditions and diseases, behaviour problems of affluence, difficult transitions to adulthood, as well as happiness. Here, "the mismatch between growing GDP and stagnant well being and declining real welfare according to some measures requires new ways of thinking about public policy and civic action" (Mulgan *et al.*, 2007: 9). To identify and categorise the different types of needs for the purposes of evaluation, we found particularly useful to follow the three approaches to social innovation proposed by BEPA (2011). Social innovation can respond to (i) social, economic, environmental and institutional demands of vulnerable groups which are not traditionally addressed by the market of by existing institutions (approach 1: SI responds to specific "social needs"); (ii) societal challenges in which the boundary between social and economic blurs, and which are directed towards society as a whole (approach 2: SI responds to broader "societal challenges"); and (iii) systemic change perspective to be reached through a process of organisational development, and changes in relations between institutions and stakeholders (approach 3: SI responds to "governance shifts", i.e. tentative experiences of creation of new relationships between public and private spheres, new role adopted by public authorities in managing the society) (BEPA, 2011: 43).</footnote>

<footnote n="17"><sup>17</sup> By 'governance shifts', we refer to inadequacies in governance shifts, where governance models are no longer adapted to changing circumstances. This may happen when governance shifts towards network-oriented and participatory-based models without designing effective mechanisms, public institutions may not be qualified, prepared or well-resourced to address the shift properly, public administration may be obsolete and bureaucracy not flexible. However, governance shifts that positively stimulate SI (e.g. efforts at networking or engaging citizens in decision making) have to be taken into consideration as well.</footnote>
<figure><img src="image_61.png" /></figure>

possibility for addressing unmet needs may create an opportunity for the emergence of SI. Unmet social needs may refer to the realms of society, economy and environment, and in SIMRA, refer to agriculture, forestry and rural development, and on the special features characterising marginalised rural areas (e.g. remoteness, isolation or depopulation – see Price et al., 2017; D3.1). However, it is not only a matter of a lack of societal well-being: in certain circumstances, an alternative social practice may improve well-being and be considered a social innovation.

The trigger may take the form of a natural or environmental disaster (flooding, drought), social, economic or financial crises, shortage of funding, deliberative or participatory movements which attempt to tackle “wicked” problems. It may also have positive connotations. For example, the trigger could be an unexpected enabling factor such as a new charitable bequest, or a new subsidy that provides unexpected resources. Thus, the trigger may emerge in connection to a single time-bound event (e.g. a three-days extreme flooding event); an accumulation of unmet needs (e.g. a prolonged period with worsening social life because of depopulation); or from a long-term process, whereby a situation becomes untenable (e.g. a critical demographic situation reached as consequence of a process of ageing and abandonment of a mountain area).

Examples of negative triggers are: consistent emigration flows because due to unemployment; abandonment of the territory; extreme climatic events such as flooding, landslides, earthquakes, and drought; uncontrolled consequences of climate change; deep recessions and economic crises; health pandemics; conflicts and wars; unbalanced demographic trends; and natural resources loss and degradation (e.g. biodiversity loss and depletion, water, air and soil pollution, pest outbreaks, invasion of alien species, deforestation).

Triggers are also connected to the concept of disturbances. “Disturbances<fnref n="18" /> can be sudden, i.e. a shock, but often gradual changes in an exogenous driver (e.g. the reduction of price support over time).” (Peerlings et al., 2014: p.). Positive events (e.g. changes toward enabling policy, effective decentralisation, subsidies for local action or a charitable bequest, positive social capital) can also “open up” the structure of opportunity, allow leaders to initiate actions towards SI, and facilitate the emergence of networks and purposeful action. This process may not involve any radical changes, but may lead to an overall improvement of the current system<fnref n="19" />. Finally, as noted above, the trigger may emerge from participatory movements or mobilisation, whereby actors perceive an opportunity as a possibility for change due to changing conditions, and ‘reframe’ a problem into an opportunity for starting an innovation.

As described in Polman et al. (2017; D2.1), environments which promote social innovation can include: i) periods of major social-political turmoil; ii) situations where particular regions or socio-ecological systems are confronted by long-standing and widely-recognised, sometimes “wicked” problems that existing policy structures have been unable to address satisfactorily; iii) situations where there has been a major, unanticipated, socio-ecological disturbance and in its wake a proliferation of responses includes SI (flood, earthquake, nuclear accident); iv) situations where there are major socially, economically, or politically structured divergences in welfare or opportunities between different groups; v) situations where there are major divergences in belief systems within countries, especially in religious and ethical beliefs; vi) situations where there is a power vacuum, or major failings or delivery gaps in state institutions (power); and (vii) situations where the state has dramatically reduced the range of services (Box 5.1 in Polman et al., 2017; D2.1). All of these situations can directly or indirectly lead to the emergence of a trigger. However, while tiggers are usually seen as “external” to the actors and independent from their perception and recognition, in the proposed evaluation framework, it matters as to how each specific situation is perceived, framed and

<footnote n="18">¹⁸ Peerlings et al. (2014) frame these concepts as: “In the context of the farm, disturbances are defined as events that disrupt a farm business (see, for example Janssen and Ostrom, 2005). Disturbances can be idiosyncratic in nature, e.g. a farm family crisis or the impact of an outbreak of an animal disease or extreme weather conditions. On the other hand, disturbances can also be structural in nature – i.e. affecting the whole farm sector or all farm businesses in a certain sector, e.g. policy reforms (see, for example Dries and Ciancian [2012] for an analysis of the impact of idiosyncratic and structural shocks on farm employment in the EU)”.</footnote>

<footnote n="19">¹⁹ Bruce (1993) uses the examples of technological innovation to describe how, “when conditions are right, a new innovation can set in motion “a continuing sequence of connected events” (citing Burke, 1978: 12).</footnote>
<figure><img src="image_62.png" /></figure>

of reconfiguration, which eventually leads to SI. From this perspective, the trigger<sup>23</sup> does not exist as an objective category, but is seen and used as an opportunity for starting actions that will lead to social innovation by agency.

Perceived unmet needs may include both individual basic (physiological and safety) and higher-level needs (belonging, knowledge, self-esteem and self-actualisation) (see Maslow, 1943; 1954 in Figure 12 and Box 6), as collective needs, such as self-determination in the case of ethnic minorities or indigenous groups. Needs are not necessarily standardised and equal for everybody, but depend upon cultural, social, economic, environmental and political contexts<sup>24</sup>.

While needs are always present in a community to varying degrees, we hypothesise different scenarios: i) needs may be recognised as salient by the general public and raise political discussion and policy action (e.g. poverty); or ii) needs may not be recognised as salient by the general public and may not be captured statistically at the regional or territorial level, yet deeply affect a minority of the population (e.g. homelessness for single parents, 'working poors', isolated and remote fractions of small villages in rural areas). In both cases<sup>25</sup>, the trigger may bring the 'need' to the fore and compel specific action to address it (see Figure 10).

SIMRA aims to identify social and societal needs and characteristics of governance shifts in the context of marginalised rural areas in EU and non-EU Mediterranean countries. In such territories, trigger-needs relationships are typically characterised by remoteness, limited access to services through public mobility, lack of employment, lack of prospects for career building, low income and poverty, population ageing, population loss, lack and loss of services, social problems (e.g. gender inequalities), and lack of IT connectivity (see Price et al., 2017, D3.1, for a full analysis).

In Table 7, we propose a preliminary and non-exhaustive list of social needs, societal challenges and governance shifts that are likely to characterise MRAs in the SIMRA target area. The list is structured around the three focal challenges and corresponding approaches to SI as suggested by BEPA (2011)<sup>26</sup>. Approach 1 refers to SI as a response to the social demands of vulnerable groups (i.e. it is grounded on specific unmet social needs); approach 2 refers to SI as a response to societal challenges directed towards society as a whole; approach 3 refers to SI as a response to governance shifts (i.e. systemic change in relations between public authority and the society).

<footnote n="23"><sup>23</sup> Triggers also lead to processes of upscaling, for example when a new social innovation itself acts as a trigger for its replication (scaling out) or for its inclusion in changes in policy at a higher level (scaling up) (Moore and Westley, 2011).</footnote>

<footnote n="24"><sup>24</sup> According to Sen (1999), improvements in well-being depend on people's capabilities as well (see Box 5).</footnote>

<footnote n="25"><sup>25</sup> In case (a), while some action may be taken, needs may simply (i) not be well addressed from the perspectives of those most vulnerable in terms of equity and effectiveness, or (ii) be considered by the general public and policy-makers as 'too big to tackle'. In case (b), the needs that are not recognised by the public as salient, ('marginalised needs' may refer to those that are not generally considered in a context analysis), may be disregarded by policy makers. However, a specific action of SI may include a statistical or qualitative analysis that specifically describes the marginalised context and raises the profile of the 'need' to the general public (e.g. violence against women, work placement for people with a disability, youth suicide, dementia among the elderly). Within SIMRA, we need to clarify how the context generates the needs and what is the difference between context and needs. For example, if the context variable is the percentage of population over 65 years, related needs may be halting or inverting the process of population ageing, lack of a dynamic social life, job prospects or opportunities that lead to youth emigration; lack of support for young families and thus lower natality rates; or loss of land based activities and traditional knowledge in agriculture and forestry. The relationship between context and needs are a core aspect for the evaluation of SI initiatives at the local level.</footnote>

<footnote n="26"><sup>26</sup> See footnote 16</footnote>
<figure><img src="image_63.png" /></figure>

## Approach 1: The social demand perspective of vulnerable groups

The first approach responds to the question “Which are the main social demands that are traditionally not addressed by the market or existing institutions<fnref n="27" /> and are directed towards vulnerable groups in society?” suggested by BEPA (2011: 37).

<table><caption>**Table 7.** The social demand perspective in MRAs. Source: SIMRA research team based on BEPA, 2011.</caption><thead><tr><td>Vulnerable groups in MRAs</td><td>Examples of social needs and demands</td></tr></thead><tbody><tr><td>Minorities or local indigenous people groups<fnref n="28" />, e.g. groups with strong and ancient cultural identities, generally not recognised as culturally distinct and excluded from the dominant culture.</td><td><ul><li>Integration</li><li>Recognition</li><li>Dignity</li><li>Respect</li><li>“Voice” and “vote” in deciding local development paths</li></ul></td></tr><tr><td>Women, e.g. low qualified and low skilled, typically engaged in the provision of child care or elderly assistance, limited by lack of rural occupational opportunities, or excluded from them for cultural reasons.</td><td><ul><li>Resources allocated for child care</li><li>Hospital assistance for childbirth and post-natal care</li><li>Options for carrier development</li><li>Recognition of their key role</li></ul></td></tr><tr><td>People with intellectual and physical disabilities, e.g. people affected by physical inabilities in areas with limited transport and physical barriers, lack of job placement opportunities, or with learning difficulties.</td><td><ul><li>Health assistance centres for therapy</li><li>Options for leisure</li><li>Social integration</li><li>Work placement</li><li>Reduced architectonical barriers</li><li>Public transport facilities</li></ul></td></tr><tr><td>Long-term unemployed, e.g. low qualified local residents and migrants, men and women, employed in the past in an industrial sector or district<fnref n="29" />.</td><td><ul><li>Employment opportunities in loco</li><li>Avoid relocation and outmigration</li><li>Take care of family and maintain personal connections</li></ul></td></tr><tr><td>Offenders, e.g. local people well known in the community but unable to reintegrate because of their criminal record and face difficulties to re-build social relations and reputation.</td><td><ul><li>Being re-accepted without prejudice</li><li>Being given a second chance</li><li>Reconstructing their social networks</li><li>Regain respect</li><li>Have employment opportunities like the others</li></ul></td></tr><tr><td>Elders, e.g. lack of access to transport (personal car or access to public transport), cannot drive or are simply too old for driving; those who face mobility barriers; are limited during</td><td><ul><li>Remain engaged in society with their values and capacities</li><li>Access to resources for health care</li></ul></td></tr></tbody></table>

<footnote n="27">²⁷ In several countries (e.g. Austria), the needs of vulnerable groups are addressed by “existing institutions” as well as by market (e.g. cooperatives, third sector). At the national level, there may also be laws that recognise minority groups, as well as specific programmes for low qualified women and other vulnerable groups. SI initiatives attempt at addressing these issues through niches initiatives, led by the voluntary engagement of civil society, e.g. social gardening with refugees and minority (migrant) women; or social enterprises aimed at long-term unemployed groups. These initiatives are partly financed through public funds or recieve other types of public support. However, despite attention, they may not always work in practice. Thus, when BEPA refers to SI in relation to “traditionally not addressed by market or existing institutions” (2011: 37), we refer to it as being traditionally not properly addressed by market or existing institutions.</footnote>

<footnote n="28">²⁸ e.g. Cimbri population in the Veneto region, North-Eastern Italy.</footnote>

<footnote n="29">²⁹ e.g. The eyewear industry in the Belluno province was flourishing up until the 1990s in the Cadore valleys, but residents lost their jobs due to the international competition of Chinese producers and the collapse of the local industries. Many people remained in the area (now marginalised) because of family relationships and duties, while others emigrated towards urban areas, thus contributing to the further decline of the social tissue.</footnote>
<figure><img src="image_64.png" /></figure>

<table><thead><tr><td>Vulnerable groups in MRAs</td><td>Examples of social needs and demands</td></tr></thead><tbody><tr><td>extreme weather events (snow, heavy rains heat); or receive declining welfare resources.</td><td><ul><li>Access to public assistance</li><li>Access to mobility options</li><li>Access to meeting and leisure centres (not only those managed by churches or small local NGOs) for active social life</li><li>Home care</li></ul></td></tr><tr><td>Children and young people, e.g. limited access to school options, cultural activities, sports facilities; engaged in family working activities such as farms, or with parents who work some distance away; limited resources allocated for school improvements and professional education and consequently limited chances of professional careers if they remain in the local area; long distance commutes to high schools and higher costs for mobility; limited internet connection in remote areas.</td><td><ul><li>Options for choosing sports</li><li>High quality school options</li><li>Options for vocational training</li><li>Access to affordable public transport and leisure centres</li><li>Internet connection</li><li>Options for building friendships and connections (cinemas, bars, swimming pools, disco dances, theatres, music concerts) (active social life)</li></ul></td></tr><tr><td>Poor families, e.g. low income families, often with one job, unstable, subject to moving in/outside of a community (sometimes not open and flexible as urban communities)</td><td><ul><li>Minimum salary</li><li>Dignity</li><li>Employment opportunity</li><li>Housing</li></ul></td></tr></tbody></table>

## Approach 2: The societal challenge perspective

The second approach responds to the question “Which examples of societal challenges<fnref n="30" /> are both social and economic in nature, and are directed towards society as a whole?”, as suggested by BEPA (2011: 38). **Societal challenges** refer to social, economic, institutional and environmental challenges that require adaptation. Examples include natural disturbances such as negative trends in the availability or access to natural resources due to climate change and resource and landscape loss and depletion, or social change, such as ageing, in- and out-migration, declining population, conflict, war and social unrest, or criminality.

Adapting to a natural disturbance may refer to survival after an environmental disaster, including the decision to remain and re-build houses and social life; adapt agricultural or forestry practices to reduced availability and access to natural resources; implement new climate-sensitive ways of producing and distributing green products and services, or finding new products and services that are sought by society. These refer to new social uses of forests for improving general health (e.g. green care), short value chains of local agricultural and forest products and specialties, community-based and rural tourism activities, or nature-based businesses.

Adapting to social change (often reported as 'social disturbances') may involve finding ways to manage the long-term care of elderly residents in MRAs during a time of public budget constraints, increasing the attractiveness of marginalised rural areas for young families, for example by guaranteeing basic public services, creating new jobs, creating cultural events and promoting a dynamic social life). It may also involve improving infrastructure and services to reduce out-migration towards urban areas, or integrating newcomers into the local economy and community. Survival after a conflict/war, adaptation to social unrest and resistance to criminality are other examples that require adaptation in marginalised rural contexts.

<footnote n="30">³⁰ Societal challenges include environmental-related challenges (e.g. climate change) that have both social and economic consequences. Thus, even if the term “ecological” is not explicitly mentioned in the question posed by BEPA, it is implicitly included as a component of societal challenges.</footnote>
<figure><img src="image_65.png" /></figure>

## Approach 3: Systemic change perspective

Governance shifts include attempts at adopting citizen engagement mechanisms, increasing information exchange and disclosure, facilitating network creation and managing new roles for the public sector (e.g. process facilitator). Governance shifts are often characterised by inadequacy, i.e. by a limited or absent capacity of the governance model to adapt to continuously fluctuating circumstances. Examples of inadequacies that are obstacles to shifts towards new models and mechanisms include the lack or ineffective involvement of civil society, community and citizens in decision-making processes and lack of voice; unbalanced representation of women in positions of responsibility (e.g. mayors, presidents, CEOs); an overwhelming bureaucracy or obsolete and rigid legal framework, brittle and inflexible public administrations (i.e. an administration with limited institutional capacity unable to address society's needs), conflicts of interest as well as corruption in public and private organisations. Thus, governance inadequacy refers to top-down, managerial systems of control (Holling and Meffe, 1996), which are brittle, inflexible, and lack the capacity to face social, environmental and economic change (societal challenges). They refer to the lack of capacity to steer governance systems involving public and private partners. In marginalised rural areas, this can be due to improper decentralisation or neo-centralisation processes (e.g. Secco et al., 2017), or, more simply, to a progressive reduction in the amount of resources allocated to peripheral (local) governments following financial and economic crises and spending reviews. Thus, governance shifts may fail to address the needs of the most vulnerable groups, leading to greater inequality and overall social instability. It is also possible for governance shifts to be positive. Shifts may be well structured and properly organised, supporting emerging SI initiatives. Both options should be considered in the evaluation.

## Box 6– Main methods that inspired the development of the SIMRA method for addressing the dimension Individual and collective needs

From the survey of methods to assess SI and its impacts, different approaches were influential. For example, Maslow's Pyramid (1943; 1954 in Figure 12), in which needs are ranked, is heuristically useful, even though extensive critique has challenged the 'hierarchization' of needs and argued that needs are perceived differently across cultures, ages, gender, class and between individualistic and collective societies (e.g. Wahba and Bridwell, 1976; Hofstede, 1984; Cianci and Gambrel, 2003).

<figure><img src="image_66.png" /><figcaption>Figure 12. Hierarchy of needs (Maslow, 1943; 1954). Source: http://blog.readytomanage.com/wp-content/uploads/2012/11/maslow-pyramid.png accessed 21.03.2017</figcaption></figure>

Needs are also connected with the concept of “quality of life” (Nussbaum and Sen, 1993) and the “substantive freedoms” developed in Sen’s “capability approach” (Sen, 1999). According to Sen (1999: 4-5) rather than guaranteeing a certain level of income or GDP per capita, development “requires the removal of major sources of unfreedom”, including poverty, tyranny, “poor economic opportunities, systematic social deprivation, neglect of public facilities” and social care. Several types of economic, social and political freedoms play a key role “in enhancing and enriching the lives that people are able to lead” (ibidem: 9). “The life that a person leads can be seen as a combination of various doings and beings, which can be generally called functionings. These functionings vary from such elementary matters as being well nourished and disease-free, to more
<figure><img src="image_67.png" /></figure>

complex doings or beings, such as having self-respect, preserving human dignity, taking part in the life of the community and so on." (Nussbaum and Sen, 1993: 3). Therefore, needs are not necessarily standardised, as they depend on cultural contexts and dominant values. Through his studies, Sen tried "to explore a particular approach to well-being and advantage in terms of a person's ability to do valuable acts or reach valuable states of being" (Sen, 1993: 30).

If we consider "freedoms as the primary ends of development"<fnref n="31" />, we can argue that guaranteeing substantive freedoms means satisfying people's needs. The five types of basic freedoms include: "1) political freedom; 2) economic facilities; 3) social opportunities; 4) transparency guarantees and 5) protective security. Each of these distinct types of rights and opportunities helps to advance the general capability of a person" (Sen, 1999: 10). They are all interconnected.

However, if we consider that "social innovation [...] seeks to enhance outcomes on societal well-being" (SIMRA definition), Sen's "capability approach" can help us to define well-being. "The capability approach to a person's advantage is concerned with evaluating it in terms of his or her actual ability to achieve various valuable functionings as a part of living. The corresponding approach to social advantage, for aggregative appraisal as well as for the choice of institutions and policy, takes the sets of individual capabilities as constituting an indispensable and central part of the relevant informational base of such evaluation" (Sen, 1993: 30). This approach differs from approaches focused on personal utility (e.g. based on pleasures or happiness), absolute or relative opulence (e.g. based on income), or absence of freedoms (Sen, 1993).

Freedom to lead different lives is reflected in a person's capability set, while the capability of a person depends on factors such as personal characteristics and social arrangements. In addition, papers and methods that have inspired development of the preliminary set of SIMRA methods and tools are: Dawson and Daniel 2010; Bock 2012; AngloAmerican 2012; McLachlan et al., 2015; SIMPACT, and Peerlings et al. (2014). More information is provided in the SIMRA Resources Web Site (see www.simra-h2020.eu/resources/).

Box 7 – What we evaluate for the dimension Individual and collective needs

* Before and after: the context of individual and collective needs (see Section on 'Perceived context')

* Trigger as a discrete event or long-running process

* Framing processes

* Identification and explanation of the relationship between trigger and needs

* Analysis of how the trigger may change perception of needs and thus compel individual or groups to action (e.g. psychology of the individual, psychology of the group, storyline of strategic framing)

* Specific unmet social needs, societal challenges and/or governance shifts to which SI responds

5.2.2. Perceived context

5.2.2.1. Perceived context: an overview

Context refers to the conditions that enable or constrain SI. In the SIMRA evaluation framework, it includes two sub-dimensions: i) a static one, based on existing context and resources; and ii) a dynamic and subjective one, based on the perceptions and framing of actors of what is an opportunity or a threat (perceived opportunities and threats). The first sub-dimension, Context, refers to the SI context set of variables (see Kluvánková et al., 2017; D2.2)<fnref n="32" />. Context typically includes the regulatory frameworks (laws, legislation and

<footnote n="31">³¹ Sen further considers freedoms as the "principal means of development" (1999: 10).</footnote>

<footnote n="32">³² The SI context includes existing enabling and constraining conditions that influence the actions of actors, and thus determine the "action arena" (Kluvánková et al., 2017; D2.2).</footnote>

<footnote n="33">³³ In the SIMRA evaluation framework, the SI context includes elements of the "resource systems" and the "resource units", as well as of the "governance systems" and "actors" described in McGinnis and Ostrom (2014) (see Kluvánková et al., 2017, D2.2; Section 5.2.2.2 of D4.2).</footnote>
<figure><img src="image_68.png" /></figure>

policy); overall governance and institutional arrangements, both formal and informal; material resources such as availability of funding, raw materials, natural resources and existing infrastructures; and intangible resources such as social memory, culture and identity, discourses, and historical background. These resources are identified as the five capitals (social, natural, human, financial, cultural) (Goodwin, 2003) or considered as dimensions of a territory, e.g. social, financial, legislative, economic, technological and natural territorial systems (Osti, 2010).

The second sub-dimension, Perceived opportunities and threats (POT), refers to the subjective understanding and perception of the context. The latter focuses on the resources or obstacles that are perceived as such by the agents and actors of a system. The existence of a specific set of assets (resources) does not imply that they are visible (identified and recognised), available and/or accessible and at the disposal of actors/agents ready to implement a SI. As described under trigger and needs (Section 5.2.1 above), context is not objectively defined, but rather, it is subject to the collective attribution of ‘opportunities’ or ‘threats’. In one case, the context could offer a set of opportunities that to others are threats. Context must be visible and ‘seen’ by an agent/actor for SI initiative to start<fnref n="34" />, and for this reason, it is case-specific at the local level.<fnref n="35" />

Perceived context refers to all the tangible and intangible resources that are available, accessible to, recognised and used by, actors/agents (or conversely, that hinder actors) in the process of reconfiguring relations and social practices and putting in place mechanisms for SI.

### 5.2.2.2. Perceived context: details for SIMRA evaluation

In the context of SIMRA evaluation, the dimension Perceived context is distinct from trigger and needs. Trigger refers to a discrete event or a situation "that causes something to start" (Cambridge Dictionary, 2017), and, in the case of SI, that causes agency to start. Perceived context refers to the structure of opportunity, external to the agency, at any scale, which integrates how actors perceive the contextual situation (e.g. the institutional environment) and which enables or constrains the action of the innovator. According to this definition, in the evaluation, it is assumed that perceived context is identified by both a context analysis (i.e. "context indicators", on various types of existing resources) and a "POT" analysis (i.e. analysis of Perceived Opportunities and Threats)<fnref n="36" />. Analysis of both sub-dimensions provides the baseline for the evaluation, i.e. the scenario before the SI, against which change can be measured

Perceived context is case-specific at the local level, because it constitutes site-specific resources (also available at other levels) that are visible, culturally and socially identified and defined; and types of access that the agent/agency has to resources which are tangible (e.g. financial, physical resources) or intangible (e.g. knowledge, discourses, symbols). For indentifying and analysing key resources in the evaluation of the dimension "Perceived context", the concept of capitals is used.<fnref n="37" />,<fnref n="38" /> Natural Capital is connected to environmental resources, Infrastructural Capital to human-constructed facilities and resources, Financial

<footnote n="34">³⁴ See footnote 22 for reference to the social movement theory (McAdam et al. 2001).</footnote>

<footnote n="35">³⁵ Power relations can shape perceived context and the kinds of relationships which develop as a result of agency and the SI initiative. However, power relations are more connected to the specific strengths and weaknesses of the agent/agency in using resources, and are thus explored under the dimension of "Agency".</footnote>

<footnote n="36">³⁶ The POT is a SWOT analysis but the components "Opportunities (O)" and "Threats (T)" are considered and seen from the specific perspective of the agency as factors external to agency, i.e.i.e. in relation to what is recognised and perceived as an enabling or constraining component by the agency itself. In evaluation, these could be identified through storytelling. Other methods will also be considered if necessary and possible.</footnote>

<footnote n="37">³⁷ When used in the contest of Sustainable Rural Livelihoods, capitals have been used to show a multidimensional approach to assets and associated to analyses of poverty. This "recognises human agency and examines the ways in which household livelihood strategies are built around protecting, substituting, increasing and using assets to produce security and achieve other goals." (Hulme and Sheperd, 2003: 414). It also contributes to the delivery of interventions that are not sector-based but recognise that interactions among capitals affect choices and opportunities.</footnote>

<footnote n="38">³⁸ In using key words connected with capitals, we are aware that this terminology over-emphasises their role and can be seen to fix them into categories. The approach does not imply support for a materialist and economist view of local development connected with SI. Rather, the approach is adopted for operational reasons, because of its relative ease of use for identifying the elements of the sets of resources for inclusion in the evaluation.</footnote>
<figure><img src="image_69.png" /></figure>

Capital to financial resources, Human Capital to human resources, and Social Capital to social resources (Table 8).

<table><caption>**Table 8.** Key components of various types of capital and their possible links to different types of resources.
Source: SIMRA research team based on various sources.</caption><thead><tr><td>Resources</td><td>Type of capital</td><td>Definition</td><td>Key components</td></tr></thead><tbody><tr><td>Environmental/ natural resources</td><td>Natural Capital</td><td>The [world's] stocks of natural assets which include geology, soil, air, water and all living things (World Forum on Natural Capital, 2017).</td><td>Natural resources such as water, soil, ecosystems, forests, pastures, energy, geology, land.</td></tr><tr><td>Human-constructed resources</td><td>Infrastructural Capital</td><td>The basic physical and organisational structures and facilities (e.g. buildings, roads, power supplies) needed for the operation of a society or enterprise" (Oxford Dictionary, 2017); or "Structures, systems, and facilities serving the economy of a business, industry, country, city, town, or area" (Dictionary.com).</td><td>Infrastructure such as airports, bridges, roads, railways, schools, hospitals, ITC infrastructures, buildings, machines, equipment.</td></tr><tr><td>Financial/ economic resources</td><td>Financial Capital</td><td>"Financial capital is any economic resource measured in terms of money used by entrepreneurs and businesses to buy what they need to make their products or to provide their services to the sector of the economy upon which their operation is based, i.e. retail, corporate, investment banking, etc." (Wikipedia)</td><td>Funds, investments, charities, short-term loans, medium-term loans, deposits, venture capital, equity, debts, leasing, financial instruments (e.g. futures, options, derivatives, investments funds, bonds), insurance.</td></tr><tr><td>Human resources</td><td>Human Capital</td><td>"The knowledge, information, ideas, skills, and health of individuals" (Becker, 1964). More recently, the definition developed a more collective-oriented meaning. "The collective skills, knowledge, or other intangible assets of individuals that can be used to create economic value for the individuals, their employers, or their community" (Dictionary.com). Or, "The skills, knowledge, and experience possessed by an individual or population, viewed in terms of their value or cost to an organisation or country" (Oxford Dictionary Online).</td><td>Skills, information, education, knowledge, health, ideas, values, social and personality attributes included creativity, motivation, competences, experiences, habits, culture.</td></tr><tr><td>Social resources</td><td>Social Capital</td><td>"Networks together with shared norms, values and understandings that facilitate cooperation within or among groups." (Healy and Côté, 2001: 41). Different types of SC: cognitive, relational and structural SC; bonding, bridging and linking SC.</td><td>Networks, relations, trust, shared norms, shared values, exchange of information, cooperation, cultural identity, social life, collaboration attitudes, solidarity, social groups, inclusion vs. exclusion, collective action, conflicts.</td></tr></tbody></table>
<figure><img src="image_70.png" /></figure>

<table><thead><tr><td>Resources</td><td>Type of capital</td><td>Definition</td><td>Key components</td></tr></thead><tbody><tr><td>Institutional/ governance resources</td><td>Institutional Capital</td><td>Despite institutional capital is referred as important to SI, not clear definition seems available<sup>39</sup>.</td><td>Government and non-governmental organisations, actors (included civil society representatives), networks, decision making rules, operational rules, property rights systems (included access rights), monitoring and sanctioning rules, security, legality, access to information, empowerment of different social groups.</td></tr></tbody></table>

McGinnis and Ostrom's framework on socio-ecological systems (2012) (see Box 8 and Kluvánková et al., 2017; D2.2) provides two useful categories of concepts: “Resource Systems” and their specific unitarian components (i.e. “Resource Units”), and “Governance Systems” and their specific unitarian components (i.e. “Actors”). The first category can be conceptually linked to Natural, Financial and Infrastructural Capital, and the second one can be linked to Institutional, Human and Social Capital.

In the SIMRA evaluation approach, we propose two sequential and complementary steps. First, identifying context indicators for natural, human-constructed, financial, human, social and institutional resources at the appropriate level of disaggregation (regional and/or local, when possible). Second, carrying out a POT analysis for the resources in Table 8, as identified by the social innovator and its network. This analysis uses different means of interpretation<fnref n="40" />, including: presence/lack of the resource; whether the resource is supportive or hinders the SI; whether the resource is identified as a resource or is not recognised. This approach allows an assessment of enabling (“presence of...”, “support to...”) and constraining (“lack of...”; “opposition to...”) conditions for SI. The social innovator can directly identify the opportunities and threats as external factors<fnref n="41" /> primarily through storytelling and narrative, whereas context indicators are based on secondary and quantitative data, when available at the appropriate level of analysis.

### Box 8 – Main methods that inspired the development of the SIMRA method for addressing the dimension Perceived context

The sub-dimension “Context” drew on: (a) the concepts of the five capitals that are at the basis of territorial systems, identified by economists and social scientists (e.g. Coleman, 1988; Becker, 1993, Goodwin, 2003; Osti, 2010); and (b) the basic components of the socio-ecological system framework (McGinnis and Ostrom, 2012; 2014). The approach and use of these specific frameworks is related to the relevance of natural resources and socio-ecological systems in the analysis of SI in marginalised rural areas (See Figure 13).

In identifying the sub-dimension “(P)OT”, reference is made to the theory of access (Ribot and Peluso, 2003). According to these authors, "[...] access analysis involves i) identifying and mapping the flow of the particular benefit of interest; ii) identifying the mechanisms by which different actors involved gain, control, and maintain the benefit flow and its distribution; and iii) an analysis of the power relations underlying the mechanisms of access involved in instances where benefits are derived" (Ribot and Peulos, 2003: 160-161).

<footnote n="39">³⁹ The Institutional Capital is referred as “the human, financial, technical, facilities, and other resources necessary to achieve an institution’s mission and goals” (http://oirap.rutgers.edu/msa/Standard-3.htm), but this meaning is not appropriate for the analysis of the institutional resources in a territory.</footnote>

<footnote n="40">⁴⁰ The order of collection of information is not necessarily the one used to illustrate the main contents of the analysis.</footnote>

<footnote n="41">⁴¹ Strenghts and Weakenesses (as internal factors with respect to the SI) are analysed in relation to other dimensions of social innovation (in particular, the dimension “Agency”).</footnote>
<figure><img src="image_71.png" /></figure>

<figure><img src="image_72.png" /><figcaption>**Figure 13.** The socio-ecological systems (SES) framework and its specific components. Source: McGinnis and Ostrom (2012) as explained in Kluvánková *et al.* (2017; D2.2).<sup>42</sup></figcaption></figure>

The concept of access to resources is multifaceted, involving access control, access maintenance and gaining access. Access can be explored in relation to access to technology and access to resources mediated by technology (Bunker, 1985; Barham *et al.*, 1994), to capital, to markets, to labour and labour opportunities, to knowledge, to authority; it can be facilitated by social identity, via the negotiation of other social relations (Ribot and Peluso, 2003: 165). In SIMRA, social innovation is often about changing access and control to common pool resources (i.e. disused hotels which reopen the use of abandoned villages, social agriculture which opens farming to people with a disability, terraces which are adopted or used by others).

Social innovation allows the reopening of previously closed areas/lands, changing access conditions for different groups of people or organisations. Reference is made to the Q methodology, which uses input from psychology (Stephenson, 1963) and action research to analyse subjective attitudes and perceptions that are difficult to quantify with other means (Barry and Proops, 1999; Brown, 1996; Watts and Stenner, 2012; Nijnik *et al.*, 2016). Its results are case and context specific and rewording is needed for different types of respondents at the national, subnational and local levels.

Other sources that supported the analysis of the dimension of Perceived context are: EU projects on social innovation such as SI-DRIVE (www.si-drive.eu/), TRANSIT (http://transitsocialinnovation.eu/; Wittmayer *et al.*, 2015) and CRESSI (www.sbs.ox.ac.uk/faculty-research/research-projects/creating-economic-space-social-innovation-cressi); and Stirling and Mayer, 2001; De Boer and Bressers, 2011; Bock 2012; AngloAmerican, 2012; Cajaiba-Santana, 2014; Katonáné Kovács *et al.*, 2016; and Dudwick *et al.*, 2006. More information is provided in the SIMRA Resources Web Site (see www.simra-h2020.eu/resources/).

<footnote n="42">42 https://blog.waterdiplomacy.org/blog2014/wp-content/uploads/2013/12/SESepanded.png.</footnote>
<figure><img src="image_73.png" /></figure>

Box 8 – What we evaluate for the dimension Perceived context

* The baseline for the evaluation

* Context indicators: natural, human-constructed, financial/economic, human, social and institutional/governance resources – in their multifaceted key components (see Table 8)

* Perceived Opportunities and Threats (POT): how resources are interpreted and used by the social innovator, in terms of presence/absence, access to/not access to, power on/not power on, supporting or obstructing resources

Note: the level of analysis is local to where the SI takes place

5.2.3. Agency

5.2.3.1. Agency: an overview

In the SIRMA evaluation framework, we consider agents (actors and drivers of their actions) and actions as key sub-dimensions of agency. Agency is "the actor's capacity to reinterpret and mobilize an array of resources in terms of cultural schemas other than those that initially constituted the array" (Sewell 1992: 19). Agency is also seen as "meaningful human behaviour, individual or collective, that makes a significant difference in the natural and/or social worlds, either by direct, unmediated action or through the mediation of tools, machines, dispositifs, institutions, or other affordances" (Moulaert et al. 2016: 169). In a context more specific to SI, agency "refers to the capacity of SI-agents to transformative change" (Haxeltine et al., 2016: 23), including by "modifying, eliminating or creating new institutions and eventually new social systems" (Cajaiba-Santana, 2014: 47).

Agency if referred to as including specific values, visions and trust, willingness to act, reflexivity and capacity for change,<fnref n="43" /> which influence how actors or groups of actors (agents) seek to change practices to respond to specific needs, but also their level of motivation and power to actually act and sustain their action toward specific goals (Sewell Jr., 1992; Janssen and Ostrom, 2006; Cajaiba-Santana, 2014). In this Section, interconnections amongst these concepts are briefly explained. Developing a framework based on agency and structure "requires gaining information about how agents make their decisions, how they forecast future developments, and how they remember the past" (Janssen and Ostrom, 2006), thus fitting into the typical, driving questions of an evaluation.

Agents and action

As described by Sewell (1992: 20), "agents are empowered to act with and against others by structures: they have knowledge of the schemas that inform social life and have access to some measure of human and nonhuman resources". SI-agents can include individual and collective human actors but also ideas, objects, activities, discourses and narratives of change (Haxeltine et al., 2016: 23). The "objects, activities, discourses and narratives of change" described in the Transformative Social Innovation framework (Haxeltine et al., 2016) are considered part of the action. Actors-agents embark on individual or collective actions, which may lead to either individual or societal benefits (see also Schouten et al., 2013).

We hypothesize that actions<fnref n="44" /> seeking societal (collective) benefit typically lead to social innovation. Action
refers to all those activities that social innovators (actors-agents, as individuals or group/s) may carry out for

<footnote n="43">⁴³ As explained by Caijaba-Santana (2014) in relation to social innovation, "...actors are conceived as purposeful, knowledgeable, and reflexive. [Taking from Giddens], the idea of 'reflexivity' implies that actors have the capacity to monitor routinely their actions by reflecting upon them and acting according to their intentions. Reflexivity stands for the continuous monitoring of the social context and the activities taking place within this context. Agents' actions have the power of changing institutions, but are at the same time constrained by institutional practices. The social innovation process requires attention to the individual persons; more specifically, to what they think, to what they value, to how they behave, and to how interrelations between actors and social systems take place." (p. 48)</footnote>

<footnote n="44">⁴⁴ Individual actions intended for individual benefits (e.g. internal reorganisation of a company proposed by the director) typically refer to technological innovations that may lead to societal benefits but do not include a reconfiguring of social practices and/or the engagement of civil society. As such they are outside the scope of the SIMRA evaluation framework.</footnote>
<figure><img src="image_74.png" /></figure>

preparing the process of reconfiguring. These could include conducting a context analysis of benchmarking, identifying and contacting potential partners/stakeholders, seeking legal information on options for types of business that could be created, finding consultants where required, introducing the idea in meetings, and raising awareness by communicating the idea (narrative building). These actions, connected with *ex-ante* transaction costs (Williamson, 1985: 20), are implemented to lay the ground for collective action, whereby networks, institutions and attitudes change during the process of reconfiguring.

## Characteristics of Agency

In some cases, SI agency may originate from a well-defined organisation where organisational aspects are established and well known. In other cases, agency may take a more distributed form, emerging from an informal group of people where the determinant characteristics of an organisation may be not identifiable at that time, but a network may be identifiable. In the first case, agency is consistent with, but not synonymous of, the concept of *organisation* as defined by Hodgson (2007: 8), involving: "(a) criteria to establish their boundaries and to distinguish their members from non-members, (b) principles of sovereignty concerning who is in charge, and (c) chains of command delineating responsibilities within the organisation". In the second case, the emphasis is on collective efforts for collective action, where Haxeltine et al. (2016) refer to "agency as distributed, rather than easily attributable to individual actors of groups", while Westley et al. (2013) refer to "strategic agency [as] typically not associated with just one individual, rather is produced through the strategies of a number of actors, each of whom takes actions that help the system progress through different stages of innovation and transformation (Garud and Karnoe, 2005, Hahn et al., 2006)".

Agency is determined by ideas and values (e.g. ethical, moral). In the commonly used sense, values are broad preferences concerning appropriate courses of action. They tend to influence attitudes and behaviour; they can be defined as "beliefs about what is right and wrong and what is important in life" (Oxford Online Dictionary, 2017). Values are linked to, and corroborated by, interrelated beliefs and assumptions on cause-effect relationships. Values guiding agency are "internal values", which can be coherent or contrast with the dominant external values of the society or historical period; and can be individual values or shared values within a community (e.g. if the agency is a group) (Polman, 2002). Thus, social innovation derives from, and is driven by shared ethical arenas, which are context-specific and a core aspect of social innovation. SI can also have a negative side, with trade-offs between other groups of persons (with different values). This is despite the SIMRA definition of social innovation of its seeking to enhance outcomes of societal well-being (Polman et al., 2017; D2.1).

Leadership, motivation, networks, learning, diverse cultural and knowledge systems, worldviews, values and perception, as well as self-organisation toward sustainability can be analysed as part of the agency guiding changes in practices through social innovation (Westley et al., 2013; McGinnis and Ostrom, 2014). Further, different actors and coalitions of actors play different roles. They can be active in supporting the social innovation, by connecting, pushing forward, and actively committing time; or inactive, by awaiting preliminary results or hesitating to commit for now. They may also reject the innovation and either oppose it during the reconfiguring phase, or fail to play a role in future actions. Actors who reject SI in the first phases could become active again in other phases. Roles can change in different situations and over time<fnref n="45" />. Finally, agency is not necessarily local but can rely on cross-scale and cross-level interactions with different organisations, foundation and research institutes (Cash et al., 2006; Westley et al., 2013).

<footnote n="45">⁴⁵ For the sake of simplicity, “inaction” and “rejection” are represented in the framework only in connection to the initial phase of the SI initiative (Figure 10).</footnote>
<figure><img src="image_75.png" /></figure>

5.2.3.2. Agency: Details for SIMRA evaluation

In this Section, we describe: i) who belongs to agency and what are the methods that can be used for data collection in the evaluation, depending on the type and structure of the agency, and the different aspects of agency during different SI phases<fnref n="46" />; ii) preparatory actions that agents carry out for starting an SI initiative.

*Who belongs to agency: agents/actors?*

Agency includes people/individuals who collectively share individual and collective values, ideas, willingness to act, reflexivity and capacity for change towards SI. Individuals may be living in the same community, or working in the same organisation (e.g. private/business, NGO, public-private partnership, etc.). In exploring the sub-dimensions of agency in relation to social innovation, leadership theory is of particular relevance, as leadership<fnref n="47" /> is expected to influence development paths and needs to be measured (see Kluvánková *et al*., 2017; D2.2).

For evaluation, it is assumed that four types of people or organisations interact: a) Innovators have an idea that may be visionary but not necessarily applicable in practice given prevailing conditions; b) followers decide to believe and take up the idea, and make it acceptable, feasible, and often amplify and implement it in its initial stages; c) transformers adopt the idea early on and contribute to network change and growth; and d) implementers realise and consolidate the idea. The final group can be internal or external to an organisation. These actors interact in the emerging process of SI.<fnref n="48" />,<fnref n="49" />

Innovators are key leaders as the first drivers of innovation. They are identifiable and either had the idea, invented it, discovered it or were attracted to it (e.g. a group of 4 people who had the idea of building a wind turbine in the Udny community, in UK and started the initiative). Innovators can be members of an organisation/association but act “autonomously” to start the innovation. In doing this they can change relationships within their community and respond to societal challenges (in the case of Udny, a marginalised rural area in UK, the abandonment of the community and the lack of economic resources for the well being of the same community). Innovators can be members of the local community (e.g. they belong to the same rural community and area where the social innovation ideas emerge), or be external. In the latter case, innovators leave a rural area to obtain qualifications and training, spend time away from the community, and then return , or they were new to the community and started a new initiative. Different types of local development approaches (endogenous, neo-endogenous and exogenous) can be associated with the different “origins” of the innovators (Box 9).

<footnote n="46">⁴⁶ Agency always includes individuals as actors, even when agency refers to an organisation. Depending on the phase of SI to be evaluated, the target population may be different (more details are provided in Parts III and IV).</footnote>

<footnote n="47">⁴⁷ Leadership refers to the capacity of an individual to lead collective action, be recognized and respected as a leader based on previous experiences and accountability (Westley *et al.* 2013). It can also refer also to enterpreneuership (Bund *et al.* 2015), administrative and political leadership (Bekkers *et al.* 2013) or community leadership (Ostrom, 2005) (Kluvánková *et al*., 2017; D2.2).</footnote>

<footnote n="48">⁴⁸ The dynamic is shown in a video: www.youtube.com/watch?v=RXMnDG3QzxE&feature=youtu.be</footnote>

<footnote n="49">⁴⁹ The Amoeba Lite model supported thinking about change and innovation (AtKisson Inc., 2015). The model uses the amoeba as a methaphor for innovation and change processes and identifies other types of people that may be involved in the innovation. “Laggards” delay adopting a change as long as possible; “reactionarists” resist change and innovation and are highly motivated to stop it from spreading; while “iconoclasts” raise the problems, and critique those who are perceived to be contributing to the problem; “curmudgeon” are fundamentally pessimistic about change; “recluse” hold back from engaging in the process of change. For SIMRA, only the four main ones have been explored, innovator, followers, transformers and implementers, even though we are aware of the importance of other actors as well, i.e. reactionary actors. However, the case studies might help to highlight the role of other actors. The model and other tools for sustainabilty transformation are available at: http://atkisson.com/acceleratorlite/. All material is under copyright.</footnote>
<figure><img src="image_76.png" /></figure>

Box 9 – Endogenous, neo-endogenous and exogenous development approaches: an overview

Local processes of social innovation that have a territorial rather than sectoral focus, and that occur 'from within' without policy support, are referred to as endogenous development approaches (Van der Ploeg, 1994; Shucksmith, 2000; Vázquez-Barquero, 2003). They emerge from a place and may gain support without policy backing their action, or base their actions as a response to top-down policies, as with social mobilisation. Endogenous approaches satisfy the requirement for relevancy, but may need to be evaluated for efficiency, effectiveness, impact and sustainability. Given that communities are not homogenous, other criteria would need to be evaluated, based on trade-offs in distribution of benefits. As High and Nemes (2007: 6) suggest, "endogenous development is aligned with a logic of evaluation that is embedded and inter-subjective, compatible with modes of knowing of the local heuristic system".

Shifts towards place-based policies, which support rural development, have been referred to as the New Rural Paradigm (OECD, 2006, pp. 55–78; p. 15) or neo-endogenous approaches (Ray, 2006). Ray (2000a p. 4) defines neo-endogenous development, as "...endogenous-based development in which extra-local factors are recognised and regarded as essential but which retains a belief in the potential of local areas to shape their future." Nemes analyses the synthesis in terms of integrated rural development, as a situation where the exogenous and endogenous institutions of rural development reinforce rather than oppore one another (Nemes, 2005; Nemes et al., 2006). While neo-endogenous approaches build on local resources and needs, they may need to be evaluated for relevancy, to ensure consistency between the needs the policy is meant to address and the actual impacts (Bosworth et al., 2016a; Dax et al., 2016).

Hypothetically, exogenous processes are unlikely to lead to social innovation because of their reliance on command and control approaches: "The danger is that centre-led projects can be disruptive of established economic and social relations (Beckman and Dissing, 2005), eroding cultural and natural values. There is therefore often an ongoing tension between the formal institutions of the political centre and the needs and established ways of doing things in developing regions" (High and Nemes, 2007: 3-4). Given common goals, ends may not match the means, and top-down approaches may work only if accompanied by consensus building at the local level, institutional innovation, and if addressing a significant need. As with most development interventions, the process of how an innovation is introduced is more important than the innovation itself, and innovative ways of introducing innovations should not be discounted in the initial stages of the evaluation of social innovation. Exogenous processes would need to be evaluated for relevancy, efficiency and equitability, with success stories perhaps demonstrating the presence of a neo-endogenous approach. As noted by High and Nemes (2007: 6), "The basic metaphor of much top-down, exogenous evaluation practice is one of measurement; based on the experimental methods of the natural."

It is likely that an EU evaluation of social innovation initiatives will: i) focus on neo-endogenous approaches, which support social innovation, or ii) consider the opportunities for scaling out and scaling up a local (endogenous) innovation through neo-endogenous approaches. As explained above, the only criteria to change would be relevancy (see Lowe et al., 1995).

Follower(s) are the first to adopt or support the idea of the innovator, they can be co-creators or identify a good idea and identify a practical approach to take it forward. They can be skilled at its promotion and dissemination at early stages (e.g. in the Amoeba Lite model, the latter are called "transformers" - AtKisson Inc., 2015). Together with the innovator, followers constitute the first clique and start to implement the idea, which is later consolidated by implementer(s). The clique represents the core of agency, together with the idea, values, willingness to act, capacity to act and reflexivity. Once started, transformers can expand the SI initiative and related network in its initial implementation steps, with the initiative later consolidated by the implementers. Implementers adopt the new idea later on and help to realise it in practice, spreading it outside the initial network. They are called mainstreamers in the Amoeba Lite model (AtKisson Inc., 2015).

Organisations, individually or as a group of people, can also be innovators. An organisation in itself can be the innovator, yet the current legal representative, e.g. director or president, might not be the original innovator and might not know the storyline of the innovation. The SI can be a product of interactions amongst individuals inside an organisation. The organisation can be: a company (e.g. micro-company, family company);
<figure><img src="image_77.png" /></figure>

a SME (with an internal structure, different offices and tasks, interrelated groups of persons with different tasks and competences; a recognised legal status); a NGO (with an internal structure, but different dynamics of decision-making, activities, roles and assignment), or a network (e.g. a Local Action Group within the EU LEADER Programme).

As represented in Figure 14 and explained in Section 6.2.2, SI has different phases of development, which range from the conception and formulation, to implementation and evaluation. In the initial phases those involved are mainly innovators and followers, while in the intermediate phases (i.e. implementation or practical realisation of the social innovation idea), the SI initiative involves other people and organisations.

During the process of SI development, networks amongst individuals and organisations, and the number of links (both internal links of the initial clique and network, and external links of the clique/network) are expected to increase and include the clique as well as direct beneficiaries. Indirect beneficiaries become involved in social innovation de facto (through its effects) once the SI has advanced to produce outcomes/impacts, and the initiative has expanded, consolidated and implemented with effects on those who directly partecipated in the project and on the community. On a temporal scale, the different phases of social innovation may start from a simple situation where only a few people are directly involved (clique), and evolve to a more complex situation, involving indirect beneficiaries within the community (see Figure 14).

<figure><img src="image_78.png" /><figcaption>**Figure 14.** Representation of who is involved in SI in different phases of SI, and the related type of network.
Source: SIMRA research team.</figcaption></figure>

Analysing agency also means addressing some critical challenges relating to what agency means (e.g. access to privileged conditions), what it is and how it is structured, and the ways in which agency (and agents) share ideas and engage in action. We hypothesise characteristics of innovators and followers to influence diverse pathways towards social innovation. These may include gender, age, employment and position, honorary positions in the community, whether they are directly affected by the trigger/need, internal or external residents of the territory (or internal or external organisation), internal or external to a public institution or an association/NGO (at local or higher levels), ethnicity, attitude as a leader or as a prospective leader, ethical and moral principles that inspired action, and experience in actively participating in voluntary initiatives and
<figure><img src="image_79.png" /></figure>

projects (see also Boltanski and Thévenot, 1991). Characteristics of human and social capitals are important,
including whether they have capabilities or skills as well as relations of trust. The evaluation method aims to
capture these identity, aspirations, awareness and capabilities of actors.

Preparatory Action

Action refers to all those activities that innovators, followers, and agents, may carry out at a preliminary stage to 'set the stage' for realising the social innovation in practice. It refers to all those actions that may be necessary for preparing the process of reconfiguring<fnref n="50" />. The action could be seen as the phase of 'organisational appropriation' (McAdam et al., 2001) whereby the ground is prepared for more systematic collective action. These actions do not refer to the activities related to the process of changing social practices, through collective action<fnref n="51" />.

The first step for understanding the dimension “Preparatory action” is to identify the activities carried out.
They can include benchmarking, identifying partnerships needed and contacting possible partners, seeking
legal and technical information for business and contractual agreements, communicating the idea to the local
community for improving awareness and building consensus, and screening potential consultants to be hired
for future programming and planning.

Initially, action is led by agents (individually or in a clique), while the enlargement of the network and
organisation of participatory activities occurs in the process of reconfiguring. Frequently, action is based on
the voluntary contribution of a few people interested and highly motivated in the SI idea. They may invest
their human (skills, experience, competence), social (relationships), and economic (money, time) resources in
such a preparatory phase. People living in small rural communities may have a strong cultural and personal
identity directly connected with their own territory and may want to do something to support their
community. As a result, action often refers a group of people who act for collective benefit and start the
preparation of what is needed for it to be achieved.

From the phase of action to the phase of reconfiguring, there might be a shift from an informal institution (e.g. a group of friends sharing a common goal) to a formal institution (e.g. an extended network of organisations signing a contract of collaboration). These dynamics of changing relations can be evaluated through Social Network Analysis. Action is not necessarily innovative (as the reconfiguring is) and can still fail, e.g. the SI idea may not progress because of problems of leadership, lack of time, emergence of latent conflicts and lack of resources for moving to the next step. However, as action prepares the set for the SI, it can emerge at a later stage.

Once the preliminary action is identified and described, the evaluation can also ask whether it is efficient and
effective with respect to the stated goals of the SI. Voluntary work can sometimes be very efficient because
engaged people are motivated and can act faster than in other circumstances. Figure 15 illustrates proposed
dimensions and sub-dimensions for the evaluation of the SI dimension of Preparatory action<fnref n="52" />.

<footnote n="50">⁵⁰ In the initial phase relationships may start to change and adaptations of governance arrangements may be possible, and attitudes of engaged innovators and followers can change. The dimension ‘Preparatory action’ and that of ‘Reconfiguring and reconfigured social practices’ partially overlap. They are kept separated for the purpose of evaluation, but in real life they are interconnected.</footnote>

<footnote n="51">⁵¹ In the evaluation framework, the latter are summed up in the SI dimension ‘Project activities’.</footnote>

<footnote n="52">⁵² The types of activities will be refined on the basis of the pilot test results, and complemented by the results of the SIMRA case studies evaluation. The aim is to collect a wide range of examples to guide future evaluations of SI in MRAs.</footnote>
<figure><img src="image_80.png" /></figure>

<figure><img src="image_81.png" /><figcaption>**Figure 15.** Proposed dimensions and sub-dimensions for the evaluation of the SI dimension ‘Preparatory action’. Source: SIMRA research team.</figcaption></figure>

**Box 10 – Main methods that inspired the development of the SIMRA method for addressing the dimension Agency**

In the survey of existing methods, we drew principally from the following work: Dawson and Daniels (2010), De Boer and Bressers (2011), Young (2011), Banerjee et al. (2013), Cajaiba-Santana (2014), the EU projects BENISI and CRESSI, De Vries (2015), Bock (2016), Dietrich et al. (2016), Neumeier (2017) and Windrum 2016. More information is provided in the SIMRA Resources Web Site (see www.simra-h2020.eu/resources/).

**Box 11 – What we evaluate for the dimension Agency**

* The initiation of the social innovation process

* Innovators and followers (leadership): who had the idea?, what were the values driving the idea?, who had the capacity to act for change?, who were the opponents? (i.e. leaders with different ideas that did not want to participate in the development or implementation of the idea or who resisted to the idea)

* Individual and collective characteristics of people involved in the social innovation (e.g. age, gender, education, motivation, reputation, job position)

* The process in terms of who is involved in its initial phases and its outcomes/impacts in terms of the direct and indirect beneficiaries (who belong to which network?)

* Networks: who is doing what (network and network of networks)?, who had the idea (individuals)?, who are the direct and indirect beneficiaries (outcome, impact)?

* Motivations for action
<figure><img src="image_82.png" /></figure>

* Contributions of the individuals and clique engaged initially (e.g. voluntary work, economic resources, skills)

* Identification and analysis of similar initiatives (benchmarking, learning from best practices)

* Idea formulation and initial external communication (value sharing, defining the vision, refining the message, writing the concept note and/or draft MoU)

* Allocation/distribution of tasks and responsibilities

* Planning of future steps

* Instruments and strategies for the identification of partners and stakeholders

* Mechanisms for convincing potential partners to join the initiative

* Organisation of preliminary informal meetings and other communication tools

* Methods designed for different target populations (innovators/followers/network/community) and different phases of social innovation

### 5.2.4. Reconfiguring and reconfigured social practices

#### 5.2.4.1. Reconfiguring and reconfigured social practices: an overview

Preliminary actions led by agents within a certain context lead to a *process of reconfiguring* in (i) governance arrangements; (ii) networks (social relationships, e.g. among public and private actors, formal and informal); and (iii) attitudes (connected to beliefs and values), i.e. in social and institutional practices. Social practices "refer to everyday practices and the way they are typically and habitually performed in (much of) a society" (Holtz, 2014: 1). They are 'types' of behaviour and understanding that are similar for different individuals at different points of time and locations (Reckwitz, 2002: 250). The process of reconfiguring leads to reconfigured social and institutional practices (i.e. new governance arrangements, new networks and new attitudes, in different combinations).

It is proposed to distinguish between the two phases of “reconfiguring” and “reconfigured” because during the reconfiguring phase it is possible to identify the factors that support, hinder or interrupt the emergence of SI. The interruption can occur at any stage since conception of the idea. It is more visible if it occurs during the action and/or the reconfiguring phase. In particular, if the interruption occurs in the action phase, and action remains action (i.e. no reconfiguring follows), then SI does not happen. Figure 12 represents this possibility by the break line between the agency and SI boxes.

If action leads to SI, then agency remains a key element of SI (and the 2 boxes “agency” and “SI” have to be considered as a unique box). The reconfiguration can occur at any levels along spatial and/or socio-economic scales: it might be well-defined at the local level, but less evident and more distributed at higher levels. During these phases, it is possible to identify what is reconfiguring, who is involved, how the process is carried, where and for how long.

As described in Howaldt et al. (2015: 31), social innovation can be “interpreted as a process of collective creation in which the members of a certain collective unit learn, invent and lay out new rules for the social game of collaboration and of conflict or, in a word, a new social practice, and in this process, they acquire the necessary cognitive, rational and organisational skills. Social innovation encompasses new practices (concepts, policy instruments, new forms of cooperation and organisation), methods, processes and regulations that are developed and/or adopted by citizens, customers and politicians, in order to meet social demands and to resolve societal challenges in a better way than existing practices.”

Governance or institutional arrangements refer to ways of implementing and operationalising the 'rules of the game' of decision-making process (how decisions are taken and by whom; how they are implemented and by whom) (Ménard, 1995; Polman, 2002), including implementation and empowerment (Kjær, 2004). In the evaluation framework, changes in governance arrangements refer to formal institutions (as policies, laws, regulations, guidelines, codes, standards), as change and adaptation of governance and institutional arrangements in relation to the role of public entities and authorities in facilitating social innovation (both internally and externally).