<figure><img src="image_83.png" /></figure>

Networks can include “rules” in terms of forms of interaction amongst actors. These forms of interaction include different forms of relationship (e.g. formal/informal, inclusive/exclusive, economic/ social). Changes in networks can be explored by looking at the: i) composition of the network; ii) structure of the network; iii) relationships; iv) dynamism of boundaries/participation; and v) coordination. The boundaries of networks, rules of (internal) governance arrangements, participation in decision-making, coordination mechanisms and multiple layers of nested enterprises are essential in designing self-organised robust institutions of relevance in marginalised rural areas, such as forest commons (e.g. Kluvánková and Gezik, 2016). In networks, both public and different types of private actors and organisations are involved, thus including both civil society representatives, such as NGOs, and the market, i.e. businesses and entrepreneurs.

Attitudes are “social rules” in the broad meaning of informal institutions, beliefs, values and discourses (North, 1990). They are connected with culture. For ease of analysis, in the evaluation framework formal institutions are included in the “governance arrangements”, and informal institutions “customs, beliefs, (social) norms, values, historical experiences, ...” are included under “attitudes” to highlight their role in social innovation (Sabatier, 1988; Pascual *et al.*, 2017). The three sub-dimensions of social practices (new governance arrangements, new networks and new attitudes) are considered as interactive elements of the same reconfiguring process and mutually influence each other. For example, a new network may imply new attitudes of actors and require new governance arrangements for managing internal and external relations.

### 5.2.4.2 Reconfiguring and reconfigured social practices: details for SIMRA evaluation

Reconfiguring is “the” key term in the SIMRA definition of SI. Reconfiguring (and the consequent reconfigured) represents the “black box” of the social innovation concept. The term refers to changes in social practices, i.e. ‘new rules of the social game’, which are represented in the evaluation framework by: a) governance arrangements, b) networks, and c) attitudes.

The term ‘social practices’ includes them all as key elements of SI in marginalised rural areas as well as in other contexts. From an evaluative point of view, the issues is one of the relative importance of these three components, i.e. whether one can be considered more important that the others in shaping the process of reconfiguring and, ultimately, in leading to social innovation. A further consideration is whether it was possible to find variables and data that relate to each of the three components, thus whether data can be easily identified, collected and studied to understand the SI process and the complex interactions and changes it implies.

Our initial assumption is that the three components of social practices should have the same importance within the evaluation. This is due to the prospect of finding appropriate variables and data for indicators to be calculated for some of these practices. For example, it is easier to identify variables and indicators on networks than on governance arrangements and/or attitudes<fnref n="53" />,<fnref n="54" />. Figure 16 shows the likely imbalance in understanding all three social practices (equally important), and the facility to identify significant variables and data to calculate indicators (possibly higher for networks, medium for governance arrangements and lower for attitudes).

<footnote n="53">⁵³ Networks can be measured more easily, but this should not impede the capturing and measuring of the other two components. We assume these three social practices are always co-existent and interact reciprocally. A change in networks will probably induce changes in attitudes (e.g. working in a network can increase collaborative attitudes) and, vice versa, acquiring more collaborative attitudes because of increased trust may lead to creating new networks. A change in attitudes can lead to new governance arrangements (e.g. a more open public sector is likely to lead to higher exchange of information). New governance arrangements can encourage network creation, and so on (in a potentially expanding “virtuous cycle”).</footnote>

<footnote n="54">⁵⁴ See additional material on existing methods provided online at www.simra-h2020.eu/resources/.</footnote>
<figure><img src="image_84.png" /></figure>

<figure><img src="image_85.png" /><figcaption>**Figure 16.** The ‘black box’ of SI in MRAs: reconfiguring and reconfigured social practices with respect to evaluation issues. Source: SIMRA research team.</figcaption></figure>

As the ‘core’ of social innovation, the evaluation of the reconfiguring process of social practices (and of the consequent reconfigured situation) has to be accurate and detailed, even if it refers to several complex aspects. Consideration of ‘who’ is important. This is in terms of: who is involved?, which represents the nodes of the network. nodes can be individuals, groups or organisations. Who is involved determines the quality of relationships, the level of trust, the intensity of the links and exchange flows of information or resources.

The time taken for the reconfiguring process to reach a reconfigured practice/s is another important consideration. This is functional to identifying the variables and mechanisms that determined the length of the process, and whether it was shorter or longer. Faster processes may be connected to types of local links with neighbours, and/or simple and user-friedly governance arrangements that speed up possible bureaucratic processes. Slower processes may be connected to a high number of network participants, unclear and complex interactions with higher jurisdictional or administrative levels, or hidden conflicts.

Where the process happens is another key question. This can help in identifying the variables that determine diverging paths of the development of SI. For example, a specific site or location can be an obstacle or a facilitating factor, due to the biophysical and socio-economic contexts. For reconfigured, the key questions are similar, as the approach is to understand the social practices before (the reconfiguring) and after (the reconfigured) the process is completed. In addition, it is important to explore which instruments are used for establishing the reconfigured social practices (e.g. whether formal types of contracts, and/or informal verbal agreements on how to proceed). The social practices of the reconfiguration process can be compared to the social practices of the reconfigured situation<fnref n="55" />, e.g. whether a network has increased or decreased, or how the

<footnote n="55">⁵⁵ The need to explore these aspects in detail is connected with the need to understand the key factors in a specific marginalised rural area/case study that support, hinder or interrupt the social innovation. Ideally, successful and</footnote>
<figure><img src="image_86.png" /></figure>

structure has changed in term of composition. How the reconfigured practice is functioning, with respect to the instruments and/or procedures (e.g. which internal rules and mechanisms, what communication tools, what procedures for taking the decisions), and where it is located (e.g. where a new network was established), are all important.

Box 12 – What we evaluate for the dimensions of Reconfiguring and Reconfigured

* The reconfiguring process of social practices and the reconfigured social practices

* Specifically, for the reconfiguring: what social practices?, who is in?, how it is carried out?, where it is carried out?, how long was the process?

* Specifically, for the reconfigured: the same as above, and in addition, which instruments are used for the formalization?, how it is functioning? and where it is located?

* Governance arrangements, networks and attitudes before and after the process

* Insights on the process of shifting from an initial to a new condition

New Governance Arrangements

In the proposed framework, governance⁵⁶ arrangements refer to institutional arrangements, according to the meaning of the term suggested by Polman (2002) and others⁵⁷. New governance arrangements are procedural and/or decisional changes and adaptations, which appear in relation to new roles of public entities/authorities as supporters of SI (e.g. change in an administrative procedure or in a phase of decision making process to support the action of the agency at local level, thus determining a change, i.e. reconfiguring, the governance arrangements). They refer to what public authorities do (change/adapt) when involved in reconfiguring practices, both as external or internal nodes of the network.

"Adequate governance" is reported as one of the main components that contribute to "create the 'natural environment' for social innovation to flourish" (BEPA 2013: 42), together with supportive policies, innovative finance, capacity building and recognition tools such as incubators and hubs. Key elements to be explored are the coordination mechanisms used in and by the SI initiative, the length of standard setting processes, the public procurement of innovation, technological and organisational innovations (BEPA, 2013). Coordination is particularly relevant (Kluvánková et al., 2017; D2.2). However, governance arrangements that are identified for evaluation have to be clearly differentiated from: i) trigger (e.g. different from the restructured governance structures or abolishment of laws for environment protection in the USA by President Trump); ii) perceived context (i.e. different from the existing legal framework that may allow or limit SI); iii) institutional/governance outcome/impact (i.e. different from a policy reform approved by means of a new law driven by social movement, e.g. law on social farming approved at national level in Italy); iv) governance elements that belong to networks (i.e. public actors that are inside the network).

interrupted SI cases would be evaluated, the latter not necessarily a SI "failure". However, with ongoing SI, evaluation might be difficult: actors/agents previously in the SI may be less likely to participate or spend time in long interviews. The implications require consideration with WPs 5 and 3 for the pilot test phase of SIMRA evaluation and CSs final selection.

⁵⁶ The term governance has two main meanings: broader as “every mode of political steering involving public and private actors, including traditional modes of government and different types of steering from hierarchical imposition to sheer information measures”, or narrower as “types of political steering in which non-hierarchical modes of guidance, such as persuasion and negotiation, are employed, and/or public and private actors are engaged in policy formulation” (Héritier 2002 - cit. by Treib et al. 2007). The framework does not exclude governance arrangements connected to hierarchical-based models. However, in most SI the narrower meaning of governance is expected to apply. The focus is on public-private interactions, e.g. in relation to special roles of public actors and institutions in defining the 'rules of the game'.

⁵⁷ Polman (2002) notes discussion by Ménard, (1995: 175) that an institutional arrangement is a way to implement and operationalise 'rules of the game'. Institutional environments define (or constrain) the environment of institutional arrangements (cf. Williamson 1996: 5). An institutional arrangement is between economic units that govern ways units can co-operate and/or compete. The arrangement may be formal or informal, temporary or long-lasting. Arrangements may involve individuals, groups of individuals co-operating, or government (alone or cooperating with others). Governance arrangements refer to “operational” for setting of rules, implementation of rules and empowerment of rules (Kjær, 2004). i.e. how decisions are taken, by whom, using which procedures and mechanisms, with which instruments decisions are implemented, who is in charge of monitoring, controlling and sanctioning, and how controlling is performed.
<figure><img src="image_87.png" /></figure>

Finally, in addition to the description of governance arrangements, “good governance” principles are also important. However, rather than assessing the quality of governance with respect to these good governance principles, which is outside the scope of SIMRA and would imply a common definition of what is ‘good’ in terms of governance in very different contexts (e.g. EU and North Africa), the evaluation questions seek to verify whether SI has positive or negative, intended or unintended impacts on one or more of the most frequently adopted good governance principles (e.g. efficiency, effectiveness, accountability, transparency, capacity, participation, sustainability).

## New networks

Over recent decades, agriculture, forestry and rural development programmes and initiatives have led to the active participation of heterogeneous types of stakeholders such as: public institutions (communitarian, national and regional authorities, and local territorial entities), entrepreneurial actors (companies and SMEs working in different economic sectors non exclusively related to agriculture), social and environmental organisations (associations, civil society organisations, and individual citizens). Networks are increasingly forming with policy makers, stakeholders and practitioners. Networks and related public-private partnerships or other types of inter-sectoral forms of aggregation are the basis of several EU programmes and initiatives. In the context of rural development, for example, the Local Action Groups (LAGs) within the EU LEADER programme are typically public-private inter-sectoral networks. Similarly, the Operational Groups of the European Innovation Partnership for Agricultural productivity and Sustainability (EIP-AGRI)58 are networks of several partners working in the agricultural sector (e.g. farmers, scientists, agri-business). Networks are also increasingly supported at the national level. In Italy, for example, a new legal instrument to specifically facilitate the creation of networks among enterprises (“network contract”)59 was introduced in 2009. Networks increasingly attract the scientific community, and there are consolidated methodologies (e.g. Social Network Analysis) and open-source software for data processing and analysis (e.g. UCINET) for exploring them, both in qualitative and quantitative ways. SNA and related methods are extensively used in the fields of natural resources management, agriculture, forestry and rural development (e.g. Bodin and Crona, 2009; Secco et al. 2014; Hauck et al. 2016; Pisani et al., in print). In social innovation, networks have a prominent role (see also Kluvánková et al., 2017; D2.2).

Different aspects of networks can be explored: i) composition of the network; ii) structure of the network; iii) relationships. Figure 17 illustrates the specific components of networks used in the SIMRA evaluation framework, considered relevant for SI.

<footnote n="58">58 EIP-AGRI was launched in 2012 to contribute to the EU's strategy 'Europe2020' for smart, sustainable and inclusive growth. More information at URL: https://ec.europa.eu/eip/agriculture/en/about</footnote>

<footnote n="59">59 This type of contractual agreement, established by National Law n. 5/2009, can be created amongst enterprises in different economic sectors, is more flexible and dynamic compared to existing formal collaborations (e.g. consortia or cooperatives). The law provides fiscal facilities and procedural simplifications for network contracts where enterprises in agriculture are involved. Business interest is demonstrated by the growth in the number of network contracts from 25 (in 2010) to c.2,700 (2016; Registroimprese, 2017). The instrument is becoming popular in agricultural and forest sectors, and in other rural development-related activities (e.g. rural tourism, such as bike renting in rural areas, social farmings, local agencies that promote eno-gastronomic events) are have one registered contract. (Secco and Pisani, 2016).</footnote>
<figure><img src="image_88.png" /></figure>

<figure><img src="image_89.png" /><figcaption>**Figure 17.** Components and sub-components to be analysed regarding networks in the reconfiguring process.
Source: SIMRA research team.</figcaption></figure>

The basic units of a network are at least two connected individuals. Individuals can be aggregated in groups, sharing the same interests and working for a common objective. Groups that are already networks can be aggregated into larger networks. Smaller and larger networks differ in their nodes: while in the first case nodes are typically individuals, in the second case, they are organisations (in which vision/approach/interests may be "represented", for example in public meetings, by individuals). For the purposes of evaluation, we can refer to the well-known concept of stakeholder<fnref n="60" />, to include both individuals or those who are invested in the organisation (such as employees and shareholders – internal stakeholders), as well as those who have other relationships to the organisation (such as workers who are not employees, suppliers, vulnerable groups, local communities, and NGOs or other civil society organisations, among others – external stakeholders) (GSSB, 2016: 28). In terms of composition of the network, involved stakeholders can belong to different sectors (e.g. multiple economic sectors, third sector) and/or to different institutional levels (multiple levels) (e.g. a regional public authority, a local farmer).

In terms of openness<fnref n="61" /> of the network, various situations are foreseen. For example, a network defined by the clique during the action is open to other new members, who can join it in the process of reconfiguring (thus

<footnote n="60">⁶⁰ Stakeholders can be defined as an “entity or individual that can reasonably by expected to be significantly affected by the organisation’s activities, products or services [in our case, by the SI initiative], or whose actions can reasonably be expected to affect the ability of the organisation [in our case, of the SI] to successfully implement its strategies and achieve its objectives” (GSSB, 2016: 28).</footnote>

<footnote n="61">⁶¹ In rural contexts, networks can be closed and be exclusive for members. e.g. in Italy, the “Regole” are community-based ancient institutions whose members can only be from families which have been local residents for centuries.</footnote>
<figure><img src="image_90.png" /></figure>

influencing it), or a network is completely fluid and fully based on participation of anyone interested in joining; or a network is created but the rules for accessing it (or withdrawing) are strict and thus participation requires a high commitment.

In terms of structure, networks can be analysed as a whole (e.g. in terms of density, centralisation (hierarchical/decentralised), isolated nodes, connectivity, legal status or organisational structure of the network), in relation to the role of nodes (e.g. centrality, brokerage; reputational power of the single nodes; legal status of the single nodes of the network) and/or in relation to the sub-groups that co-exist in the network (e.g. clique is already a network, core/periphery, ego-network) (Borgatti et al., 2013).

Finally, networks can be explored in terms of relationships, both informal and formal. Formal relationships include institutional/institutionalised agreements with public actors involved in the social practice "network. The first includes options such as oral agreements, participation in the same events (e.g. if we both participate to an event, then we talked, we have the same interests and motivation – objective, participate or not participate), social norms, relational roles (e.g. kinship, friendship, professional between colleagues), and relational cognition (affective, perceptual).

## New attitudes

A useful definition of attitude is the one provided by Gobattoni et al. (2015): "Attitude, as defined by Ajzen (2001) is an evaluative judgement of an object that can be considered as good-bad, harmful-beneficial, pleasant-unpleasant, likeable-dislikeable: this judgement is based on the subjective beliefs we form about that object, but it is also influenced by feeling states and emotions (Agarwal and Malhotra, 2005). Attitude is affected by the socio-economic context in which people live and, at the same time, attitudes are significant for understanding and predicting social behaviour (Ajzen, 2001). To this aim, it becomes really useful to understand the attitude of a community towards traditional activities in order to discern the drivers that guide people's choices in remaining in the place where they live and in adopting a sustainable life style." (Gobattoni et al., 2015: 414). Attitudes are somehow connected with beliefs. These concepts are very complex, as they deal with the most profound and personal characteristics of each of us, both individually and as members of collective entities (families, groups, associations, organisations, institutions, parties, etc.). They should be studied and explained in detail by means of philosophical and psychological approaches.

For the sake of simplicity, we selected a few concepts to be explored as proxy for such a complex issue. This simplification is necessary for a specific evaluative need: to identify amenable/measurable variables that can describe attitudes and beliefs of individuals and groups involved in the SI initiative. In the framework, we adopt the definition of beliefs taken from the Stanford Encyclopaedia of Philosophy which defines them as "What one person doubts or hopes, another might fear, or believe, or desire, or intend—different attitudes, all toward the same proposition" or more in general belief are whenever we take something to be the case or regard it as true. Most contemporary philosophers characterise belief as a "propositional attitude": "the mental state of having some attitude, stance, take, or opinion about a proposition or about the potential state of affairs in which that proposition is true—a mental state of the sort canonically expressible in the form "S A that P", where S picks out the individual possessing the mental state, A picks out the attitude, and P is a sentence expressing a proposition<fnref n="62" />. This corresponds to a representational structure of beliefs that we have used in order to assess information institutions. The belief can be on any kind of proposition or content (e.g. how our vision has changed compared to five years ago or in the case of SIMRA "Do you believe that "you live in a marginalised rural area"?"). Its strength can be assessed through different verbs or sentences (as: I believe, I hope, I fear, I doubt). Our interpretation of belief, for evaluation, is therefore a sum of expectations on the project, individual attitude, collective attitude and objective obstacles. To better explain the approach used in evaluating this specific sub-dimension, such components are described in Table 9.

<footnote n="62">⁶² For example: Ahmed [the subject] hopes [the attitude] that Alpha Centauri hosts intelligent life [the proposition], or Yifeng [the subject] doubts [the attitude] that New York City will exist in four hundred years" (Schwitzgebel, 2015).</footnote>
<figure><img src="image_91.png" /></figure>

<table><caption>**Table 9.** Beliefs, features and questions as a basis of the evaluation.</caption><thead><tr><td>Beliefs</td><td>Features</td><td>Possible questions at the basis of evaluation</td></tr></thead><tbody><tr><td>Expectations of the project</td><td>Not alterable over a short term</td><td>Which are your expectations of the project?<br/>Do you hope that ... ?</td></tr><tr><td>Individual attitudes</td><td>Not alterable over the medium term</td><td>Do you think you are a positive person?</td></tr><tr><td>Collective attitudes</td><td>Alterable over a short term</td><td>Do you believe that the common project will deliver as expected? (sum of individual answers)</td></tr><tr><td>Objective obstacles</td><td>-</td><td>Are things going well, as expected and planned?</td></tr></tbody></table>

Source: SIMRA research team.

Other concepts are connected to attitudes. Shared or individual ethical/moral values influence attitudes. Depending on the socio-economic and cultural contexts, on the historical development trajectories and also on contingent conditions, personal and collective ethical/moral values are different. Therefore, they are necessarily case-specific and *momentum*-specific, and are also critical from the point of view of evaluation. For example, these values can (or not) include the idea that e.g. social inclusion is important, climate change is a priority in the political agenda, sustainability is a must, governments should address social justice and equity, human rights have to be respected, relations are key to happiness, migrants are welcome, women are resources for the society and economy of a country ("meaningfulness of relationships", as suggested by Pascual *et al.* 2017: 15). These values are different from person to person, from place to place, from country to country and from historical period to historical period, and are typically confidential<fnref n="63" />.

Attitudes can be personal or collective, in relation to one's networks and/or the whole society. New attitudes can also be influenced by dominant or emerging discourses<fnref n="64" />, both through public opinion or that of leaders or innovators. Individual attitudes can be influenced by dominant institutions, including informal ones, e.g. social norms that guide the relationships and behaviours within a certain community (North, 1990)<fnref n="65" />. The level of commitment by individuals or groups towards new attitudes can fluctuate: the higher the level of commitment, the higher the possibility that "deep normative core" beliefs<fnref n="66" /> change; in certain cases, the shift towards new attitudes is due to monitoring systems and sanctions that may also be social-based, as in the case of robust forest commons regime, e.g. Kluvánková and Gezik, 2016).

Figure 18 illustrates the complexity of attitude with respect to SI, and the challenge of evaluating the different components.

<footnote n="63">⁶³ A challenge to questions on ethical/moral values in interviews for evaluation is that people may not disclose personal values (e.g. political party), and/or they respond differently to what they believe, with potential risks of bias.</footnote>

<footnote n="64">⁶⁴ Discourses can be seen as "an ensemble of ideas, concepts, and categorisations that are produced, reproduced, and transformed in a particular set of practices and through which meaning is given to physical and social realities" (Hajer, 1993: 8-9). Discourses relate to more than language and communication, contributing to societal constructions of reality (Keller, 2006).</footnote>

<footnote n="65">⁶⁵ In rural contexts, see for example Kluvánková and Gezik (2016) in relation to robust forest commons regimes, and Górriz-Mifsud *et al.* (2016) in relation to forest governance and social capital.</footnote>

<footnote n="66">⁶⁶ "Deep normative core beliefs" are difficult to change even over the long term, while "secondary agents" can be easily changed due to contingent interests in creating a coalition to pursue common objectives, according to the Beliefs Systems model of Sabatier (1988). "Near policy core" are in an intermediate position.</footnote>
<figure><img src="image_92.png" /></figure>

<figure><img src="image_93.png" /><figcaption>**Figure 18.** Components of the SI sub-dimension of reconfiguring social practices “New attitudes”. Source: SIMRA research team.</figcaption></figure>

**Box 13 Main methods that inspired the development of the SIMRA method for addressing the dimension of Reconfiguring and Reconfigured social practices**

In the survey of methods to assess reconfiguring and reconfigured social practices, we referred to a number of sources: Sabatier (1988), Knack (2002), Polman (2002), McIntyre (2008), Sørensen and Torfing (2009), Soma (2010), Seraj (2012), BEPA (2013), Soma and Vant (2014), Gobattoni et al. (2015), Malek and Costa (2015), McLachlan et al. (2015), Pradel-Miquel (2015), Dhondt et al. (2016), Soma (2010), Merad et al. (2013), Neumeier (2017), Kluvánková and Gezik (2016), Pascual et al. (2017), Schwitzgebel (2015). EU projects of relevance were SIMPACT, while for the Social Network Analysis, a recent example is provided in Pisani et al. (in print). More information is provided in the SIMRA Resources Web Site (see www.simra-h2020.eu/resources/).

### 5.2.5. Project activities

#### 5.2.5.1. Project activities: an overview

The process of reconfiguring leads to the reconfiguration of new activities, procedures and practices that are needed to implement and realise in practice the SI project/s deriving from the SI idea. Project activities describes the actions that characterise the new reconfigured phase (e.g. communication of SI), procedures detail how the activities should be carried out (e.g. who should communicate what to whom), while practices refer to how procedures are implemented in practice (e.g. when the news about the main outputs of the SI project should be circulated and by means of which communication channels). In short, project activities and related procedures and practices refer to what is done in the reconfigured social practices<fnref n="67" /> once the SI initiative has become operative through SI projects, to act regularly and have outputs, outcomes and impacts.

<footnote n="67">⁶⁷ As detailed in Polman et al. (2017; D2.1), reconfigured practices and activities can include new investment activities (McGinnis and Ostrom, 2014); monitoring activities (Olsson et al., 2006); resource dependency in organisations and networks (Bekkers et al., 2013; Krlev et al., 2014); allocation of resources (Bekkers et al., 2013); incubation period (Biggs et al., 2010); networking and lobbying activities (McGinnis and Ostrom, 2014; Biggs et al., 2010); and novel property rights and regimes (Ostrom, 2009; Vatn, 2005). For evaluation, activities likely to be in initial, preparatory steps of a SI initiative (SI dimension “Preparatory action” in the proposed evaluation framework) are kept separate from those likely to occur in a more mature phase of the SI initiative, when SI projects are implemented (SI dimension “Project activities”). However, many such activities are carried out throughout the SI process and its implementation phases.</footnote>
<figure><img src="image_94.png" /></figure>

Project activities are located in a different temporal phase with respect to the preparatory actions described in Section 5.2.3.2, and are somewhat different in essence and quality. These activities implement the SI in a reconfigured situation of social practices, and not for preparing SI before the reconfiguring takes place. For example, lobbying can play a relevant role in the preparatory action phase and a secondary role in the project activities phase<fnref n="68" />.

#### 5.2.5.2. Project activities: details for SIMRA evaluation

The term “Project activities”, used in the evaluation framework, encompasses not only activities per se but procedures and practices too. The term refers to all those daily working tasks, procedures and practical solutions that are implemented for managing the SI initiative in its implementation phases (i.e. the reconfigured social practices), when SI projects are put in place<fnref n="69" />. Project activities focus on SI project management and planning issues, i.e. what is done on the SI project, by whom, how and by means of which instruments, in general terms. Project activities can become recurrent, when the SI project is consolidated. Both the flows of activities and the tools used are important elements to understand in the analysis, as they can directly or indirectly affect the performance and capacity of the SI initiative in implementing its SI project/s and getting expected outcomes/impacts.

Types of project activities include: organising ad hoc training sessions for SI partners on new communication tools, periodically arranging meetings for sharing information about advancements and future projects, managing the internal decision making process, paying salaries, selecting suppliers and managing procurements, keeping in contact with local authorities not directly involved in the SI, fundraising (e.g. to design and implement a crowdfunding), managing human resources, planning future investments, searching for educational and training needs with respect to the SI objectives and projects, managing, updating and archiving documents, replying to requests of clients or inhabitants interested in the SI project, monitoring the SI project advancement and changing plans if necessary, evaluating projects' effectiveness and other relevant aspects. Project activities include all the aspects needed to manage the SI projects as defined by the reconfigured social practices.

Procedures specify how activities should be carried out, within or in relation to the external contexts by the SI initiative, in the ordinary working and/or relational activities. Procedures can exist in written form, or be simply agreed orally and adopted by everyone in their work. They also imply that different persons, with different roles and responsibilities, perform different types of tasks and are engaged in different ways in the SI management (in business, this would be managed by means, for example, of organisational flow charts).

Practices refer to the practical implementation of procedures (informally or formally). Practices may differ from procedures, since what is done in practice may be different from what should be done according to the procedures.

**Box 14 – What we evaluate when analysing the dimension Activities, procedures and practices**

* How the initiative planned for its development

* How human resources are managed (e.g. law compliance, training, equity on merits)

* How financial resources are identified, collected and managed (e.g. crowd-funding, fundraising, budget allocation, etc.)

* How infrastructural resources are managed (e.g. work places, facilities, transports)

* How external interactions are managed (e.g. communication, marketing)

* How administration is managed (e.g. administration, cloud services, accounting)

<footnote n="68">⁶⁸ The literature on social enterprises and entrepreneurship explores one means by which the third sector fosters social innovation through engagement in socially-oriented business practices (Bund et al., 2015; Krlev et al., 2014). Social entrepreneurship is a key factor for SI in marginalised rural areas (see Kluvánková et al., 2017; D2.2).</footnote>

<footnote n="69">⁶⁹ Fore more details on the evaluation approach in relation to SI process and SI project see Part III.</footnote>
<figure><img src="image_95.png" /></figure>

### 5.2.6. Outputs

#### 5.2.6.1. Outputs: an overview

Outputs are often considered together with outcomes and impacts. We consider SI to include: i) agency; ii) action; iii) a process of reconfiguring; iv) reconfigured social practices; and (v) consequent project activities, these are not the end of the model but focus on SI as a process. This is in line with Cajaiba-Santana's affirmation that, "what underlies the path of social innovation is not a social problem to be solved, but the social change it brings about" (2014: 44). Reconfigured social practices, activities and procedures lead to outputs, which eventually lead to outcomes and impacts. Outputs are the first, immediate results<sup>70</sup> in terms of products, services, capacities, that are delivered by and derive from SI, and in particular from SI projects that are implemented as a consequence of the SI process. They are identifiable and quantifiable, often tangible, and refer to the creation of opportunities for changes in interactions and behaviour. They are results of the SI initiative and within the control of the SI implementation agency.

#### 5.2.6.2. Outputs: details for the purposes of SIMRA evaluation

Outputs are immediate<sup>71</sup> identifiable results. They can be tangible or intangible, but have to be quantifiable and measurable. Products and services are typically included in the first category, and information, collaboration, ideas for new projects, and the creation of new relationships in the second category. Key questions on outputs are about the types of products and services provided (what is provided by SI projects?), e.g. inclusion of disabled people in a farm's activity, educational sessions on transition to low carbon energy systems or organic farming, e-marketing tools, communication services.

Another point to be explored is who are the direct beneficiaries of the outputs? (e.g. final users such as target vulnerable groups). Outputs should be able to satisfy the initial needs, as these are the reason why agency first started the SI reconfiguring process. These are related to the relevance of SI initiative. The key evaluative question could be: "Have the SI outputs addressed the initial social needs, societal challenges or governance shifts that characterised the marginalised rural area?" Here, the initial social needs, societal challenges or governance shifts are those that determined the SI to emerge in the marginalised rural area. This question allows the application of some traditional evaluation criteria, in terms of relevance and effectiveness of the SI initiative.

**Box 15 – What we evaluate for the dimension Outputs**

* Products and services (identifiable, both tangible and intangible)

* Beneficiaries of the SI outputs

* Satisfaction with respect to the individual, clique and community needs

* Relevance and effectiveness of the SI initiative

### 5.2.7. Outcomes/impacts

#### 5.2.7.1. Outcomes/impacts: an overview

Outputs of the reconfigured practices, activities and procedures of the social innovation initiative eventually lead to outcomes and impacts. Outcomes are effects of SI at the level of beneficiaries that often emerge over the mid-term, and become visible (i.e. measureable) between 18 and 36 months or 18 to 60 months. However, this is not only a matter of temporal scale. Rather, it is a matter of affected population: outcomes are the effects that affect 'direct beneficiaries' of the SI initiative. Direct beneficiaries are those on whom the effects of SI project(s) are targeted; they are primarily affected by the social needs, societal challenges and/or governance shifts that were at the basis of the SI, and the reasons for the SI to emerge.

<footnote n="70"><sup>70</sup> In business sciences, it refers to products or services that are delivered in the short-term (1 to 18 months).</footnote>

<footnote n="71"><sup>71</sup> In business economics, outputs are often considered immediate tangible results that become visible in the short-term (e.g. 18 months) (Rane and Deorukhkar, 2007). Another option is to refer to outputs that derive from the "first season" of activity (thus the length varies depending on the type of activity: farming, nursery, livestock, ...).</footnote>
<figure><img src="image_96.png" /></figure>

Outcomes are behavioural changes, both intended and unintended, positive and negative (Binnendijk, 2000), that produce new routines, decisions, rules and institutions. The outcomes of SI can have a broader effect than in the original rural site of emergence, shifting social arrangements and institutional relations and values at higher levels as well (e.g. national policy reform, regional or national reconfiguration of social services). In this case, when they affect 'indirect beneficiaries' of the SI initiative, they shift into the category of impacts.

Impacts are typically long-term and widespread effects of SI at the rural territory level. They should be positive, but there can be negative impacts. These are considered to be long-term changes that result from an accumulation of outcomes, leading to concrete benefits for the communities in which SI has took place (e.g. creation of goods, improved access conditions) and that have multiple drivers. In marginalised rural areas, they can be related to the enhancement of social cohesion and human well-being, competitiveness, self-organisation and resilience, higher education and skills (see Polman et al., 2017; D2.1). Outcomes and impacts of SI can be on four main domains: economy, society, environment and/or governance/institutions.

#### 5.2.7.2. Outcomes/impacts: details for SIMRA evaluation

Outcomes/impacts belong to the same category of effects of the SI, but for different groups: while outcomes affect direct beneficiaries, impacts have effects on both direct and indirect beneficiaries. They may be associated with a different length of time, with outcomes in the mid-term and impacts in the long term. However, length<fnref n="72" /> of time is associated with the direct or indirect connections between the SI initiative and the groups of beneficiaries it affects (the closer the link, the faster the effect becoming visible). Outcomes/impacts may happen immediately after the SI project is active, or become visible months or years later. Impacts are often considered "results on the ground" (Emerson et al., 2011). In impact evaluations (Gertler et al., 2016), they are also called "final outcomes" and are associated with long-term goals. Figure 19 shows where outcomes/impacts are positioned in an example of a result-chain.

<figure><img src="image_97.png" /><figcaption>**Figure 19.** A result-chain used in impact evaluation. Source: Gertler et al., 2016 (Figure 2.1, p. 25, modified).</figcaption></figure>

Impacts can be defined as follows: "Impacts result from the actions spurred by collaborative dynamics. Impacts are intentional (and unintentional) changes of state within the system context; they are alterations in

<footnote n="72">⁷² For SIMRA evaluations, and which can be adopted is used in business economics can be used, where "medium-term" is 18-36 months or 18-60 months, "long-term" is 3-5 years (www.bytestart.co.uk/planning-short-term-long-term.html).</footnote>
<figure><img src="image_98.png" /></figure>

a preexisting or projected condition that has been deemed undesirable or in need of change. Impacts may also include the added value of a new social good or technological innovation developed by collaborative action. Impacts can be physical, environmental, social, economic, and/or political. They can be specific, discrete, and short term or they can be more broadly cast, cumulative in nature, and with longer term impacts. The former is much easier to measure and confirm, the latter more challenging to verify and evaluate. [...] Preferably, the nature and extent of the impacts are consistent with the desired outcomes targeted by the collaborative partners during the iterative principled engagement process. The absence of impacts, as well as unintended impacts (both negative and positive), can occur and should be accounted for" (Emerson et al., 2011: 18).

Box 16 – Main methods that inspired developing the SIMRA method for the dimension Outcomes/impacts

In the survey of existing methods to assess various types of impacts, the Global Reporting Initiative (GRI)<fnref n="73" /> was identified as a useful model because GRI Standards refer to three different domains that can be affected by the impacts determined by an organisation: environmental impacts, social impacts and economic impacts. According to the GRI Standards, organisations that want to comply with sustainability principles and communicate their performance to an external audience (e.g. clients, public authorities, public opinion, donors) can voluntarily adopt the standards. Even when the standards were initially created for the target of private business-oriented organisations (e.g. transnational corporations protecting their reputation in the face of environmental and social movements and potential boycotting campaigns), de facto they can be adopted by any kind of organisation, including public administratrons, NGOs, public-private ventures, or network-based initiatives. The GRI model for impact identification, evaluation and reporting can be adapted to the needs in SIMRA. An agency developing a social innovation initiative in a marginalised rural area may be considered similar to an organisation, which carries out activities with different types of effects (impacts) on the environment, society and the economy in its territory. According to the GRI Standards Glossary (GSSB, 2016: 9), impact “refers to the effects an organisation has on the economy, the environment and/or society, which in turn can indicate its contribution (positive or negative) to sustainable development”. Further, impacts can be positive, negative, actual, potential, direct, indirect, short-term, long-term, intended, or unintended (GSSB, 2016: 9), and lead to consequences for the organisation’s management model, reputation, or ability to achieve its objectives. This confirms the idea that impacts determine changes in the Perceived context where the social innovation is implemented. ToSIA (Tool for Sustainability Impact Assessment of Forest-Wood-Chains) uses a data-oriented approach that is very flexible in the focus of the analysis and the selection of indicators of sustainability (Lindner et al., 2010: 2197). ToSIA calculates sustainability values as products of the relative indicator values (i.e. indicator value expressed per unit of material flow) multiplied with the material flow entering the process. Calculated sustainability values are then aggregated for the segments of the FWC or for the complete chain. The sustainability impact assessment requires carefully specified system boundaries. ToSIA includes both economic, social and environmental indicators. However, the GRI Standards, ToSIA and other inspiring tools do not fully cover all the aspects within the fourth domains of interest for SIMRA evaluation, i.e. including governance and institutional aspects, and require integration with different approaches and methods. We also referred to: TSI, SIMPACT (Dhondt et al. 2016), Sefton et al. (2002), Hornsby (2012), Clark et al. (2004), the Vancity Community Foundation and the BC Association of Farmers Markets (2013), Allee (2008), AngloAmerican (2012), Cajaba-Santana (2014), Dawson and Daniel (2010), McGinnis and Ostrom (2012), Nicholls et al. (2012), ISO 14001:2015 standards, Newton et al. (2006), Kassa et al. (2009), Sandker et al. (2010), Millennium Ecosustem Assessment (2005), TEEB (2010) and CICES categorisation of ecosystem services (Haines-Young and Potschin, 2013), OECD Better Life Index, OECD Statistics and European Values Survey (2008), EUROSTAT, Sustainable Development Goals (UN, 2015), Symes and Hoefnagel (2010), Ahmed and Ahmed (2014), Bhatt and Altinay (2013), Temel and Dorjee (2014), GECES (2014), and indicators from the OECD Statistics (2017).

<footnote n="73">⁷³ Sustainability reporting, common since 1997, is an organisation's practice of reporting publicly on its economic, environmental and/or social impacts. Through this process, an organisation identifies and discloses its significant impacts on economy, environment and society using a globally-accepted standard. Of the world's largest 250 corporations 93% report on their sustainability performance and 82% of these use GRI's Standards. (www.globalreporting.org.).</footnote>
<figure><img src="image_99.png" /></figure>

As mentioned above, outcomes and impacts of social innovation in marginalised rural areas can be on different aspects, economic, social, environmental as well as governance/institutional aspects. Depending on the type, dimension and location of the SI initiative, outcomes and impacts can be relevant to environmental aspects (e.g. a SI initiative established in a rural area for a new organic production system, or a rural community transition to low carbon energy systems), social aspects (e.g. a SI initiative established for the inclusion of newcomers or disabled people into farming practices), economic aspects (e.g. a SI initiative established to provide income diversification based on the multi-functionality of forests) or governance/institutional aspects (e.g. a SI initiative for creating new private-public coordination mechanisms and promoting coalitions to support policy reforms). SI initiatives are also expected to have outcomes and impacts on a combination of these aspects. There might be a wide range of aspects connected to each of the four domains. In the evaluation framework, each domain is considered in detail.

### Economic aspects

A social innovation initiative can have different effects on the economy and at different levels. For the purposes of SIMRA, we suggest identifying three key types of “economic actors” connected to the social innovation, to determine direct and indirect impacts on the economy. They are: 1) the SI initiative in itself (to be considered as the organisation that provides activities, products and services of social innovation), even if in some cases, during the initial steps, it may not be formally approved as a legal entity and work through an informal aggregation of persons); 2) the network of individuals and organisations directly connected to the SI initiative but not part of the organisation (e.g. satellite activities such as suppliers, direct or indirect buyers/clients); 3) actors working and living in the territory where the SI initiative is established, not directly connected to the SI, but that indirectly benefited (or were negatively affected) for the economic performance by the activities, products and services provided by the SI. The elements to be considered in the evaluation for addressing types of impacts, and the ways to measure them, depend on the type of “economic actors” identified. Whatever the unit of analysis, in SIMRA, economic outcome/impact are defined as “any change in the economy, whether adverse or beneficial, wholly or partially resulting from the activities or products or services delivered by the social innovation in the marginalised rural area” (i.e. by the specific social innovation initiative that is evaluated). Details on the types and variety of possible economic aspects on which SI might have an effect are provided in Part III.

### Box 17 – Example of existing methods that have inspired us in developing SIMRA evaluation method on economic aspects

In the case of economic aspects, we drew mainly from the MIAA framework (Methodology for Impact Analysis and Assessment), developed by Hornsby 2012. The framework provides very concrete and detailed explanation of how to be applied; the methodology aims to look at social-purpose organisations throughout the sector, and analysing their impacts in a robust and consistent fashion. To do this, it draws on measurements made at the ground level, and accordingly is accompanied by a set of Guidelines for How to Measure and Report Social Impact. Another key source was the guide to social return on investment (Nicholls et al., 2012). This framework measures and accounts for a much broader concept of value; it seeks to reduce inequality and environmental degradation and improve wellbeing by incorporating social, environmental and economic costs and benefits. Finally, the guide to the use of economic evaluation in social welfare economic evaluation, involving the systematic assessment of costs and outcomes proposed by Sefton et al., 2002 was also very useful, as well as all the OECD Frameworks for measuring well-being and progress.
More information is provided in the SIMRA Resources Web Site (see www.simra-h2020.eu/resources/).

### Social aspects

In SIMRA, we define social outcome/impact as “any change in society, whether adverse or beneficial, wholly or partially resulting from the activities or products or services delivered by the social innovation in the marginalised rural area” (i.e. by the specific social innovation initiative evaluated). Details on the types and
<figure><img src="image_100.png" /></figure>

variety of social aspects affected by SI are provided in Part III. Box 18 mentions some key methods and sources that supported developing the evaluation framework.

Box 18 – Examples of existing methods that have inspired us in developing SIMRA evaluation method on social aspects

In the case of social aspects, we drew mainly from methods and research work focused on social capital, a relevant factor to support development processes in rural areas. Many recent studies have focused on the ability of social capital to enhance development in areas characterised by socio-economic marginality (e.g., Marquardt et al., 2012). Definitions and measures of social capital often conflate inputs (networks, norms) with outputs (forms of cooperative behaviour like political participation and social organisations). In developing SIMRA evaluation framework, and the overall methodological approach, the work of Pisani et al. (in print) on social capital inspired us. By integrating qualitative and quantitative analyses, the identify the specific factors that affect economic and social processes of development, and identify and measure various dimensions of social capital, namely structural, normative-cognitive, bonding, bridging, and linking forms and assess how they combine to determine local development, in the context of EU policy and the LEADER Approach. Also Sabatini (2009) was taken into consideration, with his framework for measurement of social capital based on five different dimensions of the concept: strong family ties (i.e. bonding social capital), weak informal ties (bridging social capital), voluntary organisations (linking social capital), active political participation and civic awareness. Another key method that we decided to adopt in SIMRA is Social Network Analysis (SNA) (Banerjee et al., 2013; Borgatti et al., 2013). SNA is about understanding how the relationships between agents of a system influence or impact specific outcomes of that system. In relation to social innovation, SNA can help understand how the links (what they are and what are their characteristics) between agents (actors or institutions) encourage or hinder SI in its different phases. Other inspiring sources are: Bhatt and Altinay 2013 (how social capital in Indian social entrepreneurial ventures (SEVs) is leveraged in SI under resource constraints); McIntyre 2008 (technicques for engaging with stakeholders and sharing information to encourage social learning and co-construction of knowledge); Ban et al., 2015 (measures to assess the impact on households’ economic welfare and social capital); Dax et al., 2016 (quantitative and qualitative analysis of LEADER programme performance in some European countries, with insights on methodological instruments); Knack 2000 (a multivariate analysis to assess the correlation between different social capital index and quality of government based on performance indicators).

More information is provided in the SIMRA Resources Web Site (see www.simra-h2020.eu/resources/).

Environmental aspects

In SIMRA, we define environmental impact as "any change in the environment, whether adverse or beneficial, wholly or partially resulting from the activities or products or services delivered by the social innovation in the marginalised rural area" (i.e. by the specific social innovation initiative that is evaluated). Details on the types and variety of environmental aspects, which SI might have an effect on, are provided in Part III. Box 19 briefly refers to some of the key methods and sources that supported developing the evaluation framework.

Box 19 – Examples of existing methods that have inspired us in developing SIMRA evaluation method on environmental aspects

For addressing environmental aspects, we drew mainly from the international standards ISO 14000 on
environmental management systems of organisations (versions 1996, 2001, 2004 and 2015). According to
these standards, environmental aspects are “elements of an organisation’s activities or products or services
that interacts or can interact with the environment” (ISO 14001:2015, term 3.2.2). The environment is defined
as “surroundings in which an organisation operates, including air, water, land, natural resources, flora, fauna,
humans, and their interrelationships” (ISO 14001:2015, term 3.2.1). Surroundings in this context can extend
from within an organisation to the local, regional and global system. Surroundings can be described in terms of
biodiversity, ecosystems, climate or other characteristics. Additionally, environmental impact is “change to the
<figure><img src="image_101.png" /></figure>

environment, whether adverse or beneficial, wholly or partially resulting from an organisation's environmental aspects" (ISO 14001:2015, term 3.2.4). According to the ISO 14001:2004 (p.27), "the relationship between environmental aspects and impacts is one of cause and effect" (see Part III on cases where statistical-based impact evaluation will be applied). Other relevant approaches are two European Directives. The first one is the European Union Directive (85/337/EEC) on Environmental Impact Assessments (know as the EIA Directive), initially introduced in 1985 and later on amended three times (in 1997, 2003 and 2009). The updated version is Directive 2011/92/EU (European Commission, 2011, published on the 13th of December). The second one is the Strategic Environmental Assessment (SEA) Directive 2001/42/EC on the assessment of plans and programmes, i.e. for predicting and evaluating the impact of a strategic action (e.g. a regional planning project) on the environment. Seven key areas are required and the aspects of the environment that may be affected and thus should be taken into consideration are listed, including populations, fauna, flora, air, soil, water, humans, landscape, cultural heritage. Other lists of environmental indicators not specifically connected to data collection or evaluation methods, that were checked are the OECD Key Environmental Indicators (2013), the UN-Habitat Guidelines for Strategic Environmental Assessment (2004), and the Millennium Ecosystem Assessment (2005). We also drew from Newton et al. (2006), Lindner et al. (2010), TEEB (2010), Papageorgiou and Salmeron (2013), de Vries (2015), Brunner et al. (2016) and Verweij et al. (2016). More information is provided in the SIMRA Resources Web Site (see www.simra-h2020.eu/resources/).

## Governance/institutional aspects

In SIMRA, we define governance/institutional impact as "any change in the governance/institutional environment, whether adverse or beneficial, wholly or partially resulting from the activities or products or services delivered by the social innovation in the marginalised rural area" (i.e. by the specific social innovation initiative that is evaluated). In this domain, there is no single approach for identifying governance/institutional aspects, which depend on the types of activities, products and services provided by the SI and on the context. In addition, the literature on governance is broad, grounded on diverse meanings of the term. Governance is sometimes considered one of the most challenging 'fuzzy' words and concepts: broadly used yet often not clearly explained. These characteristics contribute to making this domain the most problematic from an evaluative point of view.

Only a few methods have been developed and tested/implemented for specifically assessing governance issues at the local level, at least in the field of natural resources management (Secco et al. 2014), i.e. the level at which the SIMRA case studies will be assessed. Details on the types and variety of governance/institutional aspects which SI might have an effect on are provided in Part III. Box 20 briefly refers to some of the key methods and sources for developing the evaluation framework.

## Box 20 – Examples of existing methods that have inspired us in developing SIMRA evaluation method on governance/institutional aspects

For identifying the main elements for analysing impacts of social innovation in MRAs on governance and institutional aspects, BEPA (2011; 2013) was a key resource. Even if it is not focused on rural areas and rural challenges specifically, they suggest key aspects of governance that are expected to foster social innovation. We can presume that there may be a double-way connection between social innovation and governance: "while the movement and the creative energy in the ecosystem comes from the actors and their connections, the administrative, economic and legal environment has to be enabling" (BEPA 2013: 21). SI actors are affected by current governance and institutional environment but are also able to influence and change it. These aspects may be relevant for fostering social innovation, and when a SI initiative is developed and implemented, related mechanisms and tools can stimulate and/or consolidate more systematic governance reforms, oriented towards the creation of new networks, adoption of new governance procedures, or transparency. For example, while "some of the most resistant barriers to social innovation are rooted in a lack of coordination between the various actors" (BEPA, 2013: 89), we assume that networking-based SI initiatives in a MRA will improve (or at least seek to improve) coordination among actors – including public administrations and policy makers. Similar considerations are valid also for other aspects mentioned by BEPA (2013), including: "collaborative, informal platforms or programmes" (p. 42), "capacity building" (p. 21),
<figure><img src="image_102.png" /></figure>

"benckmarking and impact measurement" (p. 21), "good public services" (p. 92), "modernisation of public administrations" (p. 89), "faster standard setting" (p. 92), "new information and communication technologies" (p. 93), "experimentation with new institutional models based on social innovation" (p. 93), accompanying the public sector "towards more effective, efficient and open public administrations" (p. 93), "enduring partnerships" between private and public sector (p. 55), and "making corporate social responsibility a systematic and essential element of analysis and operating mode of all businesses" (p. 55). Emerson et al. (2012) proposed the concept of "collaborative governance" as "the processes and structures of public policy decision making and management that engage people constructively across the boundaries of public agencies, levels of government, and/or the public, private and civic spheres in order to carry out a public purpose that could not otherwise be accomplished. This definition allows collaborative governance to be used as a broader analytic construct in public administration and enables distinctions among different applications, classes, and scales" (Emerson et al. 2012: 2). This concept is similar to the network governance model, but focuses more distinctly on the institutional arrangements and capacity that public institutions ought to develop and implement to ensure that network governance works in practice. Emerson et al. (2012: 10) refer to "a system in which cross-boundary collaboration [among organisations/actors/sectors/levels/scales] represents the predominant mode for conduct decision making and activity". In their view, collaborative governance includes two components: collaborative dynamics and collaborative actions. The first component consists of "cyclical or iterative interactions" where "principled engagement, shared motivation and capacity for joint action" are the key elements (ibidem). Each of them is expanded into several sub-key-elements that allow to understand the procedural arrangements for collaboration. The second one is less well developed and specified. The authors initially argue that it is "usually seen as the major outcome of a linear process and is sometimes conflated with impacts" (ibidem: 17), but later propose a list of possibile new mechanisms for collective actions and mention "leadership, diverse representation and power" as key elements (ibidem: 18). In the SIMRA framework, these mechanisms can be considered equivalent to the dimension 'Activities, procedures and practices' of the SI. Arts and Goverde (2006) adop an institutional perspective within a Governance Capacity Approach (GCA), to refer to both "indicative governance capacity" and "performative governance capacity". In the first type of governance capacity, "the key question is whether a certain policy arrangement is such that we can expect a 'capacity to govern'. This means that there are enough resources available, that the key policy actors are involved, that the rules of the game do not prohibit appropriate (change) behavior, etc." (Arts and Goverde, 2006: 80). Such governance capacity can be assessed on the basis of the notion of "congruence" assuming that "a certain level of congruence – respectively among the policy views of different actors, among the dimensions of policy arrangements and available resources and rules, and among a policy arrangement and its wider institutional context – is needed for any policy arrangement to perform (and, in contrast, a lack of congruence implies governance failure): [...] the more congruence, the more (potential) governance capacity" (Arts and Goverde, 2006: 80-81). While the institutional capacity can be assessed on the basis of congruence of a policy arrangement, the "performative governance capacity" can be assessed on the basis of the capacity of policy arrangements to balance among jurisdictional, economic-managerial and political-civic principles of good governance that reflect perceptions, needs and strategies of policy actors (Nelissen, 2002). The two types (indicative and performative) of governance capacities are clearly interlinked. Finally, for aspects related to the normative approach to governance assessement, i.e. based on the so-called principles of good governance, we drew from the method and indicators developed by Secco et al. (2014) for assessing governance at local level. These indicators focused on forest governance, but can be adapted to the context of agriculture and rural development as well as to cases of social innovation. References adopted for developing the SIMRA preliminary set of methods include: BEPA (2011), Kjær (2004), UNDP (2009), Masten and Saussier (2000), Stirling and Mayer, (2001), Jongeneel e al. (2012), Soma and Vatn (2014), Citroni et al. (2015), Duberry (2015), Gobattoni et al. (2015), Pradel-Miquel (2015), Bosworth et al. (2016b), Džinić et al. (2016), Scott and Boyd (2016), Ibrahim (2017), Van der Schoor et al. (2016), and Górriz-Mifsud et al. (2016). More information is provided in the SIMRA Resources Web Site (see www.simra-h2020.eu/resources/).
<figure><img src="image_103.png" /></figure>

### 5.2.8. Innovation and learning processes

#### 5.2.8.1. Innovation and learning processes: an overview

In Figure 8, the four types of arrows connecting social outcomes at each of the four levels show the non-linear processes that characterise policy and institutional responses and changes to social innovation, as well as interactions among actors over time. However, it should be noted that feedback might be relevant in several stages of the social innovation and not only at the end of the process<sup>74</sup>. The four types of arrows show responsiveness and the level of expected influence: the continuous black arrow refers to social outcomes at the local level, where the reaction of the local context may be faster. The grey discontinuous arrow refers to social outcomes with less directly relevant effects at the regional level, clearer and occurring over a longer period of time. The light grey discontinuous arrow refers to less direct and evident social outcome effects on the national context, with slow responsiveness and longer-term processes. Finally, the dark grey discontinuous arrow refers to even less evident social outcome effects at the international/global level. The arrow from activities via outputs to outcomes/impacts suggests also that the social innovation that occurs at the local level in a MRA could be scaled up or have effects at larger spatial scales. This feedback loop is consistent with Holling and Gunderson's (2002) examination of adaptation within nested systems.

An important aspect of social innovation is the learning process, which can occur through participatory or deliberation processes (Dryzek and Pickering, 2017); self-organising activities (McGinnis and Ostrom, 2014) and social learning (by failures) (Garmendia and Stagl, 2010; Biggs et al., 2010). These processes can support processes of scaling up or out of the social innovation. Following Moore and Westley (2011), "we use the term "scaling out" to refer to the replication of the same innovation in several different locations. In a different vein, we use "scaling up" to refer to moving an innovation into a broader system. While SI cases may not be initially supported by any policy (endogenous process), their success may lead to their replication (scaling out) or to the implementation of neo-endogenous approaches such as focused funding, policies or legislation, to scale up the initiative. Scaling up is not always possible, as SI may depend on locally specific social relationships and contexts which are not present in different spatial and socio-economic levels.

Outcomes may have multiplier effects which feed back to the reduction of needs at all three levels: local, territorial and macro, albeit with different expected levels of responsiveness (See also McAdam, 2003 on 'scale shifts'). Multiplier effects are often difficult to be quantified. In addition, SI may take different pathways, and impacts on the alleviation of 'need' may not be directly traceable in an evaluation process to the SI - due to the complexity of the social system. Evaluative challenges include confounding effects, path dependency processes and unpredictability, non intended effects. These processes are best represented heuristically by the three non-linear arrows. Despite this complexity, efforts should be carried out to reflectively assess effects of outcomes at different spatial and temporal scales, in terms of distribution of benefits and trade-offs, responding to questions such as, for whom and for what are actions geared? Whose framing (of problem and response) is privileged and whose becomes marginalised? These questions seek to highlight the risk of justifying the hollowing out (or the restructuring) of the state in relation to welfare interventions (Swyngedouw, 2005), and to maintain instead a clear focus on a normative stance towards social innovation.

#### 5.2.8.2. Learning processes: details for SIMRA evaluation

In the evaluation framework, the dimension *Innovation and learning processes* is identified through the sub-dimensions of feedback loops and multiplier and critical effects. Feedback loops occur both within the local level, and in bridging to higher levels. In the first case, sharing results within the local community through presentations or specific events can lead to raising interest, debating issues and identifying trade-offs, as well as seeking opportunities and implementation beyond the sector or the beneficiaries already reached.

Feedback on the SI project can also derive from partners, external donors or local politicians. Reflection on the results of the SI project can refer to whether the social innovation initiative has reduced the level of

<footnote n="74"><sup>74</sup> We represent learning processes in the framework in Figure 10 only in connection to final phases of social innovation.</footnote>
<figure><img src="image_104.png" /></figure>

marginality of the territory, and identified the elements that point to this reduction. Positive and negative feedback loops can also be received by participating in events outside of the territory of the case study, or by presenting the initiative to regional and national politicians.

The sub-dimension multiplier and critical innovation effects refers to the assessment of the mainstreaming, upscaling and outscaling processes, as well as of critical effects. Mainstreaming refers to the consideration of whether the SI initiative remains innovative, and whether it is in the process of becoming a 'normality'. Upscaling considers whether the SI initiative has had an impact at higher scales, including in national policy or laws. Outscaling, the evaluation identifies whether the SI initiative has been replicated, adapted or aggregated into a body of similar initiatives at the national level.

Finally, critical effects can be a component of any SI initiative. Adopting specific evaluation terminology, the evaluation seeks to identify deadweight, displacement and substitution. It does this by asking whether the effects of the SI would have arisen regardless (deadweight), and whether the effects were at the expense of another area (displacement) or another person, organisation or enterprise (substitution).

### 5.3. Final Remarks on the Evaluation Framework

The framework offers significant opportunities to address the challenges of evalutation of SI:

* It responds to the requirement of identifying the elements for analysing and assessing social innovation in practice, from its causes (e.g. trigger), its core dimensions (e.g. agency, their networks, activities, etc.) and its effects (e.g. outputs, outcomes or impacts).

* The framework was developed for "setting the scene" of the development of methods and tools, enabling reflection on the steps taken, and understanding the flows of SI initiatives. It will be valuable for use with current and completed SIs. It can inform action based research.

* It places attention on *the process of reconfiguration* (in governance arrangements, networks and attitudes) and on the broader social, political-economic, environmental and cultural contexts which provide a *perceived context* (as framed by enabling and constraining conditions) for the *action of agency*.

* It provides a comprehensive framework, which is theoretically grounded in social theory on the relationships between structure and agency in shifting practices and values, and in theory of change.

* It includes *attitudes/beliefs* as part of reconfiguring.

* It enables inclusion of *reflexivity* on what the outcomes are, for whom, and how they can be used in *learning processes* at various levels, thus leading to innovation occurring elsewhere.

* It considers the *role of the public* sector/state in enabling social innovation (potential).

* It draws on the model of Cajaiba-Santana's due to its emphasis on situating processes of social innovation within a structure-agency framework, but deepens the understanding of: i) the organisation (or self-organisation) at the level of agency (where collective action is identified as the result of different combinations of individual/public-private collective action for individual/collective benefit); ii) the process of reconfiguration (which includes both a reconfiguring and a reconfigured phase); iii) the impact of reconfiguration on attitudes/beliefs.

The framework also faces some challenges which will be a focus of improvement:

* It does not enable the representation of all aspects of real situations of social innovation, where several "reverse loops" might occur. i.e. it is linear and deterministic.

* It is not specific to social innovation in marginalised rural areas. It is more general and generalizable to SI in various contexts, which may be a positive characteristic. For the purpose of D4.2, and a specification in terms of our target (i.e. MRAs), it will be developed with the identification of the parameters that characterise what a MRA at local level (see Price et al., 2017; D3.1), where there may be significant societal and biophysical challenges to be tackled, and fewer resources and opportunities comped to urban areas).
<figure><img src="image_105.png" /></figure>

<figure><img src="image_106.png" /></figure>

# PART III – SETTING THE METHODS: HOW TO ASSESS AND EVALUATE SI IN MRAs
<figure><img src="image_107.png" /></figure>

# 6. The Fundamentals of SIMRA Methods to Assess and Evaluate SI in MRAs

## 6.1. Evaluation: A Brief Introduction

An evaluation is “a systematic and objective assessment of an ongoing or completed project, programme or policy, its design, implementation and results. The aim is to determine the relevance and fulfilment of objectives, efficiency, effectiveness, impact and sustainability” of the implemented actions (OECD, 2000<sup>75</sup>; Morra-Imas and Rist, 2009; Kandkher et al., 2010; Gertler et al., 2016). In short, evaluation is a periodic analytical assessment<sup>76</sup>, which seeks to answer ‘why’ and ‘how’ questions about specific performance and outcomes. Typically, “[impact] evaluation seeks to prove that changes in targets are due only to the specific policies undertaken” (Khandker et al., 2010: 8). These analytical assessments emphasize “reliability and usefulness of findings. Their role is to improve information and reduce uncertainty” (OECD, 1999: 6).

Typically, the evaluation of any policy, programme and project is based upon principles, standards, types of evaluation, criteria, evaluation questions and indicators. These concepts, already introduced in Secco et al., (2016; D4.1), are briefly recalled hereafter.

**Principles of evaluation** can relate to the design, implementation and delivery of the results, and include specific reference to the purposes of the evaluation (i.e. improve future policy), impartiality and independence, credibility and usefulness, participation and cooperation (OECD, 1991; 2011; 2013a; 2013b).

**Evaluation quality standards** – proposed by the OECD in 2006 – are the key pillars needed for a quality evaluation and refer to rationale, purpose, and objectives of an evaluation, evaluation scope, context, evaluation methodology, information sources, independence, evaluation ethics, quality assurance, relevance of the evaluation results and completeness (OECD, 2006)<sup>77</sup>. These standards are relevant to the development of evaluation methods in the context of participatory approaches such as those to be adopted by SIMRA in the case study areas and through the SIMRA SITT.

**Types of evaluation** refer to the methods used (qualitative, quantitative and mixed), the manager of the evaluation (internal, external), the final use (formative or summative), the use of participatory techniques (participatory, not participatory) in relation to the project, and the programme or policy cycle if the evaluation is ex-ante, ongoing or interim, final, ex-post (European Commission, 2004).

**Evaluation criteria** need to be established to evaluate whether an intervention achieved what was needed and led to the intended results, impacts. For the purposes of the definitions, the criteria are those proposed by the OECD Development Assistance Committee (OECD, 1991) and adopted by EC DG Agri. These include relevance, effectiveness, efficiency, impact and sustainability (OECD, 1991)<sup>78</sup>.

<footnote n="75">75 The DAC Principles for the Evaluation of Development Assistance (OECD, 1991) and Glossary of Key Terms in Evaluation and Results Based Management (RBM) (OECD, 2000; 2010).</footnote>

<footnote n="76"><sup>76</sup> In contrast to continuous monitoring, evaluations are carried out at discrete points in time and often seek an outside perspective from technical experts. In many cases evaluation draws on monitoring activities, although the reverse does not happen, i.e. monitoring does not draw on evaluation. "Monitoring is a continuous process that tracks what is happening within a program and uses the data collected to inform program implementation and day-to-day management and decisions. Using mostly administrative data, monitoring tracks program performance against expected results, makes comparisons across programs, and analyzes trends over time. Usually, monitoring tracks inputs, activities, and outputs, though occasionally it can include outcomes, such as progress toward national development goals." (Gertler et al. 2016: 7).</footnote>

<footnote n="77"><sup>77</sup> OECD (2006) *Evaluation Quality Standards for Test Phase Application*. Development Assistance Committee. http://www.oecd.org/dataoecd/30/62/36596604.pdf.</footnote>

<footnote n="78"><sup>78</sup> See the OECD DAC Criteria for Evaluating Development Assistance website: www.oecd.org/dac/evaluation/daccriteriaforevaluatingdevelopmentassistance.htm</footnote>
<figure><img src="image_108.png" /></figure>

1. **Relevance** refers to the extent to which the objectives of an intervention are consistent with beneficiaries' requirements, country needs and policies and global priorities.

2. *Effectiveness* is a measure of the extent to which a project reaches its objectives.

3. *Efficiency* is a measure of outputs – qualitative and quantitative – in relation to inputs, meaning that the project uses the less costly resources in order to achieve the outputs. This requires a comparison with alternatives to determine whether the most efficient process has been adopted.

4. Impacts are the positive and negative changes produced by a project, directly or indirectly, intended or unintended.

5. Sustainability refers to the probability of a continuation in the stream of benefits produced by the project after the period of external support has ended<fnref n="79" />.

From a practical perspective, to address the above mentioned criteria, the evaluation of an intervention (e.g. a SI initiative and its effects) has to start from the identification of the so-called intervention logic. This is defined as "the logical link between the problem that needs to be tackled (or the objective that needs to be pursued), the underlying drivers of the problem, and the policy options [...] available to address the problem or achieve the objective" (European Commission 2015: 10). Linking problems with possible solutions is the main feature of the theory of change (see Section 4). The theory of change is typically based on the analysis of a results chain detaining the causal sequence beginning with inputs, moving through activities and outputs, and culminating in outcomes, impacts and feedback (Morra-Imas and Rist, 2009: 167). While this chain implies linear relations which are quite simplistic, multi-causation or results model can draw greater attention to the non-linearity of cause and effect (GIZ, 2013)<fnref n="80" />.

Adopting this approach, the intervention logic (which can also be represented by means of the theory of change) is analysed by means of **evaluation questions** relating to the design, implementation and results of a SI initiative. In general, evaluations can address three types of questions:

* **Descriptive questions.** The evaluation seeks to determine what is taking place and describes processes, conditions, organisational relationships, and stakeholder views. Examples of descriptive questions for the evaluation of an SI initiative in MRAs are: (i) What are the goals of the SI initiative from the perspective of the different stakeholders of the rural context?; (ii) Where the SI initiative has been implemented?; (iii) How many marginalised people of the target area participated to the initiative?

* **Normative questions.** The evaluation compares what is taking place to what should be taking place; these questions compare the current situation with a specified target, goal or benchmark. It assesses activities and whether or not targets are accomplished. Normative questions can apply to inputs, activities, and outputs. Examples of normative questions for the evaluation of a SI initiative in MRAs are: (i) Did we meet the objective of including marginalised people in the decision-making process?; (ii) Did we reach the goal of including new farmers in social farming initiatives?; (iii) Was the process for including new beneficiaries fair and equitable?

* **Cause-and-effect questions.** They allow the determination of what difference the intervention makes. Often referred to as outcome, impact, or attributional questions, they attempt to measure what has changed because of the intervention. Cause-and-effect questions seek to determine the effects of a project, programme, or policy. They are the "so what" questions. Cause-and-effect questions ask whether the desired results have been achieved as a result of the programme (Morra Imas and Rist, 2009: 227). The evaluation examines outcomes and tries to assess what difference the intervention

<footnote n="79">⁷⁹ Sustainability refers in this context to the classical meaning adopted in project evaluation. However, in SI initiatives in marginalised rural areas, "external support" in terms of funds provided by external donors or investors is not an obviuos component of the initiative. In some cases, the SI initiative can be initially based on voluntary and free engagement of local actors and later be self-funded, or able to generate revenues for self-sustaining – without necessarily depend on external resources.</footnote>

<footnote n="80">⁸⁰ The GIZ Monitoring and Evaluation Unit does not distinguish between outputs, outcomes, impacts, but simply refers to 'results'. According to this model, any output, outcome, or impact is a goal, and anything that may be achieved, more or less directly or indirectly, is a result. More details on this issue are available in Secco et al. (2016; D4.1).</footnote>
<figure><img src="image_109.png" /></figure>

makes in outcomes" (Gertler et al., 2016: 7). Examples of cause and effects questions for the evaluation of a SI initiative in MRAs are: (i) Did the SI initiative determine an improved wellbeing of the community?; (ii) Did the government increase the financing of new SI initiatives?; (iii) What other impacts (positive or negative) did the SI initiative determine on the wider community?

On the basis of these general premises, the overall evaluation approach is presented below, including its basic constituents: framework, methods and tools.

## 6.2. The Evaluation Approach

### 6.2.1. Elements of the evaluation approach

The community of evaluators has developed a wide range of categorizations of different evaluation approaches.<fnref n="81" /> To be pragmatic, by evaluation approach in this deliverable we mean the set of options<fnref n="82" /> that we have chosen to design, test, refine and finalise an innovative methodology for assessing and evaluating social innovation in marginalized rural areas. Any evaluation approach is typically constituted by three main parts: the "evaluation process", the "evaluation methods" and the "evaluation tools" (EU EuropeAid, 2006). Each of these parts comprises elements and sub-elements, which are briefly presented in Table 10 to provide an overview of the selected set of options in SIMRA. The Table synthesizes the main steps and activities (some already implemented, others foreseen) that are necessary for finalising the SIMRA evaluation methods at the end of the project. The final manual for the evaluators (D4.3) will be structured on the basis of this Table contents.

<table><caption>**Table 10.** Elements and sub-elements of the evaluation approach adopted in SIMRA</caption><thead><tr><th>Elements of the evaluation</th><th>Sub-elements of the evaluation</th><th>Elements of the evaluation as proposed in SIMRA</th></tr></thead><tbody><tr><td colspan="3" style="background-color: #cccccc; font-weight: bold;">EVALUATION PROCESS</td></tr><tr><td rowspan="4">Subject of the evaluation</td><td>Project, programme, strategy</td><td>SI initiative in MRA</td></tr><tr><td>Sectors, themes and cross-cutting issues</td><td>Sectors: agriculture, forestry, rural development</td></tr><tr><td>Scope of the evaluation</td><td>SI initiative (SI process and SI project) and its impacts</td></tr><tr><td>Scale</td><td>A SI initiative in SIMRA is assessed at local level</td></tr><tr><td rowspan="2">Timing of the evaluation</td><td>Case of stand-alone intervention</td><td>Ex-post</td></tr><tr><td>Case of a series of interventions</td><td>Depending on the momentum of SI development (initial phases, or completed successive cycles).</td></tr><tr><td>Use of the evaluation</td><td>Evaluation users</td><td>Policy makers and public administration on different administrative levels, enterprenuers (including social enterpreneuers and relevant interest groups) (policy target groups and practice target groups)</td></tr></tbody></table>

<footnote n="81">⁸¹ Morra Imas and Rist (2009) propose a comprehensive list of different typologies of evaluation approaches such as: prospective evaluation, evaluability assessment, goal-based evaluation, goal-free evaluation, multisite evaluations, cluster evaluation, social assessment, enviromental and social assessment, participatory evaluation, outcome mapping, rapid assessment, evaluation synthesis, meta evaluations, utilization-focused evaluation, empowerment evaluation, realist evaluation, inclusive evaluation, beneficiary assessment and horizontal evaluation. BetterEvaluation, which is an international initiative to improve evaluation practice and theory, suggests different types of evaluation approach: e.g. apprentice inquiry, beneficiary assessment, case study, collaborative outcomes reporting, critical system heuristics, democratic evaluation, innovation history, most significant changes, outcome harvesting, outcome mapping, participatory evaluation, participatory rural appraisal, positive deviance, randomised controlled trials, realist evaluation, social return of investment, utilization-focused evaluationFor detailed information on the specific features of the different evaluation approaches previously mentioned, see the webpage: www.betterevaluation.org/en/approaches</footnote>

<footnote n="82">⁸² The selected set of options of the SIMRA evaluation approach are grounded on both theoretical analysis (SIMRA Deliverables 2.1, 2.2 and 3.1) and literature review (Task 4.2) and SITT stakeholders consultations (WP2) (see Part I).</footnote>
<figure><img src="image_110.png" /></figure>

<table><tr><td></td><td>Types of use</td><td>Help decision making, articulate judgements on SI and/or better know and understand SI and its effects</td></tr><tr><td></td><td>Dissemination of the evaluation</td><td>Online reports and material in the website, scientific conferences, etc. as planned by SIMRA project</td></tr><tr><td rowspan="3">The players and their roles</td><td>Evaluation manager</td><td>Responsible for the evaluation: SIMRA WP4 and WP6 Leaders</td></tr><tr><td>Evaluation team</td><td>Responsible for data collection, data analysis, formulation of judgements and reporting in relation to evaluation questions: SIMRA WP5 and CS Teams</td></tr><tr><td>Evaluation stakeholders</td><td>SITT members, SAB members, stakeholders involved in local SI CSs, other SIMRA partners, EU and non-EU Mediterranean countries representatives within SIMRA project</td></tr></table>

(PRELIMINARY) EVALUATION METHODS

<table><tr><td rowspan="6">Setting the scene</td><td>Theoretical framework: rational, logic and theoretical connections</td><td>Identification and analysis from a theoretical point of view of the rational, logic and cross-sectoral theoretical connections of SI in MRAs: WP2, WP3</td></tr><tr><td>Evaluation framework</td><td>Deconstructing SI concepts (identification of dimensions, sub-dimensions, components): WP4</td></tr><tr><td>Policies analysis</td><td>Identification of policies relevant to SI in MRAs implemented at different institutional levels: WP6</td></tr><tr><td>Complementarity with existing evaluation frameworks/methods</td><td>Identification of and possible integration with existing evaluation approaches framworks/methods/tools: WP4</td></tr><tr><td>Who?</td><td>Agency and other actors at the core of the SI initiative: innovators, followers, transformers, implementers</td></tr><tr><td>Domains of impact</td><td>SI initiative could have impacts on four different domains: economy, society, environment and governance/institution</td></tr><tr><td rowspan="4">Evaluation questions</td><td>Origins of initial set of questions</td><td>Preliminary list of questions based on SITT stakeholders consultation (with WP2), identification and analysis of existing methods that can be used in assessing SI and its impacts on four domains, creation of ad hoc questions</td></tr><tr><td>Selection of questions</td><td>Test of preliminary list of questions in 2 to 3 pioneer CSs: the initial proposed questions have to be selected on the basis of the results of pioneer case studies in WPs 5 and 6, as well as WP4.</td></tr><tr><td>Questions and evaluation criteria</td><td>Refinement and formulation of the final set of questions to be applied in all the selected CSs: WP5 and CS Teams<br>Further refinement of questions and consequent aggregation of questions to answer different evaluation criteria (e.g. relevance, efficiency, effectiveness, impact, sustainability), depending on the results of the assessment in each of the selected CSs. Both qualitative and quantitative questions are used.</td></tr><tr><td>From questions to indicators</td><td>On the basis of evaluation framework and questions, in connection with hypothesis of diverging paths (WP2), different indicators have to be developed as final elements to formulate judgement within SIMRA.</td></tr></table>
<figure><img src="image_111.png" /></figure>

<table><thead><tr><th></th><th>Indicators</th><th>Mainly focused on descriptive indicators on SI dimensions, sub-dimensions, components and their aggregations.</th></tr></thead><tbody><tr><td></td><td>Narrative</td><td>Mainly focused on in-depth interviews with key informants and policy experts</td></tr><tr><td rowspan="4">Methodological design</td><td>Initial steps</td><td>Strategy for data collecting and analysis: data collection in selected CSs in MRAs at local level<br>Selection of investigation areas: depending on the status of the SI initiative (in a large and complex SI initiative, only one SI project will be analysed)<br>Specifically designed tools: guiding questions for semi-structured interviews and questionnaires for structured interviews.</td></tr><tr><td>Design table per indicator</td><td>For each indicator a design table has to be formulated and completed based on the following elements: questions, scope of indicator, evaluation criteria which it refers to, analysis strategy, type of SI, information sources and tools.</td></tr><tr><td>Data collection</td><td>Evaluation teams (SIMRA CSs Teams) collect primary and secondary data in selected CSs in MRAs (WP5).</td></tr><tr><td>Analysis</td><td>Analysis of change: two approaches are proposed, based on the contextual conditions of each CS. One approach is based on descriptive and normative questions as well as on cause-effect questions (counterfactual before-after). Another approach is based on cause-effect questions with the application of impact assessment (counterfactual dif-in-dif, only when applicable).</td></tr><tr><td colspan="3"><strong>EVALUATION TOOLS</strong></td></tr><tr><td>Combination of tools</td><td>Toolbox</td><td>Qualitative and quantitative tools used in combination: semi-structured interviews (open questions, storytelling), structured interviews (questionnaires with open and closed questions), focus groups, survey, expert panel, case study analysis, context indicators, POT analysis, social network analysis, stakeholders analysis.</td></tr></tbody></table>

Source: EU EuropeAid, 2006 – Evaluation methods for the European Union's External Assistance.

In the following sub-section, the focus is on the elements and/or sub-elements that require clarification in relation to the activities of assessment planned for the next steps, i.e. the application of the preliminary set of methods in the pioneer CSs, before it is applied to the other selected CSs of SI in MRAs (in WP5) and policy processes (in WP6).

At this stage some key elements of evaluation are presented in outline with details to be developed after the collection and interpretation of feedback from the practical application of the methods and tools in the field.

### 6.2.2. Subject of the evaluation: the SI initiative in MRA and its impacts

In our approach, any SI initiative is considered to comprise two parts, to be evaluated in conjunction with each other. The first part is the SI process, and the second is the SI project. The SI process occurs with the emergence of the SI idea through the reconfiguration of social practices. The SI process is based on the
<figure><img src="image_112.png" /></figure>

following dimensions: perceived context, agency (actors and preparatory action), from reconfiguring to reconfigured social practices (i.e. new governance arrangements, new networks, new attitudes).

Once these dimensions have occurred the SI project emerges based on inputs, activities and outputs. These two parts determine outcomes and impacts (identified as 'SI effects' in Figure 20) on the economic, social, environmental and institutional domains.

<figure><img src="image_113.png" /><figcaption>**Figure 20.** SI in MRA initiative composed by an SI process, SI project and SI effects. Source: SIMRA research team.</figcaption></figure>

SI initiative processes are of different types: i) initial processes in the initial creation of the social innovation through to the reconfiguration of social practices, paving the way to the SI project; ii) project processes happening in the implementation of the SI project. In the Figure 1 the term "Process of SI in MRA" refers to the first of these two types.
<figure><img src="image_114.png" /></figure>

Processes are a continuous movement and it is challenging, within an evaluation study, to capture their
complex dynamics to understand how they affect the production of results. The same can be said in relation to
social practices (networks, attitudes and governance arrangements). To reduce this complexity to a
manageable evaluation, the approach is to create “pictures” of two types of processes at different moments of
time, allowing the entry of evidence of their effects in the SI initiative.

The term 'project' used in the evaluation framework does not refer exclusively to operations implemented due to external funds. A SI project can be and normally is implemented due to the use of the internal resources of the individual, organization, institution, network that have identified it. In both the cases (internal and external funding) the evaluation plays a central role in specifying what has functioned in the SI process, which are the outputs produced by the SI project, and what are the impacts determined by the SI initiative. This evaluation framework can be applied in both the cases.

The sequence of a SI process, that leads to a SI project, and subsequently leads to SI effects, is simplistic and it is adopt here only for the needs of evaluation practice. This sequence, as mentioned in Section 4 and Section 6.1, is aligned with the theory of change (ToC). According to Kusek and Rist (2004), the theory of change refers to the assumption that specific interventions will lead to desired results<sup>83</sup>. The ToC is normally based on a model organised in different components: inputs, activities, outputs, outcomes and impacts. The ToC is widely used at international level for the evaluation of projects, programmes and policies, e.g. by the Common Monitoring and Evaluation Framework (CMEF) of the Common Agricultural Policy. Moreover, after the Better Regulation Agenda<sup>84</sup>, which strongly relies on Impact Assessments (IA), all the European policies have to be designed and evaluated by using the ToC, which (despite the disadvantages of being a linear approach) has the positive elements of making clear the components of the process within the project, and its results and effects. This helps in the identification of the cause-effect relationships between an intervention (a SI in a MRA) and its effects (impacts).

The ToC normally assumes that the behavioural change of beneficiaries occurs once the production of outputs has been completed, thus determinining the variation in the outcome variables and so the impacts. We anticipate that in any SI initiative the behavioural change can happen at different moments of time, and in relation to different actors involved in the SI initiative. Different types of actors can be associated to the two main parts of the SI initiative.

Actors of SI process (i.e. innovator(s), follower(s) and transformer(s)) play different roles throughout the SI initiative, and can modify their behaviour during the SI process and due to the development of new attitudes, new networks and new governance arrangements, and before the production of outputs. The repeated interactions amongst actors can reinforce their behavioural change in the longterm.

Actors of the SI project are multiple (i.e. implementers or project partners and direct and indirect beneficiaries) and in their case different hypotheses on behavioural change can be formulated: a beneficiary can change the behaviour during the implementation of the project due to the interactions with the project implementers; at the end of the project due to the results achieved; after the conclusion of the project due to novel interactions. It is possible that the beneficiary will never change their behaviour or that they will return to previous habits and behaviours. In the case of implementers, different conditions can determine their behavioural change, depending upon whether they took part, as transformers, in the SI process or if they are totally new to the SI project.

<footnote n="83">⁸³ In the proposed framework and set of methods the assumptions is that the effects of SI depend not only on the types of action but also on the specific context, enabling factors and conditions (i.e. the same actions do not necessarily lead to the same results).</footnote>

<footnote n="84">⁸⁴ The Better Regulation Agenda is: “about designing and evaluating EU policies and laws transparently, with evidence, and backed up by the views of citizens and stakeholders. It covers all policy areas and aims for targeted regulation that goes no further than required, in order to achieve objectives and bring benefits at minimum cost”
https://ec.europa.eu/info/law/law-making-process/better-regulation-why-and-how_en
https://ec.europa.eu/info/better-regulation-guidelines-and-toolbox_en</footnote>
<figure><img src="image_115.png" /></figure>

These multiple issues qualify the differences in the use of ToC within SI initiatives when compared to a standard evaluation of projects. These hypotheses will be verified after the application of the method to SI case studies in the field.

### 6.2.3. Subject of the evaluation: SI initiative at local level

Three scales<fnref n="85" /> are of relevance for the evaluation of SI and its impacts in SIMRA are relevant: the spatial, the social and the temporal. However, while considerations of the spatial and social scales are reported hereafter, the temporal scale is discussed in Section 6.2.3 because it is related to the timing of evaluation.

#### Spatial scale

The spatial or geographic scale of analysis refers to the extent at which a certain phenomenon is analyzed within a certain territory, which is the overall size of a study area. Levels of analysis can be global, national, regional or local.<fnref n="86" /> In analysing the phenomena with respect to SI and its impacts in MRAs, spatial scale is important in several ways.

i) The identification of the geographical location of the SI initiative. This is to understand whether it is (or not) located in a marginalized rural area (and thus whether it is eligible as a SIMRA case study). For this purpose, a set of maps was developed at the regional level, generally NUTS 3 spatial resolution (Price et al., 2017; D3.1). However, SI often happens at lower spatial levels, and in particular, at the local level, where individuals, single organisations and local networks matter in projects implementation (Secco et al. 2014). The majority of the examples of SI that are included in the online catalogue in the SIMRA Web Site (Bryce et al., 2017; D3.2) are cases developed at local level, on relatively limited territory and size.

ii) SI emerges in a multi-level context. The various dimensions, sub-dimensions and components<fnref n="87" /> of SI may refer to different spatial levels, depending on the case, their effects are typically local. We hypothesize that the dimension “trigger and individual and collective needs” may occur at any spatial level (e.g. international, national, regional or local). However, it becomes salient for social innovation when its

<footnote n="85">⁸⁵ Scale refers to the spatial, temporal and institutional/jurisdictional dimensions used to measure a phenomenon, while levels are the elements of analysis located at different positions in a scale. Dimensions refer to the unstructured aspects of reality or phenomena to which scales are applied, such as space, time and power (Gibson et al., 2000; Veldkamp et al. 2011). Following this definition, the spatial scale applies to geographical areas, and the main levels are globe, regions, landscapes and patches. The temporal scale applies to time frequencies; levels are for example daily, weekly, seasonal, annual, etc. but they can also refer to the speed of a process (fast/slow) or durations (long/short). In the jurisdictional scale, the analysed dimension refers to administrations. Typically, its levels are inter-governamental (e.g., the EU), national, provincial, localities (e.g. municipalities, towns); but others are also used (based for example on aggregations of localities – such as the “Unions of Municipalities” in mountainous areas in Italy). Scales apply also to social sciences, social systems, actors and businesses as well as decision-making processes, that can be analysed by using a social scale, that typically refers to micro, meso and macro levels. Other types of scale exist (e.g. institutional, management, networks, knowledge, etc.), with related levels. The institutional scale for example refer to rules, and the level are constitutions, national laws and regulations, and operating rules. The management scale refers to plans, and levels can be strategies, projects, tasks. The network scale refers to links, and levels can be trans-society, society, kin, family. The knowledge scale can vary from general to specific and from universal to contextual (Cash et al. 2006). But those we consider relevant for the evaluation of SI and its impacts are: the spatial, the social and the temporal scales.</footnote>

<footnote n="86">⁸⁶ To analyse physical or socio-economic phenomena, in general we consider administrative boundaries of territories (Cash et al. 2006; Gibson et al. 2000). As mentioned in Price et al. (2017; D3.1), SIMRA uses the subdivision of the European Union territory provided by EUROSTAT. The EU territory is divided into five levels: three levels of Nomenclature of territorial units for statistics (NUTS) and two levels of local administrative units (LAUs). The three NUTS level are the following: NUTS1 corresponds to groups of regions within the Country (formally NUTS0); NUTS2 corresponds to the regional level at sub-national level while NUTS3 corresponds to the provincial level at sub-national level. The LAU level includes the upper LAU level (LAU1, formerly NUTS4), which is defined for most, but not all of the countries; and the lower LAU level (LAU2, formerly NUTS5), which corresponds to most detailed level of analysis i.e. the municipal one. Depending on the size, not all countries have every level of division: indeed, the upper LAU level (LAU1) is not defined for all of the countries, while the lower level (LAU2) is present in the 28 EU Member States. In non-EU Mediterranean countries (e.g. North African countries) the administrative subdivision is different.</footnote>

<footnote n="87">⁸⁷ The SI concepts have been explicated to guide the evaluation - see the evaluation framework in Part II.</footnote>
<figure><img src="image_116.png" /></figure>

effects and reactions appear at local level (Oosterlynck et al. 2013; Baker and Mehmood, 2015). Thus, while the sub-dimension “trigger” can be international (e.g. the global financial crises), its effects (e.g. loss of jobs) and related societal “needs” (e.g. support to income of poor families) are typically locally specific and thus have to be analysed at local level. Similar considerations and dynamics are valid for many other dimensions of SI (e.g. perceived context, reconfigured social practices, impacts, etc.).

The local level at which the SI occurs in a certain territory can both have clear boundaries (e.g. a group of landowners within the same municipality) or unclear boundaries (e.g. a private-public hybrid network engaged in different nearby areas but not in the same municipality). In the first case, the analysis can refer to a single municipality (LAU2); in the second, it can refer to more than one municipality (scaling up to a territorial or landscape level), and at different jurisdictional levels<fnref n="88" />. Only in the first case, when all the actors involved in the SI initiative, including the beneficiaries, are located in the same municipality our statistical unit of analysis will correspond to a LAU2 unit. In the second case, the unit of analysis will likely be an aggregation of LAU2 units, the dimension/size and borders of which will depend on the SI initiative and territory where the initiative is implemented. For simplicity, even when the analysis refers to a larger spatial unit with respect to a LAU2 unit and thus it corresponds to a spatial meso-level of analysis, in our evaluation approach it is referred to it as “local level”. This approach is mainly used for the examination of the context (context indicators and narrative analysis) and SI impacts (impacts indicators and narrative analysis) (see Part IV for more details).

A final consideration is needed of those SI initiatives that are structured into different layers, at various
jurisdictional levels. This is the case, generally, of advanced or mature and well-consolidated SI initiatives,
which started initially from one (or a few separated) local SI process(es) and/or project(s) and were able to
scale out and/or to scale up.

To scale out means they were able to replicate the same innovation in different locations, while to scale up
means to move the innovation to a broader system, e.g. bringing it to a regional or national level (see Part II,
section XV on learning processes)<fnref n="89" />. In this case, local SI initiatives are connected both with SI initiatives
located in different areas, thus creating horizontal networks, and with entities built at a higher jurisdictional
level by the horizontally networked SI initiatives. The latter is for the purpose of representing the SI initiatives

<footnote n="88">⁸⁸ For methodological reasons, the relevant spatial level of analysis is the local one. However, multi-level and trans-boundary interconnections are also considered, especially by identifying links and networks among actors who belong to different jurisdictional and spatial levels. The term trans-boundary refers for example to trans-national or trans-regional interconnections i.e. the links of SI occurring in a certain territory with other administrative/institutional levels and territories: it allows to grasp urban-rural-periurban relationships and linkages existing between organisations involved in SI initiatives located in different spaces and should be taken into consideration in evaluating SI impacts. Similarly, interconnections between sectors at different levels should also be taken into consideration. “Cross-level” interactions refer to interactions among levels within a scale, whereas “cross-scale” means interactions across different scales, for example, between spatial domains and jurisdictions. “Multilevel” is used to indicate the presence of more than one level, and “multiscale” the presence of more than one scale, but without implying that there are important cross-level or cross-scale interactions (Cash 2006).</footnote>

<footnote n="89">⁸⁹ One good example of these dynamics is the case of the National Forum of Social Farming (FNAS in the Italian acronym of *Forum Nazionale dell'Agricoltura Sociale*), in Italy. The Forum is a national network of local SI initiatives mainly focused on social and employment inclusion of disadvantaged people through their active involvement in farming activities. Some of the local SI initiatives are located in marginalised rural areas, but of course the national network also includes initiatives in urban areas or in rural areas that are not marginal or marginalised (according to the criteria defined in Deliverable 3.1). At local level, the single initiative is based on networks of many actors (e.g., farms, social cooperatives, host communities, parents' associations, experts, public authorities of the social-health sector and others). These local initiatives are networked at a higher level into regional forums of social farming (7 out of 21 Italian regions have a regional forum formally established). These Regional Forums are then networked at a higher level, into the National Forum. However, the National Forum also includes other provincial, regional or national associations (e.g., the Association "Organic-social farms of Veneto", the National Association of Organic Agriculture, and many others). The main goal of the National Forum is to represent the experience of social farming at national level. More information are available in the web site, URL: http://www.forumagricolturasociale.it/ and in the document available at: http://www.maielist.org/project.eu/fileadmin/user_upload/de/dateien/newsletter_FNAS_I Italy_eng.pdf</footnote>
<figure><img src="image_117.png" /></figure>

in the political debate at a regional and/or national context, and for obtaining greater visibility and recognition
by the public authorities and private donors.

SI initiatives are linked in vertical networks, involving different administrative/jurisdictional levels, from the local to the regional, and from the regional and to national. In these cases, the whole SI initiative can be considered a network of networks. However, from the perspective of the evaluations in SIMRA, what matters (and what can be measured) are the impacts determined by the SI initiative on the social needs and/or societal challenges in each context, i.e. again at local level. Therefore, in these complex cases, where the SI initiative as a whole is made up of different entities or networks at different levels, the main statistical unit of analysis remains the local one.

The role of the entities involved at higher hierarchical level (e.g. a national association/federation or forum) and their interactions with the local SI initiatives will be explored in two directions.

i) the survey at local level investigates whether and how these national or regional entities/organisations have influenced the emergence of the SI in that case, e.g. due to its communication activities for sharing information, training options or representativeness of the SI interests in national debates. In this sense, the national or regional entity will be understood as one of the conditions of the context (i.e. it is explored in the dimension "Perceived context" of SIMRA evaluation framework).

ii) The survey at local level investigates whether and how the local SI initiative has provided feedback and passed knowledge, best practices or skills to higher levels thus eventually determining changes, e.g. by increasing the power and capacity of the national entity to influence policy reforms or to attract funds. In this sense, the national and regional entity will be understood as one of the effects of SI in terms of further innovation and learning processes (i.e. it is explored in the dimension "Learning processes" and sub-dimensions "Feedback loops" and "Multiplier and critical effects" of SIMRA evaluation framework).

We assume that behavioural changes occuring in each SI initiative at local level can contribute to behavioural
changes at higher levels, but such contributions will be analysed and considered case by case, not in an
aggregated way. Analysis is case-specific and the total contribution cannot be measured as a "sum" of the
contributions by the local initiatives.

Social scale

The social scale considers socio-economic entities (Blalock, 1979; Krugman et al. 2013), as individuals or collective entities that act in the society or in a certain economic sector (e.g. agriculture). In sociological sciences, the phenomena that occur at the social scale can occur at three levels<fnref n="90" />: micro level refers to individuals (individual attitudes, face-to-face interactions) or small groups (family); meso level refers to larger social groups or networks (e.g. residents in a certain area, cooperative groups, a federation of companies working in the same sector), and macro level refers to institutionalized large-scale social processes (e.g. the whole population of a country).

In economics, the micro-level is typically a single company or an individual (e.g. a consumer, a producer); the meso level<fnref n="91" /> can be an association or federation of companies working in the same sector; and the macro level refers to the whole economic sector in a country.

In SIMRA, the micro-level of analysis at a social scale corresponds to a single organisation, a single person, a single NGO or association acting for SI; the meso level relates to a cooperation system (i.e. a cluster or network of entities, typically collective); and the macro-level refers to a cluster of clusters, i.e. a broader network system. The evaluation is grounded on data collection mainly based on interviews with local actors involved in the SI initiative, and in particular innovator(s) and follower(s) (i.e. agency), as well as transformer(s) and beneficiaries (see Part IV and Annex D.1).

<footnote n="90">⁹⁰ http://www.tropentag.de/2007/abstracts/full/240.pdf</footnote>

<footnote n="91">⁹¹ The meso level is not a generally accepted term in mainstream economics and various proposed definitions are not consistent. One of the most famous definitions comes from Schumpeter's interpretation, as "a structure component of a 'deeper' invisible macro structure" (Dopfer, 2012: 133).</footnote>
<figure><img src="image_118.png" /></figure>

## Summary

The focus of data collection is at the micro level of analysis, where the statistical units of analysis are individuals or single organisations, or local networks that act in SI initiatives. The focus of the examination of the SI impacts is at the meso-level of analysis, where the statistical units of analysis are the individual SI initiative (i.e. the case-study) and the territory where the SI project is implemented (that can involve more that one municipality).

In terms of data to be collected, primary data are collected at the local and/or territorial level, while secondary socio-economic, institutional and environmental data are often available at regional, national, European and other macro-regional spatial levels. Specific databases have to be identified and explored for collecting secondary data to be used for specific indicators as part of the evaluation process, if available and when necessary. More details are provided in Part IV.

### 6.2.4. Timing of the evaluation: ex-post/phases

The proposed evaluation framework analyses cause-and-effects of SI with a sequential temporal scale. Thus, we observe drivers of action, changes that have arisen, and short- and long-term consequences. In relation to the "time" factor, we hypothesise that:

i) Trigger and/or needs may occur at any temporal level: over long, short, or intermittent periods (Gibson et al., 2000); as mentioned in section 5.2.1, a trigger can be a prompt event (e.g. an earthquake that requires immediate interventions), or derive from a long process of change (e.g. a progressive decline of population that allows for strategic plans to be developed). Moreover, new triggers and needs can happen at any moment. This can be from the outset or during the whole SI process, thus affecting the SI dynamics (see Kluvánková et al., 2017; D2.2). Triggers and needs are presented only as initial elements of SI in Figure 10. Whether and how a different temporal scale of triggering is relevant in SI and its impacts have to be explored by means of SIMRA case studies evaluation, to be carried out in Work Package 5;

ii) Once triggered (i.e. inception), any initiative of SI can be described in its various phases over time, i.e. conception, formulation, realization and final evaluation. These temporal phases are very similar to the ones of traditional policy cycle (e.g Easton, 1957; Krott, 2005), allowing the evaluation of SI as a process, as a project and as a local policy initiative<fnref n="92" />. The duration of SI with respect to the *momentum* of evaluation is another key aspect. Typically, if the SI to be evaluated is in its conception phase, the evaluation can be classified as *ex-ante*. It allows the identification and description of the enabling factors of SI process or who participates in the SI process and the expected impacts (not actual impacts). If SI is in its realization phase, when the SI project starts and is implemented, the evaluation can be classified as *in-itinere*, focusing on project processes and outputs. Finally, if the SI process and/or project have ceased to exist (i.e. it is completed, ended or stopped), the evaluation can be classified as *ex-post*. In this case, and especially (but not only) if the SI is a long-lasting initiative started several years previously, SI impacts can also be sought. Each situation requires a different approach to its evaluation.

<footnote n="92">⁹² In the same way, even if they are graphically represented and conceived in our evaluation framework as sequential phases of a problem-solving model (and their interlinks as input-activities-outputs-outcomes), we are aware that in any real situation of social innovation several "reverse loops" might occur: the process is stopped and it evolved in a different direction with respect to the initial one; there might be some new triggers emerging that determine a different evolving process; the process is nor linear neither consequential; outputs are not always derived from planned activities; outcomes might have impacts at various spatial levels, etc. The process represented in Figure 10 and Figure 21 could more usefully be designed as a cycle, with outcomes/impacts spurring more action (upscaling and outscaling), but its graphical representation would have been more complex and difficult. While it would be more correct from a theoretical/conceptual point of view, it would also be less effective for the purposes of the scope of application of this framework within an evaluation (see an example in GIZ, 2013). Notwithstanding the inevitable simplifications, such a consequential approach allows us to highlight in a simple way the different elements of SI that will need to be evaluated. For a discussion on the non-linear nature of SI dynamics see Kluvánková et al. (2017; D2.2).</footnote>
<figure><img src="image_119.png" /></figure>

iii) The evaluation is divided into two main phases. In phase 1, the initiative of social innovation may be evaluated mainly in terms of configuration process; in phase 2, the initiative of social innovation may be evaluated mainly in terms of its consequences. Thus, phase 1 will focus on the evaluation of social innovation *per se*, and phase 2 will focus on the evaluation of its outputs and impacts, starting from the reconfigured social practices (i.e. a situation from which SI projects can emerge and be implemented). The reconfigured social practices represent – in our view – the passage from the SI process to the SI project. We are aware that in practice this step could be much more complex, but as pointed out in Part II, the proposed evaluation framework requires a simplification of complex interactions and interconnections (see also Figure 20) needed to facilitate the evaluators in their tasks. The two phases of evaluation, run sequentially, are presented in Figure 21. Data collection instruments and indicators used for the evaluation will be different phase by phase.

<figure><img src="image_120.png" /><figcaption>**Figure 21.** The temporal phases of social innovation and of evaluation. The reconfigured social practices represent – in our view – the passage from the SI process to the SI project. Source: SIMRA research team.</figcaption></figure>

### 6.2.5. Setting the scene: complementarity with existing evaluation frameworks/methods

Social innovation in marginalised rural areas can be co-financed by using already existing EU policies and programmes conceptualised and carried out to reduce the level or marginality of European rural contexts. The second pillar – Rural Development Programme (RDP) – of the Common Agricultural Policy (CAP) plays a central
<figure><img src="image_121.png" /></figure>

role in relation to this aim. The SIMRA evaluation framework should be integrated in such existing evaluation frameworks. Figure 22 presents a proposal for connecting the SIMRA evaluation framework to the EU Common Monitoring Evaluation Framework (CMEF) of the CAP, via the components of the theory of change (ToC). The figure shows the dimensions of SI in its upper part, the corresponding phases of the project cycle at the center (i.e. programming, identification, formulation, financing, implementation and evaluation), and how these previous elements connects to the ToC of the CMEF and to its indicators in the lower part of the figure.

In relation to this last post, we recall that the evaluation of RDP is defined within the EU Regulation No 1303/2013 by using a set of context, input, output, target, result and impact indicators. These indicators are refered to the monitoring and evaluation of RDP projects, and they do not take into account the processes, which are of central importance in SI initiatives. More specifically, context indicators of CMEF can be used for the context analysis of SI initiatives, the input and output indicators of the CMEF can be used for the analysis of the same elements in the SI projects, and the result and impact indicators of the CMEF can be used for the analysis of the same elements of SI initatitives.

In this regard, we can observe two different working hypotheses which try to respond to the following question: "When do the phases of identification and formulation of the SI project start?".

1. The identification and formulation phases of the SI project can overlap with the SI process (Figure 22, Case A). In this case, social problems determining both the process and the project are exactly the same.

2. The identification and formulation phases of the SI project start once the reconfiguration of social practices has been concluded (Figure 22, Case B). In this second case, social problems determining SI process changes during the time, thus social problems determining the SI project could be novel ones, or a sub-set of the previous ones.

In terms of evaluation, the practical consequences of these two different working hypotheses are:

1. In Figure 22, Case A, the evaluation criterium of relevance will consider the same problems both for the analysis of the SI process and project; the evaluation criterium of impact will be based on the same outcome variables – which is strinctly linked to the perceived social problems – both for the analysis of SI process and project (see additional element of specifications in relation to the outcome variable for impact analysis in the part of impact evaluation).

2. In Figure 22, Case B, the evaluation criterium of relevance will consider different social problems in the analysis of the SI process and project; the evaluation criterium of impact will be based on different outcome variables for the impact analysis of SI process and project.
<figure><img src="image_122.png" /></figure>

### CASE A

<figure><img src="image_123.png" /></figure>

### CASE B

<figure><img src="image_124.png" /><figcaption>**Figure 22.** Linking the conceptual framework of SI to the evaluation framework/theory of change typically adopted by the European Commission Common Monitoring Evaluation Framework. – Case A above and Case B below. Source: SIMRA research team.</figcaption></figure>
<figure><img src="image_125.png" /></figure>

These are only possible indications on how to adapt the SIMRA evaluation framework to an already existing evaluation framework such as the CMEF, thus making the best use of already existing indicators by integrating them with our evaluation questions and indicators, which are specific to social innovation.

6.2.6. Setting the scene: policy analysis

As specified in section 6.2.2, in evaluation of SI in MRAs scale has a fundamental role. Observations show that SI generally arises at the local level rather than at higher jurisdicational and spatial levels. At a local level it is possible to analyse several aspects of SI in detail, including why and how it has emerged in a certain context, identifying the innovator(s) and the follower(s), how the SI process works in practice and, above all, the outcomes for direct beneficiaries and the impacts for both direct and indirect beneficiaries.

Although the SI may manifest itself or be identified in a local place, SIs are often characterised by significant cross-level and cross-scale<fnref n="93" /> interplay among actors at different levels (through boundary or bridging organisations). Scale becomes relevant especially when the SI initiative under study is the result of scaling up or it is policy-supported/policy-induced. While SI cases may not be initially supported by any policy (endogenous process), their success may lead to their replication (scaling out) or to the implementation of neo-endogenous approaches such as focused funding by external donors or financiers, or adaptation, changes or development of new policies or legislation, to scale up the initiative. In the field of interest of SIMRA, several policies and programmes developed at various levels (from international to local) can support (or impede) SI. Examples of different types of policies at different jurisdictional and administrative levels include international conventions on climate change, EU rural development programmes, national legislation on network contract (e.g. in Italy), regional laws to stimulate investments and local regulations to promote employment opportunities for youth. Consequently, in policy analysis, it is necessary and typical to adopt a cross-level approach, i.e. to understand what effects have international policies at regional or local levels.

From a methodological point of view, while SI and impacts of SI can be evaluated at the case study level (typically, local), policy implications for SI need to be analysed with a cross-level approach. The two approaches require two sets of methods and tools that are complementary. At a local level the focus is on the SI process and impacts, and the evaluation is based on interviews and focus groups that will include local stakeholders and actors (e.g. innovators, clique, local policy makers). Implications of international or national policies for SI at local level can be identified by means of questions to local stakeholders and innovators, such as "What policies have most affected (negatively or positively) your SI initiative?" However, the effect perceived by local stakeholders might be different from what is planned, perceived and known about policy implications at local level by policy makers who work at higher jurisdictional levels. Thus, there is a need to integrate the analysis of policy effects as perceived and emerging at local level (to be done in SIMRA CSs evaluation), with the analysis of policy and policy processes based on documents analysis (e.g. texts of EU regulations, other policy documents) and/or policy experts' consultation, where policy experts are typically at national or regional levels. Figure 23 represents how the evaluation of SIMRA local case studies are integrated with the analysis of policy processes on SI in the evaluation approach.

<footnote n="93">⁹³ Following Cash et al. (2006), “cross-level” interactions refer to interactions among levels within a scale, whereas “cross-scale” means interactions across different scales, for example, between jurisdictions. “Multilevel” is used to indicate the presence of more than one level, and “multiscale” the presence of more than one scale, but without implying that there are important cross-level or cross-scale interactions.</footnote>
<figure><img src="image_126.png" /></figure>

<figure><img src="image_127.png" /><figcaption>**Figure 23.** Matching national and local scale (where SI generally arises) through two different evaluation instruments. Source: SIMRA research team.</figcaption></figure>

Further information on how the cross-level policy analysis is integrated in the local case study evaluation is provided in Part IV.

### 6.2.7. Setting the scene: Who?

Agency is the “starting point” as well as one of the core dimensions, on which the proposed evaluation framework and methods focuses. Agency refers to ‘who’ has initiated the SI in a certain territory, and then to next development steps. An agency’s actors are the first who are involved in data collection, starting with a focus group and moving to interviews (the first to be interviewed being innovator(s) and follower(s)). However, other actors involved in the sequential phases of the SI initiative development should also be included in the survey. Table 11 summarizes the actors targeted in the survey, in the different phases of SI. Depending on the phase, and thus on the focus of the evaluation, the sampling design and related sampling method are different. More details on this are provided in Part IV.

<table><caption>**Table 11.** Target population, related network structure and sampling design according to different phases of SI as planned for SIMRA case studies evaluation.</caption><thead><tr><td>Phases of SI</td><td>Main focus of the evaluation</td><td>Actors to be analysed (target population)</td><td>Sampling design (and sampling method)</td></tr></thead><tbody><tr><td>Initial phases (t<sub>0</sub>)</td><td>Agency during the ideation of SI, its values, ideas, etc. ("the SI egg")</td><td>Leader(s)<br/>Follower(s)<br/>(cliques of individuals)</td><td>Census (identified through stakeholders analysis)</td></tr><tr><td>Intermediate phases (t<sub>1</sub>)</td><td>The action of reconfiguring (the SI process)</td><td>Leader(s)<br/>Follower(s)<br/>Implementer(s)<br/>(network)</td><td>Census within the network (Social Network Analysis methods)</td></tr></tbody></table>
<figure><img src="image_128.png" /></figure>

<table><tr><td>Advanced phases (t<sub>2</sub>)</td><td>The effects of SI (outcomes and impacts on two groups: direct beneficiaries i.e. the SI network, indirect beneficiaries, i.e. the whole territorial community)</td><td>Leader(s)<br>Follower(s)<br>Implementer(s)<br>Direct beneficiaries<br>Indirect beneficiaries (community)</td><td>Sampling (method to be defined case-by-case, depending on availability of data)</td></tr></table>

Source: SIMRA research team.

Several key methodological issues have to be addressed during the evaluation of SIMRA case studies. for example, the sampling design, the identification of the boundaries of the statistical unit to be evaluated (i.e. who is in and who is out of the analysis, key characteristics of actors/agents to be analysed, and other issues). The entity (individual, clique or organisation) that innovates can be analysed by both semi-structured in-depth interviews and structured interviews (based on questionnaire). If the use of Social Network Analysis is anticipated (e.g. to understand the changes in the local networks and relationships among involved actors<fnref n="94" />), a census of all the actors involved is necessary. To identify the people to be interviewed, a stakeholders' analysis at the level of "Agency" needs to be performed at the beginning of the evaluation. The different aspects need to be evaluated along all three of the main phases of SI development, from t₀ to t₂, but the focus and the methods are different. For example, in the evaluation of advanced phases of SI (at t₂), the innovators' ideas are at the core of the evaluation; and vice versa, in the evaluation of initial phases of SI (at t₀), the outcomes and impacts are still embrionic (or do not exist yet). More information on this is provided in Part IV.

### 6.2.8. Setting the scene: domains of impact

It is assumed that a SI initiative may determine impacts in four different domains: economy, society, environment and governance/institutions. However, for the purposes of the evaluation, more detailed components of each domain have to be identified. The initiative can have both positive and negative impacts, on one or more of these domains; moreover, the impacts can be different on different components of the same domain. For example, with respect to the environment, a SI initiative may have positive impacts on biodiversity conservation but negative on water quality; with respect to the economy, a SI initiative may have internal positive impacts (e.g. high value added of its services/products) but external negative impacts on other actors not directly involved in its SI project (e.g. competition with business-as-usual companies providing same products or services). Similar considerations are valid also for the social and institutional domains. A SI initiative can have different impacts on different groups or entities, thus implying trade-offs. To deal with these trade-offs, the identification of the impacts that derive from a SI initiative should be as accurate as possible. For this purpose, lists of possible aspects are integrated in the questions to help respondents to qualitatively evaluate related impacts. The lists are neither exhaustive nor fixed, and other aspects can be considered. Therefore, it is useful to carry out a preliminary screen of the various aspects, in each domain in which the SI can have impacts. More information is provided in section 6.2.10.1, Part IV and Annex 1 (questions H.e.2.1-2-3-4).

### 6.2.9. Evaluation questions

The SIMRA evaluation approach seeks to answer evaluation questions based on a set of evaluation criteria. Examples of evaluation questions related to the most commonly used evaluation criteria are provided in Table 12. Questions are suggested separately for "SI process", "SI project" and the whole "SI initiative". These are preliminary suggestions and questions, in line with what is classically proposed for the evaluation of projects. They are provided as general examples of what the evaluators can use. However, other additional criteria (and related questions) are expected by the end of the SIMRA project, to be reported in Deliverable 4.3. This will be based on the application of the preliminary evaluation method to the selected SIMRA case studies, as social innovation issues may arise during the field work in the case studies.

<footnote n="94">⁹⁴ Borgatti et al. (1998) operationalise connections among actors, by identifying their position as either internal or external to the actor under study. Relations within and among actors of a SI initiative can be defined according to the following three categories: (i) relationships *external* to an individual actor, (ii) relationships *internal* to a collective actor, and (iii) relationships *external* to a collective actor.</footnote>
<figure><img src="image_129.png" /></figure>

<table><caption>**Table 12.** Examples of evaluation questions related to evaluation criteria of the SI process, SI project and SI initiative.</caption><thead><tr><th>SI Evaluation questions</th><th>Relevance</th><th>Efficiency</th><th>Effectiveness</th><th>Impact and sustainability</th></tr></thead><tbody><tr><td>SI process</td><td>Is the SI process consistent with the contextual conditions?<br/>Is the reconfiguring of social practices consistent with the aims of local policies?</td><td>How much time has the SI process required?</td><td>To what extent were/are the objectives of the SI process achieved/likely to be achieved?<br/>What were the major factors influencing the achievement or non-achievement of the objectives of the SI process?</td><td>What has happened as a result of the SI process?</td></tr><tr><td>SI project</td><td>Is the SI project consistent with the contextual conditions?<br/>Is the SI project consistent with the aims of local policies?<br/>Are the activities and outputs of the SI project consistent with the intended impacts and effects?</td><td>Were activities cost-efficient?<br/>Were objectives of the SI project achieved on time?<br/>Was the SI project implemented in the most efficient way compared to alternatives?</td><td>To what extent were/are the objectives of the SI project achieved/likely to be achieved?<br/>What were the major factors influencing the achievement or non-achievement of the objectives of the SI project?</td><td>Which are the positive and negative changes produced by the SI project directly or indirectly, intended or unintended on beneficiaries?<br/>To what extent did the benefits of a SI project continue and spread in the local community?</td></tr><tr><td>SI initiative</td><td>To what extent are the objectives of the SI initiative still valid?</td><td>Does the SI initiative display in an efficient way?</td><td>To what extent were/are the objectives of the SI initiative achieved/likely to be achieved?<br/>What were the major factors influencing the achievement or non-achievement of the objectives of the SI initiative?</td><td>What real difference has the SI initiative made to the beneficiaries?<br/>What were the major factors which influenced the achievement or non-achievement of sustainability of the SI initiative?</td></tr></tbody></table>
<figure><img src="image_130.png" /></figure>

<figure><img src="image_131.png" /></figure>

### 6.2.10. Judgement references: indicators

The method being applied for the construction of indicators has been tested and applied in similar research relating to the evaluation of social capital in local development processes (see Pisani *et al*. (in print).

In SIMRA, social innovation initiatives in MRAs, analysed at the local level, are deconstructed for evaluation purposes by using an abstraction scale which allows to investigate general, intermediate, specific, basic and then single unit concepts (Corbetta, 2014). The scale of abstraction is applied to the evaluation of social innovation in MRAs, allowing the categorisation of social innovation, its dimensions, sub-dimensions, components and indicators. The disaggregation of concepts for evaluation purposes allows the organisation of the evaluation by linking the general concept of social innovation in MRA to its intermediate concepts (dimensions of SI); the intermediate concepts (dimensions of SI) to specific concepts (SI sub-dimensions); the specific concepts to basic components (SI indicators); and finally, the basic components (SI indicators) to single unit concepts (variables to be measured). The variables are then linked to questions in the survey and tailored to five different categoryies f respondents: the key informants of the SI initiative, the clique, the network members, the project partners and the beneficiaries of SI project. All these elements are detailed in Table 13.
<figure><img src="image_132.png" /></figure>

<table><caption>**Table 13.** Structure of the evaluation practice for social innovation in MRAs.</caption><thead><tr><th colspan="2">CONCEPTUALISATION</th><th colspan="3">EVALUATION PRACTICE</th></tr><tr><th>Scale of abstraction</th><th colspan="3">Elements of evaluation</th><th>Statistical tools</th></tr></thead><tbody><tr><td>General concept</td><td colspan="3">Social Innovation in MRA</td><td>Composite indices</td></tr><tr><td>Intermediate concepts</td><td colspan="3">Dimensions</td><td>Indices</td></tr><tr><td>Specific concepts</td><td colspan="3">Sub-dimensions</td><td>Composite indicators</td></tr><tr><td>Components</td><td colspan="3">Indicators</td><td>Indicators</td></tr><tr><td colspan="5" style="text-align:center;">Survey at local level<br>(each single SI initiative)</td></tr><tr><td>Single unit concepts<br>(single questions in<br>the survey<br>questionnaire)</td><td>Questionnaire<br>1<br>Key<br>informants</td><td>Questionnaire<br>2<br>Clique</td><td>Questionnaire<br>3<br>Network</td><td>Questionnaire<br>4<br>Project<br>partners</td><td>Questionnaire<br>5<br>Beneficiaries</td><td>Variables to<br>be measured</td></tr><tr><td>Unit concept 1</td><td>X<sub>1</sub></td><td></td><td></td><td></td><td></td><td>Variables of<br>unit concept 1</td></tr><tr><td>Unit concept 2</td><td></td><td>X<sub>2-2</sub></td><td></td><td>X<sub>2-4</sub></td><td></td><td>Variables of<br>unit concept 2</td></tr><tr><td>Unit concept 3</td><td></td><td></td><td>X<sub>3</sub></td><td></td><td></td><td>Variables of<br>unit concept 3</td></tr><tr><td>Unit concept 4</td><td>X<sub>4-1</sub></td><td></td><td>X<sub>4-3</sub></td><td>X<sub>4-4</sub></td><td></td><td>...</td></tr><tr><td>Unit concept 5</td><td></td><td>X<sub>5-2</sub></td><td>X<sub>5-3</sub></td><td></td><td></td><td>...</td></tr><tr><td>Unit concept 6</td><td></td><td></td><td></td><td>X<sub>6-4</sub></td><td>X<sub>6-5</sub></td><td>...</td></tr><tr><td>Unit concept ...</td><td>X<sub>...</sub></td><td>X<sub>...</sub></td><td>X<sub>...</sub></td><td></td><td></td><td>...</td></tr></tbody></table>

Source: Pisani *et al.* (in print) – mod.

Data on the variables will be collected by means of interviews with the five categories of actors. The data collection tools are questionnaires (one for each category). The variables included in the proposed evaluation method are of different types, depending on how they are constructed. Variables within the X<sub>n</sub> type are based on a question posed either to one actor or another (i.e. one question can be posed only to one category of actor, such as key informants). Variables within the X<sub>n-n</sub> type are based on the same question posed to more than one category of actor (e.g. X2-2 and X2-4 are based on the same question posed to both clique and project partners on the same single unit concept). This approach allows to operationalise a statistical method for evaluating SI in MRAs where the different concepts are measured at diverse levels of aggregation. Different types of statistical tools will be developed and used, depending on elements to be evaluated. In particular:

i. Composite indices will be used if referred to the general concept (SI in MRAs)

ii. Indices will be used if linked to dimensions of SI in MRAs (intermediate concepts)

iii. Composite indicators will be used if connected to sub-dimensions (specific concepts)

iv. Indicators - with the same wording for both the evaluation and statistical methods - will be used for components.

The value of each indicator will be calculated from responses to one or more questions, as well as from responses across one or more of the five questionnaires. Each question has its own range of responses. For
<figure><img src="image_133.png" /></figure>

example, they can vary from [0-1] in case of a [No-Yes] answer, to a [1-10] for a Likert scale or to [0-100] in the case of a rate. Each indicator is calculated from questions, regardless of the range of responses. The methodological process for the calculation of each indicator will follow a standardised format, which will include the question or questions from where it is derived (i.e. questionnaire 1, 2, 3, 4, 5), the possible responses with coding for each response, as well as the range of the responses (see Figure 24).

<figure><img src="image_134.png" /><figcaption>**Figure 24.** Social innovation composite indices, indices, composite indicators and indicators. Source: Pisani et al. (in print) – mod.</figcaption></figure>

Some of the indicators proposed for the evaluation of social innovation and its impacts will be calculated through indices used in the Social Network Analysis (SNA), an approach that has been increasingly used in a wider set of fields of application because it highlights the central role that networks play in social relations (Borgatti et al., 2002 and 2009; Bodin et al., 2006; Ingold, 2008; Borgatti et al., 2009; Secco et al., 2014; Pisani et al., in print), and in the evaluation of local development projects and programmes (e.g. Pisani & Laidin, 2016). As Scott (2013) explains, SNA shows how actors (points or nodes) are connected through relationships (lines or ties) visually, through graphs.

Different types of relationships can be identified through direction and thickness of the lines. SNA uses also matrices of data that are “oriented”, “non-oriented”, binary (1 = presence of linkage; 0 = absence of linkage) or expressed through numerical values that identify the “strengths” of relationships. In addition, SNA uses indices that may be grouped as three main typologies: (i) indices which describe the whole network structure, (ii) indices referred to the role of actors inside the network, and (iii) measures of the subgroups (Chiesi, 1999). In the SIMRA project, SNA will be adopted to analyse (i) exchange of information; (ii) friendship; (iii) informal collaboration; (iv) formal collaboration; and (v) trust among members of the network. Each relationship will be binary (presence or absence), and non-oriented. Data will be analysed with UCINET, a software package for analysing social network data developed by Steve Borgatti, Martin Everett and Lin Freeman (2002) and with GEPHI open source software developed by Bastian and Heymann (2009). The main SNA indices that will be used will be: density (the proportion between the number of ties and all possible ties among nodes), isolated nodes (the number of nodes which have no contacts with any node), measures of centrality (when the position of the actor is measured in relation to other actors in the network), and core/periphery analysis (the core is the sub-group with the maximum density). These indices will be applied to measure relationships among actors depending on the social innovation network considered.
<figure><img src="image_135.png" /></figure>

As explained above, each indicator will have its own range that will be tied to the scale of measurement of the data. The normalisation of the indicator's values will be required because the scales of measurement used are different (Figure ZZZ). Comparison between different indicators with different ranges would be possible only after normalisation, which would transform indicators' values to a unique scale [0-1]. The values could then be comparable and aggregated at different levels of the abstraction scale. Possible normalization techniques to be used are¹: (i) normalisation through the maximum and minimum range expected in the indicator; (ii) normalisation through the higher and lower values observed in the distribution of data collected in all the case studies; (iii) normalisation by using quartiles, where maximum scores are given to values above the third quartile, a 0.5 score to values between the first and third quartile, and a null score to values that are below the first quartile; (iv) normalisation by clusters, where by looking at the distribution of values obtained, case studies are divided in three clusters and normalisation assigns maximum scores to the highest cluster, a 0.5 score for the cluster in the middle, and a null score to the lowest cluster.<fnref n="95" />

Indicators will be calculated at the lower level of the aggregation process, which means that they directly
derive from the questions (also called single unit concepts) in the survey. At this level, they refer to the
components of social innovation. At the level of sub-dimensions, the indicators are aggregated into composite
indicators (or specific concepts), at the level of dimensions the composite indicators are aggregated into
indices of social innovation (or intermediate concepts). At the higher level (general concepts) the indices are
aggregated into composite indices of SI and its impacts in MRAs. During the aggregation phase, the same
weight, by way of a simple mean, will be given to each of the different components considered, independently
of the number of indicators included. This is the simplest approach to aggregation. Evaluators and researchers
could adopt a multi-criteria analysis based on expert consultation and use it to differentiate the weights given
to the identified sub-dimensions. However, further research would be needed to ascertain the impacts of
different weighting techniques on results.

6.2.11. Methodological design: analysis of impacts

Impact evaluation “figures among a broad range of complementary methods that support evidence-based policy” being “a particular type of evaluation that seeks to answer cause-and-effect questions” (Gertler et al., 2016: 7). Unlike other types of evaluation, which can answer descriptive or normative types of questions (see Section 5.1), impact evaluations are structured around a “basic question: *What is the impact (or causal effect) of a programme on an aspect of interest?*” (ibidem: 8). Other possible, similar questions are: “*How much better off are the beneficiaries because of the programme/policy?*”; “*How would outcomes change if changed program design?*”. What is important in impact evaluation is only the effect on outcomes that the policy, programme/ or event causes directly. Therefore, impact evaluation is used to isolate the effects of a well-defined cause (e.g. a policy, a programme, a project or an event) with respect to the effects determined by other different causes. “An impact evaluation looks for the changes in outcome that are directly attributable to the programme” (ibidem: 8) (or policy, event, etc.). “The focus on causality and attribution is the hallmark of impact evaluations and determines the methodologies that can be used. To be able to estimate the causal effect or impact of a programme on outcomes, any method chosen must estimate the so-called counterfactual, that is, what the outcome would have been for programme participants if they had not participated in the programme. In practice, impact evaluation requires that the evaluator find a comparison group to estimate what would have happened to the program participants without the program” (Gertler et al., 2016: 8). Figure 25 shows a conceptual diagram of the impact evaluation as applied to a SI initiative. In practice, as mentioned, the idea is to compare a situation actually observed (i.e. the SI initiative case study to be evaluated) with a counterfactual situation (i.e. the same case study area in case there would not have been the SI initiative).

<footnote n="95">⁹⁵ In statistics, normalisation can have multiple meanings. In simple terms, normalisation of ratings identifies the adjustment of values measured on different scales to a notionally common scale. In more complex cases, normalisation can refer to sophisticated adjustments where the intention is to bring the entire probability distributions of adjusted values into alignment (Dodge, 2003).</footnote>
<figure><img src="image_136.png" /></figure>

<figure><img src="image_137.png" /><figcaption>**Figure 25.** Conceptual diagram of the impact evaluation of SI. Source: SIMRA Research team.</figcaption></figure>

In summary, impact evaluation is an assessment of the causal effect of a project, a programme or a policy on beneficiaries (with respect to non beneficiaries). It uses a counterfactual to estimate what the state of the beneficiaries would have been in the absence of the programme (the control or comparison group), compared to the observed status of beneficiaries (treatment group), and to determine intermediate or final outcomes attributable to the intervention (Giraldo, 2017 – pers. comm.).

The basic cause-effect question can be applied to many contexts and fields of analysis. As mentioned, impact evaluations are typically used “to inform policy makers on a range of decisions, from curtailing inefficient programmes to scaling up interventions that work, to adjusting programme benefits, to selecting among various programme alternatives” (ibidem: 8). Therefore, they can be used also to evaluate the impacts of a certain event i.e. in technical terms a “treatment” or an “intervention”<fnref n="96" /> (such as a social innovation initiative established in a marginalised rural area) on the economy, society, environment and governance at various spatial and administrative levels.

In SIMRA, it is proposed to use two different approaches to impacts evaluation, depending on the case study to be evaluated. In the following, both of which are summarised below.

### 6.2.12. SIMRA approach 1 to impact evaluation

The approach is based on the use of counterfactuals, typically “before-after”. The measurement of changes are based on perceptions of interviewed people and/or secondary data that only show general trends and cannot be directly and clearly correlated to the analyzed SI initiative. In this case, observable changes can be due to the SI initiative and to other, not clearly identifiable, factors (e.g. a generalized improvement in the economic conditions because of global dynamics, a new policy that was introduced in the area not targeted to SI, etc.) the effects of which cannot be isolated from those due to the SI initiative.

Approach 1 to impact evaluation will be adopted in all SIMRA case studies.

In the proposed data collection tools (semi-structured interviews for the focus on policy analysis and structured interviews for the focus on the quantitative data of SI evaluation), specific questions are included to identify the perceived changes (i.e. the outcomes and impacts, such as the context in relation to one or more

<footnote n="96">⁹⁶ Intervention/treatment/program/policy: A well defined intervention targeted to a well-defined population, whose members can in principle be (self-)assigned to/(self-) denied the intervention, with the purpose of inducing a change in a Well defined state and/or behaviour of the units exposed to the intervention (source: Giraldo, 2017).</footnote>
<figure><img src="image_138.png" /></figure>

of the four domains, after the SI initiative has been implemented). An accurate formulation of questions to capture and evaluate retrospectively the events is needed in order to correctly collect data, minimize biases and reduce the risk of collecting useless information. Statistical techniques such as the use of diaries, checklists, aided recall measures and bounding measures (Van Der Vaart et al. 1995, Belli et al. 2013, Van der Zouwen et al. 1993), can be applied to achieving this goal.

The issue of the identification of impacts (i.e. what impacts are determined by the SI initiative on the environment? What impacts on social aspects? etc.) has to be discussed in focus group in each case study (see Part IV for more details).

Any activity developed and implemented by an organisation might have impacts that can be more or less relevant, i.e. they have longterm consequences on a large portion of the territory and/or a high number of persons, or they do not significantly affect the economy, the society, the environment and the institutions because they have only temporary effects at a small scale. It would not be possible to identify and analyse all the different impacts on all the four domains in relation to a SI initiative in a MRA. To do this would be time consuming and, in certain circumstances, it would not make sense from a technical point of view. If the impacts are very limited, for example because the SI initiative is very punctual, and/or it does affect a very small group of persons in a small remote area, then it might not be necessary and correct to evaluate them in details and/or by using statistical techniques (see Approach 2 in Section 6.2.10.2). To select those impacts that are relevant enough for being evaluated in detail in SIMRA it is proposed the adoption of the concept of significant impact<fnref n="97" />. In the case of impacts on environment, the guiding idea is that the SI initiative itself (and in particular the “Agency”) has to identify all its potential environmental aspects (i.e. all those aspects which the SI is likely to impact<fnref n="98" />). However, after a first screening based on broad categories, only those classified as significant environmental aspects and impacts will be taken into account and analysed more in detail in the evaluation. A similar approach can be adopted in any domain: in relation to environmental aspects, and in relation to the economic, social and institutional. In order to facilitate the identification of the aspects, four lists (one for each domain) of possible aspects taken from the literature have been drafted. To facilitate the selection of the significant aspects and related impacts, a set of guiding criteria is proposed, which includes: 1) level of control and influence; 2) sensibility of the context; 3) frequency; 4) intensity; 5) risk. Details on both (i.e. lists of aspects and criteria for identifying significant aspects/impacts) are provided in Part IV.

Whether the lists are complete, comprehensive, clear and functional to the evaluation, the use of the significant impact concept and the criteria for its application, as well as details on how to process, analyse and interpret the data and interviews results on the outcomes/impacts will be refined after the data collection tools are tested in the selected SIMRA case studies. This will enable the selection of a quick, easy and effective way to deal with this complex and key issue. The full methodology will be finalized and reported in Deliverable 4.3.

### 6.2.13. SIMRA approach 2 to impact evaluation (in the statistical meaning of the term)<fnref n="99" />

Evaluators have to be aware that the evaluation of impacts determined by an intervention/treatment (a policy, a programme, or any event such as a social innovation initiative), in the statistical meaning of the term, is a

<footnote n="97">⁹⁷ This was introduced initially by the international standards ISO 14001:2001, recently updated into the ISO 14001:2015 version, designed for industrial organisations for implementing and keeping under control environmental management systems. However, it can be adapted to SIMRA needs. In short, this approach is based on an active role of the organisation that wants to identify and keep under control its environmental impacts and that it is evaluated (by an external, independent entity, i.e. an accredited certification body) respect to its compliance with the compulsory current environmental laws and with this voluntary commitment. The guiding idea is that the organisation has to identify, firstly, all its “environmental aspects”, but then only those that are classified as “significant environmental aspects” - i.e. those having “one or more significant environmental impact/s” - will be taken into consideration and analysed in detail during the evaluation. In order to select ‘significant environmental aspects’, organisations typically adopt a set of criteria, including: 1) level of control and influence; 2) sensibility of the context; 3) frequency; 4) intensity; 5) risk.</footnote>

<footnote n="98">⁹⁸ Lists of possible impacts are provided in Part IV of the current Deliverable 4.3.</footnote>

<footnote n="99">⁹⁹ This paragraph was written with the help of Anna Giraldo and Maria Castiglioni.</footnote>
<figure><img src="image_139.png" /></figure>

<figure><img src="image_140.png" /></figure>

time consuming, technically complex and costly procedure. Consequently, whether and when such an impact
evaluation is needed and appropriate has to be carefully considered by evaluators on a case-by-case basis. In
SIMRA, this screening and decision is part of the tasks of the CS Teams, with the support of Work Packages 4
and 5. In the following, general guidelines are provided to support them in understanding and dealing with
this important step of the evaluation. Further details are provided in Part IV.

In which cases to apply an impact evaluation in the statistical meaning of the term

According to Gertler et al. 2016 (p. 8), impact evaluations are “most effective when applied selectively to answer important policy questions, and they can be particularly effective when applied to innovative pilot programmes that are testing a new, unproven, but promising approach”. In general, this means that it is appropriate to perform an impact evaluation when programmes to be evaluated have a large scale of application (i.e. having high number of people treated, large areas affected, high amount of financial resources allocated), and thus it is likely to have significant impacts. In general, impact evaluation can be useful and appropriate when the project/intervention to be evaluated is: innovative, replicable/scalable, strategically relevant, influential and when evaluation will fill knowledge gaps (Gertler et al. 2016). It can also be used within a programme to test alternatives and improve programmes. However, in some circumstances, impact evaluation is neither ethically correct/useful, from a political point of view (e.g. costs of evaluation are too high and collected information are too specific for being valuable to guide policy reforms), nor technically feasible from a statistical point of view (e.g. the counterfactual case does not exist). Therefore, in defining whether or not to use impact evaluation in SIMRA case studies, it is necessary to evaluate the technical feasibility case-by-case. Details on this issue are provided in Part IV. The application of the method in SIMRA needs to be tested and therefore will be finalized in Deliverable 4.3.

How to design an impact evaluation in the statistical meaning of the term

This section describes the main evaluation steps and methods that can be used to apply an impact evaluation
in the case impact evaluation is considered feasible according to Error! Reference source not found.. The
methods are focused on how to find suitable comparison groups. "The basic form of impact evaluation will test
the effectiveness of a given programme. I will answer the question, *Is a given programme effective compared
to the absence of the programme?* [...] This type of impact evaluation relies on comparing a treatment group
that received a project, programme, or policy to a comparison group that did not in order to estimate the
effectiveness of the programme" (Gertler et al. 2016: 8). The quality of the comparison group makes the
difference in the accuracy and statistical correctedness of the evaluation. In designing an impact evaluation, in
the statistical meaning of the term, evaluators have to ask three key questions: i) *When the evaluation is
carried out?* ii) *What is the scope (in particular, what are the target population, the treatment and the outcome
variable)?* iii) *Which are the most appropriate statistical tecniques to apply?*

In the approach proposed, the questions posed are addressed as follows: i) in SIMRA case studies the evaluation is always retrospective (i.e. ex-post); ii) the target population, the treatment and the outcome variable/s have to be identified case-by-case; and iii) the recommended statistical technique is a combination of the propensity score matching with the difference-in-differences method. The following boxes LL, PP and FF provide details on each of these questions and motivations for answers. Practical instructions on how to deal with these issues in the case studies are provided in Part IV of this deliverable.
<figure><img src="image_141.png" /></figure>

**Box 21 – When the evaluation is carried out?**

In answering the “when?” question in SIMRA, evaluators have to consider the special nature of SI, that is a spontaneous event, that can be stimulated (or obstructed) but cannot be forced to happen (e.g. by means of a public intervention or compulsory rules). As a consequence, only retrospective evaluations<fnref n="100" /> (Gertler et al. 2016) are possible, i.e. evaluations that are carried out only once the SI initiative started, and that cannot be planned or designed in advance with respect to the SI intervention. This implies that baseline data (when existing) are typically poor, and that counterfactual estimates are more difficult.

**Box 22 – What is the scope of the evaluation?**

Three key features of any SI initiative that have to be evaluated and answer to the key question of “what?” are the following: 1) the target-population; 2) the intervention, and 3) the outcome variable.

The *target population* is a well defined set of units upon which the intervention may operate, at a given time; units can be, e.g. individuals, households, organisations, networks, villages, or other entities at higher levels of aggregation (e.g. municipality, provinces, regions, countries, etc.), the “borders” and characteristics of which should be clearly identified by the evaluators.

The *intervention (or treatment)* is the set of activities put in operation by the SI initiative to address an issue of the target population. The treatment is intended to have an effect on the outcome variable. The analyst aims to assess the effect of the treatment on that variable with respect to no intervention. In order to reduce complexity and make the evaluation feasible, an intervention should consist of a single treatment (i.e. of a single SI project)<fnref n="101" />.

Finally, the *outcome variable* is an observable characteristic (α a measurable aspect) of the units of the population, on which the intervention may apply/not apply, and in principle may have an effect on (α impact). The variable must be a characteristic of the person/unit that can be observable even in the absence of the treatment. In SIMRA, we propose that outcome variables are selected among those variables directly connected with societal well-being.

**Box 23 – Which are the most appropriate statistical techniques to apply?**

The last key question to be answered is “how?”, in relation to the most recommended techniques and methods for carrying out impact evaluations. The proposal is to perform an impact evaluation is to find a comparison group that share the same characteristics of the treated group. The comparison group should be identical to the treated group except for the treatment. The differences found in the outcome variables between the two identical groups could be then imputed to the treatment. The best way to obtain an identical comparison group is a randomization, i.e. units divided randomly to the treated or comparison group. As said before SI initiatives are spontaneous, in some sense not planned by the policy makers, and therefore a randomization is an unrealistic solution to enrol people in the treatment/comparison group.

An ex-post way to proceed is to build a fake comparison group, i.e. a comparison group that has the almost the same characteristic of the treated group and it is not affected at all by the treatment. A technique to obtain such a comparison group is the matching (Gertler et al., 2016). Called X the characteristics of the

<footnote n="100">¹⁰⁰ Vice versa, prospective evaluations are developed at the same time that the program to be evaluated is designed. This allows for assuring good baseline data, evaluating intended results and defining in advance the treatment and the comparison groups. This is NOT applicable in the case of SI in MRAs.</footnote>

<footnote n="101">¹⁰¹ This means that, in case of multiple treatments (e.g., multiple projects developed and delivered by a single SI initiative to the same target population), the treatment taken into consideration for evaluation will be the most important. See Part IV of Deliverable 4.2 for details. It is important to mention that this applies also to the first approach adopted for impact evaluation in SIMRA (i.e. the one that is not based on statistical techniques).</footnote>
<figure><img src="image_142.png" /></figure>

treated unit, a donor should have the same values of X; in other words the donor should be a clone of the treated unit. Since a perfect matching is impossible to find, a techniques that permits to consider as much characteristics of an individual and to match based on it, is the propensity score matching.

A propensity score (defined as the probability to enrol in a treatment conditional to X, Rosenbaum and Rubin, 1983) is an index that summarizes the characteristics of an individual in number (transform a multivariate problem in a univariate one). Therefore, the matching is done on the value of the propensity score. There are several methods for choosing the closest match (or matches) based on the propensity score (stratification matching, kernel matching, nearest neighbour, etc.). The choice of the best one is conducted on the basis of sensitivity analysis (Dehejia and Wahba, 2002). In general, statistical errors and inaccurate measurements due to selection bias or extraneous factors are always possible, as “perfect clones” of any treated units do not exist in practice. However, the biases connected with potential selection errors (depending for example on how the treatment group is chosen) and inaccurate measurements can be minimized, or at least reduced, by combining different methods.

One of the methods more frequently used in combination with matching techniques is the so-called difference-in-differences (DD, DID or diff-in-diff) method (Gertler et al., 2016). DD is a statistical technique typically used in quantitative research in social sciences that uses observational data to study the differential effect of a treatment on two comparable groups, a “treatment group” versus a “control group” (i.e. the counterfactual). It estimates the effect of a treatment (independent variable) on an outcome (dependent variable) by comparing the average change over time in the outcome variable for the treatment group, compared to the average change over time for the control group. Although it is intended to mitigate the effects of extraneous factors or control group selection bias, DD may still be subject to certain methodological problems and potential biases (such as reverse causality or omitted variable bias). Matching techniques, and in particular propensity score matching, can help to reduce these problems, as they allow to identify a comparison group that is as much similar as possible to the treatment group, in terms of the observable variables. The possible drawback of the matching diff-in-diff is related to the availability of data, since information on outcome variable is needed not only after the implementation of the SI intervention but also at the baseline for both groups treated and control.

### 6.2.14. In summary: SIMRA approach to impact evaluation

On the basis of previous considerations, and because of the nature of impact evaluation, an impact evaluation will be applied in SI case studies by adopting two different approaches, depending on the case. An impact evaluation based on statistical techniques will be applied only in those specific SI case studies where such a type of impact evaluation is technically feasible (if any). The identification of the cases to be evaluated by means of statistical techniques of impact evaluation will be performed case by case in the initial steps of the evaluation process, in collaboration with Work Package 5 and each Case Study Team. Those case studies that have the minimum basic characteristics for being evaluated using the proposed statistical techniques will be used as exploratory-explicatory case studies to demonstrate SI impacts in MRAs, for checking how an impact evaluation of this type should be implemented in practice. They will be used for cross-checking whether the analysis based on stakeholders' perceptions on changes (based on primary data and qualitative methods) is consistent/aligned with the findings based on the statistical techniques of impact evaluation (based on secondary data and quantitative methods). To focus the evaluation on key impacts (to reduce costs and capture the real meaning and consequences of SI), the method is based on the identification of significant aspects and impacts on the economic, social, environmental and institutional domains. The reasons for considering an impact as significant have to be explained, by means of storytelling (if secondary data are not available at the level of detail required), and/or on the basis of fact-based evidences (if secondary data exist and are accessible for free at the needed level). The questions in the SIMRA method will be submitted to all the target populations (from innovator(s) to indirect beneficiaries), and/or used to guide the discussion in the focus groups, so that triangulations will enable checks on whether the perceptions on impacts are convergent between actors.
<figure><img src="image_143.png" /></figure>

### 6.2.15. Methodological design: policy analysis

In this section<fnref n="102" />, we explain how policies supporting (or hindering) SI can be identified both at higher institutional levels, at the local level and how the analysis can be carried out. The methods of data collection and analysis of the policy implications for SI follow the Innovation System<fnref n="103" /> Approach (Rametsteiner et al. 2005; Weiss et al. 2011), and focus ex-post on actors, institutions and policies that are relevant for the SI.

A step-by-step approach is proposed for data collection (based on qualitative interpretative approach) that focuses on the analysis of policy processes and implications for SI in MRAs. The approach is based on three main steps: (i) identification and analysis of policy documents; (ii) analysis of policy implications at the local level, i.e. in SIMRA case studies of social innovation; and (iii) validation of findings by cross-checking document analysis with surveys of local actors. An additional fourth step might be necessary: matching the results from the evaluation carried out at the local level with those from an analysis carried out at higher jurisdictional levels (in case policy gaps emerge from the previous steps). The three (plus one) steps are briefly explained below. Practical instructions for evaluators (currently, for SIMRA Case Studies Teams) and a scheme to graphically represent it within the overall integration between qualitative and quantitative methods are provided in Part IV.

#### Step 1: Document analysis (desk-top research)

The first step is the identification of relevant policy documents. This step is intended to identify and examine policy assumptions, discourses, problem frames, regulatory approaches and policy instruments that support or hamper effects of SI in MRA. Relevant policies are chosen from agriculture, forestry, rural development, welfare, labour markets, technological innovation, social innovation, regional development, etc. The focus is on policy processes that are relevant at international, national and regional levels but are expected to have effects at a local level. The identification of policy documents has to be carried out by both online searching (literature analysis, institutional web sites) and interviews of local stakeholders. Documents relating to local interpretations of international, national or regional policies (e.g. brochures) should also be taken into consideration. The methodology is based on content analysis of documents. This can be done by means of classical approaches (e.g. direct identification and extraction of "citation" from the document text) and/or by means of the application of softwares specifically for text analysis (e.g. NVIVO, MAXQDATA or others). Other information is provided in Part IV.

#### Step 2: Analysis of policy implications at the case-study level (field research)

This step focuses on an analysis of policy implications based on interviews of local actors. They include innovator(s), follower(s), land owner(s), opponent/s, beneficiarie/s, people from political administration in the region/area, project partners such as businesses, consumers of SI products and services. These local stakeholders (including public authorities) may recognise the existence of policies that support SI directly or indirectly; in some cases, they may not be able to recognise the role of policies specifically targeted for social innovation, or frame strategically a social innovation against imposed policies to promote a local level response. The initial sub-step of this part of the analysis is the identification of actors (see Part IV).

Knowledge of policies at the local level refers to knowledge of available possibilities (enabling and/or constraining) and the (local) perceptions of effectiveness, which need to be validated with an analysis carried out at the regional, national or international levels. In the field research, the main methods applied are semi-structured interviews, based on open-ended questions and possibility for the interviewer/evaluator to interact

<footnote n="102">¹⁰² The methods for analysing policy implications on SI are developed jointly by Work Packages 4 and 6.</footnote>

<footnote n="103">¹⁰³ The approach analysis proposed for the case studies follows the innovation systems approach (Edquist, 1997; Asheim and Gertler, 2005). Innovation systems have been categorised into national innovation systems, regional innovation systems, local innovation systems, technological innovation systems and sectoral innovation systems (Lundvall, 1985). Examples for the application of this approach in innovations in the rural forest sector are provided in Rametsteiner et al. (2005), Kubeczko et al. (2006) and Buttoud et al. (2011). While there is no consensus on the exact definition of an innovation system, and the concept is still emerging, there is an agreement on the necessity to include dynamic influences of actors and institutions on innovation</footnote>
<figure><img src="image_144.png" /></figure>

with the interviewee in a flexible way (i.e. adapting the sequence of questions to his/her attitudes and flow of concepts, statements - storytelling). Details of the proposed structure of the semi-structured interview guides for SIMRA policy analysis are provided in Part IV.

Key questions are, for example: "Which policies supported you?" and "How the legal institutions (the policy settings and regimes) influence the emergence and maintenance (or withdrawn) of the SI case?" These questions, in evaluative terms, are not cause-effect questions (see Section 5.1). Rather, they are typically descriptive questions that refer to the evaluation of the perceived effectiveness of the policies, and not to an evaluation of policies impacts in the statistical meaning of the term (see Section 6.2.10.2). The key aspects that the interviews target are: general backgrounds and objectives of the SI; sources of ideas, spill over and upscale; the role of policies (e.g. targets, subjective perception of support by policies, types of support in terms for example of finances, information, etc.); financial situation and maintenance of the SI; coordination, cooperation and conflicts; internal evaluation of the SI initiative (e.g. identification of success or hindering factors) and future prospects. An important part of the analysis is the qualitative identification and interpretation of impacts, or more correctly, we should consider effects in order to clearly differentiate them from those impacts of SI that will be measured by means of statistical techniques).

### Step 3: Validation of findings (mixed-approach)

In this final step, the method is based on a dynamic activity of cross-checking to be performed by analyst(s). The policy analysts (evaluator(s)) have to move back and forward between the field observations (data collected in the second step based on interviews and/or questionnaires to local stakeholders and institutional authorities) and the documents. After the initial interviews, desktop research is undertaken, followed by more interviews, until "saturation". In practice, more interviews with stakeholders have to be undertaken until enough information about the case is obtained. This is called a "mixed approach" (between the desk-based text and the field). This final step is necessary for cross-checking consistency between what is stated in the policies (e.g. policy document formulated at higher institutional levels, such as EU regulations and regional laws) and what happens in practice (e.g. how policies affect the local scale, in terms of perceptions expressed by stakeholders and/or in terms of impacts to be measured by means of statistical techniques). The desk-top step of analysis helps to identify incongruences and/or inconsistencies between the policy as designed, communicated and perceived by high level policy makers, and the policy effects in practice at local level. For example, when a question such as "Why does this predefined (and so nicely formulated) regulation not produce effects in the field?" emerges during the interviews with a policy maker, it helps to identify cases of non-effectively activated/delivered policy or regulation, or inconsistencies between conflicting norms. Inconsistencies amongst policies and incongruences between policies and practices are important issues to understand. Policy-practice incongruences may have different causes. For example, the innovator(s) were not informed of the possibilities for policy support. This lack of information can be due to a lack of communication of the policy from higher institutional levels; or, it can be due to a low capacity of the local innovator(s) to interact with higher levels (e.g. they do not have direct contact with higher levels or do not know how to use the institutional channels for getting information). Another issue to consider is the way in which actors avoid, circumvent or divert regulations. Other constraints can be irrelevancy of the policy to the specific case. By means of such a complex and qualitative-interpretative based analysis, the causal relation between a policy and its effects is neither automatic nor always clearly identifiable.

### Step 4: Policy experts' consultation

Additional instruments to be used in the SIMRA evaluation method are interviews to policy experts at higher jurisdictional levels with respect to the local one. These policy experts can be, for example, representatives of public authorities working in the agriculture, forestry and rural development fields as well as on innovation at regional or national (or even international, EU) levels. They are typically identified as national experts or key informants; they can be, for example, regional rural development planner(s). Whether and how to carry out a set of interviews at these higher levels is a matter to be carefully considered case-by-case, as this would require an understanding of whether inconsistencies or incongruences exist in the specific SI case. These
<figure><img src="image_145.png" /></figure>

<figure><img src="image_146.png" /></figure>

instruments can be used mainly in the specific case that inconsistencies/incongruencies are identified, to
complement the analysis carried out at local level.

This approach has some advantages (e.g. reduction of costs and simplification of analysis) and disadvantages.
In particular, (i) it does not allow a complete understanding of the positive factors and supporting
governance/institutional mechanisms that eventually lead to congruences and consistencies (the opinions and
interpretations of the policy makers at higher institutional levels would be missing), and (ii) it would not
facilitate access to some information about the effects of SI on policy reforms. While interviewing local actors
and analysing policy document will enable evaluators to understand implications of policy on social innovation
(i.e. the cause is the policy and the effect is the social innovation), they would also need to interview policy
makers and other key informants at higher governance levels to understand whether and how SI possibly had
effects on policy and governance (i.e. the cause is the SI and the effect is the policy or governance reform at
higher level). This part is connected with the “Learning process” dimension of SI as identified in our evaluation
framework (see Part II). However, some techniques exist (e.g. Delphi method) to minimize the efforts of
possibly complement the analysis carried out at local level with this, as face-to-face interviews are not strictly
necessary and key informants can be contacted on remote (e.g., by phone or Skype). Other information is
provided in Part IV of this deliverable.
<figure><img src="image_147.png" /></figure>

# PART IV – INSTRUCTIONS FOR ASSESSMENT AND EVALUATION FOR WP5 AND WP6
<figure><img src="image_148.png" /></figure>

# 7. Instructions for the Pilot Test of the Assessment and Evaluation Methods in Pioneer SIMRA Case Studies

## 7.1. Steps of Evaluation Method

Instructions are provided for both the quantitative analysis of SI in MRAs and its effects for the case studies at the local level and the qualitative analysis for related policy processes and implications. These instructions describe the five steps on which the assessment activities are based, and directions for Work Packages 5 and 6 to conduct the pilot test on 2 to 3 selected pioneer SIMRA case studies<fnref n="104" />. The main aim of the pilot test is to check the preliminary proposed set of methods and tools, i.e. to verify whether the overall proposed evaluation approach, process, methods and tools are feasible, effective, complete, clear and connected in a logical sequence. Instructions also provide guidance on the provision of feedback to Work Package 4 for developing improvements in the evaluation framework.

The evaluation method presented in this Section includes five steps (see Figure 26). These steps are designed to integrate two paths: the qualitative-based, focused on policy analysis and its implications on SI in MRAs, and the quantitative-based, focused on the evaluation of SI initiatives and its impacts at local level. The five steps are outlined below:

1) Step 1: Consultation of experts and key informants: pioneer Case Study Teams (CS Teams) initially contact a small group of key (local) informants to provide overall and introductory information on the actors, events, characteristics and dynamics of the SI initiative in the local case study area.

2) Step 2: Survey design for the analysis of policy processes and local SI case studies: the identified experts and key informants are interviewed individually through narrative semi-structured interviews; stakeholders and project partners of the SI initiative and a sample of beneficiaries are interviewed through a structured questionnaire (see the preliminary list of questions for the assessment and evaluation of SI in Annex 1). Both the qualitative list of questions and the quantitative questionnaires aim to address the nine dimensions of the SI evaluation framework described in Section 6. As this is a pilot test, only a few actors for each category have to be interviewed.

3) Step 3: Interview data are collected and input to Excel databases, and a test on the processing and analysis feasibility carried out. Feedback on the appropriateness of each part of the data collection tools (list of qualitative and quantitative questions) are provided by pioneer CS Teams. Examples of narrative, qualitative and quantitative (mainly descriptive) indicators are drafted on the basis of data collected from the test.

4) Step 4 (optional): an additional survey of policy experts could be carried out within the analysis of policy processes, to verify the results from the local case study, if a need became apparent from the interviews (e.g. in case policy gaps are identified). Further, an impact evaluation based on statistical techniques could be set up through the selection of a control group for the evaluation of one or more pioneer case studies, if technically feasible.

5) Step 5: The common final reporting is drafted which includes the results from both test analyses.

<footnote n="104">¹⁰⁴ As previously mentioned (see Part III), these instructions represent a starting point, which the final set of SIMRA evaluation methods will be grounded on. The final integrated set of methods to assess and evaluate SI in MRAs has to be developed in Deliverable 4.3 by Month 40. The structure, contents, methods as well as the overall approach and process of evaluation - also in its practical steps - presented in the current set of instructions will be used as a basis to refine according to both the results of its application in the pioneer case studies (pilot tests) and in all the other SIMRA selected case studies of social innovation - the methods.</footnote>
<figure><img src="image_149.png" /></figure>

<figure><img src="image_150.png" /><figcaption>**Figure 26.** Steps in the assessment and evaluation process in SIMRA.</figcaption></figure>

The pioneer CSs for the pilot test of the methodology should have characteristics similar to those of the SIMRA case studies on which the evaluation method, once refined after testing, will be applied in 2018. As mentioned, rather than focusing on results and data analysis, the pilot test aims to collect information on:

* Challenging unclear or potentially repetitive questions included in the proposed set of questions

* Missing information from the proposed set of questions

* Jargon

* Difficulties in data collection (e.g. management of face-to-face interviews and/or focus groups)

* Estimated interviews length

The manual instructs CS Teams on the target population for each set of questions (Table 18), and provides a summary of the dimensions, sub-dimensions and components, presented in Section 5, as well as the codes of the questions (Table 19), fully listed in Annex 1. Feedback in a written form will be provided to other activities in Work Package 4, and for activities in Work Packages 5 and 6 during the pilot test period (October to December 2017). Detail guidelines to pioneer CS Teams for the process of feedbacks provisioning are included in Deliverables 5.1 and 5.2.