# Ex-post evaluation of the impacts of the science-based research and innovation program: a new method applied in the case of farmers’ transition to organic production in the camargue

Sylvain Quedeville, Dominique Barjolle, Jean-Claude Mouret, Matthias Stolze

► To cite this version:

Sylvain Quedeville, Dominique Barjolle, Jean-Claude Mouret, Matthias Stolze. Ex-post evaluation of the impacts of the science-based research and innovation program: a new method applied in the case of farmers’ transition to organic production in the camargue. Journal of Innovation Economics & Management, 2017, 22, pp.145-170. 10.3917/jie.022.0145. hal-01608441

HAL Id: hal-01608441

https://hal.science/hal-01608441

Submitted on 26 May 2020

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_1.png" /></figure>

Distributed under a Creative Commons Attribution 4.0 International License
# EX-POST EVALUATION OF THE IMPACTS OF THE SCIENCE-BASED RESEARCH AND INNOVATION PROGRAM: A NEW METHOD APPLIED IN THE CASE OF FARMERS' TRANSITION TO ORGANIC PRODUCTION IN THE CAMARGUE

Sylvain Quiedeville, Dominique Barjolle, Jean-Claude Mouret, Matthias Stolze

De Boeck Supérieur | « Journal of Innovation Economics & Management »

2017/1 n° 22 | pages 145 à 170

ISBN 9782807391116

Article disponible en ligne à l'adresse :

http://www.cairn.info/revue-journal-of-innovation-economics-2017-1-page-145.htm

Pour citer cet article :

Sylvain Quiedeville et al., « Ex-Post Evaluation of the Impacts of the Science-Based Research and Innovation Program: A New Method Applied in the Case of Farmers' Transition to Organic Production in the Camargue », *Journal of Innovation Economics & Management* 2017/1 (n° 22), p. 145-170.

DOI 10.3917/jie.022.0145
EX-POST EVALUATION
OF THE IMPACTS OF THE
SCIENCE-BASED RESEARCH
AND INNOVATION PROGRAM:
A NEW METHOD APPLIED
IN THE CASE OF FARMERS'
TRANSITION TO ORGANIC
PRODUCTION IN THE
CAMARGUE<fnref n="1" />

Sylvain QUIEDEVILLE

Department of Economics and Social Sciences
Research Institute of Organic Agriculture (FiBL), Switzerland
sylvain.quiedeville@fibl.ch

Dominique BARJOLLE

Institute for Agricultural Sciences
ETH Zürich, Switzerland
barjolle@ethz.ch

Jean-Claude MOURET

UMR Innovation
National Institute of Agronomic Research (INRA), France
mouret@supagro.inra.fr

Matthias STOLZE

Department of Economics and Social Sciences
Research Institute of Organic Agriculture (FiBL), Switzerland
matthias.stolze@fibl.ch

<footnote n="1">1. **Acknowledgements:** We would like to thank all the people who participated in the evaluation process of the research under review: respondents from INRA, CFR, the Tour du Valat, the traders Madar, SARL Thomas, BIOSUD and organic and part-organic farmers. We would also like to thank the European Union as this publication has been generated and funded through the IMPRESA project (grant agreement no 609448).</footnote>

<footnote>**Disclaimer:** This publication has been generated as part of the IMPRESA project funded by the European Union. The content of this publication is the sole responsibility of the implementing partners of the IMPRESA project and can in no way be taken to reflect the views of the European Union.</footnote>
ABSTRACT

This paper aims to assess the contribution, role, and impacts of the Science-Based Research and Innovation Program (ISRIP) on farmers' transition to organic production in the Camargue. Focusing on how, and to what extent, research actors have contributed to the innovation pathway, we applied a methods-mix. The Participatory Impact Pathway Analysis (PIPA) was used to uncover complex mechanisms along the innovation process; the Outcome Harvesting method to adapt PIPA to the requirements of an ex-post evaluation, and the Social Network Analysis (SNA) to emphasize actors' relationships in relation to the development process. We demonstrate that the research has contributed to change by developing co-learning interactions with farmers, although this was not critical to the success of the innovation. Rather, we highlight that agricultural policies, economic factors, testing conducted independently by farmers, and the institutional framework, are the most important and influential factors.

Keywords: Evaluation, Science-Based Research, Program Theory, Innovation Process, Ex-Post Participatory Impact Pathway Analysis, Camargue Rice Systems

JEL Codes: 032

The European Commission has developed a long-term research strategy in order to enhance the impacts of European agricultural research and innovation (European Commission, 2015a). At the same time, growing attention is paid to the effectiveness and efficiency of those research programs (a set of projects implemented by researchers) in a context of scarce financial resources. Effective results implemented by final beneficiaries of research programs need to be evidence-based in order to (1) report to stakeholders on the return on investments (CGIAR Impact Assessment and Evaluation Group, 2000) and (2) develop improvements in policies and projects (Mackay, Horton, 2003). To do so, indicators of performance and the impacts of research programs are being developed, both at EU and Member State level. The task of developing "Horizon 2020 indicators" highlights this process (European Commission 2015b), since it intends to assess the results and impacts of the EU Horizon 2020 program<fnref n="2" />.

Our paper is focused on a research program concerning a transition to organic farming in the Camargue territory (south-east France), where rice is the main crop production (circa 20,000 ha in 2013). Cultivation of rice is crucial for reducing soil salinity in Camargue by flooding paddy fields with fresh water. That is because the cumulative effect of the severe winds (the Mistral), as well as exposure to sunlight and temperature, result in strong evaporation, leading to salinity from groundwater, which in turn sterilizes

<footnote n="2">2. "Horizon 2020 is the financial instrument implementing the Innovation Union, a Europe 2020 flagship initiative aimed at securing Europe's global competitiveness" (European Commission, 2016).</footnote>
agricultural land in the Camargue (Chataigner 1997). On the other hand, the Camargue is subject to protective measures, due to severe environmental problems (e.g. pollution of the Rhône River). There is a dilemma between the need to cultivate rice to maintain farming in the Camargue and the need to protect the environment; thus the issue of the transition to organic farming has become increasingly important.

Organic rice production in the Camargue increased in the 1980s through the initiative of pioneer producers and today accounts for 10% of the total area of rice farming and for 16% of rice producers (35 out of a total of 215). In 2000, a research program for organic production in the Camargue was launched by INRA (French National Institute of Research Agriculture), together with CIRAD (French Agricultural Research and International Cooperation Organization), CFR (Centre for French Rice), and FranceAgriMer (National Institute of Agricultural and Seafood Products), with a view to fostering organic farming development in the Camargue. The Camargue Nature Reserve and the Rice Farmers Union (Syndicat des Riziculteurs) were also involved in discussions. In this paper we take account of six projects (table 1) out of eight included in the research program developed by INRA and its partners. Two projects, slightly related or unrelated to the transition to organic farming, are not considered: the valorization of rice straw, and farmers' views on sustainability.

This paper pursues 3 objectives:

* To assess how organic cropping has developed in the Camargue, as well as the level and origin of related socio-economic and environmental impacts;

* To evaluate ex-post the role of research in the innovation pathway of the transition to organic farming in the Camargue;

* To identify external factors that have played major roles as alternative catalysts or barriers within that impact pathway.

We undertook a participatory approach by involving stakeholders in the evaluation process and increasing their interest. We endeavored to enhance responsiveness during the evaluation process with a view to mobilizing changes and increasing the probability that results will be utilized (Weiss, 1997; Cousins, Whitmore, 1998). These objectives required us to follow an approach designed for ex-post impact assessment, in order to reconstruct the innovation pathway after it has occurred. There was a particular focus on the role played by the network, mainly to clarify complex relationships and evaluate the accurate contribution of the different research actors in the pathway.
<table><caption>Table 1 – Research projects under review regarding the transition to organic farming in the Camargue</caption><thead><tr><td>Years</td><td>Projects</td><td>Objectives of the different projects</td></tr></thead><tbody><tr><td>2000-2004</td><td>CEBIOCA project: "Céréaliculture Biologique en Camargue" (Organic Cereals in the Camargue)</td><td>To explore the conditions of developing organic cereals and highlight yield variability factors.</td></tr><tr><td>2005-2006</td><td>Experimentation in farming plots</td><td>To develop new crop management techniques, i.e. techniques to regulate weeds and improve fertilization management.</td></tr><tr><td>2008</td><td>"ORPESA Table": "Organic Rice Production in Environmentally Sensitive Areas". This project has been conducted by INRA and was part of the EU Leonardo Da Vinci program.</td><td>To establish professional training in order to support farmers' conversion to organic rice production.</td></tr><tr><td>2011</td><td>Experimentation in crop management techniques (new testing conducted by INRA and its partners)</td><td>To further develop techniques focusing on weed management.</td></tr><tr><td>2011</td><td>International conference on rice (held in Montpellier)</td><td>To facilitate exchanges and develop knowledge among rice producers, researchers and other actors operating at the different stages of organic rice value chains in the world.</td></tr><tr><td>Since 2012</td><td>CIRAD Experimentation</td><td>To focus on the technicality of harrows, hoes and rotavators to regulate weeds. The interest of bringing ducks onto the land (to eat weeds) is also evaluated.</td></tr></tbody></table>

Moreover, the assumption was made that the innovation process is derived from changes in organic farming issues, within the complex system of interlinked actors.

In the next section we present the theoretical background, followed by the method developed and applied for evaluating ISRIP (Impacts of Science-Based Research and Innovation Program). Finally, case study results are outlined, while the paper concludes by summarizing the main lessons learned.

# THEORETICAL BACKGROUND

## Rationale for Qualitative Evaluation

In this paper we refer to innovation as “the implementation of a new or significantly improved production or delivery method [that] includes significant changes in techniques, equipment and/or software” (OECD, 2016). Furthermore, we consider innovation as a dynamic process, in line with the concept of the Agricultural Knowledge and Information System (AKIS), which replaced the traditional linear transfer of knowledge (Gibbons et al., 1994). The linear view of innovation is, however, still predominant in evaluation practices
(Cozzens, Snoek 2010), which does not allow complex and dynamic under-
lying mechanisms to be taken into account, nor to recognize systems of
reflexive, learning and network interactions (Knickel et al., 2009).

Moreover, considering dynamic and complex innovation processes made
us focus on a qualitative analysis of the contribution, role and impacts of
the research program under review. A quantitative assessment could be
performed using a non-monetary approach or by comparing the cost of the
resources invested to economic impacts (Penfield et al., 2013). However, this
type of approach embraces a broad level of analysis in considering the inno-
vation system (Touzard et al., 2014) as a “black box”. In effect, such explora-
tions do not analyze the detailed process by which results were generated
(Colinet et al., 2014). Complex learning synergies, systems of reflexive and
institutional environment, are not consistently reflected (Hall et al., 2003)
given that a quantitative approach focuses on an “input-output” evaluation.
We refer to inputs as resources (material, financial, human) invested in a
research program (Springer-Heinze et al., 2003), while outputs are concrete
and tangible results (e.g. a new variety developed) of research activities,
representing the various actions undertaken.

## Literature Review on Qualitative Methods for Evaluating ISRIP

In this section, we seek to identify the most relevant qualitative methods for evaluating ISRIP through a literature review. These methods are critically assessed for their focus on complex mechanisms, on understanding of both the role played by the actor network and the process leading to wide changes, and for the extent to which they are participatory.

The *Public Value Mapping (PVM) of Science Outcomes* is a conceptual
tool that aims to comprehend causes of social outcomes (Bozeman, Sarewitz,
2005). Outcomes stand for changes in the behaviors, relationships, activi-
ties and/or actions of the stakeholders (Earl *et al.*, 2001). PVM presents the
strength to look at the underlying causal logic of programs by focusing on
“Knowledge Value Collectives” (KVC), which are used or developed by the
actors’ network to enhance and use scientific knowledge (Bozeman, Rogers,
2002). Furthermore, PVM makes the assumption that science is only part of
the process leading to social outcomes; thus, it explores alternative explana-
tions to the underlying causal logic of programs. This allows all factors to
be considered and attribution problems to be avoided. However, PVM only
explores the public (social) value, and other dimensions like economic and
environmental impacts are omitted. Finally, the contribution of the research
is unsatisfactorily reflected and there is no official participatory procedure.
The *SIAMPI approach* (Spaapen *et al.*, 2013) stands for Social Impact Assessment for research and funding instruments by studying productive interactions between science and society. Impacts are the effects produced by outcomes in a long-term perspective and can be intended or unintended (OECD, 2002). The method focuses on learning instead of judging and accounting, as it concentrates on productive interactions (direct, indirect and financial interactions between researchers and other involved actors), which helps to understand the “black box” of the process between research activities and achieved impacts. This, in turn, allows more relevant indicators of measurement to be defined. Nonetheless, factors linked to the research are not clearly distinguished from external causes, and the contribution of the research remains unclear, whilst both economic and environmental impacts are not tackled. Finally, the method is not of a participatory nature.

*The Outcome Harvesting (OH) method* was developed by Wilson Grau and Britt (2002) to overcome the shortcomings of the Outcome Mapping approach (Earl *et al.*, 2001) in terms of ex-post evaluating programs. Outcome Mapping aims to implement projects by constructing expected pathways. OH presents the primary focus for ex-post analysis and suggests identifying changes related to the intervention, before going further back to outputs and research activities. However, it does not rely on participatory instruments; rather OH recommends the collection of data through publicly available documents, surveys, questionnaires, and in-depth interviews.

*The Payback Framework* (Donovan, Hanney, 2011) is a logic model representing all the complexity of the research process, which helps in achieving impacts. The research process is considered as complex and non-linear since it comprises several feedback loops (e.g. an outcome can affect an activity). The model allows analysis of a research program, from the idea or invention developed in the research process, to the dissemination and adoption phase; in turn generating final outcomes and wide impacts. However, the information is collected through surveys, interviews, and the study of various documents; so that the method is not of a participatory nature. Additionally, the contribution of the research and actors within the network would merit significant additional reflections.

*The Participatory Impact Pathway Analysis, or PIPA* (Douthwaite *et al.*, 2007), is an approach inspired by program-theory (PT). PIPA looks at the detailed process generating impacts from activities, outputs and outcomes. It can also identify both expected impacts, and indicators of impacts, to better evaluate the program (Springer-Heinze *et al.*, 2003). Workshop(s) are organized with stakeholders in order to construct the expected impact pathway of the program by drawing it. The participatory nature of this approach
is intended to enhance responsiveness during the evaluation process and raise the interest of local actors (Weiss, 1997; Cousins, Whitmore, 1998). However, two main concerns with respect to the workshops are as follows: (a) the available time in workshops can be limited; (b) a power game may occur among participants, which means that the discussions can be dominated and biased by some actors (Mathie, Greene, 1997). This is particularly problematic when diversity is lacking in group discussions (Mathie, Greene, 1997). Moreover, causes with little or no link to research projects are underexplored.

Within these qualitative approaches, PIPA appears to be the most suitable for the purpose of this paper. PIPA is a participatory approach that allows local actors and changes to be mobilized while increasing interactions in the innovation network. In fact, a participatory procedure of evaluation pursues two major and interesting objectives. The first is to increase the uptake of evaluation results (Cousins, Earl, 1992) by increasing appropriation of the results by local actors. The second is to increase the autonomy of the actors (Tandon, Fernandez, 1984), in that a participatory evaluation can induce changes through learning effects and interactions, and create knowledge (Plottu, Plottu, 2009). The other approaches also present strengths: the PVM for the exploration of alternative explanations; the SIAMPI and payback framework which analyze the detailed mechanisms in the innovation process, and OH for its particular relevance to an ex-post evaluation. However, none of the approaches mentioned sufficiently address the issue of the actor network, nor do they deeply explore the factual occurrence of the different pathway links<fnref n="3" />, as well as the contribution of the research. Thus, there is a need to develop a new method to accurately evaluate ISRIP.

## METHODOLOGICAL ISSUES

The method undertaken to evaluate ISRIP in this paper was constructed on the above-mentioned approaches but was mainly derived from PIPA, which was seen as the most suitable existing method. The other approaches complemented the method in order to (1) adapt PIPA to the requirements of ex-post assessment (with OH); (2) further consider factors with little or no link to the research program and explore alternative explanations (with PVM); and (3) use complementary ways of collecting data (explicitly suggested by OH and the Payback Framework) e.g. publicly available documents, since the time available in workshop(s) can be limited.

<footnote n="3">3. A pathway's link refers to one event leading to another event.</footnote>
Furthermore, a detailed verification of the pathway links becomes imperative for participatory procedures, as discussions can be biased and scientific rigor can be questionable. Thus, three crucial supplementary tools were considered, although not used so far in the literature, or not explicitly, to assess ISRIP qualitatively (more details in the next sections): (1) the Social Network Analysis (SNA) (Scott, 2000); (2) process tracing (Bennett, 2010; Mahoney, 2012; George, Bennett, 2004) and (3) the counterfactual approach (Collins et al., 2004). We developed a step-by-step approach, which utilizes all those elements.

# First Step: Initial Screening, Identification of the Main Impacts and Collection of SNA Data

## Overview of the Case Study

In order to obtain an overview on the case study and understand general decision factors for conversion to organic farming, a preliminary screening of the case was needed. Therefore, in-depth interviews were undertaken with the actors (the most “aware” ones) at the different stages of the value chain (table 2); as well as a first workshop with key researchers and representatives of the producers (11 people attended<fnref n="4" />).

<table><caption>Table 2 – Actors interviewed and related objectives</caption><thead><tr><td>Actors interviewed</td><td>Specific objectives of the interviews</td></tr></thead><tbody><tr><td>INRA and CFR researchers</td><td>To identify the outputs produced by research activities; and gather understandings on the role of the research.</td></tr><tr><td>Natural Park of Camargue</td><td>To obtain a broad overview of the case study.</td></tr><tr><td>Traders (SARL Thomas<fnref n="5" />, the Comptoir Agricole du Languedoc and Biocamargue)</td><td>To collect overall views; and shed light on changes related to the scaling-up of the value chain.</td></tr><tr><td>15 farmers (4 organic and 7 partially-organic out of 35, and 4 conventional out of 180) chosen randomly</td><td>To identify general factors that facilitate or hinder farmers' transition; and to identify the most important impacts (socio-economic and environmental). 4 conventional farmers were selected (out of 180) to obtain external opinions. Moreover, the list of impacts was completed with the program's expected impacts.</td></tr></tbody></table>

<footnote n="4">4. Present: 6 farmers, 2 researchers from INRA, 1 speaker, and 2 researchers from the case study team.</footnote>

<footnote n="5">5. SARL Thomas is the main trader for organic rice, with around 5000 tons processed per annum.</footnote>
## SNA Data Collection and Identification of the Main Actors

We conducted a Social Network Analysis (SNA)<fnref n="6" /> with UCINET 6 software (Borgatti, Everett, and Freeman 2002). The interviewees were the same as the previous sub-step (overview of the case study), except that the conventional farmers were excluded as they do not belong to the organic network. They were interrogated during the same interviews (first round) as for the previous section (overview of the case study). We first used SNA to identify the most important actors in the organic actor network, which helped us to concentrate on the main players for the second workshop (see the second step). But of most importance was to further examine, later in step 4, what the stakeholders stated in the course of this second workshop. We calculated SNA indicators which allow for investigation into the degree of intermedia- tion of the actors (Betweenness), into the evolving intensity of relationships between pairs of actors (Degrees), and into the development of groups of actors who are strongly connected (Clustering Coefficient).

Stakeholders were asked to reconstruct the actor network during 6 peri- ods<fnref n="7" />, corresponding to significant changes within the actor system (scaling- up of the value chain, cooperation building or ending). Additionally, inten- sities of relationships around organic crop production, for each of the SNA dimensions (information flows, collaboration and financial links), were defined by stakeholders in accordance with a rating from 0 to 3⁸.

## Second Step: Stakeholder Pathway Building

A second workshop was organized with the aim of reconstructing the theory of change of the program and drawing the related pathway. On the basis of the SNA results (Betweenness) we invited the most important types of actors (20 persons attended<fnref n="9" />). However, all researchers involved in the research program were invited, as well as 7 organic and partially-organic farmers (beneficiaries) chosen randomly (out of 35).

Stakeholders were asked to identify changes (outcomes) related to the transition to organic production (e.g. adoption of new techniques to control weeds) before defining how, when and where they occurred, by identifying the

<footnote n="6">6. For more details on SNA, see Quideville, n.d. Paper submitted in 2016 in *The Journal of Agricultural Education and Extension*.</footnote>

<footnote n="7">7. In the years 1999, 2001, 2003, 2005, 2006, and 2010.</footnote>

<footnote n="8">8. We did not consider the direction of the relationships.</footnote>

<footnote n="9">9. Present: 2 researchers from INRA, 3 researchers from the CFR, 2 participants working at SARL Thomas, 1 respondent from Biocamargue, 1 moderator, 1 speaker, 7 organic and partially-organic farmers, and two assistants.</footnote>
related activities, outputs, and different milestones. This idea of reconstructing the pathway in reverse is based on the Outcome Harvesting approach.

Finally, the stakeholders drew the pathway (as expected in the PIPA approach) by linking cards representing the different pathway's events, i.e. the activities, outputs, etc. Identification of the changes was conducted in the plenary session, while the completion of the subsequent steps was undertaken within three diversified groups. However, it should be noted that, for reasons of time constraint, the choice was made not to ask stakeholders to specify which particular outcomes have triggered the different impacts. Still, we hypothesized that farmers do not necessarily feel concerned by impacts which do not occur at farm level (no such impacts were raised by farmers in face-to-face interviews during the first step).

### Third Step: Refinement of the Pathway and Collection of ISRIP Pathway Indicators

We first completed the stakeholders' pathway with relevant elements from the first step, where we took account of links which were raised by the majority of the “aware” interviewees (table 3). Then, ISRIP pathway indicators i.e. indicators for measuring each of the impacts, were defined by asking INRA researchers and studying the literature on organic Camargue rice. We asked researchers from INRA as they know the subject very well; the other stakeholders were not interviewed in order not to overburden them throughout the evaluation process. These indicators of impacts and the way they have been calculated are reported in table 5 (fifth step).

<table><caption>Table 3 – Pathway links arising from the first step, and opinions considered</caption><thead><tr><td>Type of links (components)</td><td>Opinions taken into account (from interviews conducted in the first step)</td></tr></thead><tbody><tr><td>Research activity to research activity.<br>Research activity to output</td><td>Researchers from INRA and CFR</td></tr><tr><td>At least one component on the institutionalization (scaling-up) of the organic rice value chain</td><td>Organic traders</td></tr><tr><td>Other links</td><td>Farmers</td></tr></tbody></table>

### Fourth Step: Process Tracing and Evaluation

In line with the PVM approach, which considers alternative explanations to the underlying causal logic of programs, we made the choice to use the process tracing method (the PVM approach does not mention this). This allows pathway links to be further explored and then validated. The links from the adoption of organic farming to the different impacts are also considered.
<table><caption>Table 4 – The four stages of process tracing in the method applied</caption><thead><tr><td>Stage</td><td>Explanations</td><td>Actors interviewed</td></tr></thead><tbody><tr><td>(1) Exploration of alternative explanations</td><td>a) The evaluator looks for alternative explanations to underlying mechanism(s) of the pathway links. They were defined from logical reasoning, studies on Camargue rice (Bayot et al., 2009; Mouret et al., 2005; Mouret et al., 2009; Delmotte et al., 2013; Mouret et al., 2012; Barbier et al., 2013; Mouret et al., 2004; Delmotte et al., 2011), innovation theories, SNA indicators, but also on stakeholders' statements from the first two steps<br/>b) Screening of the hypothesized pathway links (including alternative explanations) by appraising whether the necessary parts of the process (events of the pathway links and their underlying mechanisms) existed. We assessed the best evidence available from scientific papers, documents on the research program, official statistics and information, and other relevant information. SNA results were also considered for the links related to relationship issues</td><td>None in this fourth step<br/>None in this fourth step</td></tr><tr><td>(2) Data collection</td><td>For pathway links for which the necessary conditions were not satisfied, we conducted new face-to-face interviews (second round), in order to further understand these links. Links<sup>10</sup> related to research activities were studied with INRA; the others with farmers (we asked the most "aware" actors)</td><td>12 organic and partially-organic farmers (chosen randomly),<br/>1 INRA researcher</td></tr><tr><td>(3) 1<sup>st</sup> phase of elimination</td><td>If the necessary conditions were finally not satisfied, the events and related hypothesized pathway links were rejected<sup>11</sup></td><td>None</td></tr><tr><td>(4) 2<sup>nd</sup> phase of elimination</td><td>The analysis was completed by asking counterfactual questions to stakeholders, since the previous steps do not allow pathway links to be fully confirmed.<br/>Process tracing should be perceived more as a step towards ascertaining causal inferences (Mahoney 2010). We adapted the key counterfactual mechanism from quantitative methods (<sup>12</sup>) to our qualitative approach, which has already been advocated in some publications in the field of social science (e.g. King et al., 1994) and in impact evaluations of agricultural R&D (e.g. Walker et al., 2008). If the majority of the stakeholders argued that the first event of a pathway's link can be removed without calling into question the subsequent incidence (counterfactual test), the link was eliminated<sup>13</sup>.</td><td>The interviewed stakeholders were the same persons as for stage (3) "Data collection"; and the questions were asked during the same round of interviews (second round)</td></tr></tbody></table>

<footnote n="10">10. 5 and 32 links were studied by INRA and organic farmers, respectively.</footnote>

<footnote n="11">11. 20 links were rejected (all of them were alternative explanations).</footnote>

<footnote n="12">12. For further information about counterfactual: see (Pearl, 2000).</footnote>

<footnote n="13">13. Four links were rejected (including one alternative explanation).</footnote>
The rationale is, first, that other factors than research may influence the achievement of outcomes and impacts. Second, there is a need to confirm what the actors stated during workshops due to the possibility of power games influencing discussions. Process tracing (Bennett, 2010) is a tool that can be used to help establish that (1) an initial event or process took place, (2) a subsequent outcome also occurred, and (3) the former was a cause of the latter. It can be applied by two different versions; the Hoop or Smoking Gun test. We only applied the Hoop test that focuses on the necessary conditions of the different mechanisms. This test is passed if the necessary conditions are actually present. A necessary condition (e.g. better relationships among farmers around an innovation K) for an event Y (e.g. adoption of the technical innovation K) is a condition that must be fulfilled for Y to be reached; however, this condition is not necessarily sufficient. The Smoking gun test was not applied as it focuses on satisfactory conditions, in other words on factors that are sufficient to explain the occurrence of the mechanisms investigated. These strong requirements to pass the Smoking gun test would have been very difficult to identify and fulfill. Instead, the counterfactual approach was used to complete the Hoop test and fully confirm the occurrence of the pathway links. We completed four stages, which are reported in table 4. Moreover, note that we made use of a so-called "table of links", in order to organize the different information collected: origin and destination of the links, underlying mechanisms, and alternative explanations (see table 6 in appendix).

## Fifth Step: Importance of Pathway Routes and Measurement of the Impacts

### Critical Research Points and Importance of the Pathway Routes

First, we named as “Critical Research Points” (CRP) the second event of the pathway links that would not have happened without the research. To do so, we utilized the answers given by stakeholders in step 4, regarding counterfactual questions. The identification of CRP was intended to help identify the accurate role of the research in the innovation pathway. Additionally, we assessed the importance of pathway events in reaching subsequent events, with the aim of better estimating the contribution of the research per se (e.g. the extent to which the “development of crop rotation” was important to achieve the “adoption of the organic production mode”). The pathway routes that contain an outcome were studied by asking the same 12 organic and partially-organic farmers as for the fourth step (during the same interviews). The pathway routes focusing on relationships among actors were,
however, studied by the SNA; and for the other links we asked 1 researcher from INRA (still the same person and round of interviews). The scale used was; low (1), moderately important (2), and important (3).

## To Measure Impact Pathway Indicators

Another objective of the interviews conducted with farmers was to collect information to help measure impact pathway indicators. Table 5 specifies the indicators of impacts in relation to the different impacts while providing explanations as to how they were calculated.

<table><caption>Table 5 – Indicators of impacts</caption><thead><tr><td>Impacts</td><td>Indicators of impacts</td><td>Explanations (the interviews, if any, were completed in the 2<sup>nd</sup> round)</td></tr></thead><tbody><tr><td>Increase in farmers' revenue</td><td>Net margin on crop productions/ha</td><td>This indicator was calculated in the ILLIAD<sup>14</sup> project (Bassenne et al., 2014). Computation formula: (price*yield-production cost)</td></tr><tr><td>Decrease in the total rice area</td><td>Total surface (ha) under rice</td><td>Organic and partially-organic farmers were asked to describe their current crop rotation (in 2014) but also their rotation before the conversion.</td></tr><tr><td>Decrease in the use of pesticides.</td><td>Treatment Frequency Index (TFI)/ha (for all crop productions)<br/>Treatment frequency index (TFI)/ha (on average for all crop productions)</td><td>On the basis of the same interviews, the use of pesticides was estimated through the Treatment Frequency Index (TFI)<sup>15</sup>, which equals the ratio of the dose applied to the approved dose. Example: a TFI of 2 corresponds to 2 full doses applied on average/ha. The year 2014 and the year before the conversion were taken into account</td></tr><tr><td>Decrease in the use of nitrogen</td><td>N (kg)/ha (for all crop productions)</td><td>We asked farmers about their yields and crop rotations in 2014 and before their conversion. We calculated the quantity of nitrogen required (based on needs per 100kg of product)</td></tr><tr><td>Decrease in the use of fuels</td><td>L/ha (for all crop productions)<br/>L/ha (on average for all crop rotations)</td><td>Farmers were asked to report their current (in 2014) and past (before the conversion) crop management techniques with the material used. We then calculated the consumption of fuel based on the "barème d'entraide" (scoring grid) from the French Chamber of Agriculture</td></tr><tr><td>Decrease in the use of water</td><td>M<sup>3</sup>/ha (for all crop productions)</td><td>Organic and partially-organic farmers were asked to specify if their consumption of water per hectare of rice has changed because of the transition to organic farming, and to what extent</td></tr></tbody></table>

<footnote n="14">14. "Initiatives locales ou localisées, innovantes pour une alimentation durable" (local or localized initiatives, innovative for sustainable foods).</footnote>

<footnote n="15">15. TFI equals the ratio of the dose applied to the approved dose. The approved doses were found on the official website: e-phy.agriculture.gouv.fr.</footnote>
# RESULTS

## Impacts of the Transition to Organic Farming

### Significance of the Impacts

The impacts identified arise from the transition to organic farming, in other words from the outcome “adoption of the organic production mode”. In effect, stakeholders were asked precisely to describe the impacts linked to the conversion to organic farming. Additionally, the expected impacts of the research program were also related to the transition to organic agriculture. Eight impacts were identified, which we divided into three categories: the development of organic cropping, the environmental impacts and the socio-economic impacts. We understand socio-economic impacts as affecting the beneficiaries and the structure or vocation of the territory. The time span ranges from 1999 (just before the research program started) through to the year 2014.

* Development of organic cropping:

o The surface dedicated to organic rice production has steadily increased and attained 1400 ha in 2014¹⁶ (progressing from 608 ha in 2003 and 1050 ha in 2007).

o The number of organic rice producers had reached 16% of the total number of rice producers by 2014.

* Environmental impacts:

o The fall in the use of pesticides with a decrease in the Treatment Frequency Index (TFI<fnref n="17" />) of around 51% at the organic and partially-organic farm level, and 8.5% within the territory (all farms).

o The reduction in the water used was about 45% at the organic and partially-organic farm level, and 8% within the territory, primarily due to the decrease in rice surface area.

o The diminution in fuel<fnref n="18" /> consumption of about 17% at the organic and partially-organic farm level and 3% within the territory. This is the result introducing grasslands and alfalfa, which are less demanding in terms of cultivation.

<footnote n="16">16. There is a lack of statistics concerning the years between 2000 and 2003.</footnote>

<footnote n="17">17. TFI equals the ratio of the dose sprayed to the highest authorized dose. The approved doses were found on the official website: e-phy.agriculture.gouv.fr.</footnote>

<footnote n="18">18. We asked farmers to report their technical itineraries with the material used. We then calculated the consumption of fuel based on the “barême d’entraide” (scoring grid) of the French Chamber of Agriculture.</footnote>
o The decrease in nitrogen requirements of about 24% at the organic and partially-organic farm level and 4% within the territory. This is due to a decrease in the yields and the cultivation of crop production requiring few or zero units of nitrogen (grasslands and alfalfa).

* Socio-economic impacts:

o An increase in net margins per hectare (higher selling prices) of about 111% on organic crop productions, without taking account of the single payment entitlements.

o A reduction in the total surface devoted to rice (conventional and organic) of about 45% at the organic and partially-organic farm level and 8% within the Camargue, given the fact that conversion to organic rice requires an extended crop rotation to control weeds.

Origin of the Impacts

The stakeholders identified (in the second workshop) 6 incremental innovations (technical and organizational) related to the transition to organic farming, and from which the impacts arose. These innovations are part of the outcomes, in that they refer to adoption behaviors. The 6 incremental innovations raised by the stakeholders, and directly connected to subsequent impacts, are as follows:

* The development of crop rotation systems: this consists in extending and diversifying crop rotations. The rationale is to reduce risks linked to pests and weeds.

* False-seed bed techniques (mechanical): this consists in working the soil to allow the germination of weed seeds, followed by further cultivation to remove and control weeds.

* Sowing the paddy fields at a later date: the interest in deferring the sowing date is to boost rice growth through higher temperatures, which encourages weeds to be smothered.

* Increased level of water in paddy fields: this technique aims to smother weeds.

* Increased crop seeding rate: the objective is to smother weeds when they emerge.

* Organization of the organic value chain through the creation of the BIOSUD firm in 2003.
Role of the Research in the Impact Pathway

We describe how the research has played a role in the impact pathway by
assessing the different activities undertaken and their effects in terms of the
outputs and outcomes produced. Figure 1 is an illustration of the impact
pathway of the transition to organic farming in the Camargue.

<figure><img src="image_2.png" /><figcaption>Figure 1 – Simplified<fnref n="19" /> impact pathway of the conversion to organic farming in the Camargue</figcaption></figure>

Legend: orange arrow: moderately strong link; red arrow: strong link; large arrow (any color): Critical Research Points, i.e. the second events of the pathway links that would not have happened without the research.

ORPESA project: Importance of creating shared knowledge

The ORPESA training scheme (2006-2007) was completed thanks to initial
experimentations by INRA (2005-2006) and testing conducted indepen-
dently by farmers (refinement and optimization of their production systems).
INRA established professional training called the “ORPESA Table” in order
to support farmers’ conversion to organic production, mainly by sharing
knowledge on weed management and fertilization. The main interest areas
were then represented in leaflets and circulated by INRA by making these
available at the CFR for free consultation. There is clear evidence that the
“ORPESA Table” would not have occurred if the CEBIOCA project and
experimentations in farming plots had not taken place. The CEBIOCA
project (2000-2004) allowed a deepening of knowledge on yield variability

<footnote n="19">19. The links of low importance (rated as such by the stakeholders) were removed from this simplified chart.</footnote>
factors in organic production systems: it was demonstrated that weeds are a key factor<fnref n="20" />. INRA could not have participated in ORPESA without possessing significant knowledge to contribute to the exchange platform. In the same context, the trials led by producers on their own were undeniably needed. The advanced experimentations (2011) were implemented based on the recommendations made by producers in the ORPESA project; however as previously stated, they seem to have played a limited role in the transition to organic agriculture.

**The Social Network Analysis: A confirmation of the role played by INRA and BIOSUD**

The SNA<fnref n="21" /> has allowed us to confirm the important and growing influence of INRA and BIOSUD. The Betweenness score of INRA has increased by 46% (370 in 1999 compared to 542 in 2014). Regarding BIOSUD, its growing influence can be illustrated by the rise of its Clustering Coefficient by 18% between 2003 and 2005 (after it was created). Furthermore, underlying mechanisms were identified: the Degrees between farmers on the one hand and INRA and BIOSUD (previously called SARL Thomas<fnref n="22" />) on the other have significantly increased between 1999 and 2014; respectively by 80% and 40%.

Moreover, it should be noted that INRA was more acknowledged by farmers who have hosted their scientific experimentations: they rated a double score of relationships with INRA. Also, the SNA has shown there was low involvement by the CFR, which confirmed what has been stated by the stakeholders during the second workshop. In fact, the CFR is strongly linked to the Rice Farmers Union and is under pressure from some conventional rice producers committed to the Rice Farmers Union.

**Limited role played by CIRAD**

Since 2012, CIRAD has been implementing specific experimentations, which are focused on the technical use of different machines like chain harrows, as well as on the interest of bringing ducks onto the land to regulate weeds. Surprisingly, the ducks like eating weeds but have little interest in rice plants. That said, we have to emphasize the very limited scope of these experimentations. Currently, only one rice producer benefits from this research effort. The other farmers are not willing to make use of this production system,

<footnote n="20">20. Experimentations in farming plots (in 2005-2006) would not have happened without the CEBIOCA project since INRA would not have been aware of the main issues to be studied.</footnote>

<footnote n="21">21. For a more detailed SNA, refer to (Quiedville *et al*., n.d.). Paper submitted in 2016 in *The Journal of Agricultural Education and Extension*.</footnote>

<footnote n="22">22. Farmers made reference to BIOSUD when talking about SARL Thomas.</footnote>
mainly because of the time-consuming nature of the technique (soil working, meticulous management of the height of the water, duck management etc.).

### The CFR: A limiting factor in the pathway

In the workshops and in-depth interviews, the rice farmers highlighted a lack of involvement by the CFR. This is seen as a barrier for them to convert to organic rice production (lack of experimentation and very weak knowledge of brokering activities). The farmers claimed that the CFR should work in specialist areas related to organic rice, which INRA currently engages in, but CFR should also be involved. The French Rice Centre's missions are to provide information and advice to farmers, experiment with cultivation techniques, and implement a breeding program. However, the institute has not undertaken specific research for organic rice until now.

## Role of External Factors as Compared to the Research

All the farmers emphasized the crucial importance of, and unfeasibility of, converting to organic farming without extending their crop rotation. However, the solutions provided by the research for crop rotations were not well acknowledged; rather farmers underlined the greater importance of their own trials, with no direct support from research. In effect, farmers argued scientific experimentations were not very well adapted to their particular situation (different geographical and soil conditions). In particular, the INRA leaflets on how to manage weeds in organic farming systems, and created on the basis of the advanced experimentations (2011), were only moderately appreciated.

A second crucial external change is improved market access due to the creation of the packer enterprise BIOSUD in 2003. BIOSUD was created by the cooperative SudCéréales and SARL Thomas (a private trader) with a view to selling their organic products through a specialized enterprise. The decision to found the company was motivated by the growing number of organic farmers, but with no clear direct link to the research program. The foundation of the BIOSUD firm seems, in turn, to have sustained the adoption process of organic cropping by developing markets' access for farmers.

Finally, the contribution of the research is reduced by the key role played by external economic drivers. These comprise the high selling price of the organic rice compared to conventional rice (double the value) as well as the CAP payments to both convert and maintain organic surfaces. Without those external factors, the adoption process would not have taken place, the farmers reported.
# DISCUSSION AND CONCLUSION

## Discussion

### Effects of the Research Program

We identified research activities that produced significant outputs, but their influence on the innovation pathway remained partial. In other words, the research program contributed to achieving outcomes and impacts, but not in a substantial way. This is illustrated by the fact that the Critical Research Points are only situated along the linkages from activities to outputs. The perception of the farmers reflected a situation where there was a certain gap between research outputs and their individual decisions regarding the adoption of new techniques. Three main pieces of evidence can be emphasized: (1) informal testing made by farmers on crop rotations was very important in converting to organic farming; (2) the institutionalization of the supply chain was a decisive factor; (3) economic factors were fundamental.

Although outcomes and impacts were achieved, the research institutions were not critical driving forces in their achievement. The role of INRA in the actor network increased, but this has not greatly supported the conversion to organic farming, nor the achievement of impacts. Two main reasons were identified: (1) the technical innovations developed by the research were not considered to be very suitable by farmers; and (2) there was a lack of appropriate advice offered by INRA, which raises three important characteristics for the implementation of innovations:

* Communication support (orally or written) and the way in which the farmers receive the information (one to one advice, plenary sessions, etc.).

* The specific and heterogeneous local conditions in farms, which can reduce the effectiveness of generalized information to all the farms.

* The question of which institutions undertake experimentations.

The evaluation of the research program has particularly allowed stakeholders to identify barriers to the adoption of organic farming, especially the relative inadequateness of the experimentations (farms are very heterogeneous) and the insufficient involvement of the CFR. Stakeholders came to the conclusion that (1) the CFR should work in specialist areas with organic rice, which INRA currently engages in, which CFR should also be undertaking; (2) the experiments should be designed and conducted with better farmer collaboration; and (3) on-farm trials should be further developed.
Furthermore, although the ORPESA project has allowed the creation of shared knowledge and has helped in designing advanced experiments, an insufficient number of farmers participated. Moreover, workshops raised sensitive issues concerning relationships and interactions among the local actors, which have had implications, particularly in terms of scientific collaborations. The number of actors in the Camargue is very limited, forming a “social microcosm” where individuals are influenced by the locally dominant opinion. Interestingly, the CFR and INRA recently decided (after evaluation of the research program) to collaborate more closely on organic farming system issues, as desired and expressed by farmers in the second workshop. Finally, we believe that external experts should regularly assess whether and how outputs, outcomes and impacts are actually achieved in order to produce changes in the way research programs are executed, whilst there is still an opportunity to do so. In fact, the researchers seem to be too optimistic regarding the use of results by potential beneficiaries, as well as the achievement of valuable results.

## Differentiation of Results

Our results reflect a broader observation concerning agronomic research on rice. It is important to consider factors other than research. As an example, a study commissioned by IRRI (International Rice Research Institute) in 1993 in Asia showed that improved varieties by breeding process were only adopted by farmers possessing parcels in which the irrigation is well controlled (Chataigner, 1997; David, Otsuka, 1994). In that case, good production structures were needed to allow the research to be effective. In the same vein, consistent support payments, political interventions, or an appropriate institutional context are essential to create a favorable environment and permit the research to reach its full potential. Moreover, knowledge is not only generated by researchers but also by farmers and private companies (EU SCAR, 2012). The trials conducted independently by Camargue producers are a good example of this. Joint research between institutions and farmers are also important; and were particularly valued in our case study by means of on-farm trials. Studies like JOLISAA<fnref n="23" /> (Almekinders et al., 2012) demonstrated more clearly, but in an African context, that collaborative research can solve problems encountered by farmers to make desired changes possible. Lastly, in the general context of increasingly complex models of knowledge co-creation, the needs of and the varied types of knowledge must be acknowledged and reflected interactively among stakeholders (Moschitz et al., 2015; EU SCAR, 2012; Klerkx et al., 2009). Nevertheless, in our case

<footnote n="23">23. Joint Learning in Innovation Systems in African Agriculture.</footnote>
study the CFR did not acknowledge the need to develop more sustainable
systems and has disrupted the pathway the innovation was taking. This
situation shows that when farmers' needs are ignored, extension services
or institutions can be barriers to innovation. Knickel et al. (2009) found a
similar result in the context of adapting farms' systems to multifunctional
agriculture.

The Method Applied to Evaluate ISRIP

Ex-post rebuilding of the ISRIP pathway in reverse allowed us to consider all
changes, both linked and unrelated to the intervention. The second work-
shop, which aimed at reconstructing the pathway of the research, was not
strictly focused on the effects of the research program, as alternative routes
to the ISRIP pathway were also discussed. Additionally, process tracing and
counterfactual approaches were very useful complements to the mixed-
method. Indeed, they allowed the hypothesized pathway links to be accepted
or rejected; and the specific role of the research to be evaluated accurately.

Some challenges have been encountered in the second workshop with
respect to the identification of outcomes by stakeholders, and to obtain suf-
ficient stakeholder diversity. Regarding the first point, stakeholders were
asked to first define the outcomes (changes) before linking them to activi-
ties and outputs. Not providing inputs to stakeholders had allowed signifi-
cant responsibilities to be given to them, but they encountered difficulties
in identifying outcomes related to the transition towards organic farming. It
could have been more appropriate to ask about changes occurring in their
personal situations instead of more generally, in order to make the exercise
more concrete. Another possibility would have been to present preliminary
results (i.e. to present likely changes due to organic farming conversion)
based on initial face-to-face interviews, before engaging in discussions.
Furthermore, in the second workshop, there were probably too many farm-
ers compared to the number of researchers. It follows that farmers were in
a relatively dominant position in the discussions; however, a wide range of
opinions could be debated. For example, at the beginning of the workshop,
farmers advocated that the research did not produce any effects on organic
farming conversion. But INRA then displayed the leaflets they produced to
those attending, in order to defend a completely different point of view; and
the discussion could be launched on a better basis.

Moreover, the impacts were identified based on individual interviews with
the program's beneficiaries, i.e. the farmers. Researchers were not involved,
given their direct involvement in the program, but this may be viewed as a
bias in the analysis since general impacts on the environment, like a higher
level of biodiversity, were not raised. Individuals from civil society may also
have been involved, to identify potential impacts which do not directly
affect farmers. Additionally, the method applied had slight weaknesses with
respect to the identification and understanding of the mechanisms occur-
ring between outcomes and impacts. Stakeholders were not asked to specify
which particular outcomes have triggered the impacts. For future similar
studies, however, one may think about identifying in workshop(s), but only
at farm level (we assume farmers are not particularly interested in the other
impacts), the specific outcomes leading to impacts (in addition to the aggre-
gated outcome "adoption of the organic production mode"). The pathway
links that do not directly concern farmers may be investigated by a panel of
experts in face-to-face interviews.

GENERAL CONCLUSION

We demonstrated that the research played a limited role in the innovation
pathway, although we also underline that the research could not be suffi-
cient by itself and requires political, institutional and organizational support.
In particular, we stress that an ambitious agricultural policy in the form of
support payments, as well as research collaboration through on-farm trials,
is required. Furthermore, a lack of attention to the needs of farmers seems to
partly explain the failures in the innovation pathway.

It would be of interest to further explore whether our results and conclu-
sions can be confirmed in other contexts, i.e. in other territories and under
alternative conditions such as different policy or institutional schemes. In
particular, it would be interesting to further investigate under which circum-
stances extension services or institutions are becoming barriers or catalysts
for innovation and research implementation.

Moreover, the mixed-method applied appears to be suitable, but could
be further refined. As an example, we could slightly reduce its participatory
nature in order to better evaluate the impacts. Additional research is needed
to examine this and achieve a more consolidated method.

REFERENCES

ALMEKINDERS, C. et al. (2012), *A Review of IS Theories and Approaches and Their Significance for Agricultural/Rural Innovation in Small-Scale Farming in Africa*, Montpellier, France.

BARBIER, J.-M. et al. (2013), *Une plateforme de recherche-développement pour stimuler les apprentissages et accompagner l'agriculture biologique en Camargue (France)*, Paris, INRA.
BASSENNE, J.-B. et al. (2014), Organisation en réseau et durabilité systémique de deux filières alimentaires (riz biologique et petit épeautre en France), in *Actes des 8èmes Journées de recherches en sciences sociales*, Grenoble, France.

BAYOT, M. et al. (2009), Formation professionnelle et recherche participative. Actions combinées pour développer la riziculture biologique en Camargue (France), *Innovations Agronomiques*, 447-455.

BENNETT, A. (2010), Process Tracing and Causal Inference, in Rowman, Littlefield (eds), *Rethinking Social Inquiry: Diverse Tools, Shared Standards*.

BORGATTI, S., EVERETT, M., FREEMAN, L. (2013), *Analysing Social Networks*, SAGE Publications Ltd, 1st edition.

BOZEMAN, B., ROGERS, J. D. (2002), A Churn Model of Scientific Knowledge Value: Internet Researchers as a Knowledge Value Collective, *Research Policy*, 31(5), 769-794.

BOZEMAN, B., SAREWITZ, D. (2005), Public Values and Public Failure in US Science Policy, *Science and Public Policy*, 32(2), 119-136.

CGIAR Impact Assessment and Evaluation Group (2000), Impact Assessment of Agricultural Research: Context and State of the Art, in *Workshop on Impact Assessment of Agricultural Research in Eastern and Central Africa* (Uganda, November 1999). Rome. Available at: http://www.cgiar-ilac.org/files/tac_impact_2000.pdf [Accessed August 17, 2015].

CHATAIGNER, J. (1997), Activités de recherche sur le riz en climat méditerranéen, *Cahiers Options Méditerranéennes*, 24(2).

COLINET, L. et al. (2014), ASIRPA – Analyse des impacts de la recherche publique agronomique. Rapport final, Paris. Available at: https://www6.inra.fr/asirpa.

COLLINS, J., HALL, N., PAUL, L. (2004), *Causation and Counterfactuals*, Cambridge, Massachusetts, London, The MIT Press.

COUSINS, J. B., EARL, L. M. (1992), The Case for Participatory Evaluation, *Educational Evaluation and Policy Analysis*, 14(4), 397-418.

COUSINS, J. B., WHITMORE, E. (1998), Framing Participatory Evaluation, *New Directions for Evaluation*, 80, 5-23.

COZZENS, S., SNOEK, M. (2010), Knowledge to Policy., in *Workshop on the Science of Science Measurement*, Washington.

DAVID, C., OTSUKA, K. (1994), *Modern Rice Technology and Income Distribution in Asia*, International Rice Research Institute, Blackwell.

DELMOTTE, S. et al. (2013), Obstacles, Levers and Impacts of Organic Farming Development in Camargue, *Innovations Agronomiques*, 32, 213-226.

DELMOTTE, S. et al. (2011), On Farm Assessment of Rice Yield Variability and Productivity Gaps between Organic and Conventional Cropping Systems Under Mediterranean Climate, *European Journal of Agronomy*, 35(4), 223-236.

DONOVAN, C., HANNEY, S. (2011), The “Payback Framework” Explained, *Research Evaluation*, 20(3), 181-183.

DOUTHWAITE, B. et al. (2007), Participatory Impact Pathways Analysis: A Practical Application of Program Theory in Research-For-Development, *Canadian Journal of Program Evaluation*, 22(2), 127-159.

EARL, S., CARDEN, F., SMUTYLO, T. (2001), *Outcome Mapping: Building Learning and Reflection into Development Programs*, International Development Research Centre.
EDQUIST, C. (1997), *Systems of Innovation Approaches: Their Emergence and Characteristics*, in C. Edquist (ed.), *Systems of Innovation: Technologies, Institutions and Organizations*, London, Pinter, 1-35.

EU SCAR (2012), *Agricultural Knowledge and Innovation Systems in Transition: A Reflection Paper*, Brussels.

EUROPEAN COMMISSION (2015a), *Horizon 2020 Indicators*, Luxembourg. Available at: https://ec.europa.eu/programmes/horizon2020/en/news/horizon-2020-indicators-assessing-results-and-impact-horizon [Accessed March 10, 2016].

EUROPEAN COMMISSION (2015b), Towards a Long-Term Strategy for European Agricultural Research and Innovation by 2020 and Beyond. Available at: http://ec.europa.eu/agriculture/expo-milano-2015/cap-events/long-term-vision/background_en.pdf [Accessed February 23, 2016].

EUROPEAN COMMISSION (2016), What is Horizon 2020? Available at: https://ec.europa.eu/programmes/horizon2020/en/what-horizon-2020 [Accessed March 10, 2016].

GEORGE, A. L., BENNETT, A. (2004), *Case Studies and Theory Development in the Social Sciences*, Cambridge MA, The MIT Press.

GIBBONS, M. et al. (1994), *The New Production of Knowledge : The Dynamics of Science and Research in Contemporary Societies*, London, Sage.

HALL, A. et al. (2003), From Measuring Impact to Learning Institutional Lessons: An Innovation Systems Perspective on Improving the Management of International Agricultural Research, *Agricultural Systems*, 78(2), 213-241.

KING, G., KEOHANE, R. O., ROBERT, O., VERBA, S. (1994), *Designing Social Inquiry : Scientific Inference in Qualitative Research*, Princeton, Princeton University Press.

KLERKX, L. W., HALL, A. J., LEEUWIS, C., 2009. Strengthening Agricultural Innovation Capacity: Are Innovation Brokers the Answer?, in W. Universiteit, UNU-MERIT (eds), *GLOBELICS 2009, 7th International Conference, 6-8 October, Dakar, Senegal*, Georgia Institute of Technology.

KNICKEL, K. et al. (2009), Towards a Better Conceptual Framework for Innovation Processes in Agriculture and Rural Development: From Linear Models to Systemic Approaches, *The Journal of Agricultural Education and Extension*, 15(2), 131-146.

MACKAY, R., HORTON, D. (2003), Expanding the Use of Impact Assessment and Evaluation in Agricultural Research and Development, *Agricultural Systems*, 78(2), 143-165.

MAHONEY, J. (2010), After KKV: The New Methodology of Qualitative Research, *World Politics*, 62(1), 120-147.

MAHONEY, J. (2012), The Logic of Process Tracing Tests in the Social Sciences. *Sociological Methods & Research*, 41(4), 570-597.

MATHIE, A., GREENE, J. C. (1997), Stakeholder Participation in Evaluation: How Important is Diversity?, *Evaluation and Program Planning*, 20(3), 279-285.

MOSCHITZ, H. et al. (2015), Learning and Innovation Networks for Sustainable Agriculture: Processes of Co-evolution, Joint Reflection and Facilitation, *The Journal of Agricultural Education and Extension*, 21(1), 1-11.

MOURET, J.-C. et al. (2004), An Integrated Study of the Development of Organic Rice Cultivation in the Camargue (France), in *International Conference on Challenges and Opportunities for Sustainable Rice-Based Production Systems*, Torino.
MOURET, J.-C. et al. (2005), La construction d'une démarche interdisciplinaire à partir de l'émergence de la céréaliculture biologique en Camargue: le projet CEBIOCA, in *Actes du séminaire sur les recherches en agriculture biologique – Programme AgriBio de l'INRA*, 31-41.

MOURET, J.-C. et al. (2009), Production de références pour optimiser la fertilisation organique en riziculture biologique camarguaise (France), *Carrefours de l'Innovation Agronomique*, 4, 9-13.

MOURET, J.-C. et al. (2012), Démarches et méthodes pour produire des références techniques et pour accompagner les riziculteurs biologiques camarguais (France), in *1re conférence internationale. Les systèmes de production rizicole biologique*. Available at: http://www1.montpellier.inra.fr/orp2012/images/communications/mouret.pdf [Accessed August 26, 2016].

OECD (2002), *Glossary of Key Terms in Evaluation and Results Based Management*, Paris, OECD Publications. Available at: https://www.oecd.org/dac/evaluation/2754804.pdf.

OECD (2016), *Oslo Manual*, Available at: https://www.oecd.org/sti/inno/2367580.pdf [Accessed May 10, 2016].

PEARL, J. (2000), *Causality : Models, Reasoning, and Inference*, Cambridge University Press.

PENFIELD, T. et al. (2013), Assessment, Evaluations, and Definitions of Research Impact: A Review, *Research Evaluation*, 23(1), 21-32.

PLOTTU, B., PLOTTU, E. (2009), Contraintes et vertus de l'évaluation participative, *Revue française de gestion*, 192.

QUIEDEVILLE, S., Unpublished. Contribution of Social Network Analysis for evaluating Impacts of Science-Based Research and Innovation Program: The example of the farmers' conversion to organic crop production in Camargue, *The Journal of Agricultural Education and Extension*.

SCOTT, J. (2000), *Social Network Analysis: A Handbook*, London, Sage.

SPAAPEN, J. et al. (2013), *SIAMPI Final Report*. Available at: https://www.research.manchester.ac.uk/portal/files/33083211/FULL_TEXT.PDF [Accessed August 18, 2016].

SPRINGER-HEINZE, A. et al. (2003), Impact Pathway Analysis: An Approach to Strengthening the Impact Orientation of Agricultural Research, *Agricultural Systems*, 78(2), 267-285.

TANDON, R., FERNANDEZ, W. (1984), *Participatory Evaluation : Theory and Practice*, New Delhi, Indian Institute for Social Research.

TOUZARD, J.-M. et al. (2014), Systèmes d'innovation et communautés de connaissances dans le secteur agricole et agroalimentaire. *Innovations*, 43(1), 13.

WALKER, T. et al. (2008), *Strategic Guidance for Ex Post Impact Assessment of Agricultural Research*. Available at: ftp://ftp.fao.org/docrep/fao/011/i0276e/i0276e.pdf [Accessed August 22, 2016].

WEISS, C. (1997), *Evaluation : Methods for Studying Programs and Policies*, Prentice Hall; 2nd edition.

WILSON GRAU, R., BRITT, H. (2002), *Outcome Harvesting*, K. Andrews (ed.), Cairo, Egypt. Available at: http://usaidlearninglab.org/sites/default/files/resource/files/Outome Harvesting Brief FINAL 2012-05-2-1.pdf [Accessed August 17, 2015].
# APPENDIX

<table><caption>Table 6 – Blank table of links</caption><thead><tr><th colspan="2">Pathway links</th><th rowspan="2">Description of the underlying mechanism(s)</th><th rowspan="2">Hypothesized alternative explanations to the mechanism(s)</th></tr><tr><th>Origin of the link</th><th>Destination of the link</th></tr></thead><tbody><tr><td>Example: Activity 1<br>(name of the activity 1 to be specified)</td><td>Example: Output 1<br>(name of the output 1 to be specified)</td><td>(Most relevant evidences to be specified)</td><td>(Plausible alternative explanations to be specified)</td></tr></tbody></table>