# How can the past inform and benefit research-for-development interventions? The value of reflexive and participatory approaches

Marie Ferre, Genowefa Blundo Canto, Aurelle de Romemont, Berline Africa

Maagoum Soh, Aurélie Binot

<figure><img src="image_1.png" /></figure>

## ► To cite this version:

Marie Ferre, Genowefa Blundo Canto, Aurelle de Romemont, Berline Africa Maagoum Soh, Aurélie Binot. How can the past inform and benefit research-for-development interventions? The value of reflexive and participatory approaches. Anthropologie et développement, 2022, 52, pp.155-174. 10.4000/anthropodev.1808. hal-05181229

HAL Id: hal-05181229

https://hal.science/hal-05181229v1

Submitted on 23 Jul 2025

HAL is a multi-disciplinary open access archive for the deposit and dissemination of scientific research documents, whether they are published or not. The documents may come from teaching and research institutions in France or abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est destinée au dépôt et à la diffusion de documents scientifiques de niveau recherche, publiés ou non, émanant des établissements d'enseignement et de recherche français ou étrangers, des laboratoires publics ou privés.

<figure><img src="image_2.png" /></figure>

Distributed under a Creative Commons Attribution 4.0 International License
How can the past inform and benefit research-for-
development interventions? The value of reflexive
and participatory approaches

Marie Ferré, Genowefa Blundo Canto, Aurelle de Romémont, Berline Africa Maagoum Soh and Aurélie Binot

<figure><img src="image_3.png" /></figure>

OpenEdition
Journals

Electronic version

URL: https://journals.openedition.org/anthropodev/1808

DOI: 10.4000/anthropodev.1808

ISSN: 2553-1719

Publisher

Presses universitaires de Louvain

Printed version

Date of publication: 25 October 2022

Number of pages: 155-174

ISBN: 978-2-39061-270-4

ISSN: 2276-2019

Brought to you by CIRAD Centre de coopération internationale en recherche agronomique pour le développement

<figure><img src="image_4.png" /></figure>

cirad

LA RECHERCHE AGRONOMIQUE
POUR LE DÉVELOPPEMENT

Electronic reference

Marie Ferré, Genowefa Blundo Canto, Aurelle de Romémont, Berline Africa Maagoum Soh and Aurélie Binot, "How can the past inform and benefit research-for-development interventions? The value of reflexive and participatory approaches", *Anthropologie & développement* [Online], 53 | 2022, Online since 25 April 2023, connection on 27 April 2023. URL: http://journals.openedition.org/anthropodev/1808 ; DOI: https://doi.org/10.4000/anthropodev.1808

<figure><img src="image_5.png" /></figure>
<figure><img src="image_6.png" /></figure>

# How can the past inform and benefit research-for-development interventions?

## The value of reflexive and participatory approaches

Marie Ferré*, Genowefa Blundo Canto**,
Aurelle de Romémont***,
Berline Africa Maagoum Soh****, Aurélie Binot*****

Development projects are often anchored in short timeframes that limit the possibility of creating a continuum across projects and building a collective "living memory" based on the accumulation of experiences and interactions. Taking stock of the influences of projects on a place is important for research-for-development projects, which are characterised by long-term dynamics and a learning posture. Impact evaluation approaches can help capture, develop, and mobilise this "living memory". Using the case of the French Agricultural Research Centre for International Development (CIRAD), we examine how researchers at an institute with a development mandate capitalise their experiences. We explore the added value of reflexive approaches developed by CIRAD in particular the role played by participation in building collective memories at different levels. The results provide insights into how to integrate the past into the design and implementation of research interventions, and thus increase relevance of projects. It offers a methodological reflection on the creation and use of research-for-development memory.

Les projets de développement sont souvent ancrés dans une temporalité courte qui limite la possibilité de créer un continuum entre projets et de construire une « mémoire vivante » collective. Considérer les influences de projets sur un lieu donné est important pour les projets de recherche à vocation de développement, caractérisés par une dynamique de long terme et une posture d'apprentissage. Les approches d'évaluation d'impact peuvent aider à capturer, développer et mobiliser cette mémoire. Nous examinons comment les chercheurs d'un institut à mandat de développement (Centre de coopération internationale en recherche agronomique pour le développement - CIRAD) capitalisent leurs expériences. Nous explorons des approches réflexives développées par le CIRAD et le rôle de la participation pour la construction de mémoires à différents niveaux. Les résultats fournissent des indications sur l'intégration du passé dans la conception et la mise en œuvre d'interventions, et offrent une réflexion méthodologique sur la création et l'utilisation de la mémoire de la recherche pour le développement.

<footnote>* UMR-Innovation, CIRAD; marie.ferre@cirad.fr</footnote>

<footnote>** UMR-Innovation, CIRAD; genowefa.blundo_canto@cirad.fr</footnote>

<footnote>*** UMR-Innovation, CIRAD; aurelle.de_romemont@cirad.fr</footnote>

<footnote>**** UMR-Innovation, CIRAD; sohberline98@gmail.com</footnote>

<footnote>***** UMR-Astre, CIRAD; aurelie.binot@cirad.fr</footnote>
Introduction

Development projects are often anchored in a defined and definite timeframe (Schultz, 2013; Barrier, 2018) that limits the possibility for researchers and research institutions to create a continuum across projects, construct a long-term legacy based on collective learning, and build a collective memory founded on the systematic accumulation of experiences. This element of continuity is particularly important in the case of research-for-development projects that entail slow, long-term processes and a continuous learning posture. While science is in essence a cumulative exercise that evolves over time and relies on progress within the scientific community, with its aim of advancing knowledge and identifying innovative research avenues, it increasingly tends to operate in a “project mode” (Covo, 2014). The design and objectives of research-for-development projects are especially determined by diverse – and sometimes conflicting – drivers, such as external demand (including donors and policy makers), institutional strategy, gaps in the literature, the relevance and novelty of the research questions, and society’s interest in the topic. Project-bound thinking leads to a sort of *tabula rasa* approach in which every new project appears to act in a time vacuum. This makes long-term research strategies difficult (Hubert and Louvel, 2012) and encourages a focus on the present and on the ‘performance’ of projects, thereby limiting the chance of in-depth stocktaking from the past as part of the project design. In this study, we explore how researchers working with a development mission engage in stocktaking and sharing experiences of a place, and show how an innovative methodological approach contributes to foster reflections on the past and inform future research-for-development projects.

Analysing the experiences, influences, traces, and characteristics of a place that have emerged from previous interventions increases the effectiveness of research-for-development projects (which we will refer from now on as “interventions”<fnref n="1" />). This form of capitalisation can materialise in the building of a collective “memory” of dynamics and interactions between interventions and a place: that is, the social-ecological context that encompasses people, environments, and institutions (Oteros-Rozas et al., 2015), evolving over time (Fouéré, 2010). Retrieving, capturing, understanding, and remobilising this knowledge, which can be either explicit or implicit, is a challenge. It means exchanging and creating a “collective vision” of the past. It takes significant human, time, and financial resources to consider how previous interventions – both individually and aggregated with others – have shaped a place at different levels (population, landscape, natural resources, institutions, and interactions), and to reflect collectively on its “aggregated living memory”. Constructing this memory of a place helps us understand the political context, potential tensions among communities, social norms, and perceptions that may have emerged from previous interventions. This information can help orient the design of new interventions, avoid the replication of research, management, and partnership-related errors, make informed decisions, and build sustainable change for the future. At a project management level, it can help foresee potential risks, increase the readiness of teams to face issues and

<footnote n="1">¹ We use the term “intervention” as it is not restricted to project-type interventions, and can encompass various modalities and formats of research intervention.</footnote>
come up with strategies, and procure information on research scaling-up and scaling-out potential. It also allows the creation of bonds among the actors involved (researchers, other institutions, and local populations) by recollecting the pathway they have shared. Not all traces left by previous interventions can be captured and examined, however: some are intangible, while others may deal with unconscious or non-explicit representations (Baussant, 2007; Halbwachs, 1925, 1952). This poses the question of how much and what type of effort should be deployed to capture this memory and collect corresponding information during project lifetimes, while still aiming at a certain level of continuity across projects and the scientific issues addressed.<fnref n="2" />

Some authors have investigated how to institutionalise and professionalise the project management discipline in order to better align project management approaches and practices (action logics) with institutional objectives (Bresnen, 2016), learn from previous projects (Williams, 2008), and look beyond the immediate duration of a project, for instance by capturing anticipated and wider impacts from a longer-term perspective (Samset and Volden, 2016). Marija Lj. Todorović et al. (2015), however, found that systems set up to identify and transfer knowledge from past to future projects are rarely implemented. Capturing the influences of projects on a place in order to benefit project design and management is not common. Little attention is paid to reviewing, analysing, and sharing the success of projects in terms of learning beside performance (Williams, 2008), and the results of previous projects are often poorly documented, leading to a low level of knowledge management (Todorović et al., 2015). This contrasts with evidence that project success analysis would encourage the acquisition and creation of knowledge and its transfer in the project environment (Todorović et al., 2015). Furthermore, effective management of this knowledge – in the sense of being able to retain, organise, and utilise tacit, implicit, and explicit knowledge – has proved to be a driver of organisational performance, sustainability, and creativity (Omotayo, 2015).

This learning and knowledge capitalising posture is fundamental in the research-for-development sector. Several research-for-development organisations (such as INRAE, CIRAD, and CGIAR)<fnref n="3" /> have recognised the importance of “looking back” and analysing the impacts and the change processes their projects contribute to generating, with the purpose of learning from previous experience. They advocate long-term reflexive approaches and learning from how research contributes to generating diverse impacts for multiple actors, along with the contributions of other actors and factors (Joly and Matt, 2017; Thornton et al., 2017; Blundo-Canto et al., 2019; Reed et al., 2021). They focus on the use of evaluation findings as part of a strategy to feed an organisational learning process. They also value approaches that enable prospective visions of impacts from research at various levels in

<footnote n="2">² We acknowledge that not all interventions may find previous experiences useful for their design. Some may require a brand-new look at a situation without having to dig into “old” experiences, reviving old tensions, and taking the risk of becoming stuck in routines, which may diminish creativity and create fatigue among partners.</footnote>

<footnote n="3">³ INRAE is the French research institute for the sustainable development of agriculture, food and the environment; CIRAD is the French Agricultural Research Centre for International Development; CGIAR is the current name of the former Consultative Group for International Agricultural Research.</footnote>
order to improve research practices (Watts et al., 2008) and make research more trans-
formative for society (Schot and Steinmueller, 2018; Cooper, 2014). Nonetheless, the
seemingly dual mandate of research and development is often reflected in diverging visions
of whether research from these organisations should be place-based and contextual, or
whether it should produce strategic international public goods (Leeuwis et al., 2018).
Hence, the type of study that we present in this article is very valuable to enable this level
of reflexivity in a context of “development and research”.

Our final aim is to understand how researchers working in a development-mandated
research institution share and capitalise on their intervention experiences by exploring how
reflexive approaches facilitate the building of a collective memory. In particular, we exam-
ined the value of participatory, reflexive, and iterative approaches and investigated whether
those are decisive when it comes to mobilising, rebuilding, or creating a collective memory.
In order to address this question, we used the French Agricultural Research Centre for In-
ternational Development (CIRAD) and two evaluation approaches it has developed (Blundo-
Canto et al., 2019) – *ex post* (retrospective) and *ex ante* (prospective) – as a case study.
Drawing from a recollection of the path development of these approaches, the existing
knowledge associated with their application, and a survey of the CIRAD employees who
mobilised them, we attempted to capture the various forms a collective memory can take.
We therefore examined three different levels of memory: 1) an institutional memory
through a study of the path first taken in 2010 to promote new reflections within the insti-
tution; 2) group-level memories emerging from *ex post* and *ex ante* related activities
relating to building a common understanding and vision of past or future interventions; and
3) individual level memories, with reference to what individuals retain in their own practices
following their involvement in *ex post* or *ex ante* evaluation activities.

Our analysis provides valuable insights into the importance of integrating “the past” in
the design and implementation of new research-for-development interventions, how
memory is created in these interventions, and how to foster its use by individuals and
groups. Our findings open new directions for “past-aware” cultures within research-for-
development institutes, and provide recommendations on pathways for improving research
practices and the related project development and management for transformational
change. The article unfolds as follows: Part 2 describes the context of this study and the
method; Part 3 presents our findings; Part 4 discusses the results; and Part 5 offers some
conclusions.

Context setting and method

Context

CIRAD is an agricultural research organisation that develops research oriented towards
serving international development, with a focus on Southern countries. Its mission is to im-
pact development in the Global South through applied research and expertise in long-term
partnerships, thereby fostering transformational changes. To conform to this mandate
while also following societal responsibility and internal motivations to learn and improve,
CIRAD embarked on a process of institutionalising reflexivity on the contribution of research to societal impacts in 2010, fostering a debate on the role and responsibilities of research in society, and advancing research practices (Blundo-Canto et al., 2019; Hainzelin et al., 2017). This reflexive culture, which is described as a “culture of impact”, materialises in approaches and methods that support and facilitate these reflections on impacts at different levels (ex post, ex ante, in itinere). It also translates into the establishment of spaces for interactions around the role and impacts of research among researchers and partners, to stimulate learning, the sharing of experiences, knowledge, skills, visions, and ultimately the creation of a collective memory. This institutional ambition is unique for a research organisation, and it makes CIRAD an ideal case for exploring research practices that make it possible to capitalise on and share field experiences and (re)construct collective memories for the benefit of future interventions.

Over the course of the last decade, CIRAD has conducted in-depth, reflexive, and participatory retrospective analyses of innovation histories, investigating the cumulated “impacts” of sets of research interventions over long timeframes. An *ex post* evaluation method, called “*ImpresS* ex post”, has been developed (Barret et al., 2017) to enable collective reflection on the historical contribution of research, retrieving and developing memory outside the temporal and socio-territorial space of a particular intervention by reconstructing long-term innovation histories.<fnref n="4" /> This has allowed the institution to capitalise knowledge of the impacts of these innovation processes<fnref n="5" /> and understand their generation mechanisms (Faure et al., 2018; Blundo-Canto et al., 2020a). Thirteen *ex post* case studies were carried out between 2014 and 2016, and seven more were conducted in 2021, involving researchers in CIRAD and their partners in the Global South.

Based on the lessons from the *ex post* cases, CIRAD has developed an adaptive *ex ante*, or prospective, design approach, called “*ImpresS* ex ante” (Blundo-Canto et al., 2020b) that engages collective thinking among actors involved in a future research intervention on the changes and impacts that it would generate and the plausible pathways by which to achieve them. It includes an in-depth collective consideration of the present and historical context (for example, on social, economic, political, and environmental issues) and an analysis of the diverse current and past interventions that may influence the situation under consideration. The analysis of the “ecosystem” of the intervention focuses on who has already worked on the subject of interest, what actions have been implemented and how, what changes have been brought about and for whom, and what lessons have been drawn from these experiences. In addition, the analysis of current obstacles to and opportunities for change involve reflecting on the historical processes and dynamics of a place and the past and current stakeholder map that could have an influence on the change process.

<footnote n="4">⁴ ImpresS (“Impact from research in the South”) implements CIRAD’s vision to invest in examining and increasing the impact of its research activities in Southern countries. It is also the name given to the CIRAD team in charge of developing the culture of impact in CIRAD, that is to say supporting the reflection on these impacts, developing suitable methodologies, and accompanying intervention teams in their strategic planning using an “impact viewpoint”.</footnote>

<footnote n="5">⁵ The innovation process consists of a complex, multi-factorial, and uncertain system that includes technical, organisational, and institutional dimensions, and results from multi-actor and multi-factor interactions.</footnote>
Understanding how the past has shaped the current context and place while also recognis-
ing that a single intervention cannot claim to have a significant impact on its own is key to
this approach.

Both the *ex post* and *ex ante* approaches are internal processes carried out voluntarily by research teams. *Ex post* case studies are selected through an institution-wide tender. The evaluation team is composed of a methodological referee from the ImpresS team and one or two researchers with knowledge of the innovation process who have submitted a proposal for the *ex post* evaluation of a specific innovation process. The choice of internal evaluation reflects the reflective and learning objectives of the method. In the case of *ex ante* working cases, the objective is the construction of a new intervention rather than its evaluation, and it is the research team that wants to build it that leads the process, supported methodologically by the ImpresS team.

Method

We investigated institutional, group, and individual level memory and how the three con-
tribute, each in their own way, towards creating a common understanding of the influence
of interventions on a place, constructing collective memories, and informing the nature and
design of future interventions. We proceeded as follows. With regard to institutional level
memory, we focused on reconstructing the historical path at the origin of the ImpresS
*ex post* and *ex ante* institutionalised evaluation approaches and the resulting institutional
learning. Through a documentary analysis that included strategic, institutional, and scien-
tific documentation, we captured institutional and individual achievements that translate
into the emergence of an institutional memory following the mobilisation of these ap-
proaches and the capitalisation of the resulting new knowledge and evidence.

As regards to group-level memory, we analysed how the design and implementation of CIRAD's *ex post* evaluation method and *ex ante* evaluation approach foster the reconstruc- tion and creation of collective memories (from the past or towards the future) and contribute towards improving research interventions. To achieve this, we analysed peer- reviewed and grey literature, especially works that describe these approaches or are ass- ciated with the valorisation of findings from the application of these approaches.

Third, we analysed the effect of applying these evaluation approaches on individual mem-
ories. Using an online survey aimed at agents of CIRAD who participated in their
development and/or application, we captured the different types of individual changes
(cognitive and behavioural changes) implemented as a result of this application (that is,
what people retain at a personal level). The objective of the survey was to understand the
nature and the level of changes that the CIRAD researchers who mobilised the *ex ante*
and/or *ex post* approaches experienced, from changes of vision and perception, to changes
in actual research practices.
# Results

## Construction of an institutional collective memory

Institutional memory is often the result of the capitalisation of different types of knowledge and assets, some tangible and others intangible (such as agents' "know-how"), and contributes to the intellectual capital of an institution (PAHO, 2015). Various types of "institutional memory" can build within a same organisation alongside the various topics, ambitions, and objectives that drive it. Indeed, this kind of memory has the potential to shape future decisions of the organisation (Hardt, 2018) since to some extent, it represents the image of the institution and translates into the organisational culture, with which employees identify, and which influences their working practices.

Institutional memory can build across a variety of modalities. Procedures, protocols, and guides enable the institution to organise and preserve its array of knowledge and experiences (PAHO, 2015). A preliminary step in these institutional developments was the initiation of collective reflections and the establishment of spaces of exchange in order to facilitate team-building, the construction of shared ambitions, balancing the means of achieving it, and – inherent in this process – organisational learning. CIRAD has strongly focused on the development of approaches, tools, and methodologies to facilitate impact reflections and learning at different levels of the institution (see Figure 1 below for an overview of CIRAD's pathway). Second, disseminating institutional information is also a part of the process of building an institutional memory (PAHO, 2015), for example through the design of directions and the communication of institutional ambitions. CIRAD defines its strategy for implementing its mission in its "Strategic Vision". This is a general document that is created every ten years and updated every five years in a participatory manner that involves scientific partners and advisors. It is then approved by the Board, and represents the ultimate reference point for any CIRAD research programming. The 2008, 2012, and 2018 strategic visions – and the detailed programming documents deriving from them (the "OSSP"<fnref n="6" />) – reveal the growing importance attached to the contribution of research to impact. In the 2012 version, one of CIRAD's four ambitions was explicitly stated to be the development of the culture of impact in the institution, in particular through the ImpresS endeavour. This ambition feeds CIRAD's entire metabolism through the "contrat d'objectifs"<fnref n="7" />, the way research projects are built by research units, incentive credits, its training policy, and specific human resource actions. More recently, in 2018, this led to the creation of the DIMS, the "direction of impact and science marketing", which is responsible for implementing the institution's policy as regards to how research projects are built with partners. This overall dynamic can also be seen in the reports from the international visiting

<footnote n="6">⁶ The initials OSSP stand for *Objectifs de stratégie scientifique et partenariale* (Scientific strategy and partnership objectives).</footnote>

<footnote n="7">⁷ The "contrat d'objectifs" is the official way through which French ministries approve CIRAD's strategy and priority actions every five years and commit to providing public resources for its mission.</footnote>
committees commissioned by HCERES<fnref n="8" />: the 2021 report describes the culture of impact as one of CIRAD's most original and positive traits (HCERES, 2021).

PAHO (2015) defines a third way of building an institutional memory, by increasing institutional intellectual capital, which can be achieved through continuous learning processes and knowledge management. The typical arenas for capturing, disseminating, and sharing knowledge and experiences include seminars, conferences, and informal talks. Yet, these processes are diffused and unformalised. Collecting and analysing knowledge and the related evidence on research impacts may be a more efficient way of capitalising it and increasing its usability and accessibility for others. CIRAD has conducted *ex post* evaluation studies that take a retrospective look at long innovation trajectories, collecting information, stimulating people's memories, and reuniting them in a cohesive manner in order to better understand the influences of research interventions on a place and its population. The systematisation and cross-analysis of this data enabled the institution to make the ways in which its activities generate societal impacts explicit (Blundo-Canto *et al.*, 2018), thus supporting institutional learning and the systematisation of an institutional memory. It also provided the basis for a formalised web-based interface that enables this knowledge to be managed. Along similar lines, CIRAD has recently invested in the design and uptake of monitoring and evaluation systems for planned interventions. This new practice will make it possible to keep track of the dynamics of an intervention and its influences on a place.

The development of these various modalities takes time: building an institutional memory is a long-term process, and is the result of both oral and written narratives (Linde, 1999). CIRAD's pathway is made up of various types of action, achievement, and engagement activity that contribute to the construction of this memory (see Figure 1), from the first reflections on the impact with various internal and external audiences to the production and testing of new approaches that foster an impact-focused reflection of CIRAD's interventions, to building an internal assessment capacity, and to the capitalisation of new knowledge in scientific materials and methodological guides, as well as dissemination and communication materials (such as videos).

These tools and resources to support the maintenance of a constructed institutional memory and offer the chance to build upon it are key to ensuring that the memory is spread more widely over time, and that the concepts and principles are also transferred to new employees. While long-term careers in CIRAD increase the chances of this memory to diffuse and circulate within the institution, the mechanisms to facilitate the vehiculation of principles and postures across agents and their appropriation exist, including through mentoring of newcomers by senior agents. Among other things, the formalisation of accompanying intervention planning teams within various DIMS agents' missions in their reflection on the impacts of their intervention, together with the ImpresS team is another such mechanism. The ImpresS team, which was created at the start of the institution's trajectory (see Figure 1), is a hybrid group that conducts research on impact evaluation,

<footnote n="8">⁸ HCERES is the “Haut conseil d’évaluation de la recherche et de l’enseignement supérieur”. It organises an evaluation of all research institutions and universities in France every five years.</footnote>
capitalising this knowledge and developing or improving impact evaluation and impact-oriented strategic planning methods, as well as assisting with their application and uptake by research teams, supporting the development of the “culture of impact” within CIRAD.

In short, the construction and preservation of an institutional memory related to the impact of interventions on a place is mainly the result of people's engagement, learning processes, and an institutional desire and investment in understanding, equipping, and disseminating knowledge on the influence of research activities.

**The construction of group-level collective memories through *ex post* and *ex ante* evaluations**

The purpose of CIRAD's ImpresS *ex post* evaluations is to build a shared perception among the actors involved in the evaluation of the contribution of research to impacts across a long-term innovation trajectory and the mechanisms that enabled these contributions (Barret *et al.*, 2017). In order to achieve this shared perception, the method applies participatory processes with actors who have been involved in the innovation trajectory (Faure *et al.*, 2018). The memory of a place that is linked to or impacted by research is explored in different ways. First, actors who have been key contributors to the research under consideration reconstruct the narrative of the innovation process. This includes identifying the set of research interventions that contributed to or structured the innovation, the main phases of the innovation process, the various roles played by stakeholders and institutions, and the contextual factors, controversies, and events that affected it. Next, primary data are gathered on the experiences of change among the actors influenced by the innovation. Through the stimulation of personal memories, this includes a characterisation of changes in practices, behaviour, and interactions with others, and their long-term consequences or impacts. Finally, the method focuses on recalling how the research activities influenced these changes and impacts. This third means of exploring memory might be referred to as the “memory of mechanisms”, as it refers to why and how changes have taken place. This is revealed by the collective reconstruction of an impact pathway of the innovation process that maps the causal chain linking the inputs from the research, the research outputs, the changes generated by the use and transformation of these outputs, and the impacts.

The collective and individual recall of how research interventions have made a difference, while accounting for other actors and factors, makes it possible to focus on the impacts that matter most according to the actors engaged (Blundo-Canto *et al.*, 2020a), while also identifying any unintended, indirect, and negative impacts the intervention may have induced (Temple *et al.*, 2018), as well as unexpected interactions and key turning points (Faure *et al.*, 2020).

The ImpresS *ex ante* approach (Blundo-Canto *et al.*, 2020b) is a prospective evaluation approach on the potential contributions of research to impacts, which aims to foster change. It has to do with building a shared vision of a future intervention and its targeted impacts. This allows for strategic planning of the intervention and serves as a reference point as the intervention develops, and therefore as a collective memory. In this process, the key contributors formulate a common vision of the future (the ideal situation after 10
to 15 years), working out the issues to be addressed to achieve it, identifying actors that are protagonists in and/or impacted by those issues, and identifying potential strategies to overcome them or take up opportunities. Following this collective diagnosis, the group builds an “outcome map” that features the changes in knowledge, capacities, practices, behaviour, and interactions targeted by the intervention in specific groups of actors, the obstacles to these changes, and possible actions to overcome them, that can be translated into an intervention architecture or logic. This process is participatory in the sense that the key actors in the intervention come together and exchange individual or institutional views and visions in order to ultimately elucidate how they think change is going to happen and build a collective vision. This vision is inevitably based on a shared “memory” of individual past experiences in the targeted context and on a common understanding of the targeted issues and collective strategic choices for addressing them.<fnref n="9" /> The underlying assumption is that creating a collective memory such as this at the beginning of an intervention enhances the collective governance of the intervention, facilitates change-oriented monitoring and evaluation in support of a “regular memory building process”, and enables the team to pursue the vision beyond the scope of the intervention. As an example, ImpresS ex ante was applied in the framework of co-constructing a One Health project, which involves a heterogeneous group of partners with stakeholders from the agriculture and health sectors in West Africa and South East Asia. The intended impact of this project is to understand, validate, and promote the characteristics of a healthy territory and to accompany the implementation of changes in practice and interactions among actors in the context of an agro-ecology transition. The application of ImpresS ex ante helped to qualify the health status of the territory with the stakeholders, build a shared vision, and co-design indicators to monitor changes. It made it possible to determine the intended major systemic changes with all the partners, and presented a unique occasion to bring together stakeholders who were not used to collaborating on a regular basis (Binot, 2020).

ImpresS ex ante was applied in about 40 CIRAD-led interventions, 7 CIRAD value chains strategies, and 4 research partnership platforms between 2017 and 2021. Across these cases, the level of support provided by the ImpresS team varied from a few meetings and training sessions to meetings and workshops over several months.

The variety of applications of the ImpresS ex ante approach in diverse contexts and groups that are themselves characterised by various interests and backgrounds translates into a multiplicity of collective memories. The richness of the memories it helps stimulate, the sharing of various individual and common experiences, and the building of a common basis and vision that is intended to be nourished over time all contribute to a living collective memory.

<footnote n="9">⁹ The set of actors who become involved in the process will naturally shape the nature of the emerging collective memory.</footnote>
<figure><img src="image_7.png" /><figcaption>Figure 1: The path of CIRAD to building an institutional memory on the contributions of research to societal impacts</figcaption></figure>

Strategic positioning of CIRAD's ambition to establish an institutional memory along research impact reflections:

- OSSP = Objectifs de stratégie scientifique et partenaire = scientific strategy and partnership objectives

- DIMS = creation of a unit dedicated to follow and accompany project development, including reflection on the impacts of CIRAD's planned interventions
## The effect of institutional and group-level memories on individual practices

Through these processes, members of research-for-development institutions and especially users of reflexive approaches like ImpresS *ex post* and ImpresS *ex ante* are invited to revive their personal experiences of an intervention and to nurture collective memories with their own knowledge of the influences of past interventions in a particular place. Little is known, however, of what people retain from the mobilisation of these approaches and more generally from their contact with an institution that promotes reflexivity on impact from research and the sharing of experiences and practices along these lines. The survey targeted agents of CIRAD and some of their partners who participated in the development and/or application of ImpresS *ex post* and *ex ante* and partners who participated in *ex ante* training (their contact information was available in the lists of trainees). This represents a total of 327 people.

The survey sought to capture what ‘memory’ people retain at their own personal level, and how this materialises in cognitive and behavioural changes. The survey was structured as follows, in order to capture: 1) the perceptions of the institution’s ambition to develop a culture of impact culture and the trajectory; 2) the changes experienced by the respondents and the institutional changes as they perceived them; and 3) the constraints and obstacles to the development of a culture of impact. In total, we obtained 65 responses. Most of the participants (88%) had been involved with *ex ante*, and 12% only applied the *ex post* method. The participants’ level of engagement with the evaluation approaches varied.

Looking first at the cognitive changes, we find that 52% of respondents reported a change in their perception of the role and mission of CIRAD and the role of research in development, stating that they had acquired a better understanding of them. 25% reported a change in their perception of their role within CIRAD as a result of participating in these activities. Significantly, 85% stated that they had acquired new knowledge about impact, impact evaluation, and the conditions of application of the ImpresS approaches. A substantial portion of the participants also improved their capacities and skills: 35% stated that they were able to apply the ImpresS approaches autonomously, and half of the project development team (DIMS) members reported developing new skills associated with supporting intervention planning. This type of increase in capacity is conducive to diffusing reflexive postures across the institution and therefore facilitating others who nurture collective memories.

There is emerging evidence that the application of these evaluation approaches successfully stimulates a rethinking of interventions being planned and how to steer them. Indeed, a third of the respondents have modified the orientation and planning of their interventions, and 46% stated that it had positively influenced their ability to interact with actors who are potentially impacted by the intervention. In addition, the process itself enables a better understanding of the role and legitimacy of every actor as part of the intervention and of the importance of constructing a common vision among partners. Finally, 40% of the respondents also considered that applying these approaches influenced the formulation of the research questions they addressed in the intervention. However, the survey did not investigate whether these changes applied to interventions other than those that were
subject to the application of these approaches, although 40% believed that their modalities
of interaction had changed within research teams and among the different disciplines and
professions in CIRAD. This result makes it possible to facilitate the sharing of experiences
and lessons, and therefore the idea of reinforcing the construction of collective memories
and making them usable by a large number of people.

If we look at people's perception of the institution's desire to promote new principles associated with the impact of research, we find that 45% perceived a level of change in the institution's top management, but not in the institution's communications strategy. This is a challenge in the process of building institutional memory that needs to be addressed.

Overall, we found various types of changes that were adopted as a result of people's ex-
posure to these collective-memory building processes. Those that were reported mainly
pertained to changes of vision, perception, knowledge, interactions, and capacities.
Changes in actual research practices emerge as the interventions that experienced the
*ex ante* approach two or three years ago are just beginning. The results of the *ex post* eval-
uations show that changes take time to materialise, pointing to the need to strengthen the
capacity-building of multiple actors, as well as co-construction and increased engagement
with public policy actors. In short, we observed that a learning process and a dynamic ori-
ented towards the adoption of new research practices is in place, but is still at its embryonic
stage. The modification of habits, norms, mindsets, and values requires time. Nonetheless,
the conditions for these changes in practice to happen – the application of new knowledge,
capacities, visions and interactions – seem to be firmly in place. Moreover, the appropria-
tion of the reflexive, iterative, and participatory stance these evaluation approaches involve
is not easily achieved. For instance, some participants reported difficulties with setting up
participatory processes, and a certain level of resistance to interacting with a broader or
different set of actors. This is why the institution is increasingly equipping its staff by invest-
ing in capacity-building in facilitation, project management, and partnership governance. At
the same time, a community of practice between researchers and practitioners, including
research support staff, was created in 2021 to provide a space for exchanges and reflections
on how to encourage positive collaborative processes that aim to foster practice change for
multiple actors, including researchers.

Discussion

Our study shows how collective memory and the processes associated with its construc-
tion and mobilisation can take multiple forms and involve various levels of an institution.
Across the three levels of collective memory that we explored, we observed that building
collective memories is a long-term, and often complex, process that needs to be adequately
equipped and supported. The permanence of these memories is never acquired: it is an
ever-evolving process that relies on how the various groups continue to mobilising and nur-
turing it over time, embedding new values, beliefs, and norms. Our findings highlight three
crucial elements for an effective memory construction process that informs and orients
strategic planning and the implementation of new interventions.
First, the *participation of a diversity of actors* (researchers and non-researchers) appears to be key for the construction of collective memories. It enables a sharing of diverse opinions and the formulation of more systemic and plausible interventions, thus favouring the building of a *comprehensive memory*. At an institutional level, this comprehensiveness cannot be achieved without a strong commitment from its members to helping bring a diversity of views, concerns, and considerations to the table, as it is clearly reflected in the nature and scale of the various activities that make up CIRAD's institutional trajectory of memory-building. The application of retrospective evaluations also leads to the building of a collective memory that aspires to be as comprehensive as possible for the innovation process being studied. This is achieved by retrieving the individual memories of a multiplicity of actors who have been involved in the innovation trajectory in various ways and at various moments, reconstructing the innovation process, identifying the drivers and contextual events that shaped the innovation, and capturing the variety of impacts based on the experiences and perceptions of these actors. The accuracy of this memory depends mainly on an analysis of the plausibility and robustness of the impacts identified. In the method applied by CIRAD, which relies on the principle of data triangulation, this increases the credibility of the results (Plottu and Plottu, 2009). All views, observations, and experiences and their associated narratives are therefore of use to the evaluation, even if they are contradictory, and they are ultimately validated by a participatory interpretation in order to ensure that the combination of viewpoints leads to a coherent memory of the history of the innovation (Blundo-Canto et al., 2020a). However – and as is the case in any participatory process – the power dynamics and potential asymmetries within a group need to be considered in advance of the process as they may affect the weight given to specific experiences. In both ImpresS *ex ante* and *ex post*, power asymmetries are assessed on a case-by-case basis in order to address them and have meaningful participatory processes: individual interviews, focus groups, and facilitation dynamics enable every voice to be heard, and the role of the facilitator is therefore crucial. The participatory process also compensates for a lack of memory systematisation in previous interventions and a shortage of reliable secondary data (Faure et al., 2020) due to imperfect knowledge management and a lack of resources for monitoring and maintenance. Just as the participation of actors is key to building a comprehensive collective memory of the past, it is also crucial for building a collective “present” memory that will help to shape and orient the future. The systemic ambitions of the ImpresS *ex ante* approach therefore assist the definition of a relevant scope in the design of new interventions, and disclose the potential for the appropriation of the results of an intervention, leading to changes in visions, perspectives, practices, and behaviours. The two evaluation approaches therefore rely on bringing together a mix of diverse actors, and reuniting different memories and experience, that for some of them are seldom documented or heard. The added value of this collective work lies in the encouragement it offers to the sharing of views and the fostering of transparency around understanding contextual issues and their connections to previous interventions. These two
processes make it possible to reach a shared vision that incorporates understanding the past in a constructive and valuable way.<fnref n="10" />

Reflexivity is another key criterion in the building and preservation of collective memory, and is closely associated with learning processes and feeding this learning into the memory. Indeed, if it is to persist over time, the institutional collective memory associated with capturing and understanding the impact of research must rely on constantly questioning the balance and the causality links between research issues and activities and the impact or developmental ambitions. On the other hand, if this institutional memory is to remain usable and useful for its members, a continuous updating process is required that principally depends on the new learning, knowledge, and skills acquired by the members. The two evaluation approaches both contribute to a learning process at an individual and group level. ImpresS ex post allows a mutual understanding and learning of the various experiences of change and constructive exchanges to promote the creation of a shared understanding of the past. It also enables learning of the mechanisms that enable the change. Prospectively, ImpresS ex ante makes it possible to build a shared vision of future changes with a critical eye informed by past interventions and their (cumulative) impacts. It achieves this by integrating the collective history and memory of the place(s) of interest through participation in its conceptual design by diverse actors. This fosters a reflexive posture and a learning process among the participants about the current situation and how certain factors might influence the targeted future changes. In both cases, as demonstrated by the effects on individual practices, the approaches can also serve as an enabling and empowering tool by which people gain knowledge, skills, and competences.

Third, the *iterative* feature of the processes enables a continuous feedback loop process that contributes to *shaping* and *adjusting* collective memories over time. The practice of building or mobilising collective memory is in itself a dynamic process that is enriched by its use over the years. In the case of ImpresS *ex post* and ImpresS *ex ante*, feedback between the various phases of their application favours the comprehensiveness, revision, and re-orientation of the collective memory that is being constructed. The establishment of monitoring and evaluation systems as part of the intervention-building process may enable the support of continuity in the collective reflection over time as the intervention becomes more refined and is implemented, and therefore the re-use of the collective memory by the people involved in its construction and/or by others.

Looking in more detail at the individual level, which forms the basis of these memories at a group and institutional level, we find that people retain a multiplicity of elements. The individual level lies at the heart of all of this because it is the condition precedent for the construction of collective memories (which are necessarily formed from individual

<footnote n="10">¹⁰ The ImpresS *ex ante* process may seem to be more participatory than the *ex post* process. This is probably because people involved in the former project themselves into potential impacts, outcomes, etc., while in the case of *ex post*, participation is conducted at specific stages of the method. However, the application of the *ex ante* approach depends very much on the demands of the group and its motivations, while *ex post* has "non-negotiable" participatory phases. Thus, the "importance of participation" may vary, depending on the extent to which the *ex ante* is implemented.</footnote>
memories and experiences), and is the element on which collective memories must rely in order to be developed further and disseminated. Therefore, what people retain in their practices is fundamental for shaping the future of collective memories and improving research practices.

We would like to note here that other reflexive initiatives are also being studied at CIRAD alongside the two institutionalised evaluation approaches presented in this study, also contributing to shape collective memories. One example of these is a study of the *in itinere* dynamics that foster a continuous co-construction process among partners involved in the course of an intervention. Communities of practice are one means of facilitating such a process, where learning and experimenting as a collective is conceived as a process of the co-creation of meaning as the research process develops (for example, from the co-construction of questions to data collection protocols and their analysis) (Louafi *et al.*, 2021). When added to reflexivity on the conditions of collaboration, this type of approach enables the dynamic building of a collective memory over time. The *in itinere* process allows for a “live” orienting and reshaping of the collective memory through regular interactions among the major actors in the process and the continuous sharing and integration of emerging visions and ideas.<fnref n="11" /> It seeks to take the evolving complexity of the given social-ecological system into consideration in order to allow a comprehensive understanding of how the memory is built over time.

Conclusion

In this study, we investigated institutional, group, and individual level memory and how it
contributes towards the creation of a shared understanding of the influence of interven-
tions on a place, towards building collective memories, and towards informing the nature
and design of future interventions. Our findings are unique in the sense that they examine
various avenues of memory-building within a research-for-development institution, and
how, considering their inter-connections, they can benefit working practices and interven-
tion strategic planning. In particular, we presented a set of two approaches that facilitate
the reconstruction of the memory of the past (ex post) and the planning of new interven-
tions that integrate this memory in order to construct a vision of the future (ex ante). Each
approach includes reflexive, participatory, actor-centred, and iterative principles, which are

<footnote n="11">¹¹ As an example of an *in itinere* approach, the CoEx project has the objective of proposing innovative governance mechanisms that better account for the diversity of crop diversity management practices (Thomas and Louafi, 2017). It is achieved through a multi-stakeholder partnership (involving research, end-users, and *ex situ* collection managers); a multi-level approach to co-developing and experimenting with governance mechanisms that allow the coexistence of different practices; and a longitudinal monitoring of patterns in crop diversity, crop use, and seed access and exchange. The main ambition is to promote multi-scale, inter-and trans-disciplinary reflexive approaches that embed crop diversity within its wider social-ecological resource system in order to foster a joint, systemic view of the system and the interactions between its actors, resources, and institutions, and to enable the co-evolution of the various components. The *in itinere* application offers another level of learning by going beyond the “co-construction” level and allowing a posture of collective experimentation whereby knowledge production is considered in the context of its social and active role, with an immediate link to the practice and its possible use (Popa *et al.*, 2015; Louafi *et al.*, 2021).</footnote>
recognised as being key to building the collective memory. We find that the process can be
carried out for different purposes, including learning and fostering institutional change,
building evidence, and shaping the future.

Our findings seem to suggest that the construction and existence of collective memories
in an institution have an added value for present and future interventions. However, this
requires a system that enables intervention-specific knowledge and individual knowledge
to be adequately stored and accessed. This echoes issues that are commonly encountered
in the research-for-development world: deficits in monitoring and evaluation, knowledge
management, and systematisation, which among other things often mean that a person
leaving the system also leaves with part of its memory. The adoption of reflexive practices
at an organisational level and the building of collective memories therefore require re-
sources and effort (Omotayo, 2015). Adapting project-management practices along these
lines (for example by encouraging reflexive postures and the exchange of knowledge about
past interventions) may help the integration and operationalisation of collective memory at
a management level. It is hard to transform practices, however, as this often implies
transcending different perspectives and interests (Bresnen, 2016). Furthermore, despite
the clear value of using reflexive approaches in the design of new interventions, these "re-
constructed memories" come with their own questions about the permanence of lessons
from the past, the effects on partnerships following an involvement in such approaches,
and potential fatigue on the part of partners in cases where the reconstruction of the
memory does not lead to actual changes.

A number of studies have emphasised the importance of capturing the lessons learned, but they have found that in most cases they are not utilised effectively (Trevino and Anantatmula, 2008). The approaches presented in this article enable the making of a “live” use of lessons learned, past knowledge, and experiences from previous interventions. However, the responsibility for documenting and retrieving lessons does not just fall on project managers. Our results show that all the actors involved in an intervention are part of this learning environment and therefore have a role to play in building these collective memories in a way that benefits the design of future interventions. There are other obstacles, however: the common challenges of participatory processes that would affect how memory is built and transmitted to others include dealing with power asymmetries. The methods presented here address these systematically by applying approaches and tools that allow multiple voices to be heard, including context analysis, the triangulation of data sources and collection methods, and iterative analysis and interpretation. Research institutions interested in developing a collective memory-building capacity must ultimately equip their members to address these challenges by embedding reflexive processes, capacity-building, and tools for meaningful collective construction in their routines.

## Acknowledgements

We want to thank Sélim Louafi and Frédéric Goulet for participation to discussions about
the development of this manuscript, and Étienne Hainzelin for his contribution and feed-
back on the paper.
Bibliography

Barret D., Blundo-Canto G., Dabat M.H., Devaux-Spatarakis A., Faure G., Hainzelin E., Mathé S., Temple L., Toillier A., Triomphe B., 2017, "Guide méthodologique ImpresS : évaluation ex post des impacts de la recherche agronomique dans les pays du Sud", Montpellier, CIRAD, 96 p.

Barrier B., 2018, "La science en projets : financements sur projet, autonomie professionnelle et formations du travail des chercheurs académiques", *Sociologie du travail*, n° 53(4):515-536.

Baussant M., 2007, "Penser les mémoires", *Ethnologie française*, n° 37, pp. 389-394.

Binot A., 2020, "Une approche de la santé des territoires dans un cadre 'One Health' – CIRAD-ASTRE", Comité de Programme du Département BIOS, 10 September 2020, Montpellier.

Blundo-Canto G., Devaux-Spatarakis A., Martin P., de Romémont A., D'Agostino B., Mouret N., 2018, "European Evaluation Society Biennial Conference: Evaluation for more resilient societies", n° 13, Thessaloniki, Grèce, 1-5 Mai 2018, 1 p.

Blundo-Canto G., Triomphe B., Faure G., Barret D., de Romémont A., Hainzelin E., 2019, "Building a culture of impact in an international agricultural research organization: Process and reflective learning", *Research Evaluation*, n° 28(2), pp. 136-144.

Blundo-Canto G., Devaux-Spatarakis A., Mathé S., Faure G., Cerdan C., 2020a, "Using a participatory theory driven evaluation approach to identify causal mechanisms in innovation processes", *New Directions for Evaluation*, n° 167, pp. 59-72

Blundo-Canto G., de Romémont A., Hainzelin É., Faure G., Monier C., Triomphe B., Barret D., Vall É., 2020b, "ImpresS ex ante. An approach for building ex ante impact pathways in development-oriented research. ImpresS ex ante methodological guide (Second version)", Montpellier, CIRAD, 74 p.

Bresnen M., 2016, "Institutional development, divergence and change in the discipline of project management", *International Journal of Project Management*, n° 34, pp. 328-338.

Cooper S., 2014, "Transformative evaluation: organisational learning through participative practice", *The Learning Organization*, n° 21(2), pp. 146-157.

Covo G., 2014, "L'évolution du fonctionnement de la recherche vers un mode projet et ses conséquences sur l'Université: le regard de celle qui accompagne les chercheurs au quotidien", *Sciences de la société*, n° 93, pp. 40-55.

Faure G., Barret D., Blundo-Canto G., Dabat M.H., Devaux-Spatarakis A., Le Guerroué J.L., Marquié C., Mathé S., Temple L., Toillier A., Triomphe B., Hainzelin E., 2018, "How different agricultural research models contribute to impacts: Evidence from 13 case studies in developing countries", *Agricultural Systems*, n° 165, pp. 128-136.

Faure G., Blundo-Canto G., Dabat M.H., Devaux-Spatarakis A., Le Guerroué J.L., Mathé S., Temple L., Toillier A., Triomphe B., Hainzelin E., 2020, "A participatory method to assess the contribution of agricultural research to societal changes in developing countries", *Research Evaluation*, n° 29(2), pp. 158-170.

Fouéré M.-A., 2010, "La mémoire au prisme du politique", *Cahiers d'études africaines*, n° 197, pp. 5-24.

Halbwachs M., [1925] 1952, *Les cadres sociaux de la mémoire*, Paris, Les Presses universitaires de France.
Hainzelin E., Barret D., Faure G., Dabat M.-H., Triomphe B., 2017, "Agricultural research in the Global South: Steering research beyond impact promises", *Perspective*, n° 42, pp. 1-4.

Hardt H., 2018, "Who matters for memory: Sources of institutional memory in international organization crisis management", *Review of International Organizations*, n° 13, pp. 457-482.

HCERES, 2021, « Rapport d'évaluation du Centre de coopération internationale en recherche agronomique pour le développement (CIRAD) », Haut Conseil de l'évaluation de la recherche et de l'enseignement supérieur, July 2021, 42 p., https://www.hceres.fr/sites/default/files/media/downloads/hceres_rapport_cirad.pdf (accessed 20 May 2022).

Hubert M., Louvel S., 2012, "Le financement sur projet : quelles conséquences sur le travail des chercheurs ?", *Mouvements*, n° 71(3), pp. 13-24.

Joly P.-B., Matt M., 2017, "Towards a new generation of research impact assessment approaches", *The Journal of Technology Transfer*, pp. 1-11.

Leeuwis C., Klerkx L., Schut M., 2018, "Reforming the research policy and impact culture in the CGIAR: integrating science and systemic capacity development", *Global Food Security*, n° 16, pp. 17-21.

Linde C., 1999, "The Transformation of Narrative Syntax into Institutional Memory", *Narrative Inquiry*, n° 9(1), pp. 139-174.

Louafi S., Thomas M., Jankowski F., Leclerc C., Guichardaz A., Leclercq L., Baufumé S., Barnaud A., Rey A., Labeyrie V., 2021, "Collaboration in Crop Diversity Management: community of practice and responsible data production and linkages", working paper, CIRAD.

Omotayo F.O., 2015, "Knowledge Management as an important tool in Organisational Management: A Review of Literature", *Library Philosophy and Practice* (e-journal), https://digitalcommons.unl.edu/libphilprac/1238/. (accessed 19 October 2021).

Oteros-Rozas E., Martín-López B., Daw T.M., Bohensky E.R., Butler J.R.A., Hill R., Martin-Ortega J., Quinlan A., Ravera F., Ruiz-Mallén I., Thyresson M., Mistry J., Palomo I., Peterson G.D., Plieninger T., Waylen K.A., Beach D.M., Bohnet I.C., Hamann M., Hanspach J., Hubacek K., Lavorel S., Vilardy S.P., 2015, "Participatory Scenario Planning in Place-based Social-ecological Research: Insights and Experiences from 23 Case Studies", *Ecology and Society*, n° 20(4).

PAHO (Pan American Health Organization), 2015, "How to organize and preserve the institutional memory", 9 p., https://iris.paho.org/bitstream/handle/10665.2/35001/Institmemory2015_eng.pdf?sequence=1&isAllowed=y (accessed 19 October 2021).

Plottu B., Plottu E., 2009, "Contraintes et vertus de l'évaluation participative", *Revue française de gestion*, n° 192, pp. 31-58.

Popa F., Guillermin M., Dedeurwaerdere T., 2015, "A pragmatist approach to transdisciplinarity in sustainability research: From complex systems theory to reflexive science", *Futures*, n° 65, pp. 45-56.

Reed M.S., Ferré M., Martin-Ortega J., Blanche R., Dallimer M., Holden J., 2021, "Evidencing impact from research: a typology of methods", *Research Policy*, n° 50(4).

Samset K., Holden G.H., 2016, "Front-end definition of projects: Ten paradoxes and some reflections regarding project management and project governance", *International Journal of Project Management*, n° 34, pp. 297-313.
Schot J., Steinmueller W.E., 2018, “Three frames for innovation policy: R&D, systems of innovation and transformative change”, *Research Policy*, n° 47, pp. 1554-1567.

Schultz E., 2013, “Le temps d’un projet”, *Temporalités*, n° 18.

Temple L., Barret D., Blundo-Canto G., Dabat M-H., Devaux-Spatarakis A., Faure G., Hainzelin E., Mathé S., Toillier A., Triomphe B., 2018, “Assessing impacts of agricultural research for development: A systemic model focusing on outcomes”, *Research Evaluation*, n° 27, pp. 157-170.

Thomas M., Louafi S., 2017, “Adaptive governance for the coexistence of crop diversity management systems”, in *Crop diversification in a changing world: Mobilizing the green gold of plant genetic resources*, Montpellier, France, EUCARPIA Genetic Resources, 8-11 Mai 2017, 1 p.

Thornton P.K., Schuetz T., Förch W., Cramer L., Abreu D., Vermeulen S., Campbell B.M., 2017, “Responding to global change: a theory of change approach to making agricultural research for development outcome-based”, *Agricultural Systems*, n° 152, pp. 145-153.

Todorović M.L., Petrović D.C., Mihić M.M., Obradović V.L., Bushuyev S.D., 2015, “Project success analysis framework: A knowledge-based approach in project management”, *International Journal of Project Management*, n° 33(4), pp. 772-783.

Trevino S.A., Anantatmula V.S., 2008, “Capitalising from past projects: the value of lessons learned”, Paper presented at PMI® Research Conference: Defining the Future of Project Management, Warsaw, Poland, Newtown Square, PA, Project Management Institute.

Watts J., Horton D., Douthwaite B., La Rovere R., Thiele G., Prasad S., Staver C., 2008, “Transforming impact assessment: beginning the quiet revolution of institutional learning and change”, *Experimental Agriculture*, n° 44(1), pp. 21-35.

Williams T., 2008, “How do organizations learn lessons from projects – And Do They?”, *IEEE Transactions on Engineering Management*, n° 55(2), pp. 248-266.