# Indicator framework for national extension and advisory service systems. Metrics for performance and outcome measurement

V. Rasheed Sulaiman, Delgermaa Degi Chuluunbaatar, Patrice Djamen Nana, Christian Grovermann, Aiden Holley

► To cite this version:

V. Rasheed Sulaiman, Delgermaa Degi Chuluunbaatar, Patrice Djamen Nana, Christian Grovermann, Aiden Holley. Indicator framework for national extension and advisory service systems. Metrics for performance and outcome measurement. FAO, 56 p., 2022, 978-92-5-135677-7. 10.4060/cb8409en. hal-05179855

HAL Id: hal-05179855

https://hal.science/hal-05179855v1

Submitted on 23 Jul 2025

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_1.png" /></figure>

Distributed under a Creative Commons Attribution 4.0 International License
# Indicator framework for national extension and advisory service systems

## Metrics for performance and outcome measurement
<figure><img src="image_2.png" /></figure>
Indicator framework for
national extension and
advisory service systems

Metrics for performance and
outcome measurement

Prepared by

Rasheed Sulaiman V

The Centre for Research on Innovation and Science Policy

Delgermaa Chuluunbaatar

Food and Agriculture Organization of the United Nations

Patrice Djamen

French Agricultural Research Centre for International Development

Christian Grovermann

The Research Institute of Organic Agriculture

Aiden Holley

Food and Agriculture Organization of the United Nations

Food and Agriculture Organization of the United Nations
Rome, 2022
Required citation:

Sulaiman V, R., Chuluunbaatar, D., Djamen, P., Grovermann, C. and Holley, A. 2022. *Indicator framework for national extension and advisory service systems – Metrics for performance and outcome measurement.*
Rome, FAO. https://doi.org/10.4060/cb8409en

The designations employed and the presentation of material in this information product do not imply the expression of any opinion whatsoever on the part of the Food and Agriculture Organization of the United Nations (FAO) concerning the legal or development status of any country, territory, city or area or of its authorities, or concerning the delimitation of its frontiers or boundaries. The mention of specific companies or products of manufacturers, whether or not these have been patented, does not imply that these have been endorsed or recommended by FAO in preference to others of a similar nature that are not mentioned.

The views expressed in this information product are those of the author(s) and do not necessarily reflect the views or policies of FAO.

ISBN 978-92-5-135677-7

© FAO, 2022

<figure><img src="image_3.png" /></figure>

Some rights reserved. This work is made available under the Creative Commons Attribution-NonCommercial-ShareAlike 3.0 IGO licence (CC BY-NC-SA 3.0 IGO; https://creativecommons.org/licenses/by-nc-sa/3.0/igo/legalcode).

Under the terms of this licence, this work may be copied, redistributed and adapted for non-commercial purposes, provided that the work is appropriately cited. In any use of this work, there should be no suggestion that FAO endorses any specific organization, products or services. The use of the FAO logo is not permitted. If the work is adapted, then it must be licensed under the same or equivalent Creative Commons licence. If a translation of this work is created, it must include the following disclaimer along with the required citation: "This translation was not created by the Food and Agriculture Organization of the United Nations (FAO). FAO is not responsible for the content or accuracy of this translation. The original [Language] edition shall be the authoritative edition."

Disputes arising under the licence that cannot be settled amicably will be resolved by mediation and arbitration as described in Article 8 of the licence except as otherwise provided herein. The applicable mediation rules will be the mediation rules of the World Intellectual Property Organization http://www.wipo.int/amc/en/mediation/rules and any arbitration will be conducted in accordance with the Arbitration Rules of the United Nations Commission on International Trade Law (UNCITRAL).

**Third-party materials.** Users wishing to reuse material from this work that is attributed to a third party, such as tables, figures or images, are responsible for determining whether permission is needed for that reuse and for obtaining permission from the copyright holder. The risk of claims resulting from infringement of any third-party-owned component in the work rests solely with the user.

**Sales, rights and licensing.** FAO information products are available on the FAO website (www.fao.org/publications) and can be purchased through publications-sales@fao.org. Requests for commercial use should be submitted via: www.fao.org/contact-us/licence-request. Queries regarding rights and licensing should be submitted to: copyright@fao.org.
# Contents

<table><tr><td>Foreword</td><td>v</td></tr><tr><td>Acknowledgements</td><td>vi</td></tr><tr><td>Abbreviations and acronyms</td><td>vii</td></tr><tr><td>1 Introduction</td><td>1</td></tr><tr><td>2 Extension and advisory services (EAS) assessment frameworks: a critical review</td><td>5</td></tr><tr><td>3 Framework logic</td><td>9</td></tr><tr><td>A. Pluralism in EAS provision</td><td>9</td></tr><tr><td>B. EAS indicator framework characteristics</td><td>11</td></tr><tr><td>C. Exploring the performance of the EAS system</td><td>13</td></tr><tr><td>4 Theory of change</td><td>15</td></tr><tr><td>A. Impacts</td><td>16</td></tr><tr><td>B. Outcomes</td><td>16</td></tr><tr><td>C. Outputs</td><td>17</td></tr><tr><td>D. Activities</td><td>17</td></tr><tr><td>E. Inputs</td><td>18</td></tr><tr><td>5 Assumptions</td><td>19</td></tr><tr><td>6 Indicators</td><td>21</td></tr><tr><td>A. Core indicators</td><td>26</td></tr><tr><td>B. Data sources</td><td>28</td></tr><tr><td>C. Data collection experiences</td><td>29</td></tr><tr><td>7 Analytical considerations</td><td>31</td></tr><tr><td>A. Monitoring, evaluation and learning</td><td>32</td></tr><tr><td>B. Impact evaluation</td><td>34</td></tr><tr><td>8 Conclusions</td><td>37</td></tr><tr><td>References</td><td>39</td></tr><tr><td>Glossary of terms</td><td>43</td></tr></table>
Table

<table><tr><td>Table 1. Indicator framework: list of core and complementary indicators</td><td>22</td></tr></table>

Figures

<table><tr><td>Figure 1. Main categories of actors in an integrated EAS system</td><td>10</td></tr></table>

<table><tr><td>Figure 2. EAS indicator framework design</td><td>12</td></tr></table>

<table><tr><td>Figure 3. EAS indicator framework design: instruments for data collection</td><td>12</td></tr></table>

<table><tr><td>Figure 4. Theory of change of agricultural extension and advisory services</td><td>15</td></tr></table>

Boxes

<table><tr><td>Box 1. Experience with data collection by indicator level</td><td>30</td></tr></table>

<table><tr><td>Box 2. Definitions of MEL</td><td>33</td></tr></table>

<table><tr><td>Box 3. Definitions and resources on impact evaluation</td><td>36</td></tr></table>
Foreword

Agricultural innovation is key to addressing many of the complex and interrelated challenges hindering sustainable development today, such as rural poverty, hunger and climate change. Well-functioning, demand-driven and pluralistic extension and advisory services (EAS) play an essential role in the agricultural innovation system (AIS), facilitating the multi-stakeholder processes that unleash agricultural innovation (FAO, 2020). This indicator framework for EAS represents a mechanism for the holistic assessment and evaluation of EAS systems, providing much-needed evidence to support their strengthening. In many countries, EAS are now provided by a variety of actors (including public, private and civil society) on a diverse array of topics and EAS systems are often decentralized. Providers of EAS are also making increasing use of digital technologies to reach smallholders and family farmers, especially in the face of the global COVID-19 pandemic. In order to strengthen the important role of EAS in facilitating innovation for sustainable agricultural development, it is essential to examine their performance, including outcomes and impact, in the context of modern, pluralistic and increasingly digital EAS systems. However, to date there has been little to no coordinated effort to systematically collect data and generate evidence on EAS, especially taking into consideration their current facilitative role.

The Food and Agriculture Organization of the United Nations (FAO) has taken on this
challenge of developing guidelines and tools for the systematic assessment and
performance measurement of national EAS systems. Three complementary instruments
are provided: 1) FAO's EAS assessment guide; 2) FAO's EAS-Yardstick (EAS-Y) scoring tool;
and 3) the indicator framework for EAS. This publication presents the latter instrument.
The indicator framework makes use of insights from past work in the area of EAS
assessment and evaluation but is the only comprehensive EAS framework to take
account of contemporary, pluralistic services, based on a theory of change that clearly
traces performance from inputs to impact, and makes use of participatory methods of
data collection, including quantitative and qualitative data.

FAO's Research and Extension Unit believes that the application of the EAS indicator
framework and data collection tools will lead to the generation of knowledge and data
directly relevant to the strengthening of national EAS systems. Information generated
using FAO's instruments for EAS assessment and evaluation can also be used to establish
national and global EAS observatories and online dashboards. These efforts will not
only fill the EAS data gap, but also provide evidence to promote better decision-making
and more investment in EAS, strengthening their role in the AIS and thus fostering
sustainable food systems and improving livelihoods in rural areas.
# Acknowledgements

The indicator framework for national extension and advisory service systems was developed to fill the data gap and provide evidence on national EAS systems. This important work was made possible through both technical and financial support from FAO's Research and Extension Unit. The authors would like to thank Selvaraju Ramasamy, Head of the Research and Extension Unit, for his strategic guidance and support throughout the development of this framework.

The development of the indicator framework was led by Rasheed Sulaiman V of GFRAS, Patrice Djamen of CIRAD, Christian Grovermann of FiBL and Delgermaa Chuluunbaatar and Aiden Holley of FAO. For their significant contributions to the content as well as field-testing of the indicators, the authors would like to acknowledge the following interdisciplinary team of experts: Johan Blockeel, Maria Auxiliadora Briones, Sanne Chipeta, Archangel Munthali, Danilo Saavedra and P. V. K. Sasidhar.

This publication also benefited from technical review and inputs from Nevena Alexandrova, Manuela Bucciarelli, Zofia Krystyna Mroczek, Yapeng Ou, John Preissing and Puyun Yang of the FAO. We would also like to thank Maria Auxiliadora Briones, Hur Ben Correa Da Silva and Francisco Guirre of RELASER for their leadership as well as country fora, national consultants and other GFRAS focal points for contributions and support for pilot testing this indicator framework in four countries in Latin America.

Finally, acknowledgements are extended to Andrew Morris for editing and studio Pietro Bartoleschi for layout.
Abbreviations and acronyms

AgGDP agricultural gross domestic product

AIS agricultural innovation system

CIRAD Centre de coopération internationale en recherche agronomique pour le développement (The French Agricultural Research Centre for International Development)

EAS extension and advisory services

EAS-Y Extension and Advisory Services Yardstick (Scoring Tool)

FAO The Food and Agriculture Organization of the United Nations

FGD focus group discussion

FiBL Das Forschungsinstitut für biologischen Landbau (The Research Institute of Organic Agriculture)

GFRAS Global Forum for Rural Advisory Services

ICTS information and communication technologies

ITU International Telecommunication Union

MEL monitoring, evaluation and learning

NEAS national extension and advisory system

NGO non-governmental organization

OECD Organization for Economic Co-operation and Development

QCA qualitative comparative analysis

RCT randomized control trial

RELASER Red Latinoamericana de Extensión Rural (Latin American Network of Rural Extension Services)

RuLIS Rural Livelihood Information System

SSA sub-Saharan Africa

SDG Sustainable Development Goal

ToC theory of change
<figure><img src="image_4.png" /></figure>

<figure><img src="image_5.png" /></figure>
# 1 Introduction

Agricultural extension and advisory services (EAS) also known as rural advisory services (RAS) consist of all the different activities that provide the information and services needed and requested by farmers and other actors in rural settings to assist them in developing their own technical, organizational and management skills and practices to improve their livelihoods, as well as promote a more sustainable agriculture. They include the diversity of actors involved in the provision of EAS and the broad support provided to rural communities (beyond information and knowledge), embracing new functions such as facilitation, intermediation, advice and brokering (Sulaiman and Davis, 2012; Christoplos, 2010). Their role is crucial in supporting producers and other stakeholders in the agricultural and rural sector in addressing challenges and realizing new opportunities related to the changing economic, climatic, and social environment.

EAS are increasingly expected to provide knowledge brokering functions and facilitate interactions between actors and stakeholders in agricultural innovation systems (AIS). They have the potential to facilitate the identification and expression of demands for information, knowledge, and innovations relevant to producers and other actors in agricultural value chains. They are also crucial for facilitating participatory development and the effective use of innovative solutions. Furthermore, EAS can contribute to the creation and securing of decent jobs, especially for rural women and youth, and to achieving several Sustainable Development Goals (SDGs), especially SDG 1 (No Poverty), 2 (End Hunger), 5 (Gender Equality), 10 (Reduced Inequalities) and 13 (Climate Action). EAS can also contribute to other SDGs such as 3 (Good Health and Well-being) and 8 (Decent work and economic growth).

As has been demonstrated in many countries, sustainable development cannot be achieved without innovation (FAO and INRAE, 2020). This observation leads to concern over the way in which EAS are designed and implemented to support agricultural
<figure><img src="image_6.png" /></figure>

innovation. It also raises the question of how EAS can more successfully contribute to the strengthening of the AIS, thus meeting the economic, social, and environmental challenges and opportunities of transforming agricultural and food systems. Addressing these questions requires enhanced capabilities to assess the functioning and impacts of the current EAS systems.

In line with its mandate, FAO through its Research and Extension Unit has developed two sets of methodologies for assessing AIS as well as EAS systems (a sub-system of AIS). The EAS assessment guide<fnref n="1" /> was developed in order to generate evidence for informed policy and investment decisions through the identification of gaps and entry points for improving and reforming EAS and providing recommendations for systemic change. The EAS assessment provides a comprehensive look into national EAS in the context of pluralistic systems, with services provided by different types of actors including public, private, non-governmental organizations (NGOs), producer organizations, as well as non-formal advisors, on a broad set of topics including production, marketing, logistics, organization of groups, facilitation of linkages, nutrition, natural resource management, and livelihoods in general. It can also be used as a monitoring tool for investments and policies. As the outputs of the EAS assessments have been mostly qualitative, in order to complement this information with metrics,<fnref n="2" /> this indicator framework for EAS together with the EAS-Yardstick (EAS-Y) scoring tool have been developed to offer a numerical illustration of the performance and outcomes of EAS systems.

The EAS-Y scoring tool<fnref n="3" /> provides scores on the performance and outcomes of EAS systems from the perspectives of EAS experts, agents and users, thus providing an in-depth assessment that can be used for monitoring, evaluation and learning (MEL), assessing capacity needs and generating evidence to target investments, among others. To supplement primary data generated by the EAS-Y scoring tool, secondary data are used to fill in the remaining informational gaps, in particular with regards to the inputs, activities and outputs of EAS (more on this in Section 6).

This indicator framework is one of the very few attempts to examine the whole of EAS systems, in terms of the systems' key resources (human, financial, infrastructure, and so on.), EAS content, reach (including the capacity to reach EAS users by gender, farm size and age) as well as key outcomes such as improved skills and knowledge and the impacts on EAS users' livelihoods and well-being. With this framework and the accompanying

<footnote n="1">¹ Full title: *Comprehensive assessment of national extension and advisory service systems: an operational guide.*</footnote>

<footnote n="2">² A set of numbers that provides information about a particular process or activity.</footnote>

<footnote n="3">³ Full Title: *The extension and advisory service systems yardstick (EAS-Y): a scoring tool to generate evidence on performance and outcomes.*</footnote>
<figure><img src="image_7.png" /></figure>

EAS-Y scoring tool and EAS assessment guide, FAO's Research and Extension Unit has embraced the challenge of developing meaningful metrics to represent nuanced and qualitative aspects of EAS systems in addition to more in-depth qualitative data generation. This indicator framework thus provides concrete evidence to inform policy making, advocacy and investment plans, reflecting the complex reality of EAS systems.

This document is organized as follows: Section 2 starts with a brief critical review of previous efforts in the evaluation and assessment of EAS. The logic behind the proposed framework, especially the recognition of pluralism, the framework characteristics, and its focus on exploring the performance of EAS systems are presented in Section 3. This is followed by a discussion on the underlying theory of change that has informed the development of the indicator framework, presented in Section 4. Assumptions underpinning the theory of change are discussed in Section 5. The detailed list of indicators across the five levels of the framework, namely: impact, outcome, outputs, activities and input and potential sources of data are presented in Section 6. Analytical considerations for using the data generated by the indicator framework are covered in Section 7. The document ends with a brief conclusion.
見

<figure><img src="image_8.png" /></figure>
2
Extension and
advisory services (EAS)
assessment frameworks:
a critical review

Historically, the indicators used for assessing EAS have focused on purely quantitative aspects such as investments in extension (mostly in the public sector), the number and qualifications of extension staff deployed, farmer-extension ratio, the number of extension activities and clients reached, and so on. For instance, Swanson and Rajalahti (2010) used human resources (categories and academic qualifications), financial expenditures and extension activities. In other cases, different indicators have been used to measure financial expenditure for EAS (although exclusively considering public investments in EAS); for example extension intensity/ha of cropped area (Sulaiman and Sadamate, 2000), or investment in extension as a share of total agricultural gross domestic product (AgGDP) (Joshi, Kumar and Parappurathu, 2015). Yet, these indicators alone fail to address the relevance, effectiveness and efficiency of the system, much less the sustainability (Chipeta, 2019). Furthermore, the focus has been exclusively on public sector activities and investments. However, the public sector is no longer the only relevant actor in EAS as many non-public actors (such as the private sector, NGOs and producer organizations) are currently active in EAS provision in many countries.

Many efforts have gone into evaluating the outcomes and impact of EAS interventions on farmers in terms of technology adoption and productivity as well as the rate of return on extension investments, among others. In the case of technology adoption, however, Glover et al. (2019) argue that focusing only on this outcome simplifies and mischaracterizes what happens during the process of technological change and also leads to a lack of understanding of other changes that may have occurred as a consequence of efforts
<figure><img src="image_9.png" /></figure>

that seek to promote new technology. Moreover, most past studies have focused on the
evaluation of single programs (e.g. Buehren et al., 2019; Makate and Makate, 2019) or
specific extension methods (e.g. Feder, Murgai and Quizon, 2003), rather than examining
the whole EAS system. Very few studies have looked at EAS systems in their totality at
local or country levels.

In 2016, Faure et al., proposed a framework to conduct such holistic assessments of performance and impact of pluralistic agricultural extension systems adapting the Best Fit Framework by Birner et al. (2009). Faure et al. (2016) frame the assessment along an impact chain starting with the characteristics of EAS, followed by their organizational performance, their outcome at farm/ household level, and, finally, their impact on development. This is then aligned with the OECD standard criteria: relevance, efficiency, effectiveness, sustainability and impact. Despite some apparent advantages of Faure et al.'s assessment framework, this methodology is geared towards in-depth case study analysis and does not provide a set of easy-to-follow data collection tools. This limits the wide applicability of Faure et al.'s framework as well as the comparability of results, which could be enabled with the use of indicators in addition to in-depth case study analysis.

In addition to impact evaluations for single projects or programmes and more encompassing case studies of EAS systems, several important scoping studies attempting to measure overall EAS outcomes and impact also exist. This includes Birkhaesar, Evenson and Feder's (1991) seminal review of empirical studies on extension impact, Evenson's (1997) review of 57 studies on the economic impact of extension and Alston et al. (2000), who found in their meta-analysis including 289 studies a median economic rate of return on extension investments of 58 percent. Yet, despite these efforts to capture the impact of EAS interventions, there has also been criticism of the methods used and overall utility of impact evaluations for EAS. Faure et al. (2016) noted that many approaches used in impact evaluation have faced methodological challenges and have not been able to establish lines of impact that can be clearly attributed to the EAS intervention. Assessing specific impacts of EAS would require more in-depth analysis and the use of rigorous evaluation designs like randomized controlled trials (RCTs) to separate the effect of EAS from other influencing factors (Dhehibi, Werner and Qaim, 2018).

With an increasing emphasis on co-innovation, facilitation and brokering roles, Coutts,
Koutsouris and Davis (2019) noted that the emphasis of the EAS evaluation must shift
towards capturing how well the process is working in bringing together the key parties
and how this is impacting problem and opportunity definition, collaboration, innovation,
adaptation and change. Ultimately, both approaches (rigorous impact evaluations and
<figure><img src="image_10.png" /></figure>

process evaluations) have an important role to play and are highly complementary. However, many impact evaluations rarely consider changes in learning, the development of problem-solving skills, or empowerment as project impacts – even though these are key to triggering the transformation of knowledge into action (Prager and Creaney, 2017; Restrepo, Lelea and Kaufmann 2018). In line with this, Taye (2013) argues that as EAS have undergone a paradigm shift from a “transfer of technology” approach to a more inclusive, participatory and complex facilitative role, a shift is also necessary in the methodological approaches used to measure the success of EAS. According to Taye’s analysis of impact evaluations of EAS interventions in sub-Saharan Africa (SSA), in most cases positive results from EAS interventions have been reported despite stagnating agricultural productivity growth rates in SSA. It has therefore been proposed to move towards approaches aimed at “improving impact,” which implies the use of participatory MEL and learning tools to improve practice and develop internal learning systems that consider the dynamic nature of agricultural development.

In recent years, efforts have been made towards identifying and establishing more holistic mechanisms for the assessment of EAS, also including the use of predetermined indicators. For example, the global consultation on "Improving agricultural advisory systems metrics and learning" organized by the Bill and Melinda Gates Foundation (BMGF) noted the importance of tracking the performance of digital innovation for delivering EAS (ALINE, 2019). This consultation also highlighted the importance of qualitative insights in terms of adding value and drawing more meaningful lessons from proposed EAS dashboards created to collect national level indicators related to EAS performance. The DLEC (developing local extension capacity) project (which also employs the best fit framework) created a conceptual framework for developing metrics around EAS in 2019. The DLEC list included a list of 25 quantitative indicators that span systems, input, delivery, output, outcome and impact (ALINE, 2019). However, the DLEC work is focused mostly on collecting data from the public sector, with little scope for collecting information on the contributions of private, NGO and producer organizations in EAS provision. Another limitation is its continued focus on technology adoption and inclusivity at outcome level, whereas EAS contribute a wider range of outcomes beyond these. There is also a need to gain insights into client perspectives on EAS especially when outcomes must be tracked.

All of this points to the importance of developing simple and feasible indicators to assess the performance and outcomes of EAS using both qualitative and quantitative methods to capture the complex reality of EAS. These indicators should capture the diversity of EAS provision, the enabling conditions that shape pluralistic EAS provision, and factors influencing EAS performance and client perspectives. Indeed, donors and
national governments are currently more interested in methods of monitoring and assessment that directly lead to improving the practical contributions and impact of EAS rather than proving their impact. However, most efforts to date have focused solely on attributing outcomes and/or impact to EAS without much context, or EAS systems have been assessed in specific country contexts under certain conditions, with little scope for scaling up. It is therefore now essential to develop new and innovative indicators that meaningfully demonstrate the performance as well as outcomes and impact of EAS, supplemented by qualitative background information and with accompanying mechanisms to collect the required data for these indicators.
3

Framework logic

A. Pluralism in EAS provision

The chosen approach focuses on the country level, but it is also valid to conduct
assessment and analyses at (subnational) regional or local levels. At the national
level, the approach is based on the concept of pluralistic EAS, which makes it possible
to help design coherent national advisory policies as well as service provision on the
ground that are functional, coordinated, more sustainable and in line with national
development objectives (Teyssier et al., 2019).

In most countries, the national agricultural extension and advisory system (NEAS) is made up of a wide variety of actors and stakeholders whose roles are more or less specific. These actors can be grouped into three broad categories: (i) EAS users; (ii) EAS providers; and, (iii) those providing support services or contributing to the enabling environment (Figure 1).

EAS users<fnref n="4" /> include, in particular, producers in their diversity (women, men, youth) including their organizations, which can also act as providers of EAS. These different users have specific needs that are at the centre of the activities of other actors in the NEAS. The demands and needs of the EAS users can be influenced by several factors: i) the capacity of users to express themselves, ii) the capacities of EAS providers, and iii) the governance and organization of the system.

EAS providers<fnref n="5" /> are an equally diverse category. These are organizations from the private
sector (input dealers, agricultural traders, consultancy firms, and so on.), producer
organizations in some cases, agricultural professional organizations (associations

<footnote n="4">⁴ Service users can be service providers in some contexts and vice versa in other contexts. In some countries, the line between service provider and user is not always very clear and quite fluid.</footnote>

<footnote n="5">⁵ This includes only formal EAS actors, but the authors would like to acknowledge the important role of informal EAS, such as peer-to-peer advisory or lead farmers advising fellow farmers.</footnote>
<figure><img src="image_11.png" /><figcaption>FIGURE 1. Main categories of actors in an integrated EAS system</figcaption></figure>

Source: Adapted from Adolph (2011), Teyssier et al. (2019), Djamen, Davis and Sacko (2020)

of agricultural professionals) or non-governmental organizations (NGOs). Public EAS organizations (as well as non-public EAS providers with full or partial public funding) are still widely dominant in most countries. However, in recent years, it has become clear that they alone cannot guarantee adequate coverage of the diverse demands for EAS in terms of territory and subject matter. New non-state providers have emerged and are playing an increasingly important role. In some cases, these organizations are also EAS users, as one of the functions of EAS is to enhance the capacities of other service providers to provide or use EAS. The dotted boundaries separating the EAS users (inner circle) and EAS providers (middle circle) in Figure 1 indicate this.
<figure><img src="image_12.png" /></figure>

The diversification of EAS providers potentially offers users the possibility of choosing the provider they consider most suitable to provide them with quality services. However, it also brings new challenges such as competition for resources, reluctance to share information and participate in learning, the duplication of efforts and contradictory messages (Djamen, Davis and Sacko, 2013). For similar types of activities, providers of EAS can employ different extension approaches and mobilize workers with widely varying profiles and skills. This heterogeneity of profiles is likely to affect the quality of the services provided to producers and other value chain actors. Therefore, there is a need to improve coordination and develop synergies in order to enhance the specific strengths of each provider and ensure the quality of services.

The third category of actors is made up of support services and other actors with various (e.g. regulatory) functions in the enabling environment for EAS. Their activities help to guide, organize and facilitate or complement EAS provision. These support functions include:

i. development of policies and strategies;

ii. organizing, planning and coordination;

iii. provision of supportive infrastructure including media and ICTs (information and communication technologies);

iv. capacity development of professionals through education and training;

v. providing funding;

vi. generating new knowledge;

vii. providing adaptive research support; and

viii. support for MEL.

The institutions around the EAS providers in Figure 1 indicate this.

## B. EAS indicator framework characteristics

The overarching objective of this indicator framework is to provide an overall structure for assessing the current status of the national EAS system and measure the performance and outcomes of the EAS system, including the enabling environment shaping EAS delivery at the country level. The indicator framework provides a wide view of the system. Implicitly, it also seeks to generate knowledge and understanding of the structural and functional characteristics and strategic orientations of EAS systems, perspectives on demand from EAS clients and key outcomes as they perceive them (Figure 2). The results can serve as a basis for decision making, planning and management of national, regional or global initiatives, but also feed exchanges and learning between different countries.
<figure><img src="image_13.png" /><figcaption>FIGURE 2. EAS indicator framework design</figcaption></figure>

While the application of the EAS assessment guide provides rich background information, including potential sources to track indicators at input, activity and output levels, the EAS-Y scoring tool provides primary data on inputs, outputs and outcomes. Relevant quantitative indicators are collected at all levels, primarily through secondary sources. These three instruments for data collection constitute the indicator framework which is illustrated in Figure 3 below. A more detailed description of data sources can be found in sub-section 6B.

<figure><img src="image_14.png" /><figcaption>FIGURE 3. EAS indicator framework design: instruments for data collection</figcaption></figure>
C. Exploring the performance of the EAS system

Another central aspect of this framework is the assessment of the performance of the EAS system. This involves assessing the extent to which the system enjoys a favourable enabling environment, provides the range of services needed by its diverse clients, and has internal systems for the coordination, collaboration and learning needed to perform its functions effectively. The use of the framework thus makes it possible to highlight the areas of the EAS system that need to be consolidated, but also areas where adjustments are necessary to improve performance. Furthermore, the application of the framework can contribute to appraising the level of responsiveness of the system and its capacity to renew itself through the integration of ongoing developments and innovations in EAS, for example, with regards to methods and tools, extensionist capacities and governance and regulation. It is in this sense that digitalization, which appears to be a major trend in knowledge management methods and extension delivery, is important to monitor and analyse. The same applies to the professionalization and coordination of EAS providers, which are made necessary by the pluralism of providers.
<figure><img src="image_15.png" /></figure>

<figure><img src="image_16.png" /></figure>
# 4

# Theory of change

The EAS indicator framework is guided by a theory of change (ToC) that considers that access to quality extension services and their effective use (information, knowledge, advice, facilitation, capacities) by farmers (and other value chain actors) are crucial for addressing the challenges they face and thereby contributing to enhanced food and nutrition security, income generation and greater resilience (Figure 4). In this figure, the

<figure><img src="image_17.png" /><figcaption>FIGURE 4. Theory of change of agricultural extension and advisory services</figcaption></figure>
inputs are seen as the starting point that gives rise to the activities, outputs, outcomes and ultimate impact of EAS. The overall importance of these indicator levels, however, flows in the opposite direction, with impact being at the forefront in terms of social and political relevance. Therefore, the authors of this report have chosen to present the levels of this indicator framework for EAS in the following sections in such a way as to emphasize the relative importance, starting with the impact and ending with the inputs.

## A. Impacts

Impacts are the long-term results – the development objectives to which EAS contribute, directly or indirectly. The impact indicator(s) suggested here measure the contributions by EAS towards the Sustainable Development Goals (SDGs), some of which are more in line with the traditional areas of focus for EAS such as food security and income generation but they also include new challenges such as nutrition and resilience to shocks including climate change. For the impact level, no specific indicator(s) is selected as the SDGs to which an EAS contribution is expected may vary depending on country context, however most EAS have the potential to contribute to SDGs such as SDG 1 (No poverty), SDG 2 (Zero Hunger), SDG 5 (Gender Equality), SDG 10 (Reduced Inequalities) and SDG 13 (Climate Action). Although these impacts are not the sole responsibility of EAS, they are nevertheless important to measure and analyse, as they provide insights into the overall performance of the AIS which also includes and influences EAS. The impact indicators are mainly informed using secondary data.

## B. Outcomes

Outcomes are changes induced by the access to and deployment of EAS by users. This transformation can be assessed by changes in knowledge, awareness, attitudes, behaviour, practices and/or capacities depending on the focus of the intervention. Transformation is a process that depends on several factors, including the capacity of EAS users, the socio-economic and policy context, changing market conditions and the support farmers receive from EAS providers. The outcome indicators to be monitored may also include, for example, users' survey responses, indicating that they have been able to make new/promising connections following the EAS intervention. The changes induced among producers and other users must be attributable to EAS.
## C. Outputs

The outputs demonstrate the extent to which activities carried out by various EAS providers contribute to producing accessible services demanded by users. This can be assessed using indicators including the percentage and characteristics of EAS users (including women, youth and smallholders) who access the services. In addition, the geographical coverage of the services offered and share of farmers who are organized into cooperatives and organizations are also considered. It is considered that EAS can generate the expected changes only if all users who demand services can also access them. It is therefore very important for an EAS system to ensure that these services are made available and are accessible physically and at an affordable cost to the majority of people who require these services.

## D. Activities

Activities refer to actions or work carried out by EAS providers to supply various services corresponding to users' demands and needs. Broadly speaking, there are five main categories of EAS services including:

i. knowledge, technology and information sharing;

ii. decision support or advisory on farm, organizational and agribusiness management (on-farm problem solving);

iii. strengthening of farmer-based organizations and collective action;

iv. human capacity development; and

v. facilitation of innovation and co-creation, linking and brokering in rural areas and within food systems.

Activities carried out have a key role in triggering the results chain of the EAS system because they help to bring about the expected changes at the level of service users. Activities can be characterized by their type, volume and methodology used. Appropriate indicators in this area can be defined for each type of activity implemented. This indicator framework captures the number of different events, activities and knowledge products provided by EAS. Moreover, the extent of use of ICTs by EAS and specifically by extension agents can provide important information with regard to efficient knowledge and information sharing, technology use and the facilitation and brokering capacity of EAS. The theory of change considers that changes at the level of users will happen only when the type and volume of the services are relevant to their demand. In addition, the methods and tools used for the implementation of activities must be effective and efficient.
## E. Inputs

Inputs refer here to the elements that are directly mobilized or that affect the planning and implementation of EAS interventions. They can be clustered in four broad categories including:

i. capacities of EAS providers (at organizational and individual level);

ii. human and financial resources;

iii. system governance (including policies and coordination) and process related to monitoring and knowledge management; and

iv. logistics and infrastructure.

With regards to capacities, a distinction is made at the individual level between technical and functional capacities, which must be in line with the needs of EAS users. At the organizational level, capacity refers mainly to strategic objectives, the availability and mobilization of logistic and financial resources and, above all, the ability to manage and adapt the organizational evolution including renewal of EAS tools and methods. It also refers to human resource management, including the ability to create the conditions for optimal staff performance. Indeed, the performance of advisers is also strongly determined by capacities of the organizations that employ them. The theory of change considers that EAS can only generate the expected results if the capacities of the providers (at individual and organizational level) are renewed whenever necessary to adapt or even anticipate changes. This capacity can be measured by indicators such as whether a structured MEL system is in place and the percentage of extension staff that receive opportunities to strengthen their capacities.

The volume of public funding is still seen as an indicator of the importance that government and development partners attach to agricultural advisory services (also as a percentage of AgGDP). However, there is growing recognition that in many cases public funding is not sufficient, and that a new EAS funding model including direct or indirect participation of service users, needs to be considered while taking account of the principle of equity. The framework also considers the policies that shape EAS provision especially how agriculture, rural development and extension policies articulate the role of EAS providers and enable EAS provision. These policies, and in particular the extent to which they are implemented, are key to determining the governance and overall coordination of the EAS system. In addition, the existence of a national mechanism for coordination among pluralistic EAS providers is considered here. Lastly, vital EAS inputs also consist of supportive logistics and infrastructure that determine EAS performance. This includes general mobility (e.g. motorable roads) and communication indicators (mobile density and internet access in rural areas) as well as general infrastructure, such as access to electricity.
5

Assumptions

A number of assumptions underpin the EAS ToC, which helps to explain how system inputs are expected to translate into outputs that lead to the achievement of outcomes, and contribute subsequently to impacts at local, country or global levels. Some of these can be managed directly by extension systems while others are beyond their control. For example, it is assumed that there is political will at country and local levels to harness the potential of EAS for achieving development objectives. This translates into the adequate consideration of EAS in agricultural and rural development policies, planning and investments. Political will at local level is particularly relevant in terms of decentralized extension and administrative systems. There are four other critical assumptions:

i. EAS planning and implementation processes are not unduly affected by external factors beyond the control of actors running the EAS systems.

ii. EAS organizations are operational, have a good territorial coverage and the capacities to deliver relevant, cost-effective services; they effectively benefit from expected support from other actors and stakeholders of the agricultural innovation system.

iii. The removal of barriers to access EAS services is possible, and EAS users (particularly women and youth) are able to take advantage of emerging opportunities, due to effective EAS intervention.

iv. Other agricultural and rural development issues are adequately addressed by concerned actors and decision-makers are willing and able to understand EAS issues and challenges to take them into account adequately in agricultural policies and investments.
<figure><img src="image_18.png" /></figure>

<figure><img src="image_19.png" /></figure>

©FUNICA
# 6

# Indicators

A set of 40 indicators was selected for this indicator framework (not including impact level indicators, which are not explicitly selected). Out of these, 10 have been identified as "core" indicators based on their relative importance in comparison to the other indicators. In other words, these 10 "core" indicators represent the most essential information, or minimum dataset required for a snapshot of the EAS system. The 30 "complementary" indicators provide a more detailed picture of the EAS system. However, in the interest of making the indicator framework more accessible, even in the face of resource constraints to facilitate data collection, these "complementary" indicators have been clearly marked as less critical. Furthermore, the selection of indicators for this framework was guided by the objective to focus on information on the characteristics of the system, as well as factors that directly influence its operation and the results generated. The indicators were chosen on the basis of suggestions already made in several publications on the subject, including in particular the working paper by Chipeta (2019) and other relevant publications (Birner et al., 2012; Faure 2016; Davis, Baby and Ragasa, 2020; Blum, Cofini and Sulaiman, 2020). Most importantly, it factored in the provisional list of indicators developed by a multi-disciplinary group of experts who met in Rome in November 2019 to discuss methods, tools and indicators for the evaluation of agricultural innovation and advisory systems. Later the set of indicators selected was tested and operational feasibilities were discussed by national and regional experts on EAS from Africa, Asia and Latin America (see Box 1 on page 30 for more details on the testing).
<figure><img src="image_20.png" /><figcaption>TABLE 1. Indicator framework: list of core and complementary indicators</figcaption></figure>

<table><thead><tr><th>ID</th><th>Indicator</th><th>Relative importance (core/complementary)</th><th>Source</th></tr></thead><tbody><tr><td colspan="4"><strong>IMPACTS</strong></td></tr><tr><td>1</td><td>Contributions to the SDGs (1, 2, 5, 10, 13, 15, etc.)</td><td>Complementary</td><td><ul><li>World Bank World Development Indicators</li><li>National SDG reports</li></ul></td></tr><tr><td colspan="4"><strong>OUTCOMES</strong></td></tr><tr><td colspan="4"><strong>2.1 Long-term</strong></td></tr><tr><td>2.1.1</td><td>Perceived economic resilience [Score from 0 - 3]</td><td>Complementary</td><td></td></tr><tr><td>2.1.2</td><td>Perceived social resilience [Score from 0 - 3]</td><td>Complementary</td><td><ul><li>EAS-Y scoring tool report</li><li>National level reports from relevant organizations, ministries, etc.</li></ul></td></tr><tr><td>2.1.3</td><td>Perceived environmental integrity [Score from 0 - 3]</td><td>Complementary</td><td></td></tr><tr><td colspan="4"><strong>2.2 Intermediate</strong></td></tr><tr><td>2.2.1</td><td>Application of new knowledge and skills [Score from 0 - 3]</td><td>Complementary</td><td></td></tr><tr><td>2.2.2</td><td>Improved access to services [Score from 0 - 3]</td><td>Complementary</td><td><ul><li>EAS-Y scoring tool report</li><li>Agricultural census</li><li>Specific case study reports</li></ul></td></tr><tr><td>2.2.3</td><td>Empowerment [Score from 0 - 3]</td><td>Complementary</td><td></td></tr><tr><td colspan="4"><strong>2.3. Immediate</strong></td></tr><tr><td>2.3.1</td><td>Technical knowledge and skills [Score from 0 - 3]</td><td>Complementary</td><td><ul><li>EAS-Y scoring tool report</li><li>National level reports from relevant organizations, ministries</li></ul></td></tr><tr><td>2.3.2</td><td>Entrepreneurial skills [Score from 0 - 3]</td><td>Complementary</td><td><ul><li>FAO-RuLIS</li><li>Case study and project reports</li></ul></td></tr><tr><td>2.3.3</td><td>Social skills [Score from 0 - 3]</td><td>Complementary</td><td></td></tr></tbody></table>

<figure><img src="image_21.png" /></figure>

<figure><img src="image_22.png" /></figure>
<figure><img src="image_23.png" /></figure>

<figure><img src="image_24.png" /></figure>

<figure><img src="image_25.png" /></figure>

<table><thead><tr><th>ID</th><th>Indicator</th><th>Relative importance (core/complementary)</th><th>Source</th></tr></thead><tbody><tr><td colspan="4"><strong>OUTPUTS</strong></td></tr><tr><td colspan="4"><strong>3.1 Reach</strong></td></tr><tr><td>3.1.1</td><td>Farmers accessing EAS<br/>[as percentage of total farmer population]</td><td>Core</td><td rowspan="2"><ul><li>Annual report of Ministry of Agriculture</li></ul></td></tr><tr><td>3.1.2</td><td>Regions/territories covered by EAS<br/>[as percentage of total area]</td><td>Complementary</td></tr><tr><td colspan="4"><strong>3.2 Inclusiveness</strong></td></tr><tr><td>3.2.1</td><td>Smallholder farmers accessing EAS<br/>[as percentage of smallholder farmer population]</td><td>Complementary</td><td><ul><li>Annual report of EAS providers</li><li>Agricultural census</li><li>National statistics</li><li>FAO-RuLIS database</li><li>World Bank micro-datasets</li><li>Relevant project reports</li></ul></td></tr><tr><td>3.2.2</td><td>Women and young farmers accessing EAS<br/>[as percentage of women and young farmer populations, respectively]</td><td>Complementary</td><td><ul><li>Producer organization database/registry</li><li>EAS assessment report</li><li>Case study reports</li><li>Project documents</li></ul></td></tr><tr><td colspan="4"><strong>3.3 Supporting farmer organizations</strong></td></tr><tr><td>3.3.1</td><td>Farmers organized in a group, association or organization (incl. user groups, farmer interest groups, producer organizations, producer cooperatives, community organizations)<br/>[as percentage of the total farmer population]</td><td>Core</td><td></td></tr><tr><td>3.3.2</td><td>Smallholder farmers organized<br/>[as percentage of smallholder farmer population]</td><td>Complementary</td><td></td></tr><tr><td>3.3.3</td><td>Women farmers organized<br/>[as percentage of women farmer population]</td><td>Complementary</td><td></td></tr></tbody></table>

## ACTIVITIES

### 4.1 Performed key activities

<table><tr><td>4.1.1</td><td>Learning events organized for EAS users annually (e.g. webinars, FFS, trainings, field days, etc.)<br/>[number of events]</td><td>Core</td><td rowspan="4"><ul><li>Annual reports and other records from national ministries and private sector providers</li><li>Case studies</li><li>Financial reports from EAS providers</li><li>National EAS assessment report</li><li>Reports/testimonies from farmers and their organizations</li></ul></td></tr><tr><td>4.1.2</td><td>Demonstration sites and learning centres maintained<br/>[number of sites/centres]</td><td>Complementary</td></tr><tr><td>4.1.3</td><td>Knowledge products made available in various forms (both physical and virtual)<br/>[number of products]</td><td>Complementary</td></tr><tr><td>4.1.4</td><td>Activities (training, event, market fairs, etc.) oriented towards linking farmers to market and/or into value chains organized annually<br/>[number of activities]</td><td>Complementary</td></tr></table>

<figure><img src="image_26.png" /></figure>

<figure><img src="image_27.png" /></figure>
<figure><img src="image_28.png" /></figure>

<figure><img src="image_29.png" /></figure>

<figure><img src="image_30.png" /></figure>

<table><thead><tr><th>ID</th><th>Indicator</th><th>Relative importance (core/complementary)</th><th>Source</th></tr></thead><tbody><tr><td colspan="4"><strong>ACTIVITIES</strong></td></tr><tr><td colspan="4"><strong>4.2 Use of ICTs</strong></td></tr><tr><td>4.2.1</td><td>Farmers using ICT-based channels, applications and/or tools to access EAS (e.g. radio or TV programmes, phone apps, etc.) [as percentage of total farmer population]<br/>Note: If possible, disaggregate by type of ICT (e.g. radio, TV, phone app, etc.)</td><td>Core</td><td><ul><li>Farmer organizations</li><li>Reports and statistics from public and private EAS providers</li><li>Agricultural census</li><li>Ministry of Agriculture, Bureau of Statistics</li></ul></td></tr><tr><td>4.2.2</td><td>Extension personnel using ICT tools [as percentage of total extension personnel]</td><td>Complementary</td><td><ul><li>National sample survey</li><li>National databases from research institutes</li></ul></td></tr><tr><td>4.2.3</td><td>Farmers trained on digital/ICT skills/tools [as percentage of total farmer population]</td><td>Complementary</td><td>Records, organizational logs from public and private EAS providers and farmer organizations</td></tr></tbody></table>

## INPUTS

### 5.1 Funding

<table><tr><td>5.1.1</td><td>Total investments in EAS (include investment from all actors - public, private, producer organizations (POs) and NGOs, etc.) [in USD]</td><td>Core</td><td><ul><li>Reports (budget and organizational) from Bureau of Statistics, Ministry of Finance, Ministry of Agriculture and Dept. of Extension</li></ul></td></tr><tr><td>5.1.2</td><td>Public Investments in EAS [as percentage of AgGDP]</td><td>Core</td><td><ul><li>Public and private national actors, international development cooperation organizations</li></ul></td></tr><tr><td>5.1.3</td><td>Services or projects co-funded by public, private and other actors [number of services/projects]</td><td>Complementary</td><td></td></tr><tr><td>5.1.4</td><td>Overseas Development Assistance (ODA) investment in EAS [in USD]</td><td>Complementary</td><td>OECD CRS database</td></tr></table>

### 5.2 Human Resources

<table><tr><td>5.2.1</td><td>Extension personnel (include from all actors - public, private, POs and NGOs, etc.) [number of persons]<br/>Note: If possible, disaggregate by field staff, middle and senior management and by gender</td><td>Core</td><td><ul><li>National extension agencies' records, payroll, survey data</li><li>Case studies</li><li>Private sector and NGO records</li></ul></td></tr><tr><td>5.2.2</td><td>Extension staff have (annual) opportunity to strengthen their capacities (e.g. training, study tours, online course, etc.) [as percentage of total extension staff]</td><td>Complementary</td><td><ul><li>Extension organizations/ departments of public and private institutions</li><li>Human resources departments</li><li>Annual reports, records, training plans of relevant institutions</li></ul></td></tr></table>

<figure><img src="image_31.png" /></figure>

<figure><img src="image_32.png" /></figure>
<figure><img src="image_33.png" /></figure>

<figure><img src="image_34.png" /></figure>

<figure><img src="image_35.png" /></figure>

<table><thead><tr><th>ID</th><th>Indicator</th><th>Relative importance (core/complementary)</th><th>Source</th></tr></thead><tbody><tr><td colspan="4"><strong>5.2 Human Resources</strong></td></tr><tr><td>5.2.3</td><td>Lead farmers and community resource persons involved in EAS (e.g. farmer facilitators, certified lead farmers, community leaders, para-extension workers, etc. [number of persons])</td><td>Complementary</td><td><ul><li>Extension organizations/ departments of public and private institutions</li><li>Human resources departments</li><li>Extensionists</li><li>Training plan(s), annual reports and records of relevant institutions</li></ul></td></tr><tr><td colspan="4"><strong>5.3 Policy</strong></td></tr><tr><td>5.3.1</td><td>Policies that recognize the role and key challenges of EAS [number of policies]<br/>Note: include any policies that mention EAS (e.g. these could be outside the agriculture sector)</td><td>Complementary</td><td><ul><li>Government documents and policies</li><li>National EAS assessment report</li></ul></td></tr><tr><td>5.3.2</td><td>Qualitative score on to what extent policies are implemented effectively [Score from 0 - 3]</td><td>Complementary</td><td>EAS-Y scoring tool report</td></tr><tr><td colspan="4"><strong>5.4. Supportive Infrastructure</strong></td></tr><tr><td>5.4.1</td><td>Availability of roads for travel [road kms per person]</td><td>Complementary</td><td><ul><li>Plans and reports from the Ministry of Infrastructure and Development</li></ul></td></tr><tr><td>5.4.2</td><td>Mobile Density (if available, mobile density for rural areas) [per 1 000 population]</td><td>Core</td><td><ul><li>Agricultural census</li><li>National development reports</li><li>Reports from bureaus of statistics and other national ministries</li></ul></td></tr><tr><td>5.4.3</td><td>Access to internet (if available, access to internet for rural areas) [per 1 000 households]</td><td>Complementary</td><td><ul><li>OECD statistics</li><li>ITU statistics</li></ul></td></tr><tr><td>5.4.4</td><td>Electrification [as percentage of total number of villages/ households]</td><td>Complementary</td><td><ul><li>OECD statistics</li><li>ITU statistics</li></ul></td></tr><tr><td colspan="4"><strong>5.5. Internal management system</strong></td></tr><tr><td>5.5.1</td><td>National mechanism in place for coordination among pluralistic EAS providers [Y/N]<br/>Note: virtual and physical e.g. country fora, platform, etc.</td><td>Core</td><td><ul><li>Country GFRAS fora</li><li>Ministry of Agriculture and Food</li><li>EAS-Y scoring tool report</li><li>National EAS assessment report</li></ul></td></tr><tr><td>5.5.2</td><td>Qualitative score on effectiveness of the existing coordination mechanisms [Score from 0 - 3]</td><td>Complementary</td><td>EAS-Y scoring tool report</td></tr><tr><td>5.5.3</td><td>Systematic/structured MEL for EAS in place at national/central level [Y/N]</td><td>Core</td><td>Public and private providers' personnel, documents and annual reports</td></tr><tr><td>5.5.4</td><td>Qualitative score on effectiveness in implementation of MEL mechanisms [Score from 0 to 3]</td><td>Complementary</td><td>EAS-Y scoring tool report</td></tr></tbody></table>
## A. Core indicators

More detailed descriptions of the ten core indicators follow, in order to further illuminate the meaning, implications, and some potential approaches to data collection methods and use of the selected indicators.

### i. Farmers accessing EAS

Core indicator 3.1.1 assesses EAS coverage among farmers. This indicator provides a good overview of the state of farmer access to EAS, although it does not measure quality or intensity of farmer interactions with EAS. Here access is defined for an individual EAS user as having an in-person visit with an EAS agent (public, private, NGO, and so on.), training or any other in-person or virtual/mobile assistance (for example website use or chat box, call centre service use or SMS with EAS agent, and so on.) from EAS within a given recall time (usually 12 months).

### ii. Farmers organized in a group, association or organization

Core indicator 3.3.1 gauges the overall level of farmer organization, thus capturing how well farmers cooperate and collaborate on joint initiatives or collective actions (which can be as diverse as joint marketing and sales and/or purchases of agricultural goods and inputs to trainings on women's empowerment and lobbying for other social or policy issues). Farmer organization can be measured by determining the number of members of relevant organizations from secondary sources such as annual reports (of government or private EAS providers, among others) or surveying farmers directly on whether they are a member or regularly participate in farmer organizations.

### iii. Learning events organized for EAS users annually

Core indicator 4.1.1 measures the number of learning events and activities organized (annually) by all EAS providers, thus providing a broad impression of the level of EAS user engagement. This can be an important indicator to relate to others, such as 5.1.1 (total investments in EAS), to provide some insight into, for example how investments in EAS translate to direct interventions with users.

### iv. Farmers using ICTs to access EAS

Core indicator 4.2.1 measures the use of ICT tools and channels to access EAS. It can include many kinds of ICTs (radio, TV, mobile phone application, computer, and so on.)
and ideally is disaggregated by type of ICT tool used. Although the indicator may not capture frequency of use or quality of ICT tools, it is important given the growing trend of ICT use in EAS provision. This information can be helpful in determining how successful EAS interventions using ICT tools have been and to inform future programme planning using ICTs.

## v. Total investments in EAS

Core indicator 5.1.1 measures the level of investment in EAS, which has been and remains an important indicator to gauge the overall level of interest in and commitment to EAS on the part of public, private and civil society actors. Depending on the database or report where relevant statistics are presented, investments in EAS may need to be calculated by adding together multiple line items or individual EAS project or programme budgets.

## vi. Public investments in EAS as percentage of AgGDP

Core indicator 5.1.2 helps to put the level of investment in EAS into the context of the size of a country's agriculture sector. Although there is no perfect level of investment in EAS, a good reference is Blum and Szonyi's (2014) publication, which provides recommended levels of investment in extension (as a percentage of AgGDP) tailored to each country's specific conditions.

## vii. Extension personnel

Core indicator 5.2.1 provides information on human resources – a vital component of EAS provision. This indicator can be related, for example to overall investments (5.1.1), number of activities (4.1.1) and other indicators for a quick look at human resource prioritization and performance. It is also helpful to gather data on personnel disaggregated by characteristics such as level (for example field staff, middle and senior management) and gender.

## viii. Mobile density

Core indicator 5.4.2 measures access to mobile phones by the rural population, which is important to assess the feasibility of using mobile-based ICT tools for EAS delivery. As one of the most exciting new areas in EAS delivery, understanding how best to reach EAS clients via mobile devices is important in developing this capacity. However, while access to mobile devices is necessary to developing mobile-based EAS delivery, it is not sufficient, as a willingness, awareness and competence to use mobile-based EAS delivery tools among providers and users are also necessary.
ix. National coordination mechanism for EAS providers

Core indicator 5.5.1 gauges the level of collaboration and communication among pluralistic providers of EAS. It is in most cases straightforward to determine by investigating the relevant (if available) GFRAS country forum, ministry of agriculture or department of extension (website, reports, and so on.). In most cases, there is no established national mechanism for coordination among EAS actors, however, gathering data on this indicator is a first step in potentially identifying existing opportunities for establishing one.

x. Monitoring, evaluation and learning for EAS

Core indicator 5.5.3 provides insight into the EAS system's ability for critical reflection and continued improvement. It is also often easy to determine. However, it should be noted that a MEL mechanism set up for a single project or programme does not fulfil the criteria of this indicator. This indicator refers to a comprehensive, overarching MEL system that applies to most EAS programmes in one or more sectors.

B. Data sources

Although there are often significant gaps in data availability with regards to data on EAS systems, in addition to primary data sources from the application of FAO's EAS assessment guide and EAS-Y scoring tool, much of the relevant data for the indicator framework can be derived from secondary sources, such as international and national governmental sources, among others. For the impact indicators, the World Bank's World Development Indicators database and national reports on SDGs could be consulted. The EAS-Y scoring tool produces data on the outcomes of EAS from the perspectives of EAS stakeholders (especially EAS users), ranging from more immediate outcomes such as acquisition of technical knowledge to long-term outcomes like perceived economic resilience. Data for outputs such as reach, inclusiveness and farmer organization can be derived from information provided by relevant national governmental ministries, annual reports and activity reports from EAS providers, as well as from international databases such as the FAO-RuLIS (Rural Livelihood Information System) database and the World Bank's micro-datasets. Some of this information can come from research and evaluation studies on EAS, while assessment reports produced using the EAS assessment guide can also provide information on EAS outputs as well as inputs and activities.
Concerning the data required for the activity indicators, various secondary sources
can be consulted, such as government ministries and statistics offices, as well as
some global databases such as OECD Statistics and International Telecommunication
Union (ITU) statistics. Any additional required information can be acquired through key
informant interviews and from the annual or activity reports of EAS providers. Data for
the indicators under inputs, such as funding, human resources, policies, infrastructure,
can be derived from many of the same secondary sources listed above. In addition,
the application of the EAS-Y scoring tool produces data on inputs in the framework, for
example, qualitative scores are generated on the effectiveness of internal management
systems for EAS.

Furthermore, it is recommended that national ministries (of for example agriculture,
rural development, statistics) responsible for data collection in the agriculture and rural
development sector adopt the indicators (all levels, from inputs and activities to impact)
suggested here into their regular surveys to simplify data collection efforts in the future.

C. Data collection experiences

In order to identify data sources, gain a better understanding of the current level of
data availability on these indicators and to test the feasibility of this framework, a
secondary data collection exercise was carried out in six countries (Costa Rica, Dominican
Republic, Ecuador, India, Peru and Malawi) led by FAO's Research and Extension Unit.
In addition, the EAS-Y scoring tool was piloted in six countries (Brazil, Burkina Faso,
Costa Rica, Ecuador, Peru and Uganda) and the EAS assessment guide was used to
produce reports for four countries and one Indian State (Ecuador, Madagascar, Odisha
[India], Tunisia, Uganda and Ukraine). These combined data collection efforts provided
a broad impression from three continents on the state of data availability related to
EAS characteristics, performance, outputs, outcomes and impact. Overall, this exercise
confirmed the existence of the gap in data availability on pluralistic EAS, but also
demonstrated the relative ease with which such data can be found and/or collected if
concerted efforts are made.
<figure><img src="image_36.png" /></figure>

Experience with data collection by indicator level

IMPACTS

Experiences of pilot testing the indicator framework have demonstrated the availability in some cases of evidence linking EAS performance with overall impact in terms of contributions to the SDGs. For instance, in India, EAS have contributed to a significant increase in food security, farmer incomes and soil health (contributing to SDG 1, 2 and 13 targets), for example, through the implementation of the soil health card (SHC) scheme. This scheme has involved extension workers (public, private and civil society) in widespread soil testing combined with advisory services to farmers on smart practices to increase soil fertility. Early research has shown that the widespread use of SHCs (around 224 million have been distributed to farmers to date) has led to increased soil health and net improvements in income contributing to reduced poverty and increased food security (NITI Aayog, 2020; Reddy, 2018).

OUTCOMES

Pilot studies of the EAS-Y scoring tool produced results on the outcome level from the perspective of EAS users. The outcome indicators, ranging from immediate outcomes to long-term outcomes, are measured using a Likert scale and qualitative statements by participants to justify responses. The experiences from pilot studies demonstrated that participants in the scoring tool exercise need to be introduced to the concepts around EAS outcomes, especially when it comes to soft skills and potentially new ideas such as "empowerment." In this case, the use of locally relevant examples and local languages in the exercise were key. Ultimately, interesting and policy-relevant insights were gained through the application of EAS-Y at the outcome level.

OUTPUTS

For most countries, statistics are available or can be calculated with available secondary data to determine some of the core output indicators, such as the proportion of farmers having access to EAS. However, with regards to disaggregation, especially in terms of gender, farm size and other characteristics, which could give insight into the status of access to EAS by more vulnerable groups (inclusiveness), this data is often unavailable. In this case, it was only possible to calculate reliable figures in terms of gender disaggregated access to EAS for the Dominican Republic, where 27 percent of all producers and 33 percent of female producers have access to EAS. This figure can be considered encouraging, given the evidence from many countries of difficulties faced in particular by women farmers in accessing EAS (Ragasa, C., 2014).

ACTIVITIES

Comprehensive data on EAS activities were difficult to identify in the six countries where secondary data was initially collected for this framework. Some figures are available for public EAS, such as in Ecuador where all public EAS events are recorded in an online system called INFOCAMPO (for example there were a total of 223 672 events organized by public EAS in 2019) and also for India for two large public programmes, namely the Agricultural Technology Management Agency (ATMA) and Krishi Vigyan Kendra (farm science centres). However, activities organized by private or NGO providers of EAS are unknown in most cases and reliable data on activities by topic as well as use of ICTs by EAS actors is too seldom recorded.

INPUTS

It was found that data on public sector inputs such as investments in EAS and information on human resources are frequently available from country statistics offices or agriculture ministries. For example, figures on public investments in EAS as a percentage of AgGDP were determined for Dominican Republic (0.12 percent), Ecuador (0.36 percent) and India (0.2 percent)<fnref n="8" /> which are all well-below the FAO's recommended targets for these countries at 0.47, 0.64 and 1.66 percent, respectively (Blum and Szonyi, 2014). Comprehensive data on private and civil society inputs into EAS are very hard to come by, underscoring the need for better coordination and transparency between pluralistic EAS actors.

<footnote n="8">⁸ Sources for these figures are as follows: Department of Agriculture Extension (DECA) in the Dominican Republic, the Ministry of Agriculture of Ecuador and Nandi, R. and Swamikannu, N. (2019) for India.</footnote>
7

Analytical
considerations

A multitude of approaches and methodologies can be applied to the data generated by this indicator framework for analysis and the generation of evidence on EAS systems. This section provides guidance and recommended approaches on making the best use of this indicator framework for targeted strengthening of EAS. The EAS indicators should be embedded in an assessment approach and/or a monitoring, evaluation and learning (MEL) system. This helps to define the purpose and practical use of the indicators. The application of the indicator framework for EAS can stimulate joint learning and action at a systemic level. Analytical results are valuable to inform decision making among all actors in the EAS system. Regular reporting of monitoring data, case studies and transversal analysis across programmes or systems can generate accountability and comprehensive evidence to support national and global learning and facilitate the design of more effective EAS in the future. Knowledge gained through clear indicators and systematic MEL efforts can help to address not only the question of whether we are doing things right, but also if we are doing the right things. For instance, the data might confirm that greater inclusiveness – rather than the number of activities – is key to better farmer empowerment outcomes.

Analysis based on the indicator framework for EAS can be carried out with data for one
country or for a set of countries. With a single data point (data available for one country,
state or region), the analysis will be mostly descriptive with a focus on monitoring. This
implies tracking indicators across time and descriptively linking inputs, activities, outputs
and outcomes in a consistent narrative. As data points increase (data available for several
countries, states or regions), more advanced types of analysis can be performed. As
described below, these cover contribution as well as attribution analysis.
<figure><img src="image_37.png" /></figure>

The indicator framework for EAS can fulfil various analytical functions, particularly in terms of monitoring, evaluation, and assessment. It is essential to combine various sources of information and approaches to assess "what", "why" and "how" something changes (providing evidence on the extent of change and on the drivers as well as mechanisms of change). To this end, a flexible set of indicators is needed. While input, activity and output indicators are best-suited to monitoring, they are also relevant for impact evaluation. For instance, monitoring data on capacity development can be analytically linked with changes in outcome variables, such as the application of new knowledge and skills or improved access to services. Outcome indicators, especially for more immediate and intermediate results, can also be used for monitoring progress. In the case of long-term outcome indicators, and of course impact variables, impact evaluation gains prominence compared to the monitoring purpose. In the following, some fundamental distinguishing features of regular MEL and impact evaluation are provided.

## A. Monitoring, evaluation and learning

Monitoring consists of an ongoing or regular assessment that aims to provide indications to all stakeholders on the extent of progress on the implementation of an ongoing project, programme or policy. It can link EAS indicators to policy objectives and can be an integral part of a (sub-)national/international programme or initiative. Monitoring using the indicator framework should involve routine data collection and continuous tracking of progress based on core indicators, as well as complementary indicators, where possible. Moreover, monitoring as a process can be made more meaningful than the simple production of information on system performance, if participatory monitoring methods are applied, monitoring can be a way to encourage actors involved to think critically about their work. Furthermore, ongoing monitoring, which focuses on tracking implementation, covering input-, activity-, output- and outcome-level indicators, can provide a basis for good evaluations (Christoplos, Sandison and Chipeta, 2012). Evaluations, unlike monitoring, aim to appraise the results of ongoing or completed programmes, projects and policies, which generally means a stronger focus on outcomes, and in some cases impacts (see following subsection for more details on impact evaluation). Evaluations aim to identify causal linkages from inputs to results (at the outcome or impact level), examining the whole implementation process, resulting in insights that can be used for the learning function of the MEL system and thus integrated into future programme planning (Kusek and Rist, 2014).

Input, activities and output monitoring systems can be established for (sub)national systems, and together this data from national observatories can be used to track global trends in extension funding, personnel, use of ICTs, and inclusive access to EAS, among others. The EAS-Y scoring tool can be used to provide data on the outcome indicators that lend
themselves well to conducting routine outcome evaluations, for example evaluating progress in outcome areas such as the acquisition by EAS users of technical, social or entrepreneurial skills (all 0-3 scores). Moreover, comprehensive EAS assessments can be organized on a semi-regular basis (e.g. once in five years) as an essential component of a robust MEL system for EAS as they provide in-depth qualitative data as well as sources of data for the indicators.

<figure><img src="image_38.png" /></figure>

## BOX 2 Definitions of MEL

The following components make up a MEL framework:

**Monitoring:** 'A continuing function that uses systematic collection of data on specified indicators to provide management and the main stakeholders of an ongoing development intervention with indications of the extent of progress and achievement of objectives and progress in the use of allocated funds'. (OECD, 2002, p. 27-28).

**Evaluation:** 'Evaluation is the systematic and objective assessment of an on-going or completed project, program or policy, [and] its design, implementation and results. The aim is to determine the relevance and fulfilment of objectives, development efficiency, effectiveness, impact and sustainability. An evaluation should provide information that is credible and useful, enabling the incorporation of lessons learned into the decision-making process of both recipients and donors. Evaluation also refers to the process of determining the worth or significance of an activity, policy or program.' (OECD, 2002, p. 21-22)

**Learning:** 'Learning requires the development of systems and procedures to reflect and act on the information generated by monitoring and evaluation. This must be complemented by a learning environment that is inclusive and participatory. Such a learning environment should also be country-led and transparent' (Noltze et al., 2021, p. 12)

### Example of participatory MEL in practice

Ethiopia's agricultural extension performance management system (AEPMS), developed by Oxfam America and Ethiopia's Ministry of Agriculture and Natural Resources (MoANR) represents a uniquely participatory approach to MEL in agricultural extension. AEPMS uses a systematic, democratic and context-sensitive approach for continuous knowledge creation, management and action based on the principles of cost-effectiveness, timeliness and ease of use; it also generates data which allow for comparative performance analysis between different administrative levels. Data for the system are collected periodically from farmers and front-line extension workers on key aspects of extension system performance using a number of tools including: structured feedback directly from farmers on a number of areas such as quality of services, technologies, challenges, and so on and a tool to collect feedback from extension workers with a focus on improving their performance and job satisfaction. The AEPMS also uses a couple of tools to measure the outputs and outcomes of specific extension activities, for example by gauging how farmers are experiencing one or more new technologies (benefit they get from the technology introduced, challenges to adoption, and so on.) and using a micro survey tool to receive feedback from recipients of trainings and other kinds of services. Overall, the AEPMS represents a fully integrated cyclical method for collecting, analysing, reporting and collective sense making of extension data to address not only the "how" but also the "why" behind extension programming for continuous learning and improvement (MoANR, 2017).
## B. Impact evaluation

Impact evaluations are a type of evaluation which aim to provide evidence on what difference an EAS programme or system makes or has made (positive or negative) and to which EAS characteristics results can be attributed. In spite of some valid critiques of EAS impact evaluation studies (see, e.g. Faure et al., 2016; Taye, 2013) as discussed in Section 2, using the appropriate data and methods, impact evaluations can represent a useful tool in overall MEL for EAS. They need to be guided by clear evaluation questions based on a theory of change. In the context of the indicator framework for EAS, impact evaluation refers to both qualitative and quantitative research designs.

## Attribution analysis

Quantitative impact evaluations are targeted and rigorously assess impact once interventions show results (White and Raitzer, 2017). Rigorous impact evaluations involve determining cause-effect relationships, i.e., attribution analysis (Gertler et al., 2016; White and Raitzer, 2017). This approach allows the evaluation team to answer questions on whether the observed impact can be attributed to an EAS intervention by testing for statistically significant causal relationships and quantifying the magnitude of the effect of the intervention (treatment effect). Establishing the *counterfactual* (the situation if no intervention had happened) is the core challenge of attribution analysis. While the actual scenario is directly observed, the counterfactual is usually not. Consequently, sound impact evaluations require measurement with and without intervention (*treatment group and control/comparison group*), and ideally involve data collection before and after the interventions (*baseline and endline*). Baseline data is however not a must but helps to obtain more solid impact estimates. A comparison group is essential.

In the context of the indicator framework for EAS, impact evaluation with randomized assignment to groups (i.e., RCTs) is unlikely. The use of quasi-experimental methods can be considered. This would require a broad cross-country dataset including one or several result(s) of interest (*impact variable[s]*), one EAS intervention or a package of interventions (*treatment variable[s]*), and country-specific information on EAS and other characteristics (*control variable[s]*). Assessing the effect of an operational extension policy (in place vs. not in place) could be an example of such an evaluation. As random assignment is not realizable, a selection bias arises from self-selection (decision to put a policy in place). Attempting to correct for this bias entails the use of statistical methods (for example *matching or reweighting*) in order to form a comparison group (countries without extension policy) that is as similar as possible to the treatment group (countries with extension policy). While estimations with time-series information are more robust,
combination(s) of factors necessary or sufficient for realizing a particular impact and determine which factors within those combinations are most critical in which contexts. Data are calibrated to indicate if there is strong or weak evidence for each factor. Through a minimization process evaluation one can then identify and discard conditions that have little explanatory power, so patterns of relevant conditions emerge. QCA can, thus, give a nuanced understanding of how different programme and context elements lead to success or failure, supporting the identification of pathways for achieving change.

<figure><img src="image_39.png" /></figure>

BOX 3 Definitions and resources on impact evaluation

**Definition of impact:** 'The overall and long-term effect of an intervention. Impact is the longer-term or ultimate result attributable to a development intervention—in contrast to output and outcome, which reflect more immediate results from the intervention. The concept of impact is close to "development effectiveness". Examples: higher standard of living, increased food security, increased earnings from exports, increased savings owing to a decrease in imports.' (UNDP, 2002, p. 101)

**Definition of impact evaluation:** 'A type of evaluation that focuses on the broad, longer-term impact or results, whether intended or unintended, of a programme or outcome.' (UNDP, 2002, p. 101)

**Resources for impact evaluation for EAS:**

1) Review on evaluation of farmer participatory extension programmes (Knook, Brander and Moran, 2018)

2) Guide to evaluating rural extension (Christoplos, Sandison and Chipeta, 2012)

3) Impact Evaluation in Practice – Second Edition (Gertler et al., 2016)

4) Impact Evaluation of Development Interventions - A Practical Guide (White and Raitzer, 2017)

5) Broadening the range of designs and methods for impact evaluations (Stern et al., 2012)

6) Qualitative Comparative Analysis as an Evaluation Tool: Lessons From an Application in Development Cooperation (Pattyn, Molenveld and Befani, 2017)
8

Conclusions

After years of neglect, EAS are at the centre of global conversations about sustainable agriculture, resilient livelihoods and inclusive growth. However, there is very limited evidence on the properties, performance, outcomes and impact of pluralistic EAS systems, which to a large extent constrains them from receiving the due recognition and support (including financial, human resource and collaborative) they deserve from policy makers, clients and other stakeholders in the AIS. So far, most attempts to measure EAS performance, outcomes and impact have focused on public-sector EAS and production-oriented outcomes which are not sufficient, as most modern EAS systems have become more pluralistic and focused on several objectives beyond promoting adoption of new technologies to enhance food production.

In order to enhance the performance and ultimate impact of pluralistic EAS systems, it is essential to identify the key characteristics of pluralistic EAS providers, areas of low performance according to both providers and users of services and also factors contributing to this. There is also a need to enhance the collective capacities of EAS providers to facilitate better coordination of their activities. To successfully monitor and evaluate the performance of EAS, there is a need for data on multiple indicators from varied EAS stakeholders at different stages along the EAS ToC from inputs to activities, outputs, outcomes and impact.

The indicator framework presented here illustrates not only the type of data set that is needed at the national level to assess the performance, outcomes and impact of EAS, but also the tools for data collection and the potential sources of this data, including FAO's comprehensive EAS assessment guide and the EAS-Y scoring tool. Data on some of these indicators is not collected in most counties and data on some of the core indicators is available only from the public sector. Some data that is relevant to understanding the outcomes and impact of EAS is collected by other agencies in the form of special studies or in specific data sets. However, there have been no concerted
efforts made to consolidate and relate the available data to be used for EAS assessment and evaluation or collect additional data, as the importance of doing this has not been fully articulated and because the relevant indicators, sources of data and the methods of collecting and analysing the same have not been promoted. The development of this indicator framework is an attempt to fill this gap.

Lack of data on EAS is not a new issue but the increasing pluralism in EAS delivery and decentralization of EAS in different countries pose new challenges in collecting data. In light of this, efforts are needed at the national level (and subnational levels too) or at the country EAS networks (GFRAS country fora) level to establish "observatories" to collect these data at regular intervals so that better policy and investment decisions can be made to enhance the relevance and contribution of EAS to its clients. We live in a world that is increasingly data-driven and thus efforts should be made to generate meaningful data on at least the 10 core indicators every year. Beyond the secondary data sources, efforts should also be made to use tools such as EAS-Y to collect data on performance and outcomes of EAS at the national and territorial levels and to organize comprehensive EAS assessments at least once in five years.

The holistic perspective provided by the indicator framework provides a strong mechanism to foster accountability, facilitate pluralism in EAS provision, assess human resource capacities and achieve financial sustainability. Moreover, its basis on a theory of change clearly tracing performance from input to impact, incorporation of participatory methods for data collection, use of both quantitative and qualitative methods, guidance on potential data sources, and finally, proposed observatories at the national and eventually global level including digital dashboards to manage and make data widely accessible are among the strong and unique features of this indicator framework. Having shared national observatories on EAS will hopefully facilitate participatory MEL, improve the performance and the eventual impact of EAS at the national level. However, these efforts will require resources to collect, store and share data and dedicated human resources with capacities to organize this system. At the global level, these combined efforts can potentially lead to globally pooled budgets to strengthen EAS capacities where they need improvement and more collaborative actions among nations to strengthen EAS.
References

**Adolph, B.** 2011. *Rural Advisory Services Worldwide: A synthesis of actors and issues*. In Revised edn. Global Forum for Rural Advisory Services. Lindau, Switzerland.

**ALINE (Agricultural Learning and Impacts Network).** 2019. *Improving Agricultural Advisory Systems Metrics and Learning, meeting summary*. April 8–9, Addis Ababa, Ethiopia.

**Alston, J. M., Chan-Kang, C., Marra, M. C., Pardey, P. G., & Wyatt, T. J.** 2000. *A meta-analysis of rates of return to agricultural R&D: Ex pede Herculem?* (Vol. 113). Washington, DC: International Food Policy Research Institute (IFPRI).

**Befani, B.** 2016. *Pathways to change: Evaluating development interventions with QCA*. Report for the Expert Group for Aid Studies-EBA. Expert Group for Aid Studies. Report 05/16. Stockholm, Sweden: EBA.

**Birkhaeuser, D., Evenson, R.E. & Feder, G.** 1991. *The economic impact of agricultural extension: A review. Economic development and cultural change*. 39(3), 607-650.

**Birner, R., Davis, K., Pender, J., Nkonya, E., Anandajayasekeram, P., Ekboir, J., Mbabu, A., Spielman, D.J., Horna, D., Benin, S. & Cohen, M.** 2009. *From best practice to best fit: a framework for designing and analyzing pluralistic agricultural advisory services worldwide*. Journal of agricultural education and extension, 15(4), 341-355.

**Blum, M.L., Cofini, F. & Sulaiman, R.V.** 2020. *Agricultural extension in transition worldwide: Policies and strategies for reform*. Rome, FAO.

**Blum, M. L. & Szonyi, J.** 2014. *Investment requirements in extension to achieve zero hunger and adapt to climate change*. Journal of Agricultural Science and Technology. A, 4(7A).

**Buehren, N., Goldstein, M., Molina, E., & Vaillant, J.** 2019. *The impact of strengthening agricultural extension services on women farmers: Evidence from Ethiopia*. Agricultural Economics, 50(4), 407-419.

**Chipeta, S.** 2019. *Draft Report Development of indicators for Extension and Advisory Services: Background Study* (unpublished).

**Christoplos, I.** 2010. *Mobilizing the potential of rural and agricultural extension*. Food and Agriculture Organization (FAO) Rome (Italy) Office of Knowledge Exchange; The Global Forum For Rural Advisory Services (GFRAS) (also available at http://agris.fao.org/agris-search/search.do?recordID=XF2016015170).

**Christoplos, I., Sandison, P. & Chipeta, S.** 2012. *Guide to evaluating rural extension* (No. C20-38). Global Forum for Rural Advisory Services (GFRAS).

**Coutts, J., Koutsouris, A. & Davis, K.** 2019. *Evaluation of rural advisory and extension services*. The Journal of Agricultural Education and Extension, 25:2, 99-101.
Davis, K., Babu, S.C. & Ragasa, C. 2020. *Agricultural Extension: Global Status and Performance in Selected Countries*. Washington, DC, International Food Policy Research Institute (IFPRI).

Dhehibi, B., Werner, J. & Qaim, M. 2018. *Designing and Conducting Randomized Controlled Trials (RCTs) for Impact Evaluations of Agricultural Development Research: A Case Study from ICARDA's 'Mind the Gap' Project in Tunisia*. Manuals and Guidelines 1. Beirut, Lebanon: The International Center for Agricultural Research in the Dry Areas (ICARDA).

Djamen, P., Davis, K. & Sacko, L. 2020. *Strengthening Partnerships and Professionalization in Agricultural Extension in Guinea*. Developing Local Extension Capacity Project. Washington DC, USAID.

Evenson, R. 1997. Chapter 4—The economic contributions of agricultural extension to agricultural and rural development. In Improving agricultural extension. In Swanson, E. B., Bentz, R. P., & Sofranko, A. J. *Improving Agricultural Extension: A Reference Manual*, pp. 27–36. Rome, FAO. (also available at http://www.fao.org/3/w5830e/w5830e06.htm)

FAO. 2020. *Transforming agricultural research and extension systems. Unlocking the potential of agricultural innovation to achieve the Sustainable Development Goals*. Rome (also available at www.fao.org/3/ca8737en/ca8737en.pdf)

FAO & INRAE. 2020. *Enabling sustainable food systems: Innovators' handbook*. Rome (also available at https://doi.org/10.4060/ca9917en).

Faure, G., Davis, K.E., Ragasa, C., Franzel, S. & Babu, S.C. 2016. *Framework to assess performance and impact of pluralistic agricultural extension systems: The best-fit framework revisited* (Vol. 1567). Washington DC, International Food Policy Research Institute (IFPRI).

Feder, G., Murgai, R., & Quizon, J. 2003. Sending Farmers Back to School: The Impact of Farmer Field Schools in Indonesia. Washington DC, The World Bank. (also available at https://doi.org/10.1596/1813-9450-3022).

Gertler, P., Martinez, S., Rawlings, L.B., Premand, P. & Vermeersch, C.M. J. 2016. *Impact Evaluation in Practice*. Washington DC, The World Bank.

Glover, D, Sumberg J, Gile Ton, Jens Andersson & Lone Badstue. (2019). *Rethinking technological change in small holder agriculture*. Outlook on Agriculture 48 (3)-179-180.

Grovermann, C., Wossen, T., Müller, A. & Nichterlein, K. 2019. *How can Agricultural Innovation Systems Enhance the Eco-efficiency of Farming in Developing Countries? PLoS ONE*, 14(4): e0214115.

Joshi, P.K., Kumar, P. & Parappurathu, S. 2015. *Public investment in agricultural research and extension in India*. The European Journal of Development Research, 27(3), 438-451.

Knook, J., Eory, V., Brander, M. & Moran, D. 2018. *Evaluation of farmer participatory extension programmes*. The Journal of Agricultural Education and Extension, 24(4), 309-325.
**Kusek, J.D. & Rist, R.C.** 2014. *A Handbook for Development Practitioners*. Ten Steps to a Results-Based Monitoring and Evaluation System. Washington DC, The World Bank.

**Makate, Clifton & Makate, Marshall.** 2019. *Interceding role of institutional extension services on the livelihood impacts of drought tolerant maize technology adoption in Zimbabwe*. Technology in Society, Elsevier, vol. 56(C), pages 126-133.

**Ministry of Agriculture and Natural Resources (MoANR).** 2017. *Agricultural Extension Strategy of Ethiopia* (also available at http://www.moa.gov.et/documents/45198/0/FINAL+EXTENSI ON+STRATEGY+EDITED+FOR+Upload.pdf/51050623-b954-46cf-bea3-aaefece29408).

**Nandi, R. & Swamikannu, N.** 2019. *Agriculture Extension System in India: A Meta-analysis*. Agricultural Science Research Journal. 10. 473-479 (also available at https://www.researchgate.net/ publication/333842261_Agriculture_Extension_System_in_India_A_Meta-analysis).

**NITI Aayog.** 2020. *India Voluntary National Report - Decade of Action Taking SDGs From Global to Local*. NITI Aayog, Government of India, New Delhi (also available at http://niti.gov.in/sites/ default/files/2020-07/26281VNR_2020_India_Report.pdf).

**Noltze, M., Köngeter, A., Römling, C. & Hoffmann, D.** 2021. *Monitoring, evaluation and learning for climate risk management*. OECD Development Cooperation Working Paper No 92. OECD Publishing. Paris.

**The Organization for Economic Co-operation and Development (OECD).** 2002. *Glossary of Key Terms in Evaluation and Results-based Management*. Development Assistance Committee. Paris, OECD.

**Pattyn, V., Molenveld, A. & Befani, B.** 2017. *Qualitative Comparative Analysis as an Evaluation Tool: Lessons From an Application in Development Cooperation*. American Journal of Evaluation 1-20. (DOI: 10.1177/1098214017710502).

**Prager, K. & R, Creaney.** 2017. *Achieving on-farm practice change through facilitated group learning: Evaluating the effectiveness of monitor farms and discussion groups*. Journal of Rural Studies, 56: 1-11 (also available at https://doi.org/10.1016/j.jrurstud.2017.09.002).

**Ragasa, C.** 2014. *Improving gender responsiveness of agricultural extension*. In Gender in agriculture (pp. 411-430). Springer, Dordrecht.

**Restrepo, M.J., Lelea, M.A. & Kaufmann, N.A.** 2018. *Evaluating knowledge integration and co-production in a 2-year collaborative learning process with smallholder dairy farmer groups*. Sustainability Science. 13:1265-1286.

**Reddy A Amarender.** 2018. *Impact Study of Soil Health Card Scheme, National Institute of Agricultural Extension Management (MANAGE), Rajendranagar, Hyderabad – 500030, Telangana State, India*. 106 pp.

**Stern, E., Stame, N., Mayne, J., Forss, K., Davies, R., & Befani, B.** 2012. *Broadening the range of designs and methods for impact evaluations*. Department for International Development (DFID). Working Paper 38). London, UK, DFID.
**Sulaiman, R. & Davis, K.** 2012, November. *The New Extensionist: Roles, strategies, and capacities to strengthen extension and advisory services*. In Lindau, Switzerland: Global Forum for Rural Advisory Services (GFRAS).

**Sulaiman, R. V., & Sadamate, V. V.** 2000. *Privatizing Agricultural Extension in India*, policy paper 10. NCAP, New Delhi (also available at https://niap.icar.gov.in/pdf/pp10%20new.pdf).

**Surbhi, S.** 2017. Difference Between Assessment and Evaluation. keydifferences.com (also available at https://keydifferences.com/difference-between-assessment-and-evaluation.html).

**Swanson, B. E. & Rajalahti R.** 2010. *Strengthening agricultural extension and advisory services*. Washington DC, The World Bank, ARD Paper 45.

**Taye, H.** 2013. *Evaluating the impact of agricultural extension programmes in sub-Saharan Africa: Challenges and prospects*. African Evaluation Journal 1(1), Art. #19, 9 pp (also available at http://dx.doi.org/10.4102/aej.v1i1.19).

**Teyssier, J. (ed.), Rigourd, C. & Dugué, P.** 2019. *Relaunch agricultural advice and extension in sub-Saharan Africa. For new policies in line with the realities on the ground*. Paris: AFD, 121 pp. (Technical notes: AFD, 55).

**United Nations Development Programme (UNDP).** 2002. *Handbook on monitoring and evaluating for results*. Evaluation Office.

**White, H. & Raitzer, D.** 2017. *Impact Evaluation of Development Interventions - A Practical Guide*. Asian Development Bank, Manila.

**World Bank.** 2012. *Process evaluations. Results of the Expert Roundtables on Innovative Performance Measurement Tools*. Brief #4. Washington DC, The World Bank. (also available at http://documents.worldbank.org/curated/en/919551468265788025/Process-evaluations).

**World Bank.** 2019. *Enabling the Business of Agriculture 2019*. Washington DC, The World Bank.
Glossary of terms

**Actors:** individuals and organizations (for example civil society, private sector, enterprises, government ministries, non-governmental organizations, research and development institutes, extension services, universities and vocational training centres, etc.).

**Agricultural innovation:** the process whereby individuals or organizations bring new or existing products, processes or ways of organization into use for the first time in a specific context, to increase effectiveness, competitiveness and resilience, with the goal of solving a problem.

**Agricultural innovation system (AIS):** a network of actors (individuals, organizations and enterprises), together with supporting institutions and policies in the agricultural and related sectors, which facilitate the process of agricultural innovation. Policies and institutions (formal and informal) play a key role in shaping the way that the AIS actors interact, generate, share and use knowledge, as well as jointly learn.

**Evaluation:** the systematic and objective assessment of an on-going or completed project, program or policy, [and] its design, implementation and results. The aim is to determine the relevance and fulfilment of objectives, development efficiency, effectiveness, impact and sustainability. An evaluation should provide information that is credible and useful, enabling the incorporation of lessons learned into the decision-making process of both recipients and donors. Evaluation also refers to the process of determining the worth or significance of an activity, policy or program (OECD, 2002, p. 21-22).

**Extension and advisory services (EAS):** EAS consist of all the different activities that provide the information and services needed and requested by farmers and other actors in rural settings to assist them in developing their own technical, organizational and management skills and practices to improve their livelihoods, as well as promote more sustainable agriculture. They include the diversity of actors involved in the provision of EAS, and the broad support provided to rural communities (beyond information and knowledge), embracing new functions such as facilitation, intermediation, advice and brokering (Sulaiman and Davis, 2012; Christoplos, 2010).
**Learning:** the development of systems and procedures to reflect and act on the information generated by prior monitoring and evaluation. This must be complemented by a learning environment that is inclusive and participatory. Such a learning environment should also be country-led and transparent (Noltze et al., 2021, p. 12)

**Monitoring:** a continuing function that uses the systematic collection of data on specified indicators to provide management and the main stakeholders of an ongoing development intervention with indications of the extent of progress and achievement of objectives and progress in the use of allocated funds (OECD, 2002, p. 27-28).
<figure><img src="image_40.png" /></figure>
Extension and advisory services (EAS) play a key role in facilitating innovation for sustainable agricultural development. To strengthen this role, appropriate investment and conducive policies are needed in EAS, guided by evidence. It is therefore essential to examine EAS characteristics and performance in the context of modern, pluralistic and increasingly digital EAS systems. In response to this need, the Food and Agriculture Organization of the United Nations (FAO) has developed guidelines and instruments for the systematic assessment of national EAS systems. The indicator framework provides overarching guidance on EAS systems assessment, including a list of 40 indicators (10 core and 30 complementary) which cover all major aspects of EAS from inputs to impact. This indicator framework provides much needed structure to EAS assessment, taking into account contemporary, pluralistic services and is complemented by FAO's instruments for participatory data collection in EAS, including quantitative and qualitative data.

www.fao.org/research-and-extension

Food and Agriculture Organization of the United Nations
Rome, Italy

ISBN 978-92-5-135677-7

<figure><img src="image_41.png" /></figure>

9

789251 356777

CB8409EN/1/02.22