PRISMTOOLKIT FOR EVALUATINGTHE OUTCOMES AND IMPACTS
OF SMALL/MEDIUM-SIZEDCONSERVATIONPROJECTS

Dickson, I. M., Butchart, S. H. M., Dauncey, V.,
Hughes, J., Jefferson, R., Merriman, J. C.,
Munroe, R., Pearce-Higgins, J. P.,
Stephenson, P. J., Sutherland, W. J.,
Thomas, D. H. L., & Trevelyan, R., (2017)
PRISM – Toolkit for evaluating the outcomes
and impacts of small/medium-sized
conservation projects. Version 1.

0:2

TERMS OF USE

Users can reproduce content based on the information in thistoolkit for non-commercial purposes, so long as appropriate credit
is given (see front cover for suggested citation). The exceptions
to this are materials that have been reproduced/adapted fromother sources, in these cases the necessary permissions should besought from the original source.

PRISM
TOOLKIT FOR EVALUATINGTHE OUTCOMES AND IMPACTS
OF SMALL/MEDIUM-SIZEDCONSERVATIONPROJECTS

Dickson, I. M., Butchart, S. H. M., Dauncey, V.,
Hughes, J., Jefferson, R., Merriman, J. C.,
Munroe, R., Pearce-Higgins, J. P.,
Stephenson, P. J., Sutherland, W. J.,
Thomas, D. H. L., & Trevelyan, R., (2017)
PRISM – Toolkit for evaluating the outcomes
and impacts of small/medium-sized
conservation projects. Version 1. Available
from www.conservationevaluation.org

0:3

INTRODUCTION &KEY CONCEPTS

DESIGNING & IMPLEMENTING
THE EVALUATION

Acknowledgements:
Principal Authors:
Iain Dickson (Birdlife International)
Stuart Butchart (BirdLife International)
Victoria Dauncey (Fauna and Flora International)
Joelene Hughes (Royal Society for the Protection of Birds)
Rebecca Jefferson (Royal Society for the Protection of Birds)
Jenny Merriman (BirdLife International)
Robert Munroe (UNEP-WCMC)James Pearce Higgins (British Trust for Ornithology)
David Thomas (BirdLife International)
PJ Stephenson (WWF International)
Bill Sutherland (University of Cambridge/Conservation Evidence)
Rosie Trevelyan (Tropical Biology Association)

PRISM has been developed by:

3

PRISM
MODULESMETHOD FACTSHEETS& ANNEXES

Reviewers/Contributing Authors:
Tris Allinson , Helen Anthem, Tomos Avent, Will Beale, Hannah Becker,
Rick Davies, Abigail Entwistle, Kay Farmer, Tiburtius Hani, Nicolas Heard,
Mike Hoffmann, Lisa M Howes, Julia Jones, Nigel Leader-Williams,
Katie Lee Brooks, Nina Marshall, Simon Mickleburgh, Kiragu Mwangi,
Nisha Owen, Victoria Price, Dave Pritchard, Dilys Roe, Roger Safford,
Nick Salafsky, Chris Sandbrook, Sarah Thomas, David Wallis, Emily
Woodhouse, Claire Wordley.
Particular thanks also to the Zoological Society of London (ZSL), theRufford Foundation, the Conservation Leadership Programme (CLP),
participants of the PRISM expert workshop, held in April 2016, thestudents from the University of Cambridge MPhil in conservation
leadership (2016/17) and all others involved in field-testing methods,
reviewing drafts and providing additional content.

With Financial support from:

PRISM - TOOLKIT FOR EVALUATING THE OUTCOMES AND IMPACTS OF SMALL/MEDIUM-SIZED CONSERVATION PROJECTS

0:4

INTRODUCTION &KEY CONCEPTS

DESIGNING & IMPLEMENTING
THE EVALUATIONThis toolkit is divided into three main sections:

Section 1

introduces the toolkit and outlines some of the
key concepts relating to evaluation of outcomes
and impacts that are likely to be relevant to small/medium-sized conservation projects.

Section 2

is designed to guide you through a step-by-stepprocess for evaluating project outcomes andimpacts.

Section 3

provides detailed guidance and methods tomeasure outcomes and impacts relating to each of
PRISM’s five thematic modules, each one coveringa different category of conservation action.

OVERVIEW

have been used throughout the PRISM Toolkit toaid navigation. Solid colour icons indicate active
page content and greyed out indicates an inactive
state. The Contents icon is active throughout.

Icons

Contents

4

Required
steps

Examplesare shown

Tips
available

Checklists
to complete

Downloads
available

PRISM
MODULESMETHOD FACTSHEETS& ANNEXES

Sections 1 & 2 should be read before the information in Section 3.
However, we are aware that some users will already have a goodidea of what they want to evaluate and will be seeking guidanceon specific evaluation needs for certain actions. Where this is thecase, users are able to go directly to the appropriate module ofSection 3 to identify suitable methods. In the interactive version ofthis toolkit, you can use the relevant hyperlinks to navigate to therelevant pages.
Methods are laid out in a factsheet format. The factsheets include
general methods (applicable to all modules) and methods thatare specific to a particular module. Factsheets are included in theannexes and via hyperlinks within the interactive version of thetoolkit.
Whilst trying to make this resource as generically applicable aspossible, we recognise that the guidance and methods may need tobe adapted to specific projects and/or local context.
For ease of use we would highly recommend adding the “previousview” button to your PDF viewer. In Adobe Acrobat you can dothis by right clicking on the toolbar, select page navigation tools
and make sure that there is a tick next to “previous view”. This
will add a back button to the toolbar. Pressing alt+right arrowprovides the same function.

Available
as Excel

PRISM - TOOLKIT FOR EVALUATING THE OUTCOMES AND IMPACTS OF SMALL/MEDIUM-SIZED CONSERVATION PROJECTS

0:5

INTRODUCTION &KEY CONCEPTS

I understand the theory behindevaluating project outcomes/
impacts and why it is important

DECISION TREEDESIGNING & IMPLEMENTING
THE EVALUATION

PRISMMODULESMETHOD FACTSHEETS& ANNEXESNO

Go to Introduction & Key Concepts

NO

Go to STEP 1: What do you need to know?

YES

I know what I need to evaluate

YES

I have already collected
my evaluation data

NO

Go to STEP 2: Whatdata do you need to collect?

&

PRISM Modules

YES

I have analysed my evaluation
data and interpreted the resultsNO

Go to STEP 3: What can you learn from the results?

NO

Go to STEP 4: What should be done next?

YES

I have my evaluation
results and I know how theyshould be applied

5

PRISM - TOOLKIT FOR EVALUATING THE OUTCOMES AND IMPACTS OF SMALL/MEDIUM-SIZED CONSERVATION PROJECTS

0:6

INTRODUCTION &KEY CONCEPTS

Section 1

DESIGNING & IMPLEMENTING
THE EVALUATION

Introduction and Key Concepts

PRISM
MODULES

Section 2

METHOD FACTSHEETS& ANNEXES

Step by step guide for evaluating project
outcomes and impacts

What is this toolkit for?

CONTENTS

How to use this toolkit to evaluate your project’s
outcomes and impacts

Who can use this toolkit?
When to use this toolkit

STEP 1

What do we mean by evaluation?

What is the project trying to achieve?

What do we mean by outcomes and impacts?

Why do you want to evaluate?

Why focus on outcomes & impacts?
Does the project’s design allow for evaluation ofoutcomes/impacts?

What questions do you need to answer?
STEP 2

What data do you need to collect?
How will you know which outcomes/impacts canbe attributed to the project?

Unintended outcomes & impacts
Was the change recorded caused by the project?
Be realistic and focus on interpreting, rather thanjust presenting results

What do you need to know?

How will you collect evaluation data?
STEP 3What can you learn from the results?
How will you analyse your data?

Tips for effective evaluation in small/medium-sized
projects

What do the results show?
STEP 4

What should be done next?
How will you use the evaluation results?

6

PRISM - TOOLKIT FOR EVALUATING THE OUTCOMES AND IMPACTS OF SMALL/MEDIUM-SIZED CONSERVATION PROJECTS

0:7

INTRODUCTION &KEY CONCEPTS

Section 3

CONTENTSDESIGNING & IMPLEMENTING
THE EVALUATION

PRISMMODULES

Thematic Modules

Method Factsheets and Annexes

Attitudes & Awareness

General Method Factsheets
Completing a Theory of Change

METHOD FACTSHEETS& ANNEXES

Identifying why you want to evaluate
Capacity Development

Evaluation feasibility checklist
Completing a prioritisation matrix

Livelihoods & GovernanceCompleting an evaluation plan
Developing ethical protocols for an evaluation

PolicyDeveloping gender protocols for an evaluation
Entering, organising and cleaning data

Species & Habitat Management

Analysing quantitative data
Analysing qualitative data
Evaluation report template
Evaluating outcomes and impacts of projects aiming tomaintain or increase the values of ecosystem services
Questionnaires
Key informant interviews
Focus groups
Direct observation
Documentary evidence

7

PRISM - TOOLKIT FOR EVALUATING THE OUTCOMES AND IMPACTS OF SMALL/MEDIUM-SIZED CONSERVATION PROJECTS

0:8

INTRODUCTION &KEY CONCEPTS

DESIGNING & IMPLEMENTING
THE EVALUATION

Evaluation Design Factsheets
Participatory Impact Assessment

CONTENTS

PRISM
MODULESMETHOD FACTSHEETS& ANNEXES

Livelihoods & Governance Factsheets and Annexes
Participatory Impact Assessment (PIA)

Interviewing key informantso

PIA Method 1: Before and after scoring

Theory-based designso

PIA Method 2: Proportional piling

Matching designso

PIA Method 3: Tally method

Before and after designso

PIA Method 4: Matrix scoring

Decision tree for selecting an evaluation designo

PIA Method 5: Before and after impact calendars

Community-based Organisational Capacity Assessment
Attitudes & Awareness Factsheets and Annexes
Sample questions and scales for evaluating attitudes andawareness actions
Planning a successful attitudes/awareness action

Community mapping
Basic Necessities Survey (BNS)
Participatory Governance Assessment
Questions for assessment meeting

Capacity Development Factsheets and Annexes
Training evaluation form: example questions and scales

Governance Assessment scoring sheet

Organisational Capacity Assessment tool
Network health scorecard

8

PRISM - TOOLKIT FOR EVALUATING THE OUTCOMES AND IMPACTS OF SMALL/MEDIUM-SIZED CONSERVATION PROJECTS

0:9

INTRODUCTION &KEY CONCEPTS

DESIGNING & IMPLEMENTING
THE EVALUATION

Policy Factsheets and Annexes
Media tracking

CONTENTS

PRISM
MODULES

Field assessment methods for evaluating changes in habitat
statusMedia scorecards

Field methods for species monitoring

Observation checklist for documenting meetings

Sampling approaches for evaluating changes in species status

Policymaker ratings

Analytical issues when estimating abundance, distribution,
survival and reproduction

Bellwether methodology
Civil society tracking toolSpecies & Habitat Management Factsheets and Annexes
Scorecard for evaluating knowledge gaps

Sampling approaches for habitat monitoring

Glossary

Scorecard for degree of completion of an action plan
Scorecard for adequacy of an action plan
Threat reduction scoring
Scorecard for evaluating changes in species status
Scorecard for evaluating changes in habitat status
Field surveys for evaluating changes in species’ status
Remote assessment methods for evaluating changes inhabitat status

9

METHOD FACTSHEETS& ANNEXESPRISM - TOOLKIT FOR EVALUATING THE OUTCOMES AND IMPACTS OF SMALL/MEDIUM-SIZED CONSERVATION PROJECTS1:0

SECTION 1
INTRODUCTION &KEY CONCEPTS

10PRISM - TOOLKIT FOR EVALUATING THE OUTCOMES &ANDIMPACTS
IMPACTSOF OF
SMALL/MEDIUM-SIZEDSMALL/MEDIUM-SIZEDCONSERVATIONCONSERVATIONPROJECTSPROJECTS

1:0

INTRODUCTION &KEY CONCEPTS

Section 1

DESIGNING & IMPLEMENTING
THE EVALUATION

Introduction and Key Concepts
What is this toolkit for?

CONTENTSSECTION 1

PRISM
MODULESThis section introduces the toolkit and outlines some of the key
concepts relating to evaluation of outcomes and impacts that arelikely to be relevant to small/medium-sized conservation projects.

Who can use this toolkit?
When to use this toolkit
What do we mean by evaluation?
What do we mean by outcomes and impacts?
Why focus on outcomes & impacts?
Does the project’s design allow for evaluation ofoutcomes/impacts?
Unintended outcomes & impacts
Was the change recorded caused by the project?
Be realistic and focus on interpreting, rather thanjust presenting results
Tips for effective evaluation in small/medium-sized
projects

11

METHOD FACTSHEETS& ANNEXESPRISM - TOOLKIT FOR EVALUATING THE OUTCOMES AND IMPACTS OF SMALL/MEDIUM-SIZED CONSERVATION PROJECTS

1:1

INTRODUCTION &KEY CONCEPTS

WHAT IS THISTOOLKIT FOR?

DESIGNING & IMPLEMENTING
THE EVALUATION

What is this toolkit for?
Conservationists are increasingly looking to improve the ways inwhich they evaluate the outcomes and impacts of their work. Goodevaluation practice is essential for demonstrating achievements,
avoiding wasted effort and modifying projects to improve delivery.
In addition, sharing evidence of what works and what doesn’tallows other conservationists to learn from experience, makedecisions and design projects and actions that are better informedby available evidence.
Existing monitoring frameworks used by small to medium sized
conservation projects often focus on how well the project hasdelivered on actions and outputs (what the project has done);however, these do not always effectively evaluate the outcomes
and impacts of these actions (the short, medium and longterm changes brought about by the project). Equally, it is often
extremely challenging for smaller projects to establish theirlong-term conservation impact, as this may take years to become
measurable.
This toolkit aims to help practitioners overcome these challenges
by describing some of the practical approaches and methods
that can be used to evaluate the outcomes and impacts of small/medium-sized conservation projects. Overall, PRISM aims to help
practitioners to go beyond only measuring actions & outputs, andto begin to evaluate outcomes & impacts, in a way that promotes
learning, while still remaining within the capacity and resource
limits of the project team.

12

PRISM
MODULESMETHOD FACTSHEETS& ANNEXES

SPECIFICALLY, THIS TOOLKIT IS DESIGNED TO HELPYOU ANSWER THE FOLLOWING QUESTIONS:

What do you need to know?
Identify which of the project’s outcomes/impacts to focus
on in an evaluation, and the questions you will try toanswerWhat data to you need to collect?
Design your data collection and select appropriate
methods that will provide you with the data you need toanswer your evaluation questions
What can you learn from the results?
Examine your data to provide answers to your evaluationquestions, determine what lessons can be learnt andwhat the results mean in relation to the overall aim of the
project
What should be done next?
Apply results to improve practice and share results thewider conservation community

PRISM - TOOLKIT FOR EVALUATING THE OUTCOMES AND IMPACTS OF SMALL/MEDIUM-SIZED CONSERVATION PROJECTS

1:2

INTRODUCTION &KEY CONCEPTS

WHO CANUSE THISTOOLKIT?

DESIGNING & IMPLEMENTING
THE EVALUATION

Who can use this toolkit?
This toolkit has been designed primarily for use by practitioners
carrying outsmall/medium-sized conservation projects and those
who work to support these projects. Many evaluation methods
require considerable investment in time and resources, making
them unsuitable for small/medium-sized projects. This toolkit
focuses instead on some of the methods and approaches that are
more relevant for smaller projects.
There is no set definition for a small/medium-sized project. As arough guide this toolkit has been designed to apply to projects withsome or all of the characteristics listed below (although much of
the guidance it contains will be appropriate for other projects too).
•

Budget between $5,000 - $100,000•

Short timeframe (<5 years)•

Small project team, often with limited capacity inrelation to evaluation•

Limited resources available for evaluation•

Projects taking place in complex environments, where
it can be difficult to separate project outcomes andimpacts from other factors.

PRISM
MODULESMETHOD FACTSHEETS& ANNEXES

There is also no set requirement for how much a small/mediumsized project should evaluate. For some projects minimalevaluation of specific actions may be sufficient, whereas others
would be better off investing considerable time and resources inunderstanding some of the specific change(s) brought about by the
project.
A key point to remember is that small/medium-sized projects
should not expect, or be expected to carry out a full andcomprehensive evaluation of all their outcomes and impacts.
How you evaluate will depend on your time, your budget andon what you and your audience need to know. This toolkit isdesigned to help you develop and carry out an appropriate
evaluation based on these criteria – including carefulconsideration of the project and its design, identificationof the key elements to be evaluated and selection of
appropriate methods for collecting the information needed.

Photo credit: Iain Dickson

13

PRISM - TOOLKIT FOR EVALUATING THE OUTCOMES AND IMPACTS OF SMALL/MEDIUM-SIZED CONSERVATION PROJECTS

1:3

INTRODUCTION &KEY CONCEPTS

WHEN TOUSE THISTOOLKITDESIGNING & IMPLEMENTING
THE EVALUATION

When to use this toolkit
It is useful to think of conservation action as a cyclical process,where learning and experience is captured and used to informfuture decision making. Evaluating your project’s outcomes andimpacts allows you to complete the cycle by looking at what
difference the project has made, what the key lessons are andwhat actions should be taken next. Above all, evaluation should
not be seen as a standalone activity - instead it should be seen as
an essential component which allows you to complete the project
cycle.

FIGURE 1: CONSERVATION PROJECT CYCLE

Plan
project
Capture results,
learn and shareexperience

PRISM
MODULESTHIS TOOLKIT CAN BE USED AT THE FOLLOWINGPOINTS IN THE PROJECT CYCLE:
POINTS IN THE PROJECT CYCLEUSE THIS TOOLKIT TO

Plan project

Identify the key outcomes/
impacts to evaluate and how
data will be collected

Implement projectCollect the data required

Capture results, learn & share

Decide how to analyse, interpret,

experiencelearn from and communicate
results

In an ideal world, all projects would build in evaluation from thestart. We recognise however that many users may be picking
up this toolkit mid-way through a project, towards the end of a
project or after the project has ended. If this is the case, then therecommended steps (see Section 2) are still the same, althoughnote that assessments can be more challenging if evaluation is not
built into the project from the start (note also that some methods
will only apply to projects that have considered evaluation at theoutset).

Implement
project

14

METHOD FACTSHEETS& ANNEXESPRISM - TOOLKIT FOR EVALUATING THE OUTCOMES AND IMPACTS OF SMALL/MEDIUM-SIZED CONSERVATION PROJECTS

1:4

INTRODUCTION &KEY CONCEPTS

WHAT DO WE
MEAN BY
EVALUATION?

DESIGNING & IMPLEMENTING
THE EVALUATION

What do we mean by evaluation?
Evaluation involves looking at a project, or specific elements ofa project, to understand what difference it has made and what
lessons have been learnt. Evaluation can look at a project of any
size and at any stage; however this toolkit is specifically concernedwith how to evaluate the outcomes and impacts of small/mediumsized conservation projects.

PRISM
MODULESIn order to ensure that you are making the best use of your timeand resources it is usually necessary to include a step before you
begin collecting data, where you consider what questions need
to be answered, and use these to guide your data collection, yourinterpretation of results and your application of those results.
This toolkit is designed to help you design and implement anevaluation to reflect this process:Evaluating project outcomes and impacts is not just aboutmeasuring “success”. It is about measuring change - which canbe positive or negative, intended or unintended (see Figure 4).
This change can be brought about by a single project action or anumber of different actions.

STEP 1

Start by identifying what you need to evaluate,
consider the project’s design and identify thequestions that you need to answer.

What about monitoring and learning?

STEP 2

The questions from Step 1 then inform what data
you need to collect, when they will need to becollected and the methods you will use.

STEP 3

These data then inform your interpretation of
the results and the judgements you make whenanswering your questions.

STEP 4

The judgements in Step 3 are then used to develop
recommendations, inform practice and shareresults.

The term monitoring is often used alongside evaluation (often asa single phrase – Monitoring & Evaluation or M&E). Increasingly
organisations also include learning (Monitoring, Evaluation &Learning or MEL).
Rather than treating these elements separately, it is often useful
to view them as parts of the same process: monitoring refers to
the collection of data over the course of the project, evaluation iswhen you examine this data to make a judgement about an aspectof the project, and learning is when you use these judgements tomake decisions about what could or should be done next.
Evaluation needs relevant data. That said, collecting data withouta purpose in view (making judgements, applying results) is a wasteof time and resources that could be used elsewhere in the project.

15

METHOD FACTSHEETS& ANNEXESPRISM - TOOLKIT FOR EVALUATING THE OUTCOMES AND IMPACTS OF SMALL/MEDIUM-SIZED CONSERVATION PROJECTS

1:5

WHAT DOWE MEAN BYOUTCOMES AND
IMPACTS?

INTRODUCTION &KEY CONCEPTS

DESIGNING & IMPLEMENTING
THE EVALUATION

What do we mean by outcomes and impacts?
When planning to evaluate a conservation project it is useful tothink of the project as a series of different stages. Throughout thistoolkit we use the following definitions and framework as shown in
Figure 21 to refer to these stages.
Actions – What the project does to bring about change. For
example conducting biodiversity surveys, training workshops,
education/outreach campaigns.
Outputs – What is produced by the project’s actions. These areusually measured in terms of quantity and quality of delivery, forexample the number of reports produced from survey data or thenumber of individuals trained.
Outcomes – The change(s) brought about by the project’s actions.
For a conservation project these can be further divided into:•

Intermediate outcomes which show progress towards
subsequent outcomes/impacts (see Figure 2);•

Threat reduction outcomes which represent the final
change(s) that need to happen in order to impact a
conservation target (see Figure 2).

Impacts – The long-term, lasting change(s) brought about by the
project’s actions. For conservation projects these typically relate
to changes in the status of species and/or habitats.
Assumptions – The links between each of the stages above areunderpinned by assumptions. For example a project focused on
training park rangers to carry out anti-poaching patrols makes
the assumption that training park guards will result in more

16

PRISM
MODULESMETHOD FACTSHEETS& ANNEXES

effective patrols. A project will also make assumptions about
external factors that the project does not have control over, butthat may influence the project’s results if they do not remain true.
For example, in order for training to be successful, park rangers
need to have sufficient resources to carry out their work (e.g.
equipment, salary). These kind of assumptions are particularly
important for a small/medium-sized project to consider, as there
will often be a large number of factors that could influence theproject’s outcomes/impacts and they will form an integral part ofthe story you will use to explain and contextualise your results.

Are these definitions always used in the same way?
Different organisations may use different terminology when theymean the same thing. For example “actions” may be referred to as“activities” or “interventions”, “outcomes” may be referred to as“results” or “intermediate results”, and “impacts” may be referred toas “goals”, “aims” or “ultimate outcomes”. We would encourage you
not to spend too much time focusing on any differences between
the terms used in PRISM and those used in other frameworks.
What is most important is that you understand what the terms
given above represent, and that you are able to relate them to yourown project.
Please refer to the Glossary for a list of key terms and definitions
used in PRISM.

1Colours, shapes and wordings in the boxes are designed to reflect those used by theConservation Measures Partnership http://www.conservationmeasures.org/ and the Miradi
Conservation Planning software www.miradi.org/ .

PRISM - TOOLKIT FOR EVALUATING THE OUTCOMES AND IMPACTS OF SMALL/MEDIUM-SIZED CONSERVATION PROJECTS

1:5

INTRODUCTION &KEY CONCEPTS

DESIGNING & IMPLEMENTING
THE EVALUATION

PRISMMODULESFIGURE 2: DEFINITIONS OF PROJECT STAGES (ACTIONS,OUTPUTS, INTERMEDIATE OUTCOMES, THREATREDUCTION OUTCOMES AND IMPACTS) USED IN PRISM.

WHAT DOWE MEAN BYOUTCOMES AND
IMPACTS?

Actions

METHOD FACTSHEETS& ANNEXES

Intermediateoutcomes

Outputs

Impact on
Biodiversity

ActionOutputs

IntermediateoutcomeThreat reduction
outcome

Impact on
biodoversity

ExamplesExamplesExamplesExamplesExamplesAssumptions

Biodiversity surveys
Training workshops
Education/Outreachcampaigns
Advocating forchanges in policy# of reports produced
from survey data

Information in reports
used to inform decision making

Decrease in poaching due toeducation campaign

Species population
increases# number of individuals
trainedIndividuals trained apply
new skills to conservation
targetPredation of native species
eliminated after eradication ofinvasive predators

Habitat status of targetedsites improves

ExamplesPolitical events do notaffect fieldwork
Weather conditions
remain constant
Local commodity prices

Training
provided to 24
park guards

Park guards
conduct more
effective patrols

Training
park guards
to conduct antipoaching
patrols

More effective
patrols deter
poachers

Poaching
decreases

Assumptions
Park guards have adequate resources to carry out their work
Park guards are able to access the areas needed to patrol
Hunted species populations do not decrease due to disease or other external factors
Simpliﬁed example of conservation action and resulting outcomes, impacts and assumptions

17

Threat reduction
outcomes

PRISM - TOOLKIT FOR EVALUATING THE OUTCOMES AND IMPACTS OF SMALL/MEDIUM-SIZED CONSERVATION PROJECTS

Hunted species
population
increases

1:6

DESIGNING & IMPLEMENTING
THE EVALUATION

Why focus on outcomes & impacts?
Project reporting often requires information relating toactions and outputs to assess whether you have done
what you planned. These measures are useful for tracking
project progress and delivery, but they do not effectively
demonstrate what difference the project has made (seeFigure 3). The latter requires information on what haschanged as a result of the project (its outcomes & impacts).
However while actions and outputs are events thatare under our control and thus somewhat predictable,
outcomes and impacts are not, which can make measuring
these changes more challenging.

PRISM
MODULESMETHOD FACTSHEETS& ANNEXES

FIGURE 3: Links between project stages, typical easeof measurement vs utility as a predictor of impact, andthe stages that PRISM’s guidance and methods focus
on (adapted from USFWS (2014) Standard Measures ofEffectiveness and Threats for Wildlife Conservation inCentral Africa, Version 1.0).

Focus of PRISM methods

StrongWHY FOCUS ON
OUTCOMES &IMPACTS?

INTRODUCTION &KEY CONCEPTS

Predictor of impact

THREAT
REDUCTIONOUTCOMESINTERMEDIATEOUTCOMESOUTPUTSACTIONS

More effective patrols
result in decreased poaching

Trained park guards apply
new skills to conduct more effective patrols

Number of park
guards trained

Number of training
workshops

SimpleChallenging

Ease of measurement

Actions

18

Population of hunted
species increases

IMPACTWeak

Why not just focus on impacts?
Ideally every project would evaluate its impacts (seeFigure 3) as these provide the most reliable measure of
the project’s success. However in practice, impacts cansometimes be challenging to evaluate, for example someconservation impacts may not become measurable until
after the project has finished while others may be too
costly to measure with the resources available. In thesecases outcomes – shorter-term changes resulting from the
project – can be used to indicate the likelihood of future
impacts. There is a trade-off involved, in the sense thatthe further away from impacts you measure, the morechallenging it is to be certain about the conclusions you arereaching. Effective evaluation therefore requires a balance
between trying to obtain the most useful results andtackling questions that are feasible to measure.

PRISM - TOOLKIT FOR EVALUATING THE OUTCOMES AND IMPACTS OF SMALL/MEDIUM-SIZED CONSERVATION PROJECTS

Outputs

Intermediateoutcomes

Threat reduction
outcomes

Impact on
Biodiversity

1:6

INTRODUCTION &KEY CONCEPTS

DESIGNING & IMPLEMENTING
THE EVALUATION

PRISMMODULESMETHOD FACTSHEETS& ANNEXES

EXAMPLE
Example: A project is aiming to reduce chimpanzee poaching

WHY FOCUS ON
OUTCOMES &IMPACTS?

by providing training to park rangers to conduct more effective
patrols. The desired impact is to see an increase in the numberof chimps. However, since chimps reproduce relatively slowly,
this is not measurable within the project’s lifetime. Equally,
simply counting how many rangers were trained does notindicate whether the training has made patrols more effective.
Instead, it might be best to evaluate whether better patrolling
resulted in a decrease in the number of snares being set (themain cause of chimp mortality), compared to other areas,or to the number of snares present before training. In time,a reduction in the number of snares might be expected tocontribute to decreased mortality and thus to an increase inchimp numbers.

Photo credit: Emily Wroblewski (Stanford University)

19

PRISM - TOOLKIT FOR EVALUATING THE OUTCOMES AND IMPACTS OF SMALL/MEDIUM-SIZED CONSERVATION PROJECTS

1:7

INTRODUCTION &KEY CONCEPTS

DOES THEPROJECT’S
DESIGN ALLOWFOR EVALUATIONOF OUTCOMES/IMPACTS?

we

DESIGNING & IMPLEMENTING
THE EVALUATION

Does the project’s design allow for evaluation of outcomes/
impacts?
Whether or not you can effectively evaluate your project’s
outcomes/impacts will often depend on whether the project
has a clear and logical design2. This should be based on a goodunderstanding of both the conservation issue being addressed andhow the project’s actions could be expected to bring about theintended conservation impacts.
Another way of framing this is to ask the question “why do you
think this project will work?”

PRISM
MODULESIf the project has a clear and logical design this makes it much morestraightforward to come up with clear questions and hypotheses
that can be tested during the evaluation, and allows you to makeeffective judgements on the project’s success and lessons learnt.
For example:
Project design:

If the project can decrease poaching bytraining park guards, then chimpanzee
populations should recover

Questions:

Did training help park guards to prevent
poaching from occurring?

For example, the design of the chimpanzee project outlined in theprevious section could be illustrated like this:Train park
guards to
conduct antipoaching
patrols

Actionsthen

Outputs

Park guards willconduct more
effective patrols

IntermediateoutcomesthenThreat reduction
outcomes

Did chimpanzee populations recover?

More effective
patrols will
deter hunters

Impact on
Biodiversity

METHOD FACTSHEETS& ANNEXES

then

Poaching will
decrease

resultingin

Increase inchimpanzee
population

Equally if the project design lacks a clear rationale this makes it
extremely challenging to identify which aspects of the project
should be evaluated and to make any kind of judgement onwhether or not the project has been a success.

2
Although good project design is crucial for effective evaluation, please note that PRISM is
not a resource for designing a conservation project or project actions. Instead this toolkit isfocused on how to evaluate project outcomes & impacts. Several excellent resources exist for
designing and implementing conservation actions. The Open Standards for the Practise ofConservation: cmp-openstandards.org/ is one of the most useful and widely applied of these.

20

PRISM - TOOLKIT FOR EVALUATING THE OUTCOMES AND IMPACTS OF SMALL/MEDIUM-SIZED CONSERVATION PROJECTS

1:7

INTRODUCTION &KEY CONCEPTS

DOES THEPROJECT’S
DESIGN ALLOWFOR EVALUATIONOF OUTCOMES/IMPACTS?

Sustainable Wax
Palm alternatives
promoted to
stakeholders

PRISM
MODULESMETHOD FACTSHEETS& ANNEXESPROJECT A

PROJECT B

Having established that the decline of the Yellow-eared
Parrot in the Colombian Andes is linked to the overharvestingof the quindío wax palm for use in Palm Sunday celebrations,conservationists conducted an education campaign backed
by the Catholic Church to raise awareness and promotesustainable alternatives for palm fronds during Easterfestivities.

Recognising the biodiversity value of the montane forests of
the Colombian Andes, home to globally threatened species
such as the Yellow-eared Parrot, conservationists launched
an integrated conservation and development project ina nearby community. The project included components
on sustainable resource management, institutional
capacity building, governance networks and stakeholder
empowerment and well-being.

This project has a clear design informed by a solid
understanding of the underlying conservation issue(i.e. if the main threat - overharvesting of wax palm - isaddressed then the species population should improve).
Because of this it is relatively straightforward to see
how the project might be evaluated, either by focusing
on a relevant ecological characteristics of the project’s
conservation target (e.g. greater nest density of Yelloweared Parrot) or by focusing on measures to address thethreats relating to the underlying conservation issue (e.g.
a reduction in palm felling, a reduction in the use of palm
fronds during Easter festivities).

EXAMPLE:CONSIDER THE
FOLLOWING TWO
PROJECTS:

Promotesustainable
alternatives for Wax
Palm use inPalm Sundaycelebration

DESIGNING & IMPLEMENTING
THE EVALUATION

Local people
harvestfewer Wax
Palms

More Wax Palms areavailable for Yelloweared Parrot to roost,
nest and feed in

While many of these actions may be worthwhile, thisproject lacks a clear design. The project does nothave a clear conservation target and there appears
to be a limited understanding of the factors affecting
biodiversity loss and how the project’s actions willaddress these. Instead, a range of different approaches is
proposed without a coherent design
showing how these link together.
This makes it much more
Improvegovernancedifficult to come up
networks
with clear questions
Promotesustainable
for an evaluation
resource
management
to address.

?

Yellow-eared parrot
populatio increases
stabilises
Building
local
capacity

Improvestakeholder
wellbeing

21

PRISM - TOOLKIT FOR EVALUATING THE OUTCOMES AND IMPACTS OF SMALL/MEDIUM-SIZED CONSERVATION PROJECTS

Biodiversity valueof Colombian montane
forest improves

1:8

INTRODUCTION &KEY CONCEPTS

DESIGNING & IMPLEMENTING
THE EVALUATION

Unintended outcomes & impacts
Sometimes projects result in changes that were not anticipated in
the original project plan.

UNINTENDEDOUTCOMES &IMPACTS

Often evaluation is focused on the intended consequencesof a project, i.e. whether the project’s results match what was
expected. However, conservation settings are often complex andthus conservation actions frequently have unintended outcomes/
impacts, which can be positive or negative (see Figure 4).
Learning from these unintended outcomes is extremely
valuable; even when things go wrong, this information isimportant for developing successful projects in future.

PRISM
MODULESMETHOD FACTSHEETS& ANNEXES

FIGURE 4: EXAMPLE OF CONSERVATION ACTIONSHOWING A RANGE OF OUTCOMES AND IMPACTS.

Negative Unintended

Conﬂict withlocal communities
increases

Poaching increases

Hunted species
population
decreases

IntendedTraining park
guards to conduct
ranger patrols

Training provided
to park guards

Park guards
conduct more
effective patrolsMore effective
patrols deter
hunters

Poaching decreases

Hunted species
population
increases

Positive Unintended
Park guards provide
training for guards
on other parks

Actions

22

PRISM - TOOLKIT FOR EVALUATING THE OUTCOMES AND IMPACTS OF SMALL/MEDIUM-SIZED CONSERVATION PROJECTS

Outputs

Poaching decreases
in other parks

Intermediateoutcomes

Hunted species
population
increases inother parks

Threat reduction
outcomes

Impact on
Biodiversity

1:8

INTRODUCTION &KEY CONCEPTS

UNINTENDEDOUTCOMES &IMPACTSDESIGNING & IMPLEMENTING
THE EVALUATION

Conservation and impacts on human well-being
Conservation actions can have complex and socially-differentiatedimpacts on people: some people might benefit, others mightfind that they are negatively affected; or the project may offer
long-term benefits but in the short term may make people more
vulnerable. Even when project actions do not involve working
directly with people the project may still have outcomes andimpacts that affect those living and working nearby.
Where unintended outcomes/impacts occur they are often felt
most by those living closest to the project or those who rely on thearea’s natural resources. As such, it is often necessary to identify
and, wherever possible, minimise any negative social impacts.
Projects can do this by incorporating questions relating to humanwell-being into their evaluation. For example whether the project
has affected any of the following:•

Material well-being - the material circumstances of aperson’s life (such as housing, income, livelihood, healthand the environment).

•

Relational well-being - how people engage with others
to meet their needs (for example through community
networks and social institutions).

•

Subjective well-being - how people feel, and theirsatisfaction with different aspects of their life.

PRISM
MODULESMETHOD FACTSHEETS& ANNEXES

More information and methods for evaluating well-being can befound in the Livelihoods & Governance module.
For more detailed information on evaluating well-being impacts
of conservation projects you should refer to the following
publication: http://pubs.iied.org/14667IIED/.

Learning as a project outcome/impact
Unintended outcomes and impacts are particularly valuable
for capturing lessons and applying these to inform futureconservation work. In addition to evaluating the project’s
conservation outcomes, this learning should be seen as aproject outcome in itself.

Perceptions of unintended outcomes/impacts
Whilst everyone likes to tell donors about their successes, in reality
many donors are more interested in seeing the realities of projects,
and are keen to see how project leaders learn from unintended
outcomes. Even if these results are not shared externally they areinvaluable for learning and for adapting a project so that it will havea greater chance of success in the future. See Step 4. What should
be done next? for more information on how best to use and apply
the results of your evaluation.

Woodhouse, E et al. (2016) Evaluating the impacts of conservation interventions on humanwellbeing. Guidance for practitioners. IIED, London

23

PRISM - TOOLKIT FOR EVALUATING THE OUTCOMES AND IMPACTS OF SMALL/MEDIUM-SIZED CONSERVATION PROJECTS

1:9

INTRODUCTION &KEY CONCEPTS

WAS THE CHANGE
RECORDED
CAUSED BY THE
PROJECT?

DESIGNING & IMPLEMENTING
THE EVALUATION

Was the change recorded caused by the project?
As well as identifying outcomes & impacts that have occurred, it isusually the aim of evaluation to investigate whether these changes
resulted from the actions of the project, rather than from otherexternal factors. Evaluation often therefore requires not only
measuring changes that happened during the project, but also
identifying which changes were due to the project and which were
not. This process is called attribution.

PRISM
MODULESMETHOD FACTSHEETS& ANNEXES

Many evaluations only measure conditions before and after
the project: while this might be acceptable for some outcomes/
impacts, it will not distinguish changes that were caused by otherexternal factors. Thankfully, there are several options available tosmall/medium-sized projects to help in attributing change to theactions of the project itself, and to control for external factors –see the section on designing your evaluation for more information.

EXAMPLEFIGURE 5: ATTRIBUTION IN A PROJECT ACTION(ADAPTED FROM CATLEY ET AL., 2014).

Example: A project attempted to reduce poaching by
undertaking patrols to remove snares. By the end of the
project the team recorded that there had been a reduction in
the frequency of snares in the area being patrolled. However
the team quickly realised that there were a number of other

Project
factors

factors that could potentially explain the reduction in numbersof snares, for example the reduction could have been causedby the patrols or alternatively the reduction could have beencaused by falling food prices and a consequent reduction ofdemand for bushmeat; or it could be a combination of the two.

Situation before
project action

Situation after
project action

The team then collected additional data that allowed them tocompare the number of snares from neighbouring areas withsimilar characteristics to the project site that were not subject
to the patrols. This gave the team a better understanding ofwhether the observed reduction in snares could be attributed
to the project.

Non-project
factors

Catley, A., Burns, J., Abebe, D., Suji, O. (2013). Participatory Impact Assessment: A Design
Guide. Feinstein International Center, Tufts University, Somerville http://fic.tufts.edu/
assets/PIA-guide_revised-2014-3.pdf

24

PRISM - TOOLKIT FOR EVALUATING THE OUTCOMES AND IMPACTS OF SMALL/MEDIUM-SIZED CONSERVATION PROJECTS1:10

BE REALISTICAND FOCUS ON
INTERPRETING,
RATHER THANJUST PRESENTINGRESULTSINTRODUCTION &KEY CONCEPTS

DESIGNING & IMPLEMENTING
THE EVALUATIONBe realistic and focus on interpreting, rather than justpresenting results
Evaluation requires you to make judgements based on the available
information. While ideally you would have all relevant information
available to you, in the real world the amount of information
available may only provide a partial explanation of the project’s
results.
There can be a temptation to be overly scientific and to focus onpresenting quantitative results (e.g. in the form of numbers orgraphs). However, the most useful evaluations are typically thosethat carefully consider the information available, and then use
this information to develop an interpretation which explains the
project’s results.

PRISM
MODULESMETHOD FACTSHEETS& ANNEXES

A good evaluation therefore usually requires two kinds of thinking.
It requires the ability to think like a scientist when designing
the evaluation and collecting data, and then to think more like adetective or an investigative journalist to interpret results andmake judgements based on the information available.
PRISM’s step by step process is designed to help you design andcarry out your evaluation in this way. See Section 2: Step by step
guide for evaluating project outcomes and impacts for more
information.

For example the fact that long term impacts are not measurable
within the project’s timeframe means that you will often need touse information collected on outcomes to explain the likelihoodof future impacts. Equally you may find it impossible to isolate
and collect data on all the different factors, risks and assumptions
which may influence the project’s outcomes and impacts, so you
may often need to rely on qualitative evidence (e.g. interviews),
both from participants and from your own observations andunderstanding of the situation, to explain and contextualise theproject’s results.

Photo credit: Tsubasa Iwabuchi

25

PRISM - TOOLKIT FOR EVALUATING THE OUTCOMES AND IMPACTS OF SMALL/MEDIUM-SIZED CONSERVATION PROJECTS1:11

TIPS FOR
EFFECTIVE
EVALUATION IN
SMALL/MEDIUMSIZED PROJECTS26

INTRODUCTION &KEY CONCEPTS

DESIGNING & IMPLEMENTING
THE EVALUATION

PRISMMODULESMETHOD FACTSHEETS& ANNEXES

Start early

The earlier you start to think about how you will measure project outcomes & impacts, the easier
it is. Starting late in a project is still possible, but often makes it harder to demonstrate change
and to link outcomes and impacts to project actions.

Be clear about why you
want to evaluate

There are many potential ways that the results of an evaluation could be used. For example you
might want to demonstrate results externally, use results to learn internally or a combination of
both. Investing time to clarify who the evaluation’s key stakeholders are, and how the results of theevaluation will be used, will help you determine the most appropriate evaluation questions, which inturn will influence the methods and your plans for applying and communicating the results.

Don’t try to measure
everything

Measuring one useful element of a project effectively is better than measuring several things
poorly. For smaller projects the key to effective evaluation is to focus on the outcomes & impactsof the project where the most useful results will be generated with measures that are feasible toimplement.

Ensure the project has aclear design

Having a project design which clearly outlines how your actions will bring about change (goingfrom actions to outcomes and impacts) will provide a framework you can use to guide theevaluation, and will make interpretation of results much simpler.

Don’t just view
evaluation as a reporting
requirement

If fulfilling reporting requirements is the primary purpose when evaluating it is unlikely to generateenough motivation within the team to carry it out effectively. Perhaps most importantly evaluation
allows you to test and improve what you are doing; for example by informing future decisions, bothwithin the project and beyond. Sharing evaluation results helps to promote transparency , adaptive
management and to create an evidence base for conservation learning.

Remember that negative
outcomes/impacts are
as important as positive
ones

Evaluation is often driven by a desire to show that that a project was successful or worthwhile.
However, understanding and sharing what didn’t work is sometimes more important, and canhelp others to avoid unfruitful efforts. Don’t be scared to report these things to donors; mostdonors will welcome this, especially if you can demonstrate that you learned something fromthe experience.

PRISM - TOOLKIT FOR EVALUATING THE OUTCOMES AND IMPACTS OF SMALL/MEDIUM-SIZED CONSERVATION PROJECTS2:0

SECTION 2
STEP BY STEP GUIDEFOR EVALUATINGPROJECT OUTCOMES
AND IMPACTS

27

PRISM - TOOLKIT FOR EVALUATING THE OUTCOMES &ANDIMPACTS
IMPACTSOF OF
SMALL/MEDIUM-SIZEDSMALL/MEDIUM-SIZEDCONSERVATIONCONSERVATIONPROJECTSPROJECTS

2:0

INTRODUCTION &KEY CONCEPTS

Section 2

DESIGNING & IMPLEMENTING
THE EVALUATION

PRISMMODULES

Step by step guide for evaluating project
outcomes and impacts
How to use this toolkit to evaluate your project’s
outcomes and impacts

CONTENTSSECTION 2
STEP 1

What do you need to know?
What is the project trying to achieve?
Why do you want to evaluate?
What questions do you need to answer?

STEP 2

What data do you need to collect?
How will you know which outcomes/impacts canbe attributed to the project?
How will you collect evaluation data?

STEP 3What can you learn from the results?
How will you analyse your data?
What do the results show?

STEP 4

What should be done next?
How will you use the evaluation results?

28

PRISM - TOOLKIT FOR EVALUATING THE OUTCOMES AND IMPACTS OF SMALL/MEDIUM-SIZED CONSERVATION PROJECTSMETHOD FACTSHEETS& ANNEXES

2:0

HOW TO USE
THIS TOOLKITTO EVALUATEYOUR PROJECT’SOUTCOMES AND
IMPACTSINTRODUCTION &KEY CONCEPTS

DESIGNING & IMPLEMENTING
THE EVALUATION

PRISMMODULESMETHOD FACTSHEETS& ANNEXES

How to use this toolkit to evaluate your project’s outcomes andimpacts
For any project, evaluating outcomes & impacts involves finding abalance between focusing on the outcomes/impacts of the project
where evaluation will be most useful, and designing an evaluationthat is feasible to implement. So like a glass prism is designed toseparate white light so that you can see the different colours of thespectrum, the PRISM toolkit is designed to help conservationists toseparate out the different elements of a project, focus on the key
outcomes/impacts to evaluate and then use methods appropriate
for smaller projects to design and carry out an effective evaluation(see Figure 6).

Photo credit: Glenn Ehmke

29

PRISM - TOOLKIT FOR EVALUATING THE OUTCOMES AND IMPACTS OF SMALL/MEDIUM-SIZED CONSERVATION PROJECTS

2:0

INTRODUCTION &KEY CONCEPTS

DESIGNING & IMPLEMENTING
THE EVALUATIONMETHOD FACTSHEETS& ANNEXES

Conservation
project

Conservation
Problem

HOW TO USE
THIS TOOLKITTO EVALUATEYOUR PROJECT’SOUTCOMES AND
IMPACTS

PRISM
MODULES

Project
ActionConservation
ImpactOutputOutcomeConservation
Problem

Conservation
project

Project
ActionOutputOutcome

FIGURE 6

Conservation
ImpactOutcome

Project
ActionOutputIdentify
questions

Designevaluation& collect
data

OutcomeAnalyse &interpret
results

Decidewhat should
be donenext

OutcomeActions30

PRISM - TOOLKIT FOR EVALUATING THE OUTCOMES AND IMPACTS OF SMALL/MEDIUM-SIZED CONSERVATION PROJECTS

Outputs

Intermediateoutcomes

Threat reduction
outcomes

Impact on
Biodiversity

2:0

INTRODUCTION &KEY CONCEPTS

DESIGNING & IMPLEMENTING
THE EVALUATION

PRISMMODULESMETHOD FACTSHEETS& ANNEXESIn this toolkit, the process of designing and implementing anevaluation is broken down into four main steps. This section of
the toolkit has been colour-coded according to the steps outlined
below:

HOW TO USE
THIS TOOLKITTO EVALUATEYOUR PROJECT’SOUTCOMES AND
IMPACTSSTEP 1

STEP 2

STEP 3

STEP 4

WHAT DO YOU
NEED TO KNOW?

WHAT DATADO YOU NEEDTO COLLECT?

WHAT CAN
YOU LEARN FROM
THE RESULTS?

WHAT SHOULD
BE DONE NEXT?

Identify which of the
project’s outcomes/
impacts to focus on, andthe evaluation questions
that you will try toanswer.

31

Design the evaluationand select appropriate
data collection methods
that will allow you toanswer your evaluationquestions.

Analyse and visualisedata then examine thedata to find out what
can be learnt from theresults, and what theresults mean in relation
to the overall goal of the
project.

PRISM - TOOLKIT FOR EVALUATING THE OUTCOMES AND IMPACTS OF SMALL/MEDIUM-SIZED CONSERVATION PROJECTS

Apply results toimprove practice andshare results with thewider conservation
community.

2:0

HOW TO USE
THIS TOOLKITTO EVALUATEYOUR PROJECT’SOUTCOMES AND
IMPACTSINTRODUCTION &KEY CONCEPTS

DESIGNING & IMPLEMENTING
THE EVALUATION

PRISMMODULESSTEP 1

STEP 2

STEP 3

STEP 4

WHAT DO YOU
NEED TO KNOW?

WHAT DATADO YOU NEEDTO COLLECT?

WHAT CAN
YOU LEARN FROM
THE RESULTS?

WHAT SHOULD
BE DONE NEXT?

QUESTIONS ADDRESSEDQUESTIONS ADDRESSEDQUESTIONS ADDRESSEDQUESTIONS ADDRESSED

What is the project trying to

How will you know which

How will you analyse your data?

How will you use the evaluationachieve?

outcomes/impacts can be•

How should I enter, organise

results?

and clean data?

•

attributed to the project?
Why do you want to evaluate?

•

What is evaluation design?

•

What is the unit of analysis?

•

Which evaluation design•

What data analysis methods

should I use?

What questions to you needto answer?
••

What questions could

Am I measuring attribution•

or contribution?

••

How should I compare data

Which questions should I

data?

focus on?

•What kind of data isneeded?

What do the results show?

•

Can any existing data be••

Do I need to control for•

Are the methods ethically•

What sampling strategy•

What data collection

Are there any unintended
consider?

used?

•

How should I present
evaluation data?

Can results be used to
improve the project?

•

How will results becommunicated to external

methods?
How will you collect evaluation

outcomes/impacts I need to•collected using different

evaluation?

Developing
recommendations

should I use?

be considered in theaudiences?
•

Can results be shared withthe wider conservation
community?

How should I interpret
evaluation results?

bias?
sound?
should I use?
methods should I use?

32

METHOD FACTSHEETS& ANNEXESPRISM - TOOLKIT FOR EVALUATING THE OUTCOMES AND IMPACTS OF SMALL/MEDIUM-SIZED CONSERVATION PROJECTS

2:1

INTRODUCTION &KEY CONCEPTS

Step 1

STEP 1
WHAT DO YOU
NEED TO KNOW?

DESIGNING & IMPLEMENTING
THE EVALUATIONMETHOD FACTSHEETS& ANNEXES

What do you need to know?

There are many different things that could be looked at in an
evaluation and it’s likely that you will not be able to measure all ofthem. Therefore, before collecting any data it is usually necessary
to focus the evaluation to ensure that the effort and resources
used will deliver the most useful results. This involves looking
carefully at your project’s outcomes and impacts and trying to findthe right balance between trying to gain answers that are as usefulas possible while still being feasible to measure.
This section will help you to outline the project’s outcomes andimpacts then to use this outline to identify and prioritise which
outcomes/impacts you want to measure, and to come up with a setof questions that you will try to answer during the evaluation.

33

PRISM
MODULESSTEP 1QUESTIONS ADDRESSED IN THIS SECTION:•

What is the project trying to achieve?

•

Why do you want to evaluate?

•

What questions do you need to answer?
oWhat questions could be considered in theevaluation?

oWhich questions should I focus on?

o

Are there any unintended outcomes/impacts I
need to consider?

PRISM - TOOLKIT FOR EVALUATING THE OUTCOMES AND IMPACTS OF SMALL/MEDIUM-SIZED CONSERVATION PROJECTS

2:1

INTRODUCTION &KEY CONCEPTS

DESIGNING & IMPLEMENTING
THE EVALUATION

What is the project trying to achieve?
Before you begin to develop your questions you first need to makesure that the project’s design allows you to evaluate the outcomes
and impacts.

STEP 1
WHAT IS THE
PROJECT TRYING
TO ACHIEVE?

This involves checking your project’s design by examining thelinks between each stage of the project from actions and outputs
through to outcomes and impacts. The project design should show
a clear and logical path demonstrating how the project’s actions
could be expected to lead to the project’s desired impacts. The
logic underpinning the project’s design should be based on a clear
understanding of the conservation issue(s) being addressed andany relevant assumptions that have been made. This should allow
you to answer yes to the following questions:

PRISM
MODULES

Projects often find it useful to have a visual depiction outlining
the project’s design and rationale. This outline then enables
you to focus on the key outcomes/impacts to address in theevaluation. There are several different approaches you can use
for this: Theory of Change, results chains, logic models and logicalframeworks (logframes) all serve this purpose and are widely usedin conservation projects.

As a rough guide:
Approach

Description

Results chain/logic model

Flow diagram showing how the
project’s actions are expected
to lead to subsequent outputs,
outcomes & impacts

Theory of Change

Similar to a results chain but also
outlines risks and assumptions
which are relevant to theachievement of the project’s
outcomes & impacts

Logical frameworkShows information similar tothat in a Theory of Change but
presented in a table, rather than
a flow diagram

Does the project have a clear intended impact? – e.g.
improvement in status of a particular species, change instakeholder behaviour relating to a conservation target.
Is it clear how the project’s action(s) will bring about these
impacts, i.e. if we do X is it reasonable to expect that Y will
happen? – e.g. if we train park rangers, is it likely that this will leadto a decrease in poaching?
Have you considered what other factors could potentially affectthe project’s results? – e.g. if we train park rangers, are there any
additional factors that could affect the target species populations
at the project site?

34

METHOD FACTSHEETS& ANNEXESPRISM - TOOLKIT FOR EVALUATING THE OUTCOMES AND IMPACTS OF SMALL/MEDIUM-SIZED CONSERVATION PROJECTS

2:1

INTRODUCTION &KEY CONCEPTS

STEP 1
WHAT IS THE
PROJECT TRYING
TO ACHIEVE?

DESIGNING & IMPLEMENTING
THE EVALUATION

PRISMMODULESMETHOD FACTSHEETS& ANNEXES

Although PRISM refers primarily to Theory of Change we would
encourage you not to spend too much time worrying about thedifference between these different approaches. The aim of allof the above is to map out the project’s design and to provide aframework showing how your project will bring about change, andthis is then used to design and plan the evaluation.
If the links between any stages of the project are unclear, thenthe project’s design may need to be re-visited and adjusted
accordingly. If the project has already finished, then these design
factors should be recorded as something which may affect theevaluation results.
See the Open Standards: http://cmp-openstandards.org/ for more
information on developing and improving project design.
Refer to the factsheet Completing a Theory of Change for
information on how to construct a simple Theory of Change.

Photo credit: David Thomas

35

PRISM - TOOLKIT FOR EVALUATING THE OUTCOMES AND IMPACTS OF SMALL/MEDIUM-SIZED CONSERVATION PROJECTS

2:1

INTRODUCTION &KEY CONCEPTS

DESIGNING & IMPLEMENTING
THE EVALUATION

PRISMMODULESMETHOD FACTSHEETS& ANNEXES

FIGURE 7: EXAMPLE OF A SIMPLE THEORY OF
CHANGE FOR A CONSERVATION PROJECTSTEP 1
EXAMPLE OF ASIMPLE THEORYOF CHANGE FOR
A CONSERVATIONPROJECT

Informing
stakeholders
about sustainable
forestmanagement
practices

Stakeholders
are informedabout
sustainable
practices

Local people havean improved
understanding ofsustainable
practices

Local people
adopt moresustainable
practices

Training
stakeholders onsustainable
extractionof NTFP

Individuals
trainedTrained individuals
have new skills forsustainable
extraction

Trainedindividuals apply
new skills

Local people
manage forest
resources moresustainably
Forest loss
slowed/reversed

Provide direct
investment for
rattan enrichment

Households
receiveinvestment for
rattanenrichment

Forest protection
agreements in
exchange for NTFP
extraction rights

Agreements in
placeHouseholdlivelihood/
wellbeing
improves

Rattan provides
additional
income tohouseholds

Local people
have right toutilise forest
resources

Status of native forest
species (including
Edward’s Pheasant)improves

Local people areincentivised to
manage forest
more sustainably

Critical Assumptions
Price of NTFPs remain stable or increase
Weather conditions remain stable enoughfor rattan to grow
Populations of native forest species do notdecrease due to disease or other natural
factors

Actions36

Outputs

Intermediateoutcomes

Threat reduction
outcomes

Impact on
Biodiversity

PRISM - TOOLKIT FOR EVALUATING THE OUTCOMES AND IMPACTS OF SMALL/MEDIUM-SIZED CONSERVATION PROJECTS

2:1

INTRODUCTION &KEY CONCEPTS

STEP 1
WHY DO YOU
WANT TOEVALUATE?

DESIGNING & IMPLEMENTING
THE EVALUATION

Why do you want to evaluate?
Once you have reviewed the project’s design
you should set out why you want to evaluate
in the first place. The best way to do this is
to briefly spend some time to identify how
evaluation results will be used, and by whom.
Note that there will typically be a large
number of stakeholders who could potentiallybenefit from the results of the evaluation.
However for the purposes of deciding what toevaluate, it is best to stick to the stakeholders
who need to know the results of theevaluation. Typically this includes:

PRISM
MODULESMETHOD FACTSHEETS& ANNEXES

See the factsheet Identifying why you want toevaluate for advice on how to do this.

EXAMPLES OF EVALUATION USES AND USERS INCLUDE:STAKEHOLDER GROUPWHAT DO THEY NEED TO KNOW?

HOW WILL THEY USE THE INFORMATION?

Project teamWas the action successful?

Modifying the project

Why/how was the action

Demonstrating results to donors

successful?
DonorsDid the project deliver as
expected?

•••

The project team and/or the
implementing organisation
The project donor(s)Other key stakeholders whose
involvement is crucial to thesuccess of the project (e.g. key
decision makers being targeted by
the project, community members
involved in the project).

Guiding future investment andWhat lessons were learnt?

programme design

Key decision makers

What was the impact on local

Informing management/policy(e.g. Local authorities,

people?

decisions

What lessons were learnt?

Distributing results

Politicians, Law enforcement,
Private sector)
Wider conservation community

Note that this is not meant to be an exercise
for identifying all the potential ways in which
the results of an evaluation could be used,
but is instead meant to help to focus on the
key outcomes and impacts to address in theevaluation.
37

External communications

PRISM - TOOLKIT FOR EVALUATING THE OUTCOMES AND IMPACTS OF SMALL/MEDIUM-SIZED CONSERVATION PROJECTS

Using results to inform
conservation practice2:1

INTRODUCTION &KEY CONCEPTS

STEP 1
WHAT QUESTIONSDO YOU NEED TOANSWER?

DESIGNING & IMPLEMENTING
THE EVALUATION

What questions do you need to answer?
Once you have considered the project’s design and the evaluation
audience, the next task is to use this information to identify andprioritise which outcomes/impacts you want to focus on, and comeup with a set of guiding questions that you will try to answer during
the evaluation.
The process of identifying appropriate questions determines
what data need to be collected. Answering these questions thenguides the interpretation of the evaluation results and the processof deciding what to do next. A good set of evaluation questions
will therefore greatly enhance your chances of producing useful
results. Equally, evaluation questions that are poorly thought out
risk wasting valuable time and/or producing unreliable results.

PRISM
MODULESMETHOD FACTSHEETS& ANNEXES

What questions could be considered in the evaluation?
Start by asking yourself “what do I need to know in relation to theproject?”.
The table on the following page lists some of the different kindsof questions that could be asked during an evaluation, along withsome specific examples.
A good way to approach this process is to look at your project’s
outcomes & impacts (as outlined in your Theory of Change or
equivalent) and, in a small group, brainstorm potential
evaluation questions that could be addressed. At thisstage it is recommended that you keep thequestions quite broad - later you will
be able to break them down into
sub-questions to use as
you are collecting
data.

Photo credit: Iain Dickson

38

PRISM - TOOLKIT FOR EVALUATING THE OUTCOMES AND IMPACTS OF SMALL/MEDIUM-SIZED CONSERVATION PROJECTS

2:1

INTRODUCTION &KEY CONCEPTS

STEP 1TYPES OFQUESTIONSMETHOD FACTSHEETS& ANNEXESDESCRIPTION

EXAMPLES

Process questions *Is the project delivering
as expected?

•

Is the project proceeding as expected?

•

Does the project need to change anything to ensure successful
delivery?

What change(s) haveoccurred since theproject began?

•

Has the number of migrant shorebirds using the target site
increased since the project began?

•

Have household incomes increased since the start of the project?

To what extent are anychange(s) due to theproject?

•

Did the project improve the habitat quality for shorebirds at the
project site?

•

Did the project improve the income of participating households?

•

What other factors may have influenced the results?

What do any changes
mean in relation to theoverall aim of the project?

•

What do the project’s results mean for regional populations ofshorebirds?

•

What difference has the project made to the lives of participants?

What can we learn fromthe results?

•

Did the method(s) used by the project work as planned? If not, why
not?

•Were there any unintended impacts? If so, what were they?

•

Did the project have any negative outcomes? If so, how could thesebe avoided in future?

•

Should the project continue? Should it be scaled up?

•

Should the project’s approach be used in other areas?

•

Which project results could be shared with other conservationists
and/or other relevant stakeholders?

Descriptive questions

Causal questions

Learning questions

Action questions

39

PRISM
MODULESTYPE OF QUESTION

Synthesis questions* Note that process questions are notsufficient to evaluate outcomes/impacts,
but they are useful for tracking projectprogress, making changes to on-goingactions and reporting to donors

DESIGNING & IMPLEMENTING
THE EVALUATION

What should be done
next?

PRISM - TOOLKIT FOR EVALUATING THE OUTCOMES AND IMPACTS OF SMALL/MEDIUM-SIZED CONSERVATION PROJECTS

2:1

INTRODUCTION &KEY CONCEPTS

STEP 1WHICH
QUESTIONSSHOULD I FOCUS
ON?

DESIGNING & IMPLEMENTING
THE EVALUATION

Which questions should I focus on?
Although the ideal scenario might be to try to answer questions
relating to all your project’s outcomes and impacts this is rarely
possible in practice. For example some outcomes/impacts may
not be measurable with the time and resources available, whileothers may be less important to the success of the overall project.
Instead it is usually far more valuable to focus on obtaining reliableanswers to a smaller number of questions (aim for no more thanfive).

PRISM
MODULESMETHOD FACTSHEETS& ANNEXES

EXAMPLE
Example: A project aimed to increase the population of athreatened species of orchid threatened by overharvestingfor the domestic flower trade. To achieve this the project
carried out surveys to map the remaining populations oforchids, workshops with wild flower collectors on sustainable
harvesting techniques and awareness raising presentations inlocal schools.

Choosing which questions to answer in your evaluation will depend
largely on the following:••

Which questions will provide the most useful
information?
Are the questions feasible to answer?

Intended outcomes: Increased awareness of orchids among
local schoolchildren, improved knowledge of the location ofremaining populations of orchids and reduced harvesting oforchids by local people
Evaluation questions: The team decided that the evaluationquestions that would provide the most useful information were
those relating to whether the project had been successful in
reducing the harvesting rate of wild orchids by local people, as

Which questions will provide the most useful information?
Looking at the project’s outcomes/impacts, ask the following:these would show whether the project had addressed the mainthreat affecting the conservation target.

Which outcomes/impacts are most important to the success ofthe project? - Consider the project’s design, your understanding of
the underlying conservation issue and the needs of your evaluation
audience. Try to identify and focus on the outcomes and impacts
that are particularly important to the success of the project andthe questions you need to know in relation to these.

40

PRISM - TOOLKIT FOR EVALUATING THE OUTCOMES AND IMPACTS OF SMALL/MEDIUM-SIZED CONSERVATION PROJECTS

2:1

INTRODUCTION &KEY CONCEPTS

STEP 1
HOW WELLKNOWN IS THE
ACTION BEINGEVALUATED?

DESIGNING & IMPLEMENTING
THE EVALUATION

How well known is the action being evaluated? - One of the main
reasons to evaluate is that it enables the sharing of experience andevidence across the conservation sector. Equally, when prioritising
your evaluation questions you should research whether the
project’s approach has been applied elsewhere and in similar
contexts.
If the action that you are measuring has been thoroughly tested
and its outcomes and impacts are well known, then an evaluation
may be of limited use. For example if the action has been regularly
applied and tested in similar contexts, or the project is scaling up
actions that have previously been trialled at the target site, in thesecases you may be better off measuring a different action or simply
measuring project delivery, and thereby saving resources.
Conversely an evaluation may be particularly useful in thefollowing situations:-Innovative (untested) schemes-

Pilot actions that are due to be scaled up-

Actions for which there is a lack of solid evidence of
impact in the given context-

Generalisable actions that an organisation is planning to
adopt widely.

PRISM
MODULESMETHOD FACTSHEETS& ANNEXES

There are several places where you can find information ondifferent conservation approaches that have been tested and theresults of which have been shared. CMP’s Miradi-CAML database:
http://cmp-openstandards.org/tools/caml/ and Conservation
Evidence: http://www.conservationevidence.com/ are twoparticularly useful resources.

Are the questions feasible to answer?
Once you have identified and developed questions for theoutcomes and/or impacts where an evaluation would be mostuseful, the next step is to check whether these questions arefeasible to measure. Feasibility should be measured against thefollowing three criteria:
Available time - Many long-term outcomes (such as changes inbiodiversity status) can take a long time for results to show. You
need to ensure that the outcome(s) you are evaluating will bemeasurable with the time available.
Available data - Are the necessary data already available? If notthen you need to be able to collect these data over the course of
the project.
Available resources - Ensure that your plans are realistic in light of
the time, skills and resources available.
Refer to the factsheet Evaluation feasibility checklist for a set ofcriteria that you can use to judge whether a particular evaluationquestion will be feasible to evaluate.

41

PRISM - TOOLKIT FOR EVALUATING THE OUTCOMES AND IMPACTS OF SMALL/MEDIUM-SIZED CONSERVATION PROJECTS

2:1

INTRODUCTION &KEY CONCEPTS

STEP 1DESIGNING & IMPLEMENTING
THE EVALUATION

PRISMMODULESMETHOD FACTSHEETS& ANNEXES

Tip for prioritising questions - The ultimate aim of mostIf evaluation of conservation impact is not feasible then move backconservation projects is to have an impact on the status of adown the chain to look at threat reduction outcomes (e.g. reduced

particular species or habitat. So when prioritising evaluation

consumption of a hunted species following an education programme,questions, one way to approach this is to begin with your project’s

reduction in invasive species population), and if this is not possible

impacts, develop appropriate questions and then apply the criteria

move the focus to intermediate outcomes. Then when interpreting

above to see whether the questions are feasible to measure.

and applying the results these should be explained in the context of
how changes relating to these outcomes can be expected to lead to

TIP FORPRIORITISINGQUESTIONS

subsequent outcomes & impacts.

Feasible to measure

ActionOutputs

IntermediateoutcomeIntermediateoutcomeThreat reduction
outcome

Impact on
biodoversity

Focus of evaluation

Feasible to measure

ActionOutputs

IntermediateoutcomeIntermediateoutcomeThreat reduction
outcome

Impact on
biodoversity

Focus of evaluation

Feasible to measure

ActionOutputs

IntermediateoutcomeIntermediateoutcome

Focus of evaluation

42

PRISM - TOOLKIT FOR EVALUATING THE OUTCOMES AND IMPACTS OF SMALL/MEDIUM-SIZED CONSERVATION PROJECTS

Threat reduction
outcome

Impact on
biodoversity

2:1

INTRODUCTION &KEY CONCEPTS

STEP 1PRIORITISATION
MATRIX43

DESIGNING & IMPLEMENTING
THE EVALUATION

PRISMMODULES

Prioritisation matrix
A very useful exercise that can help to prioritise evaluationquestions is to complete a prioritisation matrix. This helps you
to review the usefulness and feasibility of each of the project’s
outcomes & impacts and to prioritise evaluation questions basedon the criteria outlined in this section. See below for a completed
example. Refer to the factsheet Completing a prioritisation matrix
for more information.

PRISM - TOOLKIT FOR EVALUATING THE OUTCOMES AND IMPACTS OF SMALL/MEDIUM-SIZED CONSERVATION PROJECTSMETHOD FACTSHEETS& ANNEXES

2:1

INTRODUCTION &KEY CONCEPTS

DESIGNING & IMPLEMENTING
THE EVALUATION

PRISMMODULESMETHOD FACTSHEETS& ANNEXES

Prioritisation matrix

STEP 1PRIORITISATION
MATRIX44

OUTCOME/
IMPACTEVALUATIONQUESTION(S)

WHICH STAKEHOLDERS
CARE ABOUT THIS?

IMPORTANCETO USERSIMPORTANCETO PROJECTSUCCESSHOW WELLKNOWN ISTHE ACTIONBEINGEVALUATED?

EASE OF DATACOLLECTIONCAN THIS BEANSWEREDGIVEN YOURTIME &RESOURCESPRIORITY

Status of native

To what extent have populations ofParticipating households

High

High

Low

Low

No

Eliminate – Not

forest species
improves

native forest species changed since theproject began and if so to what extent

Local authorities;
Project teampossible to measure
within project

can this be attributed to the project?

Project fundertimeline

Forest loss

Has there been a change in the rate ofParticipating households

slowed/reversed

forest loss since the project began and if
so can this change be expected to persist

Local authorities;
Project teamProject funder

Local people
manage forest
resources moresustainably

Has there been a change in the way local
people manage forest resources and if
so to what extent can this be attributed
to the project?

Participating households
Local authorities;
Project teamProject funder

High

High

Low

Med

Yes

High - Crucial toproject success andmeasurable with available time / resources

Local people areincentivised to
manage forest
more sustainably

Have the project’s actions given local
people more of an incentive to manage
forest resources sustainably and if so
can this be expected to persist?

Participating households;
Local authorities

Med

High

Low

Med

Yes

High - Crucial toproject success andmeasurable with available time / resources

Local people haveright to utilise
forest resources

Have agreements provided local people
with the right to utilise forest resources?

Participating households
Local authorities

High

High

Med

High

Yes

High - Crucial toproject success andmeasurable with available time / resources

Householdlivelihood/
wellbeing
improves

Have household livelihoods and/or
wellbeing status improved as a result of
the project’s actions

Participating households
Local community
Project Funder

High

High

Med

Low

Partially

High - Crucial toproject success andmeasurable with available time / resources

Rattan provides
additional income
to households

Has the rattan planted with support
from the project provided (or will
provide) households with additional
income?

Participating households
Local community

Med

High

Med

Med

Partially

Med – Rattan yet tobe harvested whichmeans income canonly be estimated.

Trained

Are trained individuals applying any ofProject team

Med

Med

Med

Med

Yes

individuals apply
new skills

the skills learnt?

Participating households

Low – Not asimportant as other

Med

High

Low

Low

No

Eliminate – Not
possible to measure
within project
timeline

PRISM - TOOLKIT FOR EVALUATING THE OUTCOMES AND IMPACTS OF SMALL/MEDIUM-SIZED CONSERVATION PROJECTS

questions

2:1

INTRODUCTION &KEY CONCEPTS

STEP 1IDENTIFYINGSUB-QUESTIONS &INDICATORSDESIGNING & IMPLEMENTING
THE EVALUATION

Identifying sub-questions & indicators
Once you have your list of priority evaluation questions it will often
be necessary to break each of these down into a number of sub
questions that will suggest what data need to be collected (seebelow for an example). Then for each sub-question you will need
to identify indicators that will allow you to know whether you haveanswered each question.

PRISM
MODULESMETHOD FACTSHEETS& ANNEXES

Indicators should be simple and specific, and can be numerical(quantitative) or non-numerical (qualitative). The table below
outlines potential indicators for the high priority questions listed in
the matrix above. See the summary tables in the module chaptersfor further examples of potential indicators as well as guidance on
how to identify appropriate indicators for each module.

An indicator is a variable that provides a reliable means of measuring aparticular outcome or impact.

OUTCOMEEVALUATION QUESTION

SUB- QUESTIONSINDICATORS

Do local people now monitor

Number of monitoring patrols

areas covered by agreements?

carried out by households

Do local people report

Number of infractions reported

infractions to the forest
Local people manage forest

Has the project changed theresources more sustainably

way local people manage forest
resources?

protection department?
Are local households converting

50% reduction in annual arealess of the forest for pastureof land converted to pasturethan before the project?

compared to before the project

Do the forest protection

Key informant interviews

department feel that thereport that forest protection

forest is being managed more

department staff feel that moresustainably than before?

sustainable management istaking place

45

PRISM - TOOLKIT FOR EVALUATING THE OUTCOMES AND IMPACTS OF SMALL/MEDIUM-SIZED CONSERVATION PROJECTS

2:1

INTRODUCTION &KEY CONCEPTS

STEP 1IDENTIFYINGSUB-QUESTIONS &INDICATORSDESIGNING & IMPLEMENTING
THE EVALUATION

It is important not just to choose questions & indicators that are
the easiest to use, for example because the data already exist orother people have used them in the past. Instead, always thinkback to why you are doing the evaluation in the first place, andconsider whether the questions and indicators selected will give
you the information that you want. In many cases, it may be betterto have a more ambiguous qualitative answer than to persist
with using an indicator that may be misleading, for the sake ofgenerating a number. For example, if an action claims to improve
local participation in land use governance, a common indicatoris attendance at community meetings, but this does not capture
whether people had a voice in the meetings or whether theircontributions were taken into account in decision-making, forwhich more qualitative data is needed.

PRISM
MODULES

Indicators for project actions & outputs
It is generally good practice (and often the minimum requirement
for reporting to donors) for all projects to develop indicators andto record information on their outputs and the delivery of actions.
These are typically recorded in the form of quantity and quality ofdelivery (e.g. number of stakeholders reached, number of surveys
carried out, immediate reaction to a training workshop). As PRISM
is focused on evaluation of outcomes & impacts, guidance on
developing indicators and collecting information on outputs anddelivery of project actions is not covered in detail here. However,when designing your project it is important to consider how thisinformation will be collected, when and by whom.

Note that many standardised lists of indicators exist for conservation
actions, however it is almost always far more useful to develop
indicators yourself based on your project’s specific outcomes/
impacts and the evaluation questions you are trying to answer.

46

METHOD FACTSHEETS& ANNEXESPRISM - TOOLKIT FOR EVALUATING THE OUTCOMES AND IMPACTS OF SMALL/MEDIUM-SIZED CONSERVATION PROJECTS

2:1

INTRODUCTION &KEY CONCEPTS

STEP 1
ARE THERE ANY
UNINTENDEDOUTCOMES/IMPACTS I NEEDTO CONSIDER?

47

DESIGNING & IMPLEMENTING
THE EVALUATION

Are there any unintended outcomes/impacts I need to consider?
Capturing information on unintended outcomes & impacts isimmensely valuable (see Section 1), however because there areso many ways in which a project or action could potentially bringabout unintended outcomes & impacts it is often difficult to know
where to start. There are two main ways you can approach this:1.

Work internally within the project team before,
during and after the project to think about
unintended outcomes/impacts. This can include
an initial risk assessment and then having points
during the implementation of the project where
the team meet to discuss possible unintended
outcomes/impacts and how data on these can becollected. Exploring assumptions in the project’s
Theory of Change can also help to identify
unintended outcomes/impacts.

2.

Include a participatory element in your evaluation
where you ask project stakeholders what they feelthe main outcomes and impacts of the project willbe/are/were.

PRISM
MODULESMETHOD FACTSHEETS& ANNEXES

If possible you should aim to use a combination of both of theseapproaches. If you decide that the outcomes/impacts identified
are significant enough to warrant further investigation, you canbuild questions into your evaluation that are designed to capture
relevant information on them. In any case it is usually good practice
to include questions in your evaluation that are open-ended inorder to explore why, and how, changes have occurred and what
they mean in the context of the project.

PRISM - TOOLKIT FOR EVALUATING THE OUTCOMES AND IMPACTS OF SMALL/MEDIUM-SIZED CONSERVATION PROJECTS

2:1

INTRODUCTION &KEY CONCEPTS

WHAT DO YOU NEED TO KNOW?
CHECKLIST

48•

Have you outlined your project’s outcomes &impacts in a Theory of Change (or equivalent)?

•

Have you identified your evaluation uses &users?

•Have you identified potential evaluationquestions for the project’s outcomes/impacts?

•

Have you prioritised the most importantquestions to answer, based on usefulness &feasibility?

•

Have you developed indicators that will allow
you to know when you have answered eachquestion?

•

Have you considered potential unintended
outcomes/impacts?

DESIGNING & IMPLEMENTING
THE EVALUATION

PRISMMODULESMETHOD FACTSHEETS& ANNEXES

WHAT DO YOU NEED TO KNOW? – TIPS FOR REDUCING TIME/COSTS
POTENTIAL WAY OF REDUCING COSTS

CHALLENGES

SOLUTIONSReduce number of evaluationquestions

Evaluation may no longer meet theneeds of audience

Carefully prioritise evaluationquestions

PRISM - TOOLKIT FOR EVALUATING THE OUTCOMES AND IMPACTS OF SMALL/MEDIUM-SIZED CONSERVATION PROJECTS

2:2

INTRODUCTION &KEY CONCEPTS

Step 2

STEP 2
WHAT DATA DO
YOU NEED TO
COLLECT?

DESIGNING & IMPLEMENTING
THE EVALUATION

PRISMMODULES

What data do you need to collect?

This is where you design your evaluation in a way that will allow
you to answer your evaluation questions and to select appropriate
methods that will allow you to collect the data required.

STEP 2QUESTIONS ADDRESSED IN THIS SECTION:•
For this step note the difference between evaluation design anddata collection methods. Your data collection methods are thetools you will use to collect data (questionnaires, interviews etc.).
Your evaluation design outlines how the data collection will be
structured in a way that will allow you to answer your evaluationquestions.

•49

METHOD FACTSHEETS& ANNEXES

How will you know which outcomes/impacts can be
attributed to the project?
oWhat is evaluation design?

oWhich evaluation design should I use?

o

Am I measuring attribution or contribution?

How will you collect evaluation data?
oWhat kind of data are needed?

oCan any existing data be used?

o

Do I need to control for bias?

o

Are the methods ethically sound?

oWhat sampling strategy should I use?

oWhat data collection methods should I use?

PRISM - TOOLKIT FOR EVALUATING THE OUTCOMES AND IMPACTS OF SMALL/MEDIUM-SIZED CONSERVATION PROJECTS

2:2

INTRODUCTION &KEY CONCEPTS

DESIGNING & IMPLEMENTING
THE EVALUATION

How will you know which outcomes/impacts can be attributed
to the project?

STEP 2
HOW WILL YOU
KNOW WHICH
OUTCOMES/IMPACTS CAN BE
ATTRIBUTED TO
THE PROJECT?

What is evaluation design?
In any area where a project is implemented, changes will take placeover time. Some of these changes will have occurred as a result of
the project, while other changes may have nothing to do with the
project and would have happened regardless of whether or not
the project existed. Therefore it is usually the aim of an evaluation
to try to find out whether any observed changes were due to theproject.
In order to find this out your data collection needs to be designed
in a way that will allow you to distinguish the amount of changethat can be claimed by the project, or by a particular project action,from the total change that takes place (i.e. how much of a particularoutcome or impact is the direct result of the project’s actions). Thisis known as attribution.

Project
outcomes/
impacts50=

Total changes

–

PRISM
MODULESMETHOD FACTSHEETS& ANNEXES

A good evalaution design should allow you to address the following
questions:•

What difference did the action/project make?

•How did it make this difference?

•

What other factors were relevant?

Evaluations that consider these questions produce stronger andmore useful findings. This is because the emphasis is on finding out
what difference the project has made, rather than just looking at
what has happened. On the other hand a flawed evaluation design
can lead to results that can be misinterpreted or can cast doubt on
the validity of the results.

Changes not due
to the project

PRISM - TOOLKIT FOR EVALUATING THE OUTCOMES AND IMPACTS OF SMALL/MEDIUM-SIZED CONSERVATION PROJECTS

2:2

INTRODUCTION &KEY CONCEPTS

DESIGNING & IMPLEMENTING
THE EVALUATION

EXAMPLE OF A BAD EVALUATION DESIGN
A project measured the height of all the trees in a patch of

STEP 2

forest, then erected a fence around all the trees and measured
them again after a period of three years. The project found thatmost of the trees were taller at the end than they were at the

HOW WILL YOU
KNOW WHICH
OUTCOMES/IMPACTS CAN BE
ATTRIBUTED TO
THE PROJECT?

start. The project concluded that the fence had made the trees
taller.
In this example the main flaw in the evaluation design wasthat the project gave no basis for comparing the fenced trees’
growth with that of non-fenced trees in other patches of forest,
nor did it consider other factors that may promote tree growth.

PRISM
MODULESMETHOD FACTSHEETS& ANNEXESThe approach you choose for addressing attribution will depend
largely on the availability of the following:
Theory of Change – A strong Theory of Change gives you aframework with which to test whether particular outcomes/
impacts have been achieved. Evaluations produce stronger
findings if they not only test the links between actions and impacts,
but also test the steps between intermediate outcomes. See
PRISM Step 1 for more information.
Counterfactual – A counterfactual is an estimate of what would
have happened without the project. What a counterfactual gives
you is a measure of the results that would have been observed
if the project had not taken place and all other conditions had
remained the same. In practice these can be real world scenarios
based on a control group/control site or constructed fromparticipants’ or experts’ knowledge.
Baseline – A starting point, or a baseline, in order to assess changeover time. If the project did not establish a baseline at the outset
this can sometimes be constructed retrospectively (ex-post) fromsecondary data (e.g. local records) or carefully designed surveys
that make use of participant recall or expert opinion.
The following resource contains useful guidance on
how to construct a baseline retrospectively: http://
siteresources.worldbank.org/INTPOVERTY/
Resources/335642-1276521901256/premnoteME4.pdf .

51

PRISM - TOOLKIT FOR EVALUATING THE OUTCOMES AND IMPACTS OF SMALL/MEDIUM-SIZED CONSERVATION PROJECTS

2:2

INTRODUCTION &KEY CONCEPTS

STEP 2WHICH
EVALUATIONDESIGNSHOULDI USE?

DESIGNING & IMPLEMENTING
THE EVALUATION

PRISMMODULESMETHOD FACTSHEETS& ANNEXES

Which evaluation design should I use?
The most rigorous option is to view the evaluation as anexperiment, for example by treating some patches of an invasiveplant in one manner and other patches in another, then comparing
the results. However this kind of approach is often challenging
for smaller projects. Small/medium-sized conservation projects
must therefore often consider other options which are feasible toimplement but still deliver meaningful results.
The table below outlines some lower cost approaches for
attributing outcomes & impacts to project actions. Each of thefollowing methods is outlined in more detail in the dedicatedevaluation design factsheets.

Photo credit: Tsubasa Iwabuchi

52

PRISM - TOOLKIT FOR EVALUATING THE OUTCOMES AND IMPACTS OF SMALL/MEDIUM-SIZED CONSERVATION PROJECTS

2:2

INTRODUCTION &KEY CONCEPTS

DESIGNING & IMPLEMENTING
THE EVALUATION

PRISMMODULESMETHOD FACTSHEETS& ANNEXESTABLE: POTENTIAL EVALUATION DESIGNS FOR SMALL/MEDIUM-SIZED PROJECTS.
DESIGN TYPEDESCRIPTION

EXAMPLESADVANTAGESCHALLENGES

Participatory Impact Assessment

Asking project participantsquestions about what changes haveoccurred and then discussing thereasons for the answers given

Asking project participants to scoreindicators relating to forest resource
use and discussing the reasons forthe answers given

Relatively cheap, does not require abaseline study or control group

Participants’ memories can changeover time

Asking certain key individuals
about what they believe caused
any observed change and why/howthese changes happened

Asking national park staff if they feelthe project improved species status
within the project target area

Relatively cheap, does not require abaseline study or control group

Checking that results support the
project’s Theory of Change (ToC)

Checking achievement ofintermediate outcomesUtilises pre-existing Theory of
ChangeContribution analysis

Often less resource intensive
and not subject to many of thechallenges associated withtraditional matching designs (seebelow)

Interviewing Key Informants

Theory-based designs

Matching designs

Before - after designs

Usually the best way ofunderstanding how the project has
affected participants
Key informants need to be familiar
with the project and the outcome/
impact being evaluatedUseful for verifying results collectedfrom other methods
Requires a thorough understandingof the mechanisms that drive change
and the species/habitats impacted(e.g. well-developed ToC)
Potential to focus only on expected
impacts and ignore unintended or
negative impacts

Comparing the results of the action
to a real world control group/control
site that is not subject to the action.
Control group/site selected to be as
similar as possible to the group/sitetargeted by the project

Comparing the trends in status of
species within an area that received
project support with a similar areathat did not receive project support

If done properly, matching designs
eliminate many of the potential
sources of bias that could affect thevalidity of an evaluation

Often higher cost & require moreresources than other designs

Measuring the situation before
the action and then again after theactionComparing the knowledge/
awareness of participants before
and after a training workshop

Can be useful if the outcome/impact
is short term or is part of a simple
causal chain

Does not provide firm proof of
attribution

Often more resource intensive andsubject to a number challenges
relating to logistics and ethics

Can only imply (rather than prove)that change occurred due to theproject

53

PRISM - TOOLKIT FOR EVALUATING THE OUTCOMES AND IMPACTS OF SMALL/MEDIUM-SIZED CONSERVATION PROJECTS

2:2

INTRODUCTION &KEY CONCEPTS

STEP 2WHICH
EVALUATIONDESIGNSHOULDI USE?

DESIGNING & IMPLEMENTING
THE EVALUATION

Selecting an appropriate design involves looking at the following
three factors. Remember that every project is different, therefore
it is up to you and your team to think about these factors and tochoose the evaluation design that is most appropriate for yourparticular project.
Nature of the action being evaluatedoComplexity of the actionoWhat project outcomes/impacts will be measurable?

o

Are there gaps or inconsistencies in the project’s Theoryof Change that would make evaluation difficult?

Nature of the evaluationo
Who are the evaluation users and what do they want to
know?
o

PRISM
MODULESMETHOD FACTSHEETS& ANNEXES

The PRISM module chapters each contain further guidance on
which evaluation designs are suitable for particular conservation
outcomes & impacts. Refer to the evaluation design factsheets formore information on each design type.
Remember that none of the designs outlined in this section isinherently better than any of the others. In choosing a design you
need to think carefully about your particular situation and choose
a design or a combination of designs that is appropriate for yourneeds.
See also the Decision tree for selecting an evaluation design for
more help in selecting a design appropriate for your project.

DESIGN APPROACHES FOR EACH OF PRISM’S MODULES
AWARENESS& ATTITUDES

What are the key questions being evaluated?

CAPACITYBUILDINGLIVELIHOODS &GOVERNANCE

POLICY

SPECIES &HABITATMANAGEMENT

Participatory
ImpactAssessment

Available resources and constraintsoExisting evidence available (e.g. baseline data)oResources (e.g. time, money, expertise within the team/organisation)

Interviewing
Key
Informants
Theory-based
designs
Matching
Designs
Before-After
Designs

Table: Design approaches for each of PRISM’s modules. Solid circles = most appropriate,
empty circles = sometimes appropriate, no circle = less appropriate.

54

PRISM - TOOLKIT FOR EVALUATING THE OUTCOMES AND IMPACTS OF SMALL/MEDIUM-SIZED CONSERVATION PROJECTS

2:2

INTRODUCTION &KEY CONCEPTS

STEP 2AM I MEASURINGATTRIBUTION OR
CONTRIBUTION?

DESIGNING & IMPLEMENTING
THE EVALUATION

Am I measuring attribution or contribution?
In some cases it will be possible to attribute an outcome/impact
directly to the project’s actions. However, in many conservation
project settings, and particularly for smaller projects, achievementof outcomes & impacts will often be dependent on other actions,projects, policies or programmes. This is often referred to as
contribution.

PRISM
MODULES

EXAMPLE - PROJECT OUTCOME THAT CAN BE
DIRECTLY ATTRIBUTED TO THE PROJECT:
The amount of firewood used by participating households
reduced after the introduction of fuel-efficient stoves.
Households that were not involved in the project were stillusing the same amount of firewood as all households in the

Although the two terms are presented as separate categories it
is perhaps most useful to think of this as a spectrum from “effects
directly and solely attributable to the project” at one extreme,to “effects achieved indirectly and/or in combination with otherinfluences” at the other.
Many projects struggle when trying to evaluate an action that
contributed towards a wider outcome/impact. However thebasic questions are still the same - what difference did the action/
project make? How did it make this difference, and what otherfactors were relevant? Although you may not be able to quantify
accurately the amount of change the project was responsible
for, you can still learn an enormous amount by considering thesequestions. All the design options listed in this section can be used
to evaluate an outcomes/impacts that can be directly and solely
attributed to the project as well as outcomes/impacts to which the
project made an indirect or partial contribution.

community had done before the project. No other actions weretargeting firewood use, meaning that any change(s) in this after
the introduction of stoves could likely be directly attributed tothe project.

EXAMPLE - CONTRIBUTING TO A WIDER OUTCOME/
IMPACT:
A new environmental law was passed, upholding a ban ondevelopment in a national park. The project team had carriedout meetings and extensive lobbying & campaigning in supportof the new law. The passing of the law had also been supported
by several other NGOs, community groups and businesses.
This meant that it was likely that the project contributed
to a portion of the overall change, rather than being solely
responsible for it.

55

METHOD FACTSHEETS& ANNEXESPRISM - TOOLKIT FOR EVALUATING THE OUTCOMES AND IMPACTS OF SMALL/MEDIUM-SIZED CONSERVATION PROJECTS

2:2

INTRODUCTION &KEY CONCEPTS

STEP 2
HOW WILLYOU COLLECTEVALUATION
DATA?

56

DESIGNING & IMPLEMENTING
THE EVALUATION

How will you collect evaluation data?
When you have identified an appropriate design for yourevaluation you then need to identify methods that will allow you tocollect the data required to answer your evaluation questions. Thissection outlines several key considerations that are relevant to alldata collection methods before describing how PRISM’s thematic
modules can be used to select methods that are relevant to specifictypes of conservation actions.

PRISM
MODULESMETHOD FACTSHEETS& ANNEXES

Some methods will only deliver one kind of data while others canbe adapted to provide both. Where possible it is recommended totry to gather data (both quantitative and qualitative) using a rangeof different methods and then to “triangulate” the findings. This
enables conclusions to be drawn from smaller sample sizes. See
Step 3. What can you learn from the results? for more information
on triangulation and how to combine and analyse data frommultiple sources.

What kind of data are needed?
When selecting methods you will need to decide whether you
need data that are quantitative, qualitative or both. Quantitative
data is numerical (numbers or answers to closed questions)while qualitative data is non-numerical (observations, answers
to open questions, written, audio, visual or video evidence).
Quantitative data are more straightforward to measure using
standard statistical analysis methods. However qualitative data
can provide extremely useful insights and information that cannot
be captured by quantitative data, for example the underlying
reasons, opinions and motivations behind a change in participant
behaviour. Increasingly, many evaluations collect both quantitativeand qualitative data: this is called a mixed methods approach.

PRISM - TOOLKIT FOR EVALUATING THE OUTCOMES AND IMPACTS OF SMALL/MEDIUM-SIZED CONSERVATION PROJECTS

2:2

INTRODUCTION &KEY CONCEPTS

DESIGNING & IMPLEMENTING
THE EVALUATION

PRISMMODULESMETHOD FACTSHEETS& ANNEXES

DIFFERENCES BETWEEN QUANTITATIVE, QUALITATIVE AND MIXED DATACOLLECTION (ADAPTED FROM WOODHOUSE ET AL. 2016)

STEP 2
HOW WILLYOU COLLECTEVALUATIONDATA?

TYPE OF DATAMOST USEFUL FOR

SPECIFIC TOOLS AND METHODS

Quantitative

Measuring magnitude of changes

Structured questionnaires (closed questions) –(numeric)

Collecting objective, measurable datainterviews or self-completed

Statistical analysis

Field survey data (transects, point counts etc.)

Reporting to external interest groups (donors,

Direct measurements

governments)

Some participatory methods which involve ranking
and scoring

Qualitative

Contextual knowledge

Focus group discussions(narrative based)

Processes of change

Field observation notes

Providing basis for quantitative questions

Key informant interviews

Uncertain outcomes, outcomes that are difficult

Participant observation

to measure (e.g. social relations, politicalchange),Adaptive management and understanding
mechanisms
Mixed method (quantitative &Combining strengths of both approaches for

Combining the collection of different data

quantitative)

different insights and audiences

types within single methods (e.g. qualitative andTriangulation of understandings based on

quantitative questions within a household survey), or

information from different approaches

using a range of methods within the evaluation (e.g.

Reporting to external interest groups (donors,

focus groups and key informant interviews prior to agovernments)

household survey).
Some specific tools described in this toolkit use mixed

Woodhouse, E et al. (2016) Evaluating theimpacts of conservation interventions
on human wellbeing. Guidance forpractitioners. IIED, London

57

methods e.g. PIA methods, Basic Necessities Survey

PRISM - TOOLKIT FOR EVALUATING THE OUTCOMES AND IMPACTS OF SMALL/MEDIUM-SIZED CONSERVATION PROJECTS

2:2

INTRODUCTION &KEY CONCEPTS

STEP 2CAN ANY EXISTINGDATA BE USED?

DESIGNING & IMPLEMENTING
THE EVALUATION

Can any existing data be used?
Start planning your data collection by reviewing the extent towhich existing data can be used. For example data from official
statistics, programme monitoring data, project records (project
proposal, a theory of change, minutes from relevant meetings,etc.), formal policy documents, programme implementation plansand progress reports may all be helpful. Data that can be used asa baseline to track change over time are particularly useful. While
making use of existing data can save a lot of time and effort, it isessential that data are of sufficient quality to still provide useful
information in relation to your evaluation questions. If data are
not of sufficient quality it may compromise the validity of theevaluation results.

PRISM
MODULESMETHOD FACTSHEETS& ANNEXES

Do I need to control for bias?
Bias refers to errors during data collection or interpretation that
can affect the reliability of the evaluation results. For example
it may be difficult to compare reliably the results of bird surveys
conducted by two observers with very different levels of
ability. With most methods some level of bias is unavoidable. By
identifying the sources of this it is possible to work out whether itwill have a significant effect on the reliability of your conclusions
and whether or not it can be minimised, reduced or measured andallowed for during analysis. If bias cannot be controlled in theseways then a different method should be used. The table³ on thefollowing page covers some (but by no means all!) potential sourcesof bias and strategies for addressing them.

³Cambridge handbook of ecological survey methods https://sunsetridgemsbiology.
wikispaces.com/file/view/Biodiversity+Handbook.pdf

58

PRISM - TOOLKIT FOR EVALUATING THE OUTCOMES AND IMPACTS OF SMALL/MEDIUM-SIZED CONSERVATION PROJECTS

2:2

INTRODUCTION &KEY CONCEPTS

DESIGNING & IMPLEMENTING
THE EVALUATION

PRISMMODULESMETHOD FACTSHEETS& ANNEXESTABLE: EXAMPLES OF POTENTIAL SOURCES OF BIAS AND STRATEGIES FOR MINIMISING THEM

STEP 2TARGETPOTENTIAL SOURCES OF BIAS

STRATEGIES FOR MINIMISING/REDUCING BIAS

Species & habitats

Some species/habitats may be easier to survey
than others

Record as much detail as possible and use the same
methods, approach, analysis etc. across years, surveys& sitesObservers have differing levels of expertiseDO I NEED TO
CONTROL FOR
BIAS?

Variation in observer effort
Local conditions may affect data collection (e.g.
weather)
Stakeholders

Stakeholders with a particular viewpoint (e.g.
favourable/ unfavourable to the project) are
more likely to be involved in the evaluation
Some stakeholders may be easier to reach thanothersLoaded or leading questions make certain
responses more likely
Observers/facilitators have differing levels of
experience
Some groups may be less likely to respond (e.g.
based on age, gender, social status, etc.)
Participants’ memories can change over time
Participants may be unwilling to share certain
pieces of information

59

PRISM - TOOLKIT FOR EVALUATING THE OUTCOMES AND IMPACTS OF SMALL/MEDIUM-SIZED CONSERVATION PROJECTSAgree & record definitions (e.g. sample size, type
population unit) beforehand
Calibrate observers against each other before &during data collection
Identify potential sources of bias and develop asampling strategy that works to minimise these(see section on developing an appropriate samplingstrategy)
Use the same observers/facilitators if possible, orcalibrate observers against each other beforehand
Test the survey methods beforehand with a sample ofthe target audience
Use a timeline with participants to help them recall
earlier conditions

2:2

INTRODUCTION &KEY CONCEPTS

STEP 2
ARE THEMETHODSETHICALLY
SOUND?

60

DESIGNING & IMPLEMENTING
THE EVALUATION

Are the methods ethically sound?
Before implementing any data collection method it is importantthat you go through an ethical review procedure to ensure that you
have thought about potential issues of this kind that the chosen
methods may raise in relation to the species, habitat or stakeholder
group being targeted. Failing to consider ethics can have severe
consequences for the validity of evaluation results and runs therisk of having a negative effect on the project target, while also
potentially posing serious reputation risks for the organisation(s)responsible for the project.

PRISM
MODULESMETHOD FACTSHEETS& ANNEXES

GenderMany conservation actions will have outcomes and impacts that
affect male and female stakeholders differently. Building gender
into your evaluation can therefore provide extremely useful
information and insights that would not be captured otherwise.
Refer to the factsheet: Developing gender protocols for an
evaluation for more information.

Refer to the factsheet: Developing ethical protocols for an
evaluation for more information.

PRISM - TOOLKIT FOR EVALUATING THE OUTCOMES AND IMPACTS OF SMALL/MEDIUM-SIZED CONSERVATION PROJECTS

2:2

INTRODUCTION &KEY CONCEPTS

STEP 2
WHAT SAMPLINGSTRATEGY SHOULDI USE?

DESIGNING & IMPLEMENTING
THE EVALUATION

PRISMMODULESMETHOD FACTSHEETS& ANNEXES

What sampling strategy should I use?
Adapted from Mwebaze, T (2014) - “How To”: Choose Sampling
Techniques for Evaluations: https://ecduganda.files.wordpress.
com/2014/08/how-to-choose-sampling-techniques-forevaluations.pdf
Sampling is the process of selecting units (e.g. people, sites, species)
from a population of interest, studying these in greater detail,and then drawing conclusions about the larger population. The
evaluator collects data from a subset – a sample – and uses theseto make inferences about the entire story. For this to be reliablehowever the key thing is that the characteristics of the sample
must reflect the characteristics of the population targeted by theaction.

Photo credit: M. K. Poulsen

61

PRISM - TOOLKIT FOR EVALUATING THE OUTCOMES AND IMPACTS OF SMALL/MEDIUM-SIZED CONSERVATION PROJECTS

2:2

INTRODUCTION &KEY CONCEPTS

DESIGNING & IMPLEMENTING
THE EVALUATION

PRISMMODULESMETHOD FACTSHEETS& ANNEXESTABLE: SAMPLING STRATEGIES FOR EVALUATIONSTEP 2
WHAT SAMPLINGSTRATEGY SHOULDI USE?

62

SAMPLING METHODDESCRIPTIONBEST WHEN

Simple random
sampling

All units in the population have an equal opportunity of being selected, using a methodthat selects units completely at random.

The whole population isavailable.

Stratified sampling(random withintarget groups)

Divide the population into meaningful homogenous or similar groups based on a given
characteristic (e.g. age, gender, race, socioeconomic status), and then select a random
sample from within each group.

There are specific subgroups to investigate (e.g.
demographic groupings/
target sites).

Systematic sampling(every nth
person)Samples are drawn by starting at a randomly selected element in the sampling frame
and then taking every nth element. This is easier to perform and hence is less subject
to researcher errors than simple random sampling.

A stream of subjects isavailable (e.g. people in thestreet).

Cluster sampling

The sample is selected in stages, first selecting groups of elements, or clusters (e.g. city
blocks, census tracts, schools), and then selecting individual elements from within eachcluster (e.g. randomly or by systematic sampling).

Population groups are
separated and access to all ofthem is difficult, e.g. in remotecommunities.

Quota sampling

The population is divided into groups according to important population
characteristics. Then people familiar with the population choose individuals they
believe can adequately represent that population’s characteristics (e.g. their
viewpoints).

Time is limited, a samplingframe is not available anddetailed accuracy is notimportant.

Purposive sampling

Sampling with a purpose in mind. Usually one or more specific target groups would
be pre-defined. Purposive sampling can be very useful for situations where you needto reach a targeted sample quickly and where proportionality in sampling is not the
primary concern.

You are studying particulargroups.

Snowball sampling(asking forrecommendations)

A type of purposive sampling where existing participants recruit further subjects fromamong their acquaintances. A few potential respondents are contacted and asked
whether they know of anyone else with the same required characteristics for theevaluation.

You are targeting subjects
with something in common(e.g. hunters in a community).

PRISM - TOOLKIT FOR EVALUATING THE OUTCOMES AND IMPACTS OF SMALL/MEDIUM-SIZED CONSERVATION PROJECTS

2:2

INTRODUCTION &KEY CONCEPTS

DESIGNING & IMPLEMENTING
THE EVALUATION

What data collection methods should I use?
This section provides details on how to select appropriate datacollection methods from PRISM’s thematic modules.

STEP 2
WHAT DATACOLLECTIONMETHODSSHOULD I USE?

With most outcomes/impacts there will be range of different
methods that could potentially be used to collect data. The PRISM
module chapters are designed to guide you towards methods thatare appropriate and feasible for the outcome/impact you are trying
to measure.
Start by taking each of the outcomes/impacts you identified insteps 1 & 2 and see which of PRISM’s modules they relate to (seetable on following page). You can use the framework and summary
information within each module to guide you towards methods formeasuring change in relation to the outcome/impact of interest.
The first four modules focus on project stakeholders andspecifically on different dimensions of behaviour change by
stakeholders.

PRISM
MODULES

while the module chapters are designed to cover a wide range ofoutcomes and impacts, they do not cover all the possible outcomes
and impacts that a small/medium-sized project may have. If theparticular outcome or impact you are trying to measure does notfeature in the module chapters it is still worth looking throughthe methods, as many of them can be adapted to suit particularscenarios and contexts.
Conservation actions will typically have multiple outcomes andimpacts, some of which may relate to different modules. For
example, training park rangers to conduct patrols may haveoutcomes that relate to capacity development, attitudes &awareness and/or species & habitat management. When selecting
methods it is important to focus on the primary outcomes andimpacts that any action is expected to achieve, and to look formethods for each of them in the relevant module chapter.

The fifth module, Species & Habitat Management, relates toactions that are directly focused on restoring and/or reducing
threats to species and habitat(s), while also covering how toevaluate the impact of conservation research and how to measurechanges in species & habitat status.
Projects looking at ecosystem services should refer to thefactsheet: Evaluating ecosystem services.
These modules have been selected because they cover a rangeof outcomes and impacts that are commonly aimed for by small/medium-sized conservation projects. Please remember that,63

METHOD FACTSHEETS& ANNEXESPRISM - TOOLKIT FOR EVALUATING THE OUTCOMES AND IMPACTS OF SMALL/MEDIUM-SIZED CONSERVATION PROJECTS

2:2

INTRODUCTION &KEY CONCEPTS

DESIGNING & IMPLEMENTING
THE EVALUATION

PRISMMODULESMETHOD FACTSHEETS& ANNEXESTABLE: THEMATIC MODULES SHOWING OUTCOMES AND IMPACTS COVEREDSTEP 2
WHAT DATACOLLECTIONMETHODSSHOULD I USE?

POSSIBLE MODULE

DESCRIPTIONSPECIFIC TOOLS AND METHODS

Attitudes and awarenessMeasuring changes in levelsof stakeholder awareness andattitudes and whether these leadto changes in behaviour that
benefit conservation.

•
•
•
•

Messages received by target audience
Audience has desired attitude
Audience has desired awareness/ knowledge
Audience adopts desired behaviour

Capacity development

Measuring changes in the abilityof people, organisations orsociety to perform better, solve
problems or manage their affairs
successfully.

•
•
•
•
•
•

Individuals have improved skills
Individuals have greater confidence
Individuals apply new skills
Organisational performance improves
Networks and alliances have been formed/strengthened
Conservation actions are implemented more effectively

Livelihoods and governance

Measuring changes in relation
to people’s livelihoods, wellbeing status and the governancearrangements that affect the livesof project stakeholders.

••

Uptake of new or improved practices/ livelihoods activities
Progress towards secure tenure or resource rights forcommunities
Community institutions have increased capacity
Governance arrangements give increased recognition/ respect
Well-being goals met
Damaging occupation/ livelihood abandoned or reduced

Measuring changes to rules,
regulations and agreements thatgovern conservation targets.

•
•
•
•

Improved policy influence in government or multilateral
institutions
Improved policy influence in private sector
Environmental movement strengthened
New/improved policy and/or associated practice implemented.

Measuring changes in relation to
the direct management of species
or habitats.

•
•
•
•
•
•

Knowledge improved
Action plan(s) developed & disseminatedThreats reduced
Recovery of species/habitats promoted
Species status improved
Habitat status improved

PolicySpecies and habitat
management

64

•
•
•
•

PRISM - TOOLKIT FOR EVALUATING THE OUTCOMES AND IMPACTS OF SMALL/MEDIUM-SIZED CONSERVATION PROJECTS

2:2

INTRODUCTION &KEY CONCEPTS

DESIGNING & IMPLEMENTING
THE EVALUATION

Module chapter structure
Each module chapter contains an introduction, a diagrammatic
outcome/impact framework, a table of indicators & methods andwritten summaries for each stage in the framework.

STEP 2
MODULE CHAPTERSTRUCTURE

Each outcome/impact framework diagram illustrates how actions
lead to particular outcomes and impacts within each module (seeFigure 8). These frameworks are designed to:1.

demonstrate how your project’s actions and outcomesmight relate to overall conservation impact;2.

identify which stage of outcome or impact you want tomeasure;3.

help to identify methods that are appropriate for yourparticular actions.

PRISM
MODULESMETHOD FACTSHEETS& ANNEXES

Most projects will not measure change across all possible
outcomes and impacts in these frameworks; once you haveidentified the stage in the framework that is most relevant to
the outcome/impact you are trying to evaluate, you can use
the information in the table and written summaries to selectappropriate methods. The frameworks help to explain how theoutcome/impact being measured relates to expected overall, longterm conservation impact.
Methods are laid out in a factsheet format. Factsheets are locatedin the appendices and via hyperlinks within the interactive toolkit.

FIGURE 8: EXAMPLE FRAMEWORK (TAKEN FROMPRISM CAPACITY DEVELOPMENT MODULE)

3. Individuals
apply newskillsCAPACITY
DEVELOPMENTACTIONSe.g. Training,
Mentoring
Providing information,
Forming alliances& networks

1. Individuals
have improved
skills
4. Organizational
performance
improves

6. Conservation
actions are
implemented more
effectively

Threats decrease

Improvement inbiodiversity status

2. Individuals
have greater
confidence
5. Networks &alliances formed /strengthened

Actions

65

PRISM - TOOLKIT FOR EVALUATING THE OUTCOMES AND IMPACTS OF SMALL/MEDIUM-SIZED CONSERVATION PROJECTS

Outputs

Intermediateoutcomes

Threat reduction
outcomes

Impact on
Biodiversity

2:2

INTRODUCTION &KEY CONCEPTS

STEP 2COMPLETINGYOUR EVALUATION
PLAN

66

DESIGNING & IMPLEMENTING
THE EVALUATION

PRISMMODULES

Completing your evaluation plan
Once you have identified your evaluation questions and methods
it is important to plan and map out how data will be collected,
specifically when it will be collected, who will collect it and an
estimate of how much it will cost.
See Factsheet: Completing an evaluation plan for a simple templatethat you can use, modify or adapt to your particular project to map
out and plan your evaluation data collection.

PRISM - TOOLKIT FOR EVALUATING THE OUTCOMES AND IMPACTS OF SMALL/MEDIUM-SIZED CONSERVATION PROJECTSMETHOD FACTSHEETS& ANNEXES

2:2

INTRODUCTION &KEY CONCEPTS

WHAT DO YOU NEED TO KNOW?
CHECKLIST

PRISM
MODULESMETHOD FACTSHEETS& ANNEXES

WHAT DATA DO YOU NEED TO COLLECT? – TIPS FOR REDUCING TIME/COSTS
POTENTIAL WAY OF REDUCING COSTS

CHALLENGES

SOLUTIONS•

Have you reviewed potential evaluation
designs?

Use smaller sample sizes- May result in reduced accuracy- Check at regular intervals
to ensure sufficient data arebeing collected•

Have you selected an appropriate design foryour project?

Make more use of existing data

- Existing data may beinsufficient to answer
evaluation questions- Only use if quality,
relevance and accessibilityof data are sufficient•

Have you decided what kind of data you willneed (quantitative, qualitative or both)?

- People’s memories can changeover time- Use a systematic approachfor collecting data e.g.
PIA, interviewing key
informants, focus groups•

67

DESIGNING & IMPLEMENTING
THE EVALUATION

Have you considered potential sources of bias
and how these can be minimised, reduced orcontrolled for?

•Have you developed a sampling strategy?

•

Have you developed ethical and gender
protocols?

•

Have you selected appropriate methods from
PRISM’s modules?

Reconstruct baseline data
(e.g. by using participant recall)

Simplify evaluation design(e.g. one that does not require abaseline/counterfactual)- Often a trade-off between
design simplicity and quality/
validity of results- Review design options andensure that the chosen
option will still produce
useful results

Use less resource intensive datacollection methods(e.g. focus groups/direct
observation instead ofquestionnaires)- Often a trade-off between lowintensity methods and quality/
validity of results- Once methods have beenchosen, return to theevaluation questions andask again: will the chosen
methods collect the dataneeded to answer these?

PRISM - TOOLKIT FOR EVALUATING THE OUTCOMES AND IMPACTS OF SMALL/MEDIUM-SIZED CONSERVATION PROJECTS

2:3

INTRODUCTION &KEY CONCEPTS

Step 3

STEP 3

DESIGNING & IMPLEMENTING
THE EVALUATION

PRISMMODULES

What can you learn from the results?

This is the stage of the evaluation where you analyse and interpret
your data and then interpret the data to find out what you canlearn from the results, and what the results mean in relation to theoverall aim of the project.

STEP 3QUESTIONS ADDRESSED IN THIS SECTION:•

WHAT CAN YOU
LEARN FROM THE
RESULTS?

•

How will you analyse your data?
oHow should I enter, organise and clean data?

oWhat is the unit of analysis?

oWhat data analysis methods should I use?

o

How should I compare data collected using
different methods?

oHow should I present evaluation data?

What do the results show?
o68

METHOD FACTSHEETS& ANNEXESPRISM - TOOLKIT FOR EVALUATING THE OUTCOMES AND IMPACTS OF SMALL/MEDIUM-SIZED CONSERVATION PROJECTSHow should I interpret evaluation results?

2:3

INTRODUCTION &KEY CONCEPTS

DESIGNING & IMPLEMENTING
THE EVALUATION

How will you analyse your data?

STEP 3
WHAT CAN YOU
LEARN FROM THE
RESULTS?

How should I enter, organise and clean data?
Adapted from Evaluation toolkit – Pell Institute: http://toolkit.
pellinstitute.org/evaluation-guide/analyze/enter-organize-cleandata/
Once you have collected your data it will typically be in “raw” form,
for example:•

completed hardcopy questionnaires;•field survey data;•

recordings of interviews or focus groups;•

video recordings of observations.

PRISM
MODULESMETHOD FACTSHEETS& ANNEXES

What is the unit of analysis?
The first thing to that will determine how you analyse your data will
be your unit of analysis. A unit of analysis is the who, or what, thatis being analysed. For conservation projects the unit of analysis
could be sites, habitats, communities or individuals. Note that thisis different from your unit of data collection, for example you mightcollect data from individuals, but for analysis these individuals aregrouped according to specific criteria (e.g. gender, age, job).
Examples:Action: Restore mangrove habitat at three coastal
sites.
Evaluation question: Has the project successfully
restored habitat at the target sites?
Unit of analysis: Site.

Typically, raw data are not very illuminating in this form. So before
you begin analysis, you will need to enter, organise and clean yourdata.
Refer to the factsheet Entering, organising and cleaning data for
more information on how to do this.

Action: Develop capacity of park staff to conductanti-poaching patrols.
Evaluation question: Has the project raised thecapacity of park staff to carry out more effective
patrols?
Unit of analysis: Individual park staff.

69

PRISM - TOOLKIT FOR EVALUATING THE OUTCOMES AND IMPACTS OF SMALL/MEDIUM-SIZED CONSERVATION PROJECTS

2:3

INTRODUCTION &KEY CONCEPTS

DESIGNING & IMPLEMENTING
THE EVALUATION

PRISMMODULES

What data analysis methods should I use?

STEP 3
WHAT DATAANALYSIS
METHODSSHOULD I USE?

70

Options for analysing quantitative data
Analysing quantitative data involves examining numeric data (datacollected in the form of numbers) to look for patterns and trends
that can help you to answer your evaluation questions. Refer to theFactsheet: Analysing quantitative data for more information andexamples.

PRISM - TOOLKIT FOR EVALUATING THE OUTCOMES AND IMPACTS OF SMALL/MEDIUM-SIZED CONSERVATION PROJECTSMETHOD FACTSHEETS& ANNEXES

2:3

INTRODUCTION &KEY CONCEPTS

DESIGNING & IMPLEMENTING
THE EVALUATION

Options for analysing qualitative data
Analysing qualitative data involves turning written or other kindsof narrative data (such as interviews and field observation notes)
into results.

STEP 3

Refer to the factsheet Analysing qualitative data for guidance on
how to code and analyse qualitative data.

WHAT DATAANALYSIS
METHODSSHOULD I USE?

Qualitative evaluation data such as notes from focus group
discussions are often quite detailed and variable in structure.
Qualitative data analysis does not happen in a linear way; it isoften not a quick process but it can produce instructive and useful
results. The process of analysis involves critically reading andinterpreting your data, perhaps through a series of iterations, toreach a shared understanding.

Mixed methods analyses - combining and comparing
quantitative & qualitative data
Adapted from Introduction to mixed method evaluations(Bamberger 2012): https://www.interaction.org/sites/default/
files/Mixed%20Methods%20in%20Impact%20Evaluation%20(English).pdf

PRISM
MODULES

understanding by integrating different ways of knowing. Most
evaluations will collect both quantitative data (numbers) andqualitative data (text, images), however it is important to plan inadvance how these will be combined.

FIVE MAIN REASONS FOR COMBINING DATA SOURCES:
Enriching

Using qualitative data sources to identify issues
or obtain information about variables thatcannot be obtained by quantitative approaches

Exploring

Generating hypotheses from qualitative data
to be tested through the quantitative data

Explaining

Where one set of options sheds light on
unexpected findings derived from another setof options

Exemplifying

Providing a case study to illustrate by example
what is happening

Triangulating

Confirming or rejecting results from one data
source using results from another

Often in an evaluation you will want to use a mixture of datafrom both quantitative and qualitative sources. Using such a
combination of data types can improve an evaluation by ensuringthat the limitations of one type are balanced by the strengths ofanother. For example it is common to use quantitative data totest whether a change has occurred, then to use qualitative datato try to explain how the change happened. This will improve

71

METHOD FACTSHEETS& ANNEXESPRISM - TOOLKIT FOR EVALUATING THE OUTCOMES AND IMPACTS OF SMALL/MEDIUM-SIZED CONSERVATION PROJECTS

2:3

INTRODUCTION &KEY CONCEPTS

DESIGNING & IMPLEMENTING
THE EVALUATION

How should I compare data collected using different methods?
Adapted from Equal Access Participatory Monitoring &Evaluation
toolkit: http://betterevaluation.org/toolkits/equal_access_participatory_monitoring

STEP 3HOW SHOULD ICOMPARE DATACOLLECTEDUSING DIFFERENTMETHODS?

In many evaluations data will be collected from a range of different
sources. Triangulation is the process that involves combining
multiple methods and perspectives with various types of data
sources in order to cross-check the results of your research. This
forms a crucial step in the evaluation process as it helps to reduce
the risk of bias associated with using a single data source.
Triangulation could involve any of the following:••

Using different participatory techniques to measure thesame indicator and then comparing the results. If theresults are similar they are more likely to be accurate.

•

Comparing the themes in your field observations withthe themes in interviews and noting any changes overtime. Ask participants to explain these changes.

•

72

Drawing on qualitative data to cross-check the findings
from your analysis of quantitative data and to indicate
how widespread certain impacts and outcomes are.

PRISM
MODULESMETHOD FACTSHEETS& ANNEXES•

Comparing the perspectives of people who havedifferent points of view (for example community
members, staff of local government agencies, andconservation professionals familiar with the species/
site.

•

Checking interview data against programme
documentation and other written evidence that canverify what participants reported.

FIGURE 10: TRIANGULATION USING MULTIPLE DATASOURCES

Field
surveys

Questionnaire
surveys

Checking the consistency of what people say about thesame topic over time.

PRISM - TOOLKIT FOR EVALUATING THE OUTCOMES AND IMPACTS OF SMALL/MEDIUM-SIZED CONSERVATION PROJECTSKey informant
interviews

2:3

INTRODUCTION &KEY CONCEPTS

DESIGNING & IMPLEMENTING
THE EVALUATION

How should I present evaluation data?
In order for data to be understood it needs to be displayed in a waythat allows you to identify trends and patterns within the data.

STEP 3HOW SHOULD
I PRESENTEVALUATIONDATA?

The way you visualise your data will depend on the kind of data you
have and what you are trying to show. See the Table right for some
options. A key point to make is that you will not need to visualise allof your project’s results.
Refer to www.betterevaluation.org/plan/describe/visualise_data
and http://labs.juiceanalytics.com/chartchooser/index.html for
more information on each of the options listed in table.

PRISM
MODULESMETHOD FACTSHEETS& ANNEXESTABLE: OPTIONS FOR VISUALISING DATAWHAT ARE YOU TRYING TO SHOW?

USE THIS TOOLKIT TO

Relationship among data points

Scatterplot
Matrix chart
Network diagram

Compare sets of values

Bar chart
Block histogram
Bubble chart
Dot plot

Changes over timeLine graph
Stacked graphSee parts of a whole

Pie chart
Icon array
Treemap

Analyse a text

Word cloudPhrase net
Word tree

See the world

Species/habitat maps
Geotagging
GIS mapping

73

PRISM - TOOLKIT FOR EVALUATING THE OUTCOMES AND IMPACTS OF SMALL/MEDIUM-SIZED CONSERVATION PROJECTS

2:3

INTRODUCTION &KEY CONCEPTS

DESIGNING & IMPLEMENTING
THE EVALUATION

PRISMMODULESMETHOD FACTSHEETS& ANNEXES

What do the results show?

STEP 3
WHAT DO THE
RESULTS SHOW?

How should I interpret evaluation results?
This involves looking at the results to determine what lessons
can be learnt and what the results mean in relation to the overall
success of the project.
Interpreting your results is part science, part art. It requires theability to think critically to make judgements in relation to yourevaluation questions, based on the information you have gathered,your understanding of the risks, assumptions and external factorsthat may influence the project’s outcomes and impacts and any
potential sources of bias (see the controlling bias section of Step 2)
There is a temptation to be overly scientific and to focus too muchon quantitative results (e.g. numbers or graphs). However, the mostuseful interpretations are typically those that carefully consider
the information available, and then use this information to provide
an interpretation of what happened to explain the project’s results.
For example, the fact that long term impacts are not measurable
within the project’s timeframe means that you will often need touse information collected on intermediate outcomes to explain
the likelihood of future impacts. Equally you may find it impossible
to isolate and collect data on all the different factors, risks andassumptions that may influence the project’s outcomes andimpacts, so you may often need to examine qualitative evidence(interpretations) both from participants and from your own
observations and understanding of the situation, to explain andcontextualise the project’s results.

Photo credit: Jeremy Madeiros

74

PRISM - TOOLKIT FOR EVALUATING THE OUTCOMES AND IMPACTS OF SMALL/MEDIUM-SIZED CONSERVATION PROJECTS

2:3

INTRODUCTION &KEY CONCEPTS

DESIGNING & IMPLEMENTING
THE EVALUATION

EXAMPLE

PRISM
MODULESMETHOD FACTSHEETS& ANNEXES

households with a valuable secondary source of income byproviding, at current market rates, the equivalent of 10 days’

STEP 3
WHAT DO THE
RESULTS SHOW?

Project: A team in the Quang Tri province of Vietnam aimed

paid work per year. Despite this, participatory evaluation

to protect and restore forest in the buffer zone of Dakrong

showed that many households regarded the economic benefits

Nature Reserve. The project encouraged households to signof rattan as relatively limited, with the main limiting factors

10 year forest protection agreements that, in exchange forbeing the long time it takes rattan to grow large enough topreserving the forest, provided households with the rightharvest (5-10 years) and the availability of local rattan markets.

to extract rattan and other non-timber forest products, andprovided additional funding to enrich a one hectare patch with

Combining rattan enrichment with forest protection contracts

rattan seedlings for future harvesting.

proved popular among participating households, with results
suggesting that this approach helps to give participating

Evaluation questions:households an increased feeling of ownership over the land
covered by the agreements and an increased incentive to•

How much income is expected by rattan cultivation?

•

Has the project had an impact on stakeholder livelihoods/

contracts led to a change in behaviour where participating

wellbeing?

households now actively monitor the areas under contract.

manage the forest in a sustainable way. Furthermore the•Has the project given local people more of an incentive tomanage forest resources more sustainably?

•

To what extent has the project brought about a change inthe way local people manage forest resources?

Contracts were found to be very popular with interviewed
forest protection department staff, who see them as having
made their work a lot easier by compelling households tomonitor the areas under contract.
Interpretation: The team interpreted the results as suggesting

Evaluation methodology: The team used a mixed methods

that pairing rattan enrichment with long-term forest protection

approach, combining an economic evaluation (to estimate

contracts can help to deliver positive results where contracts

future income from rattan), Participatory Impact Assessment

provide an immediate benefit to households by giving them an(to evaluate householders’ perceptions of the project’s impact)

incentive to conserve the land under contract and to adopt new

and Key Informant Interviews with key community members

behaviours that prevent further degradation. This evaluationand employees of the local forest protection department.

suggests that this approach may offer significant advantages
over a project that only provides support for rattan enrichment

75

Results: The economic evaluation suggested that, once readywhere benefits to households will not be felt until rattan has

for harvesting, rattan has the potential to provide participatinggrown large enough to harvest.

PRISM - TOOLKIT FOR EVALUATING THE OUTCOMES AND IMPACTS OF SMALL/MEDIUM-SIZED CONSERVATION PROJECTS

2:3

INTRODUCTION &KEY CONCEPTS

STEP 3
WHAT DO THE
RESULTS SHOW?

DESIGNING & IMPLEMENTING
THE EVALUATION

PRISMMODULES

Capturing learning
As well as the project’s results it is also important to capture
learning, by documenting it properly. During the project, everyone
may think that their learning is well understood, successfully
shared and built into adaptive responses, and there may be little
incentive for them to be conscientious about recording it, as longas simply “getting on with the project” is the priority. But such a
record will be an important part of the legacy of the project, andsome discipline about this should be encouraged.
Ideally evaluations should be a two-way process, providing an
extremely useful opportunity for information exchange on thecontext, situation or community the project is working in.

WHAT CAN YOU LEARN FROM THE RESULTS?
CHECKLIST

76•

Have you analysed your data?

•

Have you interpreted your results to answer
your evaluation questions?

PRISM - TOOLKIT FOR EVALUATING THE OUTCOMES AND IMPACTS OF SMALL/MEDIUM-SIZED CONSERVATION PROJECTSMETHOD FACTSHEETS& ANNEXES

2:4

INTRODUCTION &KEY CONCEPTS

Step 4

STEP 4
WHAT SHOULD BE
DONE NEXT?

77

DESIGNING & IMPLEMENTING
THE EVALUATION

PRISMMODULESMETHOD FACTSHEETS& ANNEXES

What should be done next?

This is where you use the judgements you have made whileinterpreting your results to make decisions about what should bedone next. If the project is on-going then you will usually want to
use the results to make improvements to the project. Results andrecommendations could also be shared with relevant stakeholders
to inform their decision making. Finally, you should also consider
whether the evaluation results could be shared more widely to
inform the work of other conservationists.

STEP 4QUESTIONS ADDRESSED IN THIS SECTION:•

How will you use the evaluation results?
oDeveloping recommendationso

Can results be used to improve the project?

o

How will results be communicated to externalaudiences?

o

Can results be shared with the wider
conservation community?

PRISM - TOOLKIT FOR EVALUATING THE OUTCOMES AND IMPACTS OF SMALL/MEDIUM-SIZED CONSERVATION PROJECTS

2:4

INTRODUCTION &KEY CONCEPTS

STEP 4
HOW WILLYOU USE THE
EVALUATIONRESULTS?

DESIGNING & IMPLEMENTING
THE EVALUATION

PRISMMODULESHow will you use the evaluation results?
Once interpreted, the results will then need to be written up sothat they can be communicated and, ultimately, applied. How you
do this will depend on which of the following you are trying to do:1) use results to learn internally within the project; 2) communicate
results to external audiences, or 3) share results to inform thewider conservation community. When beginning the process ofapplying your results, a good place to start is by reminding yourself
who the evaluation stakeholders are and how the results will be
useful to them (see Step 1).

Developing recommendations
Evaluations often make recommendations about how a project or
project action can be improved, how the risk of project failure canbe reduced and whether a project should continue.
When developing recommendations, it is good practise to try andinvolve stakeholders in developing and/or reviewing them as thiswill help contribute to the use of the evaluation findings. If possible,
aim to follow up recommendations with evaluation users, e.g.
through annual reviews.

78

PRISM - TOOLKIT FOR EVALUATING THE OUTCOMES AND IMPACTS OF SMALL/MEDIUM-SIZED CONSERVATION PROJECTSMETHOD FACTSHEETS& ANNEXES

2:4

INTRODUCTION &KEY CONCEPTS

DESIGNING & IMPLEMENTING
THE EVALUATIONCan results be used to improve the project?
Adapted from – CDC Brief 7: Applying policy evaluation results-http://www.cdc.gov/injury/pdfs/policy/Brief%207-a.pdf

STEP 4
CAN RESULTS BE
USED TO IMPROVE
THE PROJECT?

One of the most significant benefits of an evaluation is that it
enables you to adapt and improve the action or project being
evaluated. This should be seen within the context of the project
cycle (see Section 1: when to use this toolkit) where yourevaluation results enable you to complete the cycle by informing
decision making. If possible, you should aim to re-visit the project’s
Theory of Change (or equivalent) and use your evaluation results
to make changes that take into account the lessons learnt from theevaluation.
Unlike communicating evaluation results for external audiences(which is usually done towards the end of the project), internallearning from results can occur at any stage of the project cycle.

PRISM
MODULESMETHOD FACTSHEETS& ANNEXES

Evaluation results can be used for internal learning in thefollowing ways (among others):
providing continuous feedback on project
implementation;-

identifying gaps in project implementation;-

eliminating or modifying project actions that are notproducing the desired results;-

adding further actions;-

identifying training and technical assistance needs;-

providing support for long-term planning;-building support for future project actions;-building support for acquiring resources;-

increasing communication between projectstakeholders.

Methods for applying your results internally can include:reflection meetings;

79-

developing an improvement action plan;-annual reviews.

PRISM - TOOLKIT FOR EVALUATING THE OUTCOMES AND IMPACTS OF SMALL/MEDIUM-SIZED CONSERVATION PROJECTS

2:4

INTRODUCTION &KEY CONCEPTS

DESIGNING & IMPLEMENTING
THE EVALUATION

How will results be communicated to external audiences?
Much of the guidance in this section has been informed by
the Better Evaluation page: report and support use: http://
betterevaluation.org/plan/reportandsupportuse

STEP 4
HOW WILLRESULTS BE
COMMUNICATEDTO EXTERNAL
AUDIENCES?

80

Identify requirements
When planning how you will communicate your results it is usefulto re-visit the section in step 1 where you identified the main uses
and users of the evaluation results. You can then design reporting
to address each use and each user.
Communicating with donors – A report presented to a funder/
donor will often be one of the main (and in some cases the only)
communication outputs of a project. Reporting should not be seen
as a box-ticking exercise to fulfil funding requirements. It should
instead be seen as an opportunity to communicate your findings
to show both what has been achieved and what has been learnt.
Many projects are reluctant to report results of actions that
produce unintended/negative results. The overwhelming majorityof funders however understand that conservation project actions
can be subject to a number of external influences, and they are
happy for projects to report unintended negative results, provided
that learning can be demonstrated. To get the most out of thereporting process you should aim to maintain communication withyour donors, make use of any reporting guidelines/templates, anddevelop reporting media that will be useful both to you and to thedonor.

PRISM
MODULESMETHOD FACTSHEETS& ANNEXES

Reporting timescales - A common challenge is that reporting
timeframes often require teams to report before the project’s longterm outcomes and impacts become measurable. In this situation
it will be necessary to report on relevant intermediate outcomes
and include in your report an explanation of how these act aspredictors of outcomes and impacts further down the project’s
Theory of Change.

Design reporting media
Options for reporting media include:
Written materials (e.g. reports, news bulletins, website
communications) - The most important information should comefirst, to allow readers to access easily the things they are mostinterested in. This will often be the findings and recommendations,which should thus appear early in the report. Less crucial details,
such as the evaluation background and methodology, belong in
an appendix or can even be posted elsewhere (e.g. online) for
reference.
See the factsheet Evaluation report template for a template thatcan be used for an evaluation report.
Presentations (e.g. posters, Powerpoint, video) – Presentation
audiences are likely to be most interested in only a portion of
the full evaluation report, such as the key findings or a lesson
learnt about evaluation methods. Thus, it is wise to focus thepresentation on that portion, while making the full report available
to anyone interested.

PRISM - TOOLKIT FOR EVALUATING THE OUTCOMES AND IMPACTS OF SMALL/MEDIUM-SIZED CONSERVATION PROJECTS

2:4

INTRODUCTION &KEY CONCEPTS

STEP 4
HOW WILLRESULTS BE
COMMUNICATEDTO EXTERNAL
AUDIENCES?

81

DESIGNING & IMPLEMENTING
THE EVALUATION

PRISMMODULESCreative (e.g. infographics, cartoons, photographic reporting)- Presenting your report in a creative manner may be the mosteffective means to get your information across if the context
allows for it. You may consider working with an artist or a graphic
recorder to produce creative displays.

Ensure that results are simply presented and are accessible to awide range of different users
Techniques for increasing accessibility:•

Use plain language•

Remove visual elements that don’t contribute to the main
message•Use visual techniques to draw attention to certain bits of
information•

Use descriptive chart titles to highlight key pieces of
information•

Use the one – three – twenty five principle where thereport takes the following format: a 1 page outline, a 3 page
executive summary and 25 pages to present the findings andmethodology.

PRISM - TOOLKIT FOR EVALUATING THE OUTCOMES AND IMPACTS OF SMALL/MEDIUM-SIZED CONSERVATION PROJECTSMETHOD FACTSHEETS& ANNEXES

2:4

INTRODUCTION &KEY CONCEPTS

STEP 4
DEVELOP A
NARRATIVE

DESIGNING & IMPLEMENTING
THE EVALUATION

Develop a narrative
However you synthesise your results your evaluation will have amuch wider reach if you are able to build a narrative that can beeasily understood by readers who may not be familiar with the
project. Remember that a good narrative does not just present
results, but uses these results to tell a story.
A simple narrative can be built using your evaluation questions:

PRISM
MODULES

EXAMPLE 2
What change(s) occurred? - Over the course of the project,
the perceived impact of Rattan on local livelihoods did notchange significantly
To what extent were any change(s) due to the project? - A
Participatory Impact Assessment found that local people
felt that Rattan was growing too slowly to deliver a regular

EXAMPLE 1
What change(s) occurred? - Over the course of the project,
the number of poaching snares found per patrol went from anaverage of 3 to 0.5
To what extent were any change(s) due to the project? - Anevaluation design, which measured the difference between
patrols at the project site and a similar area not targeted by the
project suggested that the majority of the observed change
in the number of snares being found was due to the project’s
training and outreach work

improvement to livelihoods
What do these results mean in relation to the project’s
overall aim? - These results mean that using Rattan as a sole
incentive is unlikely to deliver the project’s overall aim ofpreventing further deforestation
What should be done next? - In light of these results werecommend that projects that intend to use Rattan as anincentive for forest protection should either find a way tomake Rattan grow faster, e.g. through training on cultivation
techniques) or provide additional incentives that delivers agreater perceived benefit to stakeholders

What do these results mean in relation to the project’s
overall aim? - These results are likely to have positive
implications for the project’s goal of increasing hunted species
populations by reducing poaching
What should be done next? - In light of these results
we recommend that the project’s actions be scaled up toincorporate similar, neighbouring areas

82

METHOD FACTSHEETS& ANNEXESPRISM - TOOLKIT FOR EVALUATING THE OUTCOMES AND IMPACTS OF SMALL/MEDIUM-SIZED CONSERVATION PROJECTS

2:4

INTRODUCTION &KEY CONCEPTS

DESIGNING & IMPLEMENTING
THE EVALUATIONCan results be shared with the wider conservation community?

STEP 4
CAN RESULTS BE
SHARED WITH
THE WIDERCONSERVATIONCOMMUNITY?

Why is it important to share results?
In addition to your main evaluation users, you should give thought
to whether your evaluation results can be shared in support ofevidence-based conservation more generally. This is particularly
relevant for projects that are carrying out:•

innovative/novel actions;•actions that have not been applied in a similar setting
before;PRISM
MODULESMETHOD FACTSHEETS& ANNEXES

position to assess the likelihood of that action working for theirown project. It also enables actions to be compared, to assess
which of them are likely to work best in a given context.

EXAMPLE
Sharing results publicly can have wide-ranging and unexpected
benefits for other conservationists. In early 2007, theConservation Evidence journal published a paper on CriticallyEndangered Echo parakeets in Mauritius. Chicks in nest holes
were dying after being covered in slime from giant African land•evaluation results which produced particularlyinteresting/unexpected findings;

snails, so conservationists put copper bands around nest
trees, which appeared to reduce chick deaths as the snails
were reluctant to cross the copper. Conservationists in the UK

Sharing results and recommendations outside your organisation
allows the conservation community to learn from your successes
and failures, improving how well conservation works overall andenables your project to extend its impact far beyond its particularlocation or context.

read this paper and decided to apply it to a different problem– the herbivory of the Critically Endangered red helleborine
in the UK. In May and June 2007 they tested the use of cageswith copper bands around the plants, and the plants withcopper bands were not damaged by slugs and snails. Thesefindings were published in the Conservation Evidence journal

It is particularly important to share the results of projects that did
not work as intended, to prevent others from wasting resources on
actions that are unlikely to be successful. Sharing ‘negative’ results
is increasingly common: in the Conservation Evidence journal, 31%of the interventions whose success could be judged did not deliver
the hoped-for result.
As the success of conservation interventions may often becontext-specific, having more publicly available studies of a given
conservation action puts planners of future projects in a better83

in late 2007.

Williams, D.R., Child, M.F., Dicks, L.V., Ockendon, N., Pople, R.G., Showler, D.A., Walsh, J.C.,
zu Ermgassen, E.K.H.J. & Sutherland, W.J. (2017) Bird Conservation. Pages 95-244 in: W.J.
Sutherland, L.V. Dicks, N. Ockendon & R.K. Smith (eds) What Works in Conservation 2017.
Open Book Publishers, Cambridge, UK

PRISM - TOOLKIT FOR EVALUATING THE OUTCOMES AND IMPACTS OF SMALL/MEDIUM-SIZED CONSERVATION PROJECTS

2:4

INTRODUCTION &KEY CONCEPTS

STEP 4HOW CAN I SHAREMY RESULTS?

DESIGNING & IMPLEMENTING
THE EVALUATION

How can I share my results?
There are several ways in which you can share the results of yourwork with the wider conservation community. One route is to
publish in traditional academic journals. However, this process isoften time consuming and expensive, and results may be rejected
for lack of scientific interest or lack of novelty.
A more straightforward option is to publish in the dedicatedpractitioner journal run by the Conservation Evidence project
http://www.conservationevidence.com/collection/view , whichaims to share globally the experience of those in the front lineof conservation practice on the effectiveness of particularconservation actions. The journal is free to publish in, and is also
open-access, so is free for other practitioners to read. To publish inConservation Evidence you must have:

METHOD FACTSHEETS& ANNEXES

Full details, author guidelines and a template article can be foundat http://www.conservationevidence.com/collection/view.
A further such resources is the CMP Conservations Actions& Measures Library (CAML), an open-source library of results
chains for the most common conservation actions: http://cmpopenstandards.org/tools/caml/

WHAT SHOULD BE DONE NEXT?
CHECKLIST•

Have you developed recommendations?

•Have you developed a plan for applying resultsinternally?

•

directly and quantitatively tested a conservation action;•measured the results appropriately;••

made some kind of comparison, such as with the earlier
situation or with a control site. There is no need for the
study to be scientifically novel.

Do you know how you will communicate
results to external audiences•

Have you considered whether yourrecommendations and results should
be shared with the wider conservation
community?

Studies are welcome in the journal on all aspects of conservation
action and do not need to be scientifically novel. Examining theconsequences of small interventions, such as changing the wording
on a sign or comparing different techniques for eradicating aninvasive species, is of particular interest. Broad programmes, suchas a large scale projects with multiple interventions, will usually
be of less interest, as they are harder to interpret. Conservation
Evidence does not include studies solely on monitoring methods,
species ecology or threats to biodiversity.
84

PRISM
MODULESPRISM - TOOLKIT FOR EVALUATING THE OUTCOMES AND IMPACTS OF SMALL/MEDIUM-SIZED CONSERVATION PROJECTS3:0

SECTION 3
THEMATIC MODULESMODULE 1
AWARENESS &ATTITUDES

85

MODULE 2
CAPACITY
DEVELOPMENTMODULE 3
LIVELIHOODS &GOVERNANCE

PRISM - TOOLKIT FOR EVALUATING THE OUTCOMES &ANDIMPACTS
IMPACTSOF OF
SMALL/MEDIUM-SIZEDSMALL/MEDIUM-SIZEDCONSERVATIONCONSERVATIONPROJECTSPROJECTSMODULE 4
POLICY

MODULE 5
SPECIES & HABITATMANAGEMENT

3:0

INTRODUCTION &KEY CONCEPTS

Section 3

CONTENTSSECTION 3

DESIGNING & IMPLEMENTING
THE EVALUATION

Thematic Modules

Attitudes & Awareness

Capacity Development

PRISM
MODULESThis section provides detailed guidance and methods tomeasure outcomes and impacts relating to each of PRISM’s fivethematic modules, each of which covers a different category ofconservation action.
Use the framework and summary information within each module
to guide you towards specific methods for measuring change in
relation to the outcome/impact you are trying to evaluate.

Livelihoods & Governance

PolicySpecies & Habitat Management

86

METHOD FACTSHEETS& ANNEXESPRISM - TOOLKIT FOR EVALUATING THE OUTCOMES AND IMPACTS OF SMALL/MEDIUM-SIZED CONSERVATION PROJECTS

3:1

INTRODUCTION &KEY CONCEPTS

DESIGNING & IMPLEMENTING
THE EVALUATION

PRISMMODULESMETHOD FACTSHEETS& ANNEXES

Attitudes & Awareness

Summary
In this chapter we provide guidance on how to evaluate the impact
of project actions that aim to affect attitudes and awareness of
stakeholders.

ATTITUDES &AWARENESS

SPECIFICALLY THIS MODULE COVERS:

FACTSHEETS & ANNEXES

What do we mean by attitudes and awareness?

•

What to consider when evaluating attitudes &awareness outcomes/impacts.

Sample questions and scales for evaluating attitudes
and awareness actions•

Planning a successful attitudes/awareness action

Understanding how your project actions aim to change
attitudes and awareness.
Methods for evaluating attitudes & awarenessoutcomes/impacts:
1. Messages disseminated to/received by target
audience(s).
2. Audience has desired attitude.
3. Audience has desired awareness/knowledge.
4. Audience adopts desired behaviour.

87

PRISM - TOOLKIT FOR EVALUATING THE OUTCOMES AND IMPACTS OF SMALL/MEDIUM-SIZED CONSERVATION PROJECTS

3:1

INTRODUCTION &KEY CONCEPTS

ATTITUDES &AWARENESS
WHAT DOWE MEAN BY
ATTITUDES AND
AWARENESS?

DESIGNING & IMPLEMENTING
THE EVALUATION

What do we mean by attitudes and awareness?
Actions aimed at influencing attitudes and awareness, and which
ultimately lead to behaviour change, are important in many
conservation projects. When planning the evaluation, during thedesign of the action, it is crucial to consider the ethical requirement
for these actions (see factsheet Developing ethical protocols for an
evaluation). Attitudes and awareness actions may be essential forachieving the biodiversity target, for example, previous research
may have identified negative attitudes or low awareness as integral
to a threat: actions in this area can be a way of engaging withthe community and can be a source of unexpected information.
However, projects that target attitudes & awareness without
identifying a specific need, risk wasting time, being unethical andnot providing useful data.
Refer to the factsheet Planning a successful attitudes/awarenessaction for more information.

PRISM
MODULESMETHOD FACTSHEETS& ANNEXES

Attitudes

The concept of attitude has had many definitions but is generally
accepted to be the evaluative judgement that a person makes
about a particular object or situation, with a key aspect being theevaluation.
Attitudes are made up of three main parts:•

Knowledge (cognitive)•

Emotion (affective)•

Behaviour

Hence an attitude represents a summary of what a person
knows about something, how they feel about it and how they act.
Attitudes can be positive or negative and can vary in strength(how resistant the attitude is to change, how strongly it influences
the way the individual thinks about other information, and how
important it is in guiding the individual’s behaviour). Researchsuggests that stronger attitudes are primarily driven by theemotional (affective) component.

EXAMPLE88

A person may have a negative attitude towards bats because

attitude may make it difficult to persuade them that bats areof: bats carrying diseases in their country (knowledge),

worth conserving. However, the negative attitude may also

Halloween or vampire stories heard as a child (emotion) and/not be a conservation concern - someone may think negatively

or, the person having regularly been required to sweep up bat

about going inside a cave because bats may be present, but

droppings at their home or church (behaviour). A negative

they may never do any harm to bats.

PRISM - TOOLKIT FOR EVALUATING THE OUTCOMES AND IMPACTS OF SMALL/MEDIUM-SIZED CONSERVATION PROJECTS

3:1

INTRODUCTION &KEY CONCEPTS

DESIGNING & IMPLEMENTING
THE EVALUATION

Attitudes can be explicit or implicit. Explicit attitudes operate
on a conscious level with the holder being aware of their attitude
and adjusting their manner accordingly, while implicit attitudes
act at a subconscious level without the person being aware of thereason for the attitude they hold. Dual attitudes may also be held,
where people hold an old implicit attitude that co-exists with a new
explicit attitude (or vice-versa).

PRISM
MODULESMETHOD FACTSHEETS& ANNEXES

CHALLENGES WHEN MEASURING ATTITUDES
Although someone may be explicitly concerned about climatechange, they may drive a short distance to work if it is raining
rather than get out an umbrella or get their shoes wet; they
may take a short flight to see friends or family rather thandrive; or they may use a lot of energy for heating their home.

ATTITUDES &AWARENESS
WHAT DOWE MEAN BY
ATTITUDES AND
AWARENESS?

In these scenarios the individual’s attitude to climate change

Linking attitudes and behaviours
The link between attitudes and behaviours is potentiallycontentious. It is generally accepted that attitudes can influence
behaviour, but when evaluating an attitude-behaviour link it isessential to determine the correct target attitude to measure (it is
not always obvious) and acknowledge that there may also be manyother influential factors involved, for example perceived barriers
to action, habits, identity and social norms (see box on “Challenges
when measuring attitudes” below).
The factors influencing behaviour may differ between situations
and it is necessary to consider the numerous factors affecting
behaviour. Several behavioural theory models have beendeveloped to try to explain how attitudes link to behaviour, butthere is no single model that fully explains the attitude-behaviour
relationship.

89

has been surpassed by their attitude towards getting wet on
the way to work, or spending time with friends/family rather
than using slower forms of transport, or the cost and trouble ofinstalling home insulation.
Although it seems logical to promote positive environmental
behaviours by linking a person’s attitude towards climatechange with behaviours such as car sharing, public transportor home insulation, these scenarios show how difficult it canbe to determine the specific attitudes that are influencing agiven behaviour. Therefore it can be difficult to identify which
attitude a project action should address and what needs to bemeasured to evaluate the action.

PRISM - TOOLKIT FOR EVALUATING THE OUTCOMES AND IMPACTS OF SMALL/MEDIUM-SIZED CONSERVATION PROJECTS

3:1

INTRODUCTION &KEY CONCEPTS

DESIGNING & IMPLEMENTING
THE EVALUATION

Awareness

Awareness raising actions within a conservation project may be
carried out for a variety of reasons. For example, awareness raising
initiatives may be used to:

ATTITUDES &AWARENESS
AWARENESS•

provide information and raise awareness within aproject team about the social context the project isworking within;•

integrate the project and staff within the local
community;•

initiate/support a desired behaviour change to reach aparticular project outcome;•inform the community about general, local or global
conservation priorities not specific to the project but
potentially gaining support for future conservation
action.

A note of caution arises from the “knowledge deficit model”. Thisis the idea that a negative behaviour is performed because theindividual is unaware of its negative consequences, and that if
this “knowledge deficit” is corrected, the behaviour will change.
However, increasing awareness or knowledge of an issue does
not necessarily lead to a change in attitudes or behaviours, since
there are normally many variables affecting whether a particularbehaviour occurs or not.

90

PRISM
MODULESMETHOD FACTSHEETS& ANNEXES

Social marketing
Conservation projects are increasingly using social marketing
approaches to influence behaviours in order to achieveconservation goals. Emerging as a discipline in the 1970s, social
marketing is most widely used in the health sector, where it
influences behaviours such as smoking and alcohol use. Social
marketing applies the ideas and methods used in marketing to“sell” attitudes, values and behaviours that benefit society at theindividual, local, national or broader scale, specifically aiming tochange behaviours, not just the attitudes or awareness about
an issue. The timeframe of intervention and behavioural effectwould limit the possibilities for use of social marketing within mostsmall/medium-sized conservation projects. An intervention aimed
only at changing attitudes or raising awareness would not be
considered as a social marketing intervention.
Social marketing in conservation may be used to promote positive
conservation behaviours where none existed before, or to reduce
the occurrence of negative behaviours. In social marketing there
is a strong emphasis on evaluation and particularly on the use ofquantitative data.

PRISM - TOOLKIT FOR EVALUATING THE OUTCOMES AND IMPACTS OF SMALL/MEDIUM-SIZED CONSERVATION PROJECTS

3:1

INTRODUCTION &KEY CONCEPTS

DESIGNING & IMPLEMENTING
THE EVALUATION

PRISMMODULESMETHOD FACTSHEETS& ANNEXES

What to consider when evaluating attitudes & awarenessoutcomes/impacts

ATTITUDES &AWARENESS
WHAT TOCONSIDER WHENEVALUATING
ATTITUDES &AWARENESSOUTCOMES/IMPACTS

Check project design
Evaluating the effects of interventions designed to influence
attitudes and awareness is a critical part of a project and should beplanned at the design stage. The effects of these interventions may
be seen both within the project or among the audience, and may
occur over a variety of timescales.
It is important to identify the appropriate audience for themessage, which may be the individuals whose behaviour could
influence the success of the conservation goal, an influential peergroup around these people, or influential groups in the wider
community.
If possible you should aim to develop a robust Theory of Change
to associate the intervention with the target outcome. Although
it may not be possible to evaluate the effect of the intervention
on the desired biodiversity outcome, intermediate outcomescan be evaluated, provided there is a clear link to the eventual
conservation target.
Timing of events is also important. For example, it is necessary
to consider the timing of the evaluation in relation to othercommunity activities, avoiding clashes with other activities or work
e.g. so that events maximise attendances, and to consider timing of
the message in relation to the target behaviour.

Photo credit: Iain Dickson

91

PRISM - TOOLKIT FOR EVALUATING THE OUTCOMES AND IMPACTS OF SMALL/MEDIUM-SIZED CONSERVATION PROJECTS

3:1

INTRODUCTION &KEY CONCEPTS

DESIGNING & IMPLEMENTING
THE EVALUATION

PRISMMODULES

Attributing outcomes & impacts to actions targeting attitudes
and awarenessmixing of the people e.g. students who sit in two different
classes but who meet during playtimes, attitudes and/or
awareness can spread between the treatment populationand the control population, reducing the differences
seen between the two groups at the post intervention
assessment.

The most appropriate evaluation designs for attitudes andawareness are given below:
1)

ATTITUDES &AWARENESSATTRIBUTINGOUTCOMES& IMPACTS
TO ACTIONSTARGETING
ATTITUDES AND
AWARENESS

Matching designs - Identifying two or more groups of people,
one of which is subject to the project’s actions while theother/s is/are not, then recording any changes. Evaluating
the effect of an intervention is achieved through assessing
attitudes or levels of awareness in each situation both prior to
the intervention and post intervention. A single comparisonof groups post intervention may also show some differences
between the groups, but this is less robust as there is no
information on an attitudes/awareness baseline for thegroups or on how similar the groups were to start with.
Data to demonstrate similarities in target audiences for thedifferent populations need to be collected in the pre-test. A
time series design can also be employed to assess persistenceof the attitudes or awareness within the target population.

Issues:•92

METHOD FACTSHEETS& ANNEXES•

Pre-testing can sensitise people to the attitudes/awarenessyou are interested in and can affect the results they give insubsequent surveys, e.g. if they know you are interested ina particular species because of some questions in the firstsurvey, they may answer more positively about this species
in the second survey.

•

Depending on the intervention, withholding activities
from one population while undertaking them with anothermay be perceived to be unethical. This might be solved byundertaking the activity with the control population at alater date, after testing is complete.

It is normally impossible to find two groups with exactly
matching characteristics outside of a rigorous scientificexperiment. However, similar groups may be found bylooking at broad characteristics such as type of employment,
ethnicity, age or gender. If it is impossible to find similargroups of people, it may be better to match people withineach group and evaluate their similarities/differences rather
than assessing the whole group. If there is the potential for

PRISM - TOOLKIT FOR EVALUATING THE OUTCOMES AND IMPACTS OF SMALL/MEDIUM-SIZED CONSERVATION PROJECTS

3:1

INTRODUCTION &KEY CONCEPTS

2)

ATTITUDES &AWARENESSATTRIBUTINGOUTCOMES& IMPACTS
TO ACTIONSTARGETING
ATTITUDES AND
AWARENESS

PRISM
MODULES

Before-after designs
Time series design. Assessments of the attitudes/
awareness of the target audience are carried out prior to theintervention, shortly after the intervention and several weeks
after the intervention. This series of assessments records
any change in attitudes/awareness and the persistence ofthat change. People are likely to respond well immediately
post intervention, but the response may erode over time as
they revert to their usual social life and habits. This kind ofassessment is best performed using a comparison with a “nointervention” control group.
Single case before-after. If long-term follow up is notpossible, recording baseline information about people’s
attitudes/awareness before an intervention and repeating
a similar assessment after the intervention can give somemeasure of whether the information has been understoodby the audience, and whether it has influenced attitudes andawareness in the short-term.

Issues:••93

DESIGNING & IMPLEMENTING
THE EVALUATION•

3)METHOD FACTSHEETS& ANNEXES

With the single case before-after design, there is no
information about whether the attitude/awareness persistsin the longer term, so it is difficult to link it to any permanent
effect.

Participatory Impact Assessment - Asking people whether
they perceive that a change in their attitudes and awarenesshas occurred between the time before and the time after theintervention. The assessment can be made post-intervention
and can provide an indication of change.

Issues:•

PIA relies on people accurately and honestly recalling
their attitudes and awareness before the intervention. Theintervention may not have addressed an attitude they had
ever thought about, so there may be bias associated with
their perception of your desired results. Their recall also may
be influenced by input from a peer group.

Pre-testing can sensitise people to the attitudes/awarenessyou are interested in and can affect the results they give insubsequent surveys.
Events apart from the intervention may affect people’s
attitudes/awareness in relation to a topic, for example if
the national news media pick up a story about an issue
similar to the one being addressed by the local project, this,rather than the project intervention, may be the cause of an
influence on people’s attitudes/awareness.

PRISM - TOOLKIT FOR EVALUATING THE OUTCOMES AND IMPACTS OF SMALL/MEDIUM-SIZED CONSERVATION PROJECTS

3:1

INTRODUCTION &KEY CONCEPTS

ATTITUDES &AWARENESSDESIGNING & IMPLEMENTING
THE EVALUATION

Identifying questions and indicators for evaluating attitudes
and awareness outcomes/impacts
The type of questions asked and the manner in which they are
asked depends on the target attitude. It is important to make surethat the indicator actually measures what you need to monitor
and is not just an indicator used in other projects in slightly similar
circumstances. Learning from other projects is valuable but the
information you collect needs to be specific to your project. Many
indicators may be unique to your situation, and choosing these isjustified if well-planned.

PRISM
MODULES•

Retrieval. People store memories in different ways.
Differences exist between short-term and long-term
memory, with long-term memories being episodic (aboutlife events), semantic (facts about the world) or procedural(how to do actions, e.g. ride a bicycle). Memories are subject
to decay, where they are increasingly likely to be forgotten
as time passes, and interference, where the more frequently
something happens, the more difficult it is to distinguish thememory of an individual event. For example, if you were to
ask what someone had for dinner two weeks ago they may
have trouble remembering specifically, due to decay andbecause they had eaten several meals since, so potentiallyconfusing the information.

•

Ability to judge and estimate. People compensate forimprecise or incomplete memories by estimating. A concept
called the ‘availability heuristic’ has shown that the morememories a person has on a subject, the more common
they think it is. The reverse is also true, where people thinksomething is rare if they cannot remember many events. Thistranslates into problems for trying to remember or assess
attitudes relating to events. For example, if you ask someone
to name six times they did something, such as ways they saveenergy, they may find it easy and therefore have a positive
attitude about themselves and their ability to save energy.
However if you asked them to recall 12 times they did it,they would find it harder and therefore have a more negative
attitude towards their energy saving abilities. These attitudes
may affect their answers to subsequent questions. People
also have a tendency to estimate rather than remembering

Considerations for selecting indicators for attitudes include:

IDENTIFYINGQUESTIONS AND
INDICATORS FOREVALUATING
ATTITUDES AND
AWARENESSOUTCOMES/IMPACTS

94•

Sensitive data. Asking people to declare their feelings,
thoughts and opinions on a topic can be difficult, particularly
if the topic concerns sensitive or illegal behaviour. Culturaldifferences between interviewer and respondent may
mean that there are unknown sensitivities about a topic,
there may be social pressures to present a particular
opinion to outsiders and/or there may be consequences
for the respondent outside the scope of the project. It is
crucial to seek guidance on the topic, establish whether any
sensitivities exist and adapt the intervention accordingly.
Special techniques exist for assessing sensitive questions
where the respondents’ information is anonymous, forexample the Randomized Response Technique. These
techniques require audience buy-in and trust in theinterviewer to ensure that the respondent does not feel they
are being tricked.

METHOD FACTSHEETS& ANNEXESPRISM - TOOLKIT FOR EVALUATING THE OUTCOMES AND IMPACTS OF SMALL/MEDIUM-SIZED CONSERVATION PROJECTS

3:1

INTRODUCTION &KEY CONCEPTS

DESIGNING & IMPLEMENTING
THE EVALUATION

accurately, so when asking someone for example how often
they collect wood in the forest, the person may think of it as adaily task, forgetting all the days when they have not done it.
•

ATTITUDES &AWARENESSIDENTIFYINGQUESTIONS AND
INDICATORS FOREVALUATING
ATTITUDES AND
AWARENESSOUTCOMES/IMPACTS

95

Attitude construction. Historically there was an idea that
attitudes were pre-existing, stable, predictable and resistantto persuasion. However a more modern outlook suggests
they are not stable, can frequently change and, importantly,
may be constructed on the spot. People may form an opinion
on a topic as a result of being asked about it. This constitutes
an attitude, but it may not be a strongly held one or one thatis influential on the desired behaviour, so an assessment
has to be made of whether this is relevant data for theintervention.

PRISM
MODULESMETHOD FACTSHEETS& ANNEXES

Identifying indicators for awareness is less difficult than for
attitudes, but still holds several challenges. Again, the type ofquestions asked and the manner in which they are asked depends
on the target answer. Considerations for selecting indicators forawareness include:•

The type of awareness required. The awareness someone
has about a topic can be locally focused or more general. For
example, does the intervention intend to improve knowledge
about the lion population in the local area or about lions ingeneral?

•External context. Consider whether another organisation
or event informs the target audience about the topic outside
the scope of the intervention but within the timescale ofassessment. This is particularly important with time-series
designs where measurements a few weeks or months after
the intervention may be affected by events external tothe project. For example a coincidental television or radio
programme focusing on the same topic as the conservation
project may influence awareness in a way that the project
actions haven’t been able to do.

PRISM - TOOLKIT FOR EVALUATING THE OUTCOMES AND IMPACTS OF SMALL/MEDIUM-SIZED CONSERVATION PROJECTS

3:1

INTRODUCTION &KEY CONCEPTS

ATTITUDES &AWARENESSIDENTIFYINGQUESTIONS AND
INDICATORS FOREVALUATING
ATTITUDES AND
AWARENESSOUTCOMES/IMPACTS96

DESIGNING & IMPLEMENTING
THE EVALUATION

Identifying indicators for behaviour change also presents
challenges. As discussed earlier, the link between attitudes/
awareness and behaviour change is typically indirect and nonlinear, affected by other factors such as social norms, perceived
barriers and actual barriers, which all influence whether a person
undertakes a desired action – their intention to undertake it is only
one factor, and the intention may be overridden by these otherfactors. Behavioural change following an attitude or awarenessintervention may, therefore, be difficult to observe without otherinterventions to deal with the other factors. However on occasions
behaviour change can occur directly and in a short time frame,for example, a strong marketing exercise promoting the plightof a species may cause a measurable response in the form of
increased donations or increased volunteer activity to support theconservation of the species.
Considerations for selecting indicators for behaviour change
include:•The behaviour required. The desired behaviour can be thestopping of a negative behaviour (for example reducing
the exploitation of a species), or the pursuit of a positive
behaviour (for example growing more appropriate plants
for wildlife). Monitoring and evaluating whether a behaviour
has stopped often requires different methods from thoserequired for evaluating whether a behaviour has started.
For example, people may be reluctant to admit they were
doing something that negatively affected biodiversity as they
may think this makes them look bad, while people may bevery willing to say they are doing something positive to help
biodiversity as they may think this will make them look good.

PRISM
MODULESMETHOD FACTSHEETS& ANNEXES•The scope and scale of the behaviour. A behaviour may
be locally specific to the biodiversity target or more
broadly related to conservation. For example very specificindicators may be identified if the aim was to get a beachclean (e.g. do people assist in events to remove rubbish, arebins used, is there less rubbish over a particular period).
However, if people are being encouraged to take up a moregeneral behaviour, for example supporting conservation
organisations, they may become active members of arange of organisations in different areas and support themin different ways (e.g. financially, politically or practicallythrough volunteering), so it is difficult in these circumstances
to identify how many people have changed their behaviour.

•

External context. As with the other indicators, if anotherorganisation or event informs the target audience about the
topic of interest outside the scope of the intervention but
within the timescale of assessment, this can affect the ability
to attribute change. This is particularly important with timeseries designs where the third measurement a few weeks
or months after the intervention may be affected by events
between intervention and assessment.

PRISM - TOOLKIT FOR EVALUATING THE OUTCOMES AND IMPACTS OF SMALL/MEDIUM-SIZED CONSERVATION PROJECTS

3:1

INTRODUCTION &KEY CONCEPTS

DESIGNING & IMPLEMENTING
THE EVALUATION

PRISMMODULESMETHOD FACTSHEETS& ANNEXES

Understanding how your project actions aim to change
attitudes and awareness
The Attitudes and Awareness Framework illustrates the route for
how interventions focussed on attitudes and awareness may leadto a positive change in the conservation target for a project.

ATTITUDES &AWARENESSUNDERSTANDING
HOW YOURPROJECT ACTIONSAIM TO CHANGE
ATTITUDES AND
AWARENESS

FIGURE 1: ATTITUDES AND AWARENESS OUTCOME
AND IMPACT FRAMEWORKACTIONS TO
INFLUENCE
ATTITUDES AND
AWARENESS:
e.g. Personal communication,
public relations exercises,
environmental
education,social marketing

2. Audiencehas desired
attitude

4. Audience adopts
or ceases
targetbehaviour(s)1. Messages
received by target
audience

3. Audience has
desired awareness /knowledge

Actions

97

Outputs

Intermediateoutcomes

Threat reduction
outcomes

Impact on
Biodiversity

PRISM - TOOLKIT FOR EVALUATING THE OUTCOMES AND IMPACTS OF SMALL/MEDIUM-SIZED CONSERVATION PROJECTS

Threat(s) reduced

Improvement inbiodiversity status

3:1

INTRODUCTION &KEY CONCEPTS

ATTITUDES &AWARENESSFRAMEWORKNOTESDESIGNING & IMPLEMENTING
THE EVALUATION

Framework notes
This framework is a simple Theory of Change that may workover different timescales. Within the project’s lifetime, a social
marketing exercise to reduce a damaging behaviour caused byattitudes that are not strongly held could bring about a change
in behaviour and subsequent improvement in the biodiversity
target quite rapidly. Over a longer timescale, changing something
like farming or hunting practices could take many years of effort
before an improvement in the target species population or habitatis seen, so the complete Theory of Change may not be observed
within the lifetime of the project.

PRISM
MODULES

Although it may be difficult to be sure that the attitudes orawareness being targeted by the intervention are crucial for
affecting the behaviour, well-executed preliminary investigations,detailed project planning, and piloting the evaluation methods
should greatly increase the effectiveness of the evaluation.

Where a rapid effect of an intervention on attitudes can beobserved, a project may detect a measureable improvement inbiodiversity status as an indicator of intervention impact. However,
it is much more likely that the project will be focused on one small
section of the framework presented.
For attitude and awareness actions, the observation of
behavioural change is usually the most desirable measure of
impact, particularly as people’s self-reported attitudes may differ
from their actions. However, given short timescales, observing
behavioural change may not achievable within the lifetime ofa small to medium-sized project. Furthermore, attitude andawareness actions are commonly part of a complex programme ofactivities that include policy, capacity, livelihoods and governanceactions. For example, laws may be put in place to stop a negative
behaviour, but if there is insufficient capacity or willingness toenforce the laws, influencing people’s attitudes and awarenessabout the behaviour would be necessary.

98

METHOD FACTSHEETS& ANNEXESPRISM - TOOLKIT FOR EVALUATING THE OUTCOMES AND IMPACTS OF SMALL/MEDIUM-SIZED CONSERVATION PROJECTS

3:1

INTRODUCTION &KEY CONCEPTS

DESIGNING & IMPLEMENTING
THE EVALUATION

PRISMMODULESMETHOD FACTSHEETS& ANNEXESTABLE 1: EXAMPLES OF INDICATORS AND SUGGESTED METHODSOUTCOMES/ IMPACT

EXAMPLE INDICATORSMETHODS

1.

Project changes in response to audience feedback

Questionnaire

Increased awareness of project by audience

Key informant interview

Increased awareness of project by audience peers

Direct observation survey

Messages received by target
audience

ATTITUDES &AWARENESS

Documentary evidence
Participatory photo evaluation

METHODS FOREVALUATING
ATTITUDES &AWARENESSOUTCOMES/IMPACTS

2.

3.

4.

Audience has desired attitude

Audience has desired awareness/knowledge

Audience adopts desired behaviour

Change in target attitude by audience

Questionnaire

Change in associated, but not target, behaviour by audience

Key informant interview

Change in target attitude by audience peers

Direct observation survey

Change in target awareness by audience

Questionnaire

Change in target awareness by audience peers

Key informant interview

Change in occurrence of target behaviour

Questionnaire
Direct observation survey
Indirect evidence

99

PRISM - TOOLKIT FOR EVALUATING THE OUTCOMES AND IMPACTS OF SMALL/MEDIUM-SIZED CONSERVATION PROJECTS

3:1

INTRODUCTION &KEY CONCEPTS

ATTITUDES &AWARENESSINFORMATIONRELEVANT TO
ALL STAGES IN
THE ATTITUDES& AWARENESS
FRAMEWORK100

DESIGNING & IMPLEMENTING
THE EVALUATION

PRISMMODULESMETHOD FACTSHEETS& ANNEXES

Information relevant to all stages in the attitudes & awarenessframework
There are several methods for evaluating attitudes & awarenessoutcomes. These vary in robustness, cost, effort and timescale.

and “wildlife”. A better way of approaching this would be to ask
questions that specify the object in question, for example (feeling)“I think ladybirds are lovely to look at” combined with (behaviour)“I have let a ladybird walk on my hand”.

Most evaluations relating to attitudes & awareness currently
use questionnaires but applied in different ways. Questions inquestionnaires can either be open, allowing the respondent scopefor providing free text answers, or closed, where the respondentis asked to select from prescribed answers, or a mix of closed andopen options.

Novel evaluation methods. Traditional methods may not beappropriate in situations where literacy is low. There are also
more engaging ways to interact with an audience that can be more
fun. Examples of these include inviting people to draw pictures
representing scenes and examining prominence of specific objects,
or photo/video evaluation techniques where people are provided
with a camera and asked to photograph/film things representing
a proposed topic. Comparisons can be made in the subjects of thedrawings or photos before and after the intervention, or between
groups involved in the intervention and groups who are not.

Piloting of any method is essential to assess its validity. Piloting isa small-scale test of what will happen in the full evaluation. Piloting
assesses whether the approach for the evaluation is realistic
and is going to achieve the desired goal of accurately measuring
attitudes and awareness. A good pilot test will reveal problems
that can be addressed before the main evaluation takes place(e.g. do people get bored and give up because the questionnaire
is too long? Are some questions unnecessary? Is the selectedinterview approach a good method?). The pilot can also be used
for specific feedback on the evaluation process, to assess whether
a range of responses is being given, whether people are replying
with the required information and whether the evaluation canbe shortened. Importantly, the pilot can also help to reveal
whether the questionnaire is “internally valid”. An internally valid
questionnaire is one where the questions are understood by the
respondent, the questions gather the data required by the project,
and there are no alternative explanations for the results observed.
For example, asking someone what they feel about the statement“I like wildlife” is open to many interpretations of the words “like”

PRISM - TOOLKIT FOR EVALUATING THE OUTCOMES AND IMPACTS OF SMALL/MEDIUM-SIZED CONSERVATION PROJECTS

3:1

INTRODUCTION &KEY CONCEPTS

DESIGNING & IMPLEMENTING
THE EVALUATIONThe numbers below relate to the boxes in Table 1 and Figure 1.
Refer to the Species & Habitat Management Module for methods
for evaluating changes in biodiversity status.

Outcome 1.

ATTITUDES &AWARENESSOUTCOME 1.
MESSAGESDISSEMINATED
TO/RECEIVEDBY TARGET
AUDIENCE(S)

Messages disseminated to/received by target
audience(s)Potential data collection methods at this level include
Documentary evidence, key informant interviews &questionnaires. Potential measures of impact include:•
qualitative records of information gained;•

PRISM
MODULES

Main issues:•With regard to audience attitudes and awareness,any evaluation at this stage provides a very shortterm measure with no indication of the persistenceof the intervention effect. This means it is difficult tobe certain that the intervention will have an effect on
behaviour.
•Social media “likes” or website hits give some indicationof interest in the general population, but they are very
simple measures that may not engender any particularchange in behaviour within the population. Signing up
to organisations, events or campaigns demonstrates
interest in the topic, although this doesn’t necessarily
always mean it is positive.

•

The social media measures are likely to be biased to theyounger portion of the population and therefore may
not necessarily indicate that the message is reaching
the target audience (unless those young people are thetarget!).

project progress records highlighting changes in project
action.

Social media activity can provide data on spread of information,
while the distribution and occurrence of information materials
can give some indication of the spread of information throughout
a community. This information can be captured in the form of anactivity log or activity tracking sheet(s).
Potential measures include:•
location of leaflets/materials distributed;

101•

number of Facebook “likes”;•

number of hits on a website;•

sign-up rates to a campaign or organisation.

METHOD FACTSHEETS& ANNEXESPRISM - TOOLKIT FOR EVALUATING THE OUTCOMES AND IMPACTS OF SMALL/MEDIUM-SIZED CONSERVATION PROJECTS

3:1

INTRODUCTION &KEY CONCEPTS

Outcome 2.

DESIGNING & IMPLEMENTING
THE EVALUATION

PRISMMODULESbecause their explicit attitude opposes the implicit
attitude that may actually affect their actions. For
example, if general attitudes are being assessed ata meeting, people who think in opposition to theprevailing attitude may not be willing to speak up or
agree with the main group attitude, in order to appear
to be in agreement with their friends, colleagues orelders.

Audience has desired attitude

Evaluation at this level looks at the uptake of the message by thetarget audience. This should be assessed in the medium to longterm to validate persistence of the attitude.
Potential measures include:•
questionnaires on attitude held by audience;

ATTITUDES &AWARENESSOUTCOME 2.
AUDIENCE
HAS DESIRED
ATTITUDE••

Participatory photo evaluation to determine how
people perceive the subject of interest, for example if
asked to take photos that represent the protected area,
do the photos hold positive or negative content;•

A change in attitude does not necessarily determine achange in behaviour.

•

The context within which the information about theattitude is gathered may affect the results of theassessment. For example if carrying out a survey
in the local Wildlife Office, people may be more
attuned to presenting wildlife-friendly and positive
management attitudes than if in their own home; andfilling out a questionnaire may elicit different resultsfrom those obtained by an interview. In a before &after comparison, timing in relation to the seasons may
affect attitudes about plants or animals that are only
seasonally present in the area.

•

With assessment of peers it may be difficult to separate
the attitude previously held from that gained from thetarget audience member.

in the medium or long-term, questionnaires or key
informant interviews to assess the attitude held bypeers (family, friends, neighbours, colleagues) due
to dissemination of the message by members of theaudience.

Refer to the factsheet Sample questions and scales for evaluating
attitudes and awareness actions for some specific examples of thetype of questions that can be used to evaluate changes in attitudes.
Main issues:•
If no baseline information has been recorded (i.e. whensurveys are only carried out after the action), it will bedifficult (but not impossible) to assess any change inattitude attributable to the intervention.
•

102

METHOD FACTSHEETS& ANNEXES

People may express an attitude they do not actively
hold, either to meet perceived social expectations or

PRISM - TOOLKIT FOR EVALUATING THE OUTCOMES AND IMPACTS OF SMALL/MEDIUM-SIZED CONSERVATION PROJECTS

3:1

INTRODUCTION &KEY CONCEPTS

Outcome 3.

DESIGNING & IMPLEMENTING
THE EVALUATION

Audience has desired awareness/knowledge

PRISM
MODULES•

Other awareness-raising activities, for example
television or radio shows, may affect results if there isa gap between intervention and assessment, making it
difficult to attribute all knowledge, except the specificsituation awareness, to the intervention.

•

With assessment of peers it may be difficult to separate
the awareness previously held from that gained fromthe target audience member.

•

The potential for negative side effects, for example
informing people about the uses of various species,
may mean the species gets exploited even more.
For example if the project is trying to stop people
harvesting a particular plant and selling it at market,
raising awareness may cause more people to go out tofind and harvest the plant.

Evaluation at this level looks at the uptake of the message by thetarget audience, and should be assessed in the medium to longterm to validate persistence of the awareness.

ATTITUDES &AWARENESSOUTCOME 3.
AUDIENCE
HAS DESIRED
AWARENESS/KNOWLEDGE

Potential measures include:•
quizzes, questionnaires or non-literary assessmentsof awareness held by the audience, for example asking
people to draw a woodland scene before and afteran event, then comparing the details included in the
pictures;•in the medium or long term, key informant interview
assessments of the awareness of peers (family, friends,
neighbours, colleagues) about the message resulting
from dissemination by the members of the audience.

Refer to the factsheet Sample questions and scales for evaluating
attitudes and awareness actions for some specific examples of
the type of questions that can be used to evaluate changes inawareness/knowledge.
Main issues:•
An absence of baseline information will make it difficult
to assess any change in awareness attributable to theintervention.
•

103

METHOD FACTSHEETS& ANNEXESA change in awareness does not necessarily determine achange in behaviour.

PRISM - TOOLKIT FOR EVALUATING THE OUTCOMES AND IMPACTS OF SMALL/MEDIUM-SIZED CONSERVATION PROJECTS

3:1

INTRODUCTION &KEY CONCEPTS

Outcome 4.

ATTITUDES &AWARENESSOUTCOME 4.
AUDIENCE
ADOPTS DESIRED
BEHAVIOURDESIGNING & IMPLEMENTING
THE EVALUATION

Audience adopts desired behaviour

Changing attitudes and/or awareness is one of the mostcommonly used strategies for changing behaviour. It is importantto remember that changes to attitudes do not necessarily leadto changes in behaviour, so if you are using this as your main
unit of measurement for assessing impact, you must be clear
about what evidence or theory you are relying on to suggest thatchanging attitudes will lead to behaviour change and the desiredconservation effect. What makes this challenging to evaluate isthat, while a change in knowledge/attitudes/values can happen
relatively quickly (sometimes immediately) after an intervention,changes in behaviour take longer to become apparent.
Indicators of behaviour change may also be evaluated throughquestionnaires that assess people’s reports of their own or others’
behaviour, or through direct observation of the behaviour andindirect information from other sources. Measuring behaviourchange is generally a much stronger predictor of conservation
success than changes in attitudes/awareness, so if at all possible,
efforts should be made to evaluate changes in behaviour.

PRISM
MODULES•Key informant interviews or focus groups to discuss theoccurrence of the target behaviour.

Refer to the factsheet Sample questions and scales for evaluating
attitudes and awareness actions for some specific examples of
the type of questions that can be used to evaluate changes inbehaviour.
Main issues:•
People misreporting the occurrence of the behaviour.
•

The potential for a negative behaviour side effect of
the intervention, for example people swapping one
resource use behaviour (collecting wood from theforest) for another behaviour (burning toxic rubbish
indoors).

Potential measures include:•changes in attitudes and awareness will most commonly
be evaluated through questionnaires and interviews;•

objective observation of behaviour through observation
surveys (direct) or related signs (indirect). For example
it is possible to see whether people change their use ofcars by doing traffic surveys, or whether there is a risein volunteer work for local charities by interviewing
charity staff;

104

METHOD FACTSHEETS& ANNEXESPRISM - TOOLKIT FOR EVALUATING THE OUTCOMES AND IMPACTS OF SMALL/MEDIUM-SIZED CONSERVATION PROJECTS

3:1

INTRODUCTION &KEY CONCEPTS

ATTITUDES &AWARENESS
EXAMPLESDESIGNING & IMPLEMENTING
THE EVALUATION

PRISMMODULESMETHOD FACTSHEETS& ANNEXES

EXAMPLE OF AN INDIRECT INDICATOR

EXAMPLE OF EVALUATION DESIGN

An intervention focused on increasing recycling behaviour by

It is can be difficult to establish the attitudes that influence

raising awareness and changing attitudes. In order to evaluatea particular behaviour, and the other factors that may bethis intervention it would be possible to ask people how much

involved in stopping someone with the desired attitude

or what they were recycling. However the evaluation could

performing the targeted behaviour. For example, the required

be strengthened by also counting the number of bins being

action may be to stop people visiting a forest where they are

put out for recycling before and after the intervention, andimpacting a local species or a valuable habitat. Interviews andpossibly by examining records of recycling volumes calculated

surveys may reveal that the reason people are visiting theby the organisation involved in collection. These would give

forest is to collect wood. Finding an alternative wood source

an indication of whether the change in attitudes/awarenessmay be one proposed solution. If the alternative wood source isresulted in a change in behaviour.

provided and people are willing to use it, they may neverthelessstill visit the forest to exercise their dogs. The dog walking
hadn’t been the explicit primary aim of visits to the wood, but
it may come to be so, since the dogs still need walking and the

EXAMPLE OF NEGATIVE BEHAVIOUR CHANGE

105

owners’ attitude to their pets’ health overrides their attitude
towards the wildlife or habitat. Despite the project’s success in

A project carried out an awareness raising campaign within thefinding the new wood source, it now has a different problem torange of a threatened species of beetle. One of the campaign

deal with and must start again. Projects in complex situations

interventions explained how rare the beetle was, and thatlike these can be improved with good planning and design,

it was under threat mostly from being harvested to be soldbut also proper evaluation can demonstrate what has worked

to collectors, who would pay a lot of money for a specimen.

and the value of information gained, in this case why the

Very unfortunately the campaign had the negative impact ofwood collection was considered the primary forest use, how

increasing the harvesting rates, by informing people how theythat challenge was met and the additional lessons that were

could profit from harvesting.

learned.

PRISM - TOOLKIT FOR EVALUATING THE OUTCOMES AND IMPACTS OF SMALL/MEDIUM-SIZED CONSERVATION PROJECTS

3:2

INTRODUCTION &KEY CONCEPTS

DESIGNING & IMPLEMENTING
THE EVALUATION

PRISMMODULESMETHOD FACTSHEETS& ANNEXES

Capacity Development
SummaryThis chapter contains methods for evaluating the outcomes andimpacts of capacity development actions being implemented bysmall/medium-sized conservation projects. Areas covered:

CAPACITY
DEVELOPMENTSPECIFICALLY THIS MODULE COVERS:

FACTSHEETS & ANNEXES

What do we mean by capacity development?

•

Example questions and scales for training evaluation

What to consider when evaluating capacity
development actions•

Organisational capacity assessment tool•

Network health scorecard

Understanding how capacity development actions
benefit conservation targets
Methods for evaluating capacity development
outcomes & impacts:
1. Improved skills/ knowledge
2. Improved empowerment/ confidence
3. Individuals apply new skills
4. Organisational performance improves
5. Networks and alliances have been formed/
strengthened
6. Conservation actions are implemented more
effectively

106

PRISM - TOOLKIT FOR EVALUATING THE OUTCOMES AND IMPACTS OF SMALL/MEDIUM-SIZED CONSERVATION PROJECTS

3:2

INTRODUCTION &KEY CONCEPTS

DESIGNING & IMPLEMENTING
THE EVALUATION

What do we mean by capacity development?
There are many definitions of capacity development or capacity
building, all of which are very broad. Here we define capacity
development as:
Processes that improve the ability of people, organisations, or society toperform better, solve problems or manage their affairs successfully.

CAPACITY
DEVELOPMENT
WHAT DOWE MEAN BYCAPACITY
DEVELOPMENT?

107

Most projects tend to focus on just one level of capacity
development – usually the individual or organisation level.
Projects that develop individual capacity may still be contributing
to organisational capacity, since organisations need individuals
with the capacity to achieve their aims. Typical aims of individual
capacity development include increased skills and knowledge,
while typical aims of organisational capacity development include
improved governance, fundraising or strategic planning. It is usefulto understand how the different levels of capacity development
are interconnected, as shown in Figure 1.

PRISM
MODULESMETHOD FACTSHEETS& ANNEXES

FIGURE 1: CATEGORIES OF CAPACITYDEVELOPMENT ADAPTED FROM UNESCO IHE.
Capacity development covers a variety of actions, including:•training – to build skills and knowledge;•building confidence;•

gaining practical experience or placements;•

mentoring and coaching;•building networks and partnerships or alliances;•providing resources such as funding, tools, or equipment.

PRISM - TOOLKIT FOR EVALUATING THE OUTCOMES AND IMPACTS OF SMALL/MEDIUM-SIZED CONSERVATION PROJECTS

Enabling
environment• Policies• Social capital• Legal & regulatory

Organisational• Strategic planning• Management• Resources

Individual• Knowledge & skills• Conﬁdence• Attitudes

3:2

CAPACITY
DEVELOPMENT
WHAT TOCONSIDER WHENEVALUATINGCAPACITY
DEVELOPMENTACTIONS108

INTRODUCTION &KEY CONCEPTS

DESIGNING & IMPLEMENTING
THE EVALUATION

PRISMMODULESMETHOD FACTSHEETS& ANNEXES

What to consider when evaluating capacity development
actions

4.

How will you be sure that the changes you aremeasuring are the result of your actions rather thanother factors?

Check project design
Before carrying out capacity development actions it is importantto understand and assess what capacity needs exist among theindividuals or groups you will be working with. This may require an
initial data-gathering phase. The results will inform your Theory of
Change, and ideally these groups should be involved in designing
your Theory of Change.

5.

What resources and length of time do you need to do
this?

You also need to understand how ready any organisation (orindividual) is to adopt the new knowledge or skills into theirpractice. You may need to incorporate some preparatory work
into your action plan to understand this and to modify the plan
accordingly.
An evaluation programme should be designed as part of youroverall project planning in the beginning rather than at the end.
You can use the Theory of Change process to choose measures –or indicators – that will assess the change – or outcome – that eachor your project steps aims to bring about. Questions to ask are:1.

What is the conservation goal that you wish to achievethrough capacity development?

2.

How will your activities lead to this goal – and how will
you measure the changes you are bringing about – whatindicators will you use?

3.

Do you need a baseline (using the indicators in 2 above)?

Practitioners often ask for tools to measure their capacity
development outcomes & impacts, whereas the most importantfirst step is to ask what you are trying to achieve, and therefore
which indicators are best suited to measure the change you areaiming to bring about. This module includes examples of indicatorsand methods that you can adapt for your own purposes, and wesuggest that you involve your team in this process. Indicatorsdeveloped by projects and organisations are likely to be more
relevant and useful to their own practices. Moreover, research has
shown that organisations are more likely to change how they dothings based on the results of their own evaluations.

PRISM - TOOLKIT FOR EVALUATING THE OUTCOMES AND IMPACTS OF SMALL/MEDIUM-SIZED CONSERVATION PROJECTS3:2

CAPACITY
DEVELOPMENTATTRIBUTINGOUTCOMES
AND IMPACTS
TO CAPACITYDEVELOPMENTACTIONSINTRODUCTION &KEY CONCEPTS

DESIGNING & IMPLEMENTING
THE EVALUATION

Attributing outcomes and impacts to capacity development
actions
You need to be able to show that the changes you are measuring
are a result of your capacity development activities as opposed toother factors. Indeed, you need to consider whether your capacity
development actions brought about any change at all! In principle
you could use a matching design, comparing the capacities ofindividuals or groups targeted by the project with those who were
not. However, in practice it is rarely possible to find a control groupthat allows you to make a meaningful comparison, and so othermethods are a better use of time and resources.
For most evaluations of capacity development actions, datacollection will be carried out in the form of a before and after
design. This shows differences that your capacity action has made,by measurements taken before you begin and then again at theend. For example using pre- and post-course questionnaires to test
for an increase in knowledge or level of confidence resulting from a
training course.

PRISM
MODULESKey issues to consider when evaluating capacity development
actions•
Decide on the timeframe over which you willmeasure your impact. The interval between capacity
development actions and end results in conservation
can be very long, and you need to select the timeframe
appropriate to your project.
•

Results may be influenced by several different
organisations or projects, and will evolve over time.
Your project will need to understand this wider context
and then measure and report on the outcomes that area result of your capacity development activities within
the context of these other influences.

•

You should aim to ensure that there is good ownershipof the process among the groups or individuals you areworking with – ideally they should be involved both in
the design of the capacity development activities and in
the evaluation process.

The fact that most capacity development actions target specificskills and capabilities means that they are more suited to before& after designs than the other modules included in this toolkit,
particularly when carried out shortly after the action itself has
been carried out. These measurements can also be carried out at
a later date to track longer term outcomes & impacts, however inthis case it is essential to consider that other factors will come intoplay with time. See the factsheet on before-after designs for more
information.

109

METHOD FACTSHEETS& ANNEXESPRISM - TOOLKIT FOR EVALUATING THE OUTCOMES AND IMPACTS OF SMALL/MEDIUM-SIZED CONSERVATION PROJECTS

3:2

INTRODUCTION &KEY CONCEPTS

DESIGNING & IMPLEMENTING
THE EVALUATION

PRISMMODULESMETHOD FACTSHEETS& ANNEXES

FIGURE 2: CAPACITY DEVELOPMENT OUTCOME/
IMPACT FRAMEWORKCAPACITY
DEVELOPMENTUNDERSTANDINGHOW CAPACITYDEVELOPMENTACTIONS BENEFITCONSERVATIONTARGETS

3. Individuals
apply newskillsCAPACITY
DEVELOPMENTACTIONSe.g. Training,
Mentoring
Providing information,
Forming alliances& networks

4. Organizational
performance
improves

6. Conservation
actions are
implemented more
effectively

2. Individuals
have greater
confidence
5. Networks &alliances formed /strengthened

Actions110

1. Individuals
have improved
skills

Outputs

Intermediateoutcomes

Threat reduction
outcomes

Impact on
Biodiversity

PRISM - TOOLKIT FOR EVALUATING THE OUTCOMES AND IMPACTS OF SMALL/MEDIUM-SIZED CONSERVATION PROJECTS

Threats decrease

Improvement inbiodiversity status

3:2

INTRODUCTION &KEY CONCEPTS

CAPACITY
DEVELOPMENT
FRAMEWORKNOTESDESIGNING & IMPLEMENTING
THE EVALUATION

PRISMMODULESMETHOD FACTSHEETS& ANNEXES

Framework notes•
How will your capacity development lead toconservation outcomes? The Theory of Change (orequivalent) approach that we propose for projects
implementing PRISM (see factsheet Completing a
Theory of Change) asks you to be clear about the link
between your actions and the conservation impacts you
expect to happen as a result. By doing so, you explain
what assumptions you are making that lead you to
expect that your actions will bring about that desired
result. For example, if you are developing skills andconfidence among natural resource managers, your
Theory of Change (or equivalent) will show just how
and why you expect this to lead to improvements inbiodiversity status.
•

Outcomes 1, 2, and 3 all relate to dimensions ofindividual capacity development (as does stage 5 insome cases). Stages 4 and 5 relate to dimensions oforganisational capacity development, while outcome 6
relates to the effect these changes in capacity have on
the delivery of conservation actions.

•

Outcomes 1 and 2 can be evaluated immediately after
the capacity development action – for example byusing before-and-after questionnaires. Outcomes 3, 4,
5 and 6 are more complex to evaluate and can only beevaluated some time after the capacity development
action has taken place.

Photo credit: Jenny Merriman

111

PRISM - TOOLKIT FOR EVALUATING THE OUTCOMES AND IMPACTS OF SMALL/MEDIUM-SIZED CONSERVATION PROJECTS

3:2

INTRODUCTION &KEY CONCEPTS

DESIGNING & IMPLEMENTING
THE EVALUATION

PRISMMODULESMETHOD FACTSHEETS& ANNEXESTABLE 1: EXAMPLES OF INDICATORS AND SUGGESTED METHODS AT EACH STAGE (NUMBERS CORRESPOND TOOUTCOMES AND IMPACTS LISTED IN THE FRAMEWORK ABOVE (FIGURE 1).

CAPACITY
DEVELOPMENTMETHODS FOREVALUATINGCAPACITY
DEVELOPMENTOUTCOMES &IMPACTSOUTCOMES/ IMPACT

EXAMPLE INDICATORSMETHODS

1.

Individuals have improved skills•

Number & percentage of individuals that demonstrate anincrease in desired knowledge and/or skills

Before & after questionnaire
feedback form

2.

Individuals have greater confidence•

Number & percentage of individuals that demonstrate anincrease in confidence to apply their knowledge or to solvea particular problem

Before & after questionnaire
Training evaluation form

3.

Individuals apply new skills•

Number & percentage of individuals that demonstratethat they are applying their acquired skills or knowledge
Number & percentage of individuals that can describechanges they have made to their practice as a result of the
support received
Number and description of cases where individuals haveapplied new skills

Before & after questionnaire

Organisation demonstrating enhanced capacity ingovernance, leadership, etc.
Organisation has improved strategic planning
Organisation demonstrates improved fundraisingcapability

Organisational capacity selfassessment tool

Organisation demonstrates enhanced capacity in building
and maintaining relationships with stakeholders
Number of organisations/alliances collaborating
Number and percentage of supported CSOs
demonstrating improved support for citizens communities
to take collective action

Organisational capacity selfassessment tool

Evidence of threat reduced or habitat managed asintendedThreat reduction scoring

•

•

4.

Organisational performance
improves

•
•
•

5.

Networks and alliances have been
formed/strengthened

•
•
•

6.

112

Conservation actions are
implemented more effectively•

PRISM - TOOLKIT FOR EVALUATING THE OUTCOMES AND IMPACTS OF SMALL/MEDIUM-SIZED CONSERVATION PROJECTS

Network Health Scorecard

3:2

INTRODUCTION &KEY CONCEPTS

Outcome 1.

CAPACITY
DEVELOPMENTOUTCOME 1.
INDIVIDUALSDEMONSTRATE
THAT THEYHAVE ACQUIREDTHE DESIREDKNOWLEDGE AND/OR SKILLS113

DESIGNING & IMPLEMENTING
THE EVALUATION

Individuals demonstrate that they haveacquired the desired knowledge and/or skills

Your Theory of Change and discussions on project aims will alreadyhave revealed what kind of knowledge and skills your project aims
to develop. Therefore, the methods and indicators you use will bebased on measuring changes in these specific areas. For example,
you may wish to increase skills in a particular survey technique orenhance knowledge of anti-poaching laws and regulations.

PRISM
MODULESMETHOD FACTSHEETS& ANNEXESA powerful measure is a follow-up survey at a later date to evaluate
whether people are still applying the skills – i.e. to see whether any
change in skill level has persisted. This is covered under outcome 3
below.
Refer to the factsheet Example questions and scales for training
evaluation for some examples of the type of questions that can beused to evaluate skills and knowledge.

The most common method used to evaluate changes in knowledge
and skills is a before and after questionnaire or feedback form.
Questionnaires will ask individuals about their level of knowledge
or skills before and then after the training. It is important to notethat self-assessments are not objective (an individual may assess
their skill levels to be much higher or lower than they actually are).
A paired before and after test partially accounts for this, since itis the same individual making both evaluations. A second typeof question could be a factual one – on the individuals’ level ofknowledge on the topic you are training in – such as knowledgeof policy or of threats to a habitat. Again it is preferable forthese questions to be asked before and then after the training.
However, there will be situations when this is not appropriate –the individuals participating may feel threatened by being testedon their knowledge, and this may create a poor environment for
the training. Note that testing for practical skill levels before andafter training can be time consuming, costly and in many cases is
not suitable, which is why questionnaire surveys are typically usedinstead.

PRISM - TOOLKIT FOR EVALUATING THE OUTCOMES AND IMPACTS OF SMALL/MEDIUM-SIZED CONSERVATION PROJECTS

3:2

INTRODUCTION &KEY CONCEPTS

Outcome 2.

CAPACITY
DEVELOPMENTMETHODS FOREVALUATINGCAPACITY
DEVELOPMENTOUTCOMES &IMPACTSDESIGNING & IMPLEMENTING
THE EVALUATION

Individuals demonstrate that they haveacquired the confidence and/or motivation to
apply the desired knowledge and/or skills

Confidence and motivation are an integral part of capacity
development. An individual may gain new skills through a training
course, but they also need to have the confidence or motivation to
apply these skills later. Before and after questionnaires, or if thisis not possible, post-training feedback forms. are normally used toevaluate changes in confidence and motivation. Measuring levelsof confidence through self-assessments should avoid “leading
questions” but instead ask questions that elicit the respondent’s
own view of their confidence. Sometimes it is good to use several
methods to triangulate your results, you can check whether you
are getting a consistent answer. Carrying out surveys some timeafter training is preferable, in order to evaluate whether any
change in levels of confidence/motivation has persisted –this iscovered in outcome 3 below.
Refer to the factsheet Example questions and scales for training
evaluation for some examples of the type of questions that can beused to evaluate confidence and motivation.

114

PRISM
MODULES

Outcome 3.

METHOD FACTSHEETS& ANNEXES

Individuals apply new skills or knowledge

This is a longer-term measure which is usually based on the samequestions and indicators used for evaluating outcomes 1 and 2,
but in this case is examining whether the change you have brought
about persists into the longer term. It requires following up withindividuals after the capacity development action has takenplace. Indicators will tell you whether individuals are using theskills and knowledge they gained through your action. The mostcommon method for doing this is to use a follow-up questionnaire,preferably paired with the questionnaire you used for measuring
immediate changes, because then you can compare the same
indicators over time.
Refer to the factsheet Example questions and scales for training
evaluation for some examples of the type of questions that can beused to evaluate the application of skills.
The Most Significant Change method (MSC) is another, more
descriptive technique for exploring the ways in which new skills or
knowledge are applied.
Details concerning MSC and its use can be can be downloaded
here: http://www.mande.co.uk/docs/MSCGuide.pdf

PRISM - TOOLKIT FOR EVALUATING THE OUTCOMES AND IMPACTS OF SMALL/MEDIUM-SIZED CONSERVATION PROJECTS

3:2

INTRODUCTION &KEY CONCEPTS

Outcome 4.

DESIGNING & IMPLEMENTING
THE EVALUATION

Organisational performance improves

Many projects focus on developing the capacity of organisations
rather than developing the technical capacity of individuals. This
is often the case where the main barrier to achieving conservation
goals is not a lack of technical capacity among individuals but
rather an organisational lack of capacity in areas such as planning,leadership or fundraising.

CAPACITY
DEVELOPMENTMETHODS FOREVALUATINGCAPACITY
DEVELOPMENTOUTCOMES &IMPACTSThe way you evaluate changes in organisational capacity will
depend on the scale of the organisation.
For evaluating changes in the organisational capacity ofcommunity based organisations (CBOs) refer to the factsheet:
Community-based Organisational Capacity Assessment which
details a simple tool for evaluating changes in the capacity of CBOs

PRISM
MODULES

Other resources
For another list of organisational capacity indicators andassociated tools, look up the following resource from BOND (a UKmembership body for NGOs). They group organisational capacity
issues under the following headings:•

Improved internal organisation;•

Improved programme management;•

Improved external relationships.

BOND – Assessing effectiveness in building the capacity oforganisations & institutions: https://my.bond.org.uk/sites/default/
files/impact-builder/CapacityDevelopment.pdf

For evaluating changes in the organisational capacity of NGOs,
please refer to the factsheet: Organisational Capacity Assessment
Tool
As for the other evaluations, we recommend that you carry out theassessment before and after the capacity development action. If
the action is only targeting specific areas of organisational capacity(e.g. leadership, fundraising) then the assessment can be restrictedjust to these areas.

115

METHOD FACTSHEETS& ANNEXESPRISM - TOOLKIT FOR EVALUATING THE OUTCOMES AND IMPACTS OF SMALL/MEDIUM-SIZED CONSERVATION PROJECTS

3:2

INTRODUCTION &KEY CONCEPTS

Outcome 5.

DESIGNING & IMPLEMENTING
THE EVALUATION

Networks and alliances have been formed/
strengthened

Evaluation at this stage involves looking at the networks & alliancesthat have been formed/strengthened as a result of the project’s
actions. For example, alliances may have been formed with otherNGOs, government institutions or with local community groups/
networks.

CAPACITY
DEVELOPMENTMETHODS FOREVALUATINGCAPACITY
DEVELOPMENTOUTCOMES &IMPACTS

Before evaluating the effect the project has had on networks it is
crucial that you first clarify what it the main aim of the network
is. E.g. strengthening communication with local or national
government, promoting/strengthening advocacy work or
promoting good practise in a particular sector.
Evaluations can be made using Questionnaire surveys,
administered to those within the network. Other options include
completing a network health scorecard (See Network healthscorecard) before and after the project. The partnerships &networks section of the Organisational Capacity Assessment
tool, used before and after the project, can also be applied in thiscontext.

Outcome 6.

Conservation actions are implemented more
effectively

Evaluation of this stage involves looking at whether improvements
in capacity then led to subsequent changes in the delivery ofconservation actions. Most capacity development actions are
aimed at creating the conditions for further conservation actions,

116

PRISM
MODULESMETHOD FACTSHEETS& ANNEXES

which in many cases will not form part of the project. For example
a project training park rangers to carry out more effective patrols
will not be involved in ensuring that the skill delivered in the
training are applied in other day to day management activities ofthe park.
This makes evaluating these outcomes/impacts beyond thereach of most small/medium-sized projects carrying out capacity
development actions. However it may sometimes be possible,
particularly if the capacity development action was targeting a
specific practise or behaviour linked to a conservation threat,to evaluate whether or not the capacity delivered had an effecton the delivery of actions relating to that threat. In these casesa threat reduction assessment (See factsheet Threat reduction
scoring) can be used to gauge the effectiveness of the deliveryof the capacity delivered. Although care will need to be taken to
ensure that any change is due to the capacity delivered by the
project and not due to factors not due to the project.
Further reading
Capacity for Conservation: http://capacityforconservation.org/
M&E of Capacity Building, is it really that difficult?: https://www.
intrac.org/wpcms/wp-content/uploads/2016/09/Praxis-Paper-23Monitoring-and-Evaluating-Capacity-Building-Nigel-Simister-withRachel-Smith.pdf
PILAC – Manual on Training Evaluation: http://www.jica.go.jp/
project/cambodia/0601331/pdf/english/5_TrainingEvaluation.pdf

PRISM - TOOLKIT FOR EVALUATING THE OUTCOMES AND IMPACTS OF SMALL/MEDIUM-SIZED CONSERVATION PROJECTS

3:3

INTRODUCTION &KEY CONCEPTS

DESIGNING & IMPLEMENTING
THE EVALUATION

Livelihoods & Governance

LIVELIHOODS &GOVERNANCE

PRISM
MODULES

SummaryThis chapter contains methods for evaluating the outcomes
and impacts of conservation projects attempting to improvestakeholder livelihoods and/or improve governance. Areas
covered:SPECIFICALLY THIS MODULE COVERS:

FACTSHEETS & ANNEXES

What do we mean by livelihoods & governance?

•

What should I consider when evaluating livelihoods &governance outcomes and impacts?
Understanding how your project actions aim to change
livelihoods & governance issues to benefit conservation
Methods for evaluating livelihoods & governanceoutcomes/impacts:

Participatory Impact Assessment (PIA) methods•

PIA Method 1: Before and after scoring•

PIA Method 2: Proportional piling•

PIA Method 3: Tally method•

PIA Method 4: Matrix scoring•

PIA Method 5: Before and after impact calendars

1. Uptake of new or improved practices/ livelihoods
activities•

2. Progress towards secure tenure or resource rights
for communities

Community Based Organisation Capacity
Assessment•

Community Mapping•

Basic Necessities Survey (BNS)•

Participatory Governance Assessment•

Participatory photo evaluation

3. Governance arrangements give increased
recognition/ respect
4. Community institutions have increased capacity
5. Well-being goals met
6. Damaging occupation/ livelihood abandoned or
reduced117

METHOD FACTSHEETS& ANNEXESPRISM - TOOLKIT FOR EVALUATING THE OUTCOMES AND IMPACTS OF SMALL/MEDIUM-SIZED CONSERVATION PROJECTS

3:3

INTRODUCTION &KEY CONCEPTS

DESIGNING & IMPLEMENTING
THE EVALUATION

What do we mean by Livelihoods & Governance?

LIVELIHOODS &GOVERNANCE
WHAT DOWE MEAN BY
LIVELIHOODS &GOVERNANCE?

Livelihoods
Livelihoods are the means by which people seek to achieve theirlife’s goals and other conditions which contribute to well-being(see Section 1). Understanding people’s livelihoods is important toknowing how to evaluate a project’s impact on the relationshipsthat people have with their environment and the choices theymake.
The ‘Sustainable Livelihoods Approach’ (SLA) is one framework
for understanding the complexity of people’s livelihoods, and theimpacts that conservation and development projects can have on
different people’s livelihoods and well-being.
4

A livelihood comprises the following elements:•

Capabilities are what people can do or what they can be– this is often strongly influenced by their entitlements;•

Entitlements refer to people’s human rights, which
include economic, social, cultural and political rights;•

Assets are the resources people have access to, andinclude both material and social resources;•

Activities are everything that people do with theircapabilities and assets as they strive to achieve desired
outcomes.

PRISM
MODULES

People require a range of assets to achieve positive livelihood
outcomes. Natural resources are just one set of the assets on
which livelihoods depend – if they are depleted or degraded,people’s livelihoods will be threatened. No single category ofassets on its own is sufficient for people to survive, to live fulfilled
lives, and to achieve the varied livelihood outcomes that different
people seek. This is particularly true for poor people, whose accessto any given category of assets tends to be very limited.
People’s livelihoods are usually influenced by external factors
outside their direct control, and they are dependent on wider
policies, institutions and processes. These affect the complex
social, cultural, economic and political context within whichpeople pursue their livelihood strategies, and they may constrain
or broaden people’s options. A livelihood is sustainable when
it can cope with and recover from trends, stresses and shocks;
can maintain and enhance its capabilities and assets, both now
and in the future; and can do so without undermining the natural
resource base of present and future generations.
Livelihood strategies are the combination of activities thatpeople choose to undertake (or those that are left open to themwhen the opportunity to pursue others is constrained) in order toachieve their livelihood goals or outcomes. Livelihood outcomes
are what people achieve as a result of their livelihood strategies.
Different people will be motivated by achieving different livelihood
outcomes – not all of us want to achieve the same things.

4
DFID (1999) Sustainable Livelihoods Guidance Sheets, No.2 – Framework. DFID. http://
www.eldis.org/vfile/upload/1/document/0901/section2.pdf

118

METHOD FACTSHEETS& ANNEXESPRISM - TOOLKIT FOR EVALUATING THE OUTCOMES AND IMPACTS OF SMALL/MEDIUM-SIZED CONSERVATION PROJECTS

3:3

INTRODUCTION &KEY CONCEPTS

DESIGNING & IMPLEMENTING
THE EVALUATIONFIGURE 1: COMPONENTS OF A LIVELIHOODS
FRAMEWORKLIVELIHOODS &GOVERNANCE
WHAT DOWE MEAN BY
LIVELIHOODS &GOVERNANCE?

119

Livelihood
Platform
Capital Assets• Natural Capital• Human Capital• Physical Capital• Financial Capital• Social Capital• Cultural Capital• Political CapitalInﬂuence and Accessmodiﬁed by

PRISM
MODULESMETHOD FACTSHEETS& ANNEXES

Figure 1: Components of a livelihoods framework. From: Bennett,
N.J and Dearden, P. (2014). Why local people do not support
conservation: Community perceptions of marine protected area
livelihood impacts, governance and management in Thailand.
Marine Policy, Volume 44, Pages 107–116. doi:10.1016/j.
marpol.2013.08.017

In context
ofResultingin

Policies, Institutions
and Processes

History, Trends
and Shocks

Livelihood
Strategies• Government (State& Local)• Private Sector• NGOs and CBOs
• Laws and Policies• Governance• Management• Culture and Norms• Social Relations(Gender, Age,Class, Ethnicity)• History• Politics• Markets• Population• Migration• Climate• Ecology• Environmental
Change• Demography• Disease• Conﬂict• Technology

NR-Based activities• Fishing andCollection• Agriculture andPlantations• Livestock• Eco-tourism

PRISM - TOOLKIT FOR EVALUATING THE OUTCOMES AND IMPACTS OF SMALL/MEDIUM-SIZED CONSERVATION PROJECTSNon NR-Based• Rural Trade• Other services• Manufacture• Remittances• Other transfers

With effects
onLivelihood Outcomes
Socio-Economic• Working days
• Wealth/poverty• Well-being• Capabilities• Diversiﬁcation• Food Security
Environmental• Habitats• Biodiversity• Productivity – ﬁsheries• Ecosystem services• Sustainable natural
resource base

3:3

INTRODUCTION &KEY CONCEPTS

DESIGNING & IMPLEMENTING
THE EVALUATION

PRISMMODULESMETHOD FACTSHEETS& ANNEXES

Governance
CHARACTERISTICS OF GOOD GOVERNANCE

LIVELIHOODS &GOVERNANCE
WHAT DOWE MEAN BY
LIVELIHOODS &GOVERNANCE?

A wide range of stakeholders, with various levels of power andauthority, may be involved in decisions on how land and natural
resources are managed. Governance is the means by whichsociety defines and makes decisions on goals and priorities, andthe mechanisms and roles through which different stakeholders
work to achieve them. It includes guiding principles, policies, laws,social norms, and the institutions, at all levels, involved in making
decisions. These aren’t only government or legal institutions –governance may involve informal institutional arrangements suchas voluntary codes of conduct for private businesses.

Good governance is participatory, consensus oriented,
accountable, transparent, responsive, effective and efficient,equitable and inclusive and follows the rule of law. It assuresthat corruption is minimised, the views of minorities are taken
into account, and that the voices of the most vulnerable insociety are heard in decision-making.
UNESCOGood governance has eight major characteristics (see box).
There is evidence that good governance helps to provide
conditions for sustainable management of environmental
resources (as well as delivering a range of other social and rightsrelated outcomes). For local communities, participation in decisionmaking can help ensure that their knowledge is acknowledged andvalued, their needs are recognised, and that they have the power,rights and capacity to control and make decisions on use andaccess to land and resources.

Photo credit: John Bett120

PRISM - TOOLKIT FOR EVALUATING THE OUTCOMES AND IMPACTS OF SMALL/MEDIUM-SIZED CONSERVATION PROJECTS

3:3

INTRODUCTION &KEY CONCEPTS

DESIGNING & IMPLEMENTING
THE EVALUATION

What should I consider when evaluating livelihoods &governance outcomes and impacts?
Attributing outcomes and impacts to livelihood and governanceactions
The main guidance on this is located in the section on Choosing
an evaluation design. The designs likely to be of most value to
livelihoods & governance actions are:

LIVELIHOODS &GOVERNANCE

1)

WHAT SHOULD ICONSIDER WHENEVALUATINGLIVELIHOODS &GOVERNANCE
OUTCOMES AND
IMPACTS?

3)

METHOD FACTSHEETS& ANNEXES

Before and after designs – The most commonly applieddesign. However because of the complexity of social
responses to interventions on livelihoods, and the long-term
nature of projects and their impacts, there are real difficulties
with using this approach to attribute change to the livelihoods& governance actions (although if conducted alongside aparticipatory design, these difficulties can be overcome tosome extent).

Participatory Impact Assessment - This is an under-usedapproach in conservation, but for small projects aiming toevaluate complex interventions and likely to impact on a
range of indicators (such as those involving livelihood orgovernance interventions), participatory designs can beeffective, affordable and practical. This type of approach canbe used to explore stakeholders’ perspectives of the impactof the project on indicators selected by the project, or it canbe used in a very open way to explore what matters locallyand how the project has impacted more locally definedindicators. (see Catley et al., 2013: http://fic.tufts.edu/assets/
PIA-guide_revised-2014-3.pdf for more information)

2)

121

PRISM
MODULES

Theory-based designs - This approach depend on a very goodunderstanding of the mechanisms which drive change incommunities. It has potential limitations in that it is perhaps
less likely to detect issues which project implementers are
not aware of but which may be extremely important to local
stakeholders.

PRISM - TOOLKIT FOR EVALUATING THE OUTCOMES AND IMPACTS OF SMALL/MEDIUM-SIZED CONSERVATION PROJECTS

3:3

LIVELIHOODS &GOVERNANCE
IDENTIFYINGQUESTIONS AND
INDICATORS FORLIVELIHOODS &GOVERNANCE
OUTCOMES/IMPACTSINTRODUCTION &KEY CONCEPTS

DESIGNING & IMPLEMENTING
THE EVALUATION

Identifying questions and indicators for livelihoods &governance outcomes/impacts
For some outcomes (e.g. uptake of new and improved resource
management practices) the questions and indicators can beobjective (i.e. they are based on fact, and aren’t influenced by anindividual’s emotions, opinions, or personal feelings). For example,
the simplest question would be one that focuses on measuring
the number of people who have adopted a new/improved/more
efficient resource management/harvesting practice. The typeand scale of the project - and the interests of the users of theevaluation - will determine whether the questions and indicatorsneed to be targeted at the community level, the household levelor the individual level. In some cases it might be appropriate todisaggregate information even further and have questions andindicators that measure separately the different rates of uptake by,for example, women compared to men; richer people compared topoorer people; young people compared to elders etc. A judgement- depending on the nature of the intervention (and the types ofpeople it targets) and the interests of the users of the evaluation will be needed to determine exactly how to focus the more genericindicators given in Table 1.
For other outcomes (e.g. stakeholder well-being) the use of
pre-defined indicators is likely to be problematic, since wellbeing relates to the perspective of the individual (their tastes,
values, feelings, etc.) and different individuals/households will
have different interpretations of what well-being means tothem. In these cases it will be usually be necessary to ask project
participants what changes in their lives they expect to occur (or if
the project has already taken place, what changes have occurred).

122

PRISM
MODULESMETHOD FACTSHEETS& ANNEXES

This is true also of many livelihoods outcomes, as impacts ofprojects can be complex and indirect, and the choices people makewill vary – e.g. a small-scale horticulture project may increase farm
output, but increased production may be used to feed the family,
sold to raise cash to pay for school fees, or stored to reduce risk intimes of drought.
Well-being is conceptualised in three interacting dimensions:material/objective wellbeing; relational wellbeing; and subjective
wellbeing (see Section 1); and these should form the basis ofquestion and indicator selection. The Social Assessment forProtected Areas (SAPA)5 initiative measures household well-being
against the following indicators6:-

food security (material well-being);
assets, for example quality of housing, ownership of aradio or TV (material well-being);
influence on decision-making at village level (relational
well-being);
feeling of security (subjective well-being);
the question “how’s life?” (overall well-being).

Participatory Impact Assessment methods (PIA Methods 1-5)
typically include a stage during which indicators are described andrefined with the participants.

Franks, P and Small, R (2016). Social Assessment for Protected Areas (SAPA). Methodology Manualfor SAPA Facilitators. IIED, London. http://pubs.iied.org/14659IIED.html .

5Projects use different indicators according to their objectives, expected outcomes, and the specificsocial context. For examples see the indicators in some of the following project documents:•
http://awsassets.panda.org/downloads/wwf_livelihood_booklet_final__july8_08.pdf•
http://portals.wi.wur.nl/files/docs/ppme/LMP_Indicators.pdf•
http://careclimatechange.org/files/toolkit/CBA_Framework.pdf
6

PRISM - TOOLKIT FOR EVALUATING THE OUTCOMES AND IMPACTS OF SMALL/MEDIUM-SIZED CONSERVATION PROJECTS

3:3

INTRODUCTION &KEY CONCEPTS

DESIGNING & IMPLEMENTING
THE EVALUATION

PRISMMODULESMETHOD FACTSHEETS& ANNEXES

Understanding how your project actions aim to change
livelihoods & governance issues to benefit conservation

FIGURE 2: LIVELIHOODS & GOVERNANCE OUTCOME& IMPACT FRAMEWORKLIVELIHOODS &GOVERNANCE
UNDERSTANDING
HOW YOURPROJECT ACTIONSAIM TO CHANGE
LIVELIHOODS &GOVERNANCE
ISSUES TO BENEFITCONSERVATIONLIVELIHOODSACTIONSe.g. Developing
alternative / diversiﬁed
livelihoods, Maximisingvalue of resources,
Introducing /developing newtechnology

1. Uptake of new
or improved practices/ livelihood activities

2. Progress towards
secure tenure or
resource rights
for communities
5. Wellbeing
goals met

GOVERNANCE
ACTIONS
3. Governancearrangements giveincreasedrecognition/respecte.g. Securing tenure,building capacity ofcommunity organisations,Supporting recognitionof communityorgs

4. Community
institutions haveincreased capacity

Actions

123

Outputs

Intermediateoutcomes

Threat reduction
outcomes

Impact on
Biodiversity

PRISM - TOOLKIT FOR EVALUATING THE OUTCOMES AND IMPACTS OF SMALL/MEDIUM-SIZED CONSERVATION PROJECTS

6. Damaging
occupation/livelihood
abandoned
or reduced

Improvement inbiodiversity status

3:3

INTRODUCTION &KEY CONCEPTS

DESIGNING & IMPLEMENTING
THE EVALUATION

PRISMMODULES

Framework notes
Figure 2 illustrates some of the potential pathways by which
actions addressing local livelihoods and governance of natural
resources may help to achieve biodiversity impacts. This is a
generic framework for such actions, and any individual small
project is likely to address (or may need to address) only one or twoof the pathways illustrated.

LIVELIHOODS &GOVERNANCE
FRAMEWORK
NOTES

124

In all cases, the overall objective and impact that is sought is thesame – to provide local people with the means to make theirliving and achieve their well-being objectives with less negative
impact (and where possible more positive impact) on the targeted
environment and its biodiversity.
You will need to think carefully about what you can assess to givean indication of the impact of your project. In the short-term it
may only be possible to assess levels of uptake of new practices(outcome 1), or direct changes in tenure, institutional capacity or
governance arrangements (outcomes 2, 3 and 4). You may also beable to assess the impact on people’s well-being (outcome 5), since
many impacts of this kind (especially those related to relationaland subjective dimensions) may be achieved in the short-term.
Material changes to well-being may take longer. In the longer term
you should aim to assess changes in the behaviours that werecausing harm to the targeted biodiversity or ecosystem (outcome6); and ultimately to assess improvements in the conservation
status of habitats or species. At whatever outcome stage in theframework you undertake your assessment, it is important to test
(or be open about) the assumptions in your own Theory of Change(or equivalent) so that it is clear how (or whether) the livelihoods
and well-being impacts have led, or are expected to lead, toconservation outcomes.
PRISM - TOOLKIT FOR EVALUATING THE OUTCOMES AND IMPACTS OF SMALL/MEDIUM-SIZED CONSERVATION PROJECTSMETHOD FACTSHEETS& ANNEXES

3:3

INTRODUCTION &KEY CONCEPTS

DESIGNING & IMPLEMENTING
THE EVALUATION

PRISMMODULESMETHOD FACTSHEETS& ANNEXESTABLE 1: EXAMPLES OF INDICATORS AND SUGGESTED DATA COLLECTION METHODSOUTCOME/ IMPACT

EXAMPLE INDICATORSMETHODS

1.

Number of groups/households/individuals carrying out newpractices and/or using new technologies

Documentary evidence

Uptake of new or improved
practices/ livelihood activities

Questionnaires
Key informant interviews

LIVELIHOODS &GOVERNANCE
METHODS FOREVALUATINGLIVELIHOODS &GOVERNANCE
OUTCOMES/IMPACTS

Participatory Impact
Assessment (PIA)
2.

Progress towards secure tenure
or resource access rights forcommunities

Change in policy, rights or responsibilities – e.g. levelsof devolution of authority and decision-making to local
communities

Participatory Impact
Assessment (PIA)
Community mapping

Formal recognition of customary tenure
Documentary evidence
Number of groups/households/individuals with secure land
title or secure resource rights
Key (community) stakeholders engaged in processes to securerights/tenure; a formally agreed action plan3.

Table 1 summarises available
methods for assessing theoutcomes in the impact
pathway in Figure 2, including
methods for data collection.

125

Governance arrangements giveincreased recognition / respect

Change in selected indicators (individually or as part of ascore sheet) relevant to the attributes of good governance, forexample:o
Number of people involved in decision making, andtheir level of influenceoAccountability of representatives and effectivenessof reporting backoInclusion of different members of societyoEffective and fair enforcement of laws

PRISM - TOOLKIT FOR EVALUATING THE OUTCOMES AND IMPACTS OF SMALL/MEDIUM-SIZED CONSERVATION PROJECTS

Community or focus groupmeetings; (supplemented bydocumentary evidence)
Participatory Governance
Assessment
Questionnaires

3:3

INTRODUCTION &KEY CONCEPTS

DESIGNING & IMPLEMENTING
THE EVALUATION

PRISMMODULESMETHOD FACTSHEETS& ANNEXESTABLE 1: EXAMPLES OF INDICATORS AND SUGGESTED DATA COLLECTION METHODS CONTINUEDLIVELIHOODS &GOVERNANCE

OUTCOME/ IMPACT

EXAMPLE INDICATORSMETHODS

4.

Community institution has key organisational capacity
requirements for sustainable NRM – this will vary according to
the context (see capacity module). Capacity may be assessedagainst a suite of indicators (e.g. using tool for Organisational
Capacity Assessment) or in relation just to the particulardimension(s) addressed by the project.

Community Based
Organisational Capacity
Assessment

Well-being has material, relational and subjective dimensions.
These are very context, project, and culture specific (andmay also change over time). Specific indicators need to be
developed with the community (see above). For example, they
may include:oNumber of meals skipped by an individual in theprevious week (food security)oNumber of households with an effective strategy forcoping with unexpected shockoFeelings of security regarding access and resources(e.g. land) in the next 10 yearsoNumber of person-days of employment createdo
Net gains in income (per person/day) and number of
local beneficiaries

Participatory Impact
Assessment (PIA)

Number of groups/households/individuals abandoning or
reducing damaging practices

Participatory Impact
Assessment (PIA)

5.

Community institutions haveincreased capacity

Well-being goals met

METHODS FOREVALUATINGLIVELIHOODS &GOVERNANCE
OUTCOMES/IMPACTS

6.

Damaging occupation/ livelihood
abandoned or reduced

Participatory Impact
Assessment (PIA)

Basic Necessities Survey (BNS)Questionnaires
Focus groups
Direct observation(e.g. of housing quality)
Documentary evidence

Questionnaires
Direct observation
Threat reduction scoring126

PRISM - TOOLKIT FOR EVALUATING THE OUTCOMES AND IMPACTS OF SMALL/MEDIUM-SIZED CONSERVATION PROJECTS

3:3

INTRODUCTION &KEY CONCEPTS

Outcome 1.

DESIGNING & IMPLEMENTING
THE EVALUATION

Uptake of new or improved practices/
livelihoods activities

Evaluation of uptake involves looking at the extent to which aparticular practice or livelihood activity that has been promoted by
the project has been taken up by the project’s target audience.
Projects might adopt one or more of the following approaches:

LIVELIHOODS &GOVERNANCE
OUTCOME 1.
UPTAKE OF NEW
OR IMPROVED
PRACTICES/LIVELIHOODSACTIVITIES•Providing an alternative resource to the one that isbeing exploited, for example by developing fish ponds as
an alternative to lake fishing.

•

Providing alternative occupations which substitute forthe income obtained from unsustainable resource use.

•

Adding to or enhancing the value of the resource ornatural habitat to local communities, so that they are
incentivised to protect it and manage it sustainably,
e.g. through development of ecotourism, or providing
new markets for resources so that communities benefit
from their conservation – e.g. payments for ecosystemservices.

•

Promoting the adoption of technologies which support
more efficient, less damaging use of resources.
Examples include fuel-efficient stoves, technologies
that reduce post-harvest losses such as drying or
freezing of fish or solar-drying of forest mushrooms.

A simple method is to use documentary evidence (E.g. Project
records, direct and indirect observation surveys) or direct

127

PRISM
MODULESMETHOD FACTSHEETS& ANNEXES

observation surveys to verify whether, and the extent to which, aparticular practice has been taken up.
Another simple method is to use a questionnaire survey athousehold or individual level to assess the level of uptake. A
simple ‘yes/no’ response to a question asking about adoption of
the new activity will provide evidence of impact. If information isdisaggregated according to one or more socio-economic variables(age, gender, level of education, wealth, household size, etc.) thiswill help you understand how the introduced activity has been
adopted, and show the way to increase levels of adoption in future
initiatives. Providing opportunities for qualitative responses(e.g. regarding the reasons for uptake) will provide additional
information relevant to understanding why your project has or has
not had the impact you expected.
See the Basic Necessities Survey (BNS factsheet for an example
of a household survey template with the kind of questions you
could include to assess uptake under a programme to introduce
fuel-efficient stoves to improve post-harvest smoking of fish andreduce collection of fuelwood.
Participatory Impact Assessment (PIA) methods can be usedvery effectively to compare before/after situations, and they use
the community’s judgement to attribute uptake to the project. In
addition to looking at whether uptake has occurred (yes/no) thesecan also include questions looking at why uptake has/has not
happened (e.g. looking at motivation, factors limiting uptake, etc.)
and whether there is an expectation of any continuation of uptake
in the future.
Key informant interviews can also be used to evaluate uptake.

PRISM - TOOLKIT FOR EVALUATING THE OUTCOMES AND IMPACTS OF SMALL/MEDIUM-SIZED CONSERVATION PROJECTS

3:3

INTRODUCTION &KEY CONCEPTS

DESIGNING & IMPLEMENTING
THE EVALUATION

Main issues:•
Documentary evidence: Try to make sure that theobservations/documents are independently verifiable.
Project reports and records may suffice if alternatives
aren’t available, but records kept by a third party are
preferable.

128

Outcome 2.

METHOD FACTSHEETS& ANNEXES

Progress towards secure tenure or resource
access rights for communities

Secure tenure and rights of access to and use of land and resources
can provide people with an incentive to protect or manage theseresources sustainably in the long term.

•

Direct observation surveys: Not all uptake will be
immediately and easily visible, especially if it takes place
inside the home, or within closed family compounds (e.g.
uptake of fuel efficient stoves). Choose this method forchanges that can be easily observed.

Evaluating security of tenure or resource rights generally
requires a written, published record showing who (community
or individuals) owns the land or has rights of access and rights toharvest or otherwise use resources. Evaluation of these outcomesgenerally involves the use of documentary evidence.

•

Questionnaire surveys: If your project has targeted a
specific sector of the community (e.g. hunters, women,
ethnic minorities) then select your sample from thetarget group – you don’t need to take a random sample
from the entire community.

Participatory Impact Assessment (PIA) (e.g. participatory mapping)can also be used to assess changes in land ownership or in rights toresources.

LIVELIHOODS &GOVERNANCE
OUTCOME 2.
PROGRESSTOWARDS SECURE
TENURE ORRESOURCE ACCESSRIGHTS FORCOMMUNITIESPRISMMODULES•Key informant interviews: Relying on key informants
for a quantitative assessment of the level of uptake may
be unreliable, especially for medium-sized projects ordispersed/large communities. However, key informants
can provide useful insights into who (which socialgroups in a community) has or has not taken up the
livelihood activity, what the barriers are, whether the
livelihood has been adapted, etc.

•

Participatory Impact Assessment: PIA methods canbe used very effectively to compare before/after
situations, and they use the community’s judgement toattribute uptake to the project.

Main issues:
Documentary evidence: To assess project impact you willprobably be looking for new documents assigning tenure or
resource rights (i.e. dated after your project start date). Even
where such documents exist you will need to use interviews
with key informants to gather evidence that changes are
attributable to the project. It may take time to secure rights
to land and resources, and especially for small projects, themain impact of the project may have been simply to initiate
a process. Impact can be assessed through documentary
evidence of this process – creation and membership ofcommittees, records of meetings, interim decisions, etc.

PRISM - TOOLKIT FOR EVALUATING THE OUTCOMES AND IMPACTS OF SMALL/MEDIUM-SIZED CONSERVATION PROJECTS

3:3

INTRODUCTION &KEY CONCEPTS

Outcome 3.

DESIGNING & IMPLEMENTING
THE EVALUATION

Governance arrangements give increased
recognition/respect

Recognition of the legitimacy of community institutions, devolving
management responsibility to them and/or involving them inmanagement decisions at an appropriate level, helps to ensure that
resources are managed sustainably and in partnership (outcome 3).

LIVELIHOODS &GOVERNANCE
OUTCOME 3.
GOVERNANCE
ARRANGEMENTS
GIVE INCREASEDRECOGNITION/RESPECT

Your project may be aiming to address one or several
characteristics of governance. Table 2 below describes somepossible indicators and methods which could be used to assesschange relating to eight commonly recognised characteristics ofgood governance. These may apply at landscape level (e.g. landuse decisions within an administrative district), site level (e.g.
decision-making in a protected area) or community level (e.g.
how a community council makes decisions on resource use andaccess). Governance could be assessed against just one of thecharacteristics, or against combination of them that is relevant tothe specific context.

PRISM
MODULES

Main issues:•
Governance is a multi-dimensional concept, andperceptions of what is acceptable or good governancecan be culturally very specific. For example, insome cultures patronage is normal and expected;
religious beliefs or social convention may exclude theparticipation of women; and it may be disrespectful
to challenge top-down decisions made by traditional
leaders. Many of the indicators of governance also
require a subjective assessment, and the assessors’ own
cultural background is likely to affect the way they viewand report outcomes. It may be important to have theintended changes and their indicators clearly articulated
at the start of the project, and to involve the community
and affected stakeholders in this.

The Participatory Governance Assessment method uses ascore card to assess governance of an organisation in relation
to a number of these attributes. If repeated (before and after a
project) the results can be displayed graphically to show change ingovernance, and interviews (e.g. with a focus group) can discuss thereasons for any significant changes and identify those attributable
to the project.

129

METHOD FACTSHEETS& ANNEXESPRISM - TOOLKIT FOR EVALUATING THE OUTCOMES AND IMPACTS OF SMALL/MEDIUM-SIZED CONSERVATION PROJECTS

3:3

INTRODUCTION &KEY CONCEPTS

DESIGNING & IMPLEMENTING
THE EVALUATION

PRISMMODULESMETHOD FACTSHEETS& ANNEXESTABLE 2: POSSIBLE INDICATORS AND METHODS FOR MEASURING GOVERNANCE OUTCOMES/IMPACTSATTRIBUTE OF GOOD GOVERNANCEPOSSIBLE INDICATORSMEASURES / SOURCES OF INFORMATION /METHODSPARTICIPATIVE

Change in levels of participation (in decision-making processes) by, forexample:•women and men;•minorities; those who are vulnerable.

Documentary evidence (e.g. of meetings andparticipation, e.g. attendance lists; minutes –who spoke?).

Number and quality of forums/meetings/consultation meetings
between communities and authorities. “Quality” might be measured interms of levels of attendance, representation, levels of participation in
discussion, opportunities to present, documentation and transparencyof decisions, language used and accessibility, availability of information(before and after), etc.

Direct observation (e.g. of participation levels
at meetings - who was present? Who spoke?
How was the meeting organised - seating
arrangements; choice of chairperson etc.?
What information was made available? Whatlanguage was used? How was the meeting
documented and by whom?).

The degree to which civil society participation in decision-making iseffectively organised

Documentary evidence (e.g. of consensusbuilding among civil-society forums).

Change in the extent to which indigenous and local knowledge isrespected and applied to management decision-making.

Documentary evidence (e.g of management
plans; management decisions and theiracknowledgement, application of local
knowledge).

CONSENSUS-ORIENTED

Change in the extent to which the points of view of different
stakeholders are effectively mediated.

Documentary evidence: Likely to be subjective– e.g. based on records of meetings, decisionmaking processes, and the basis/justification
for any decisions.

ACCOUNTABLE

Change in the extent to which institutions and representatives(government and CBOs) report back to their members andconstituencies.

Documentary evidence
Questionnaires.
Focus groups.

LIVELIHOODS &GOVERNANCE
OUTCOME 3.
GOVERNANCE
ARRANGEMENTS
GIVE INCREASEDRECOGNITION/RESPECT130

PRISM - TOOLKIT FOR EVALUATING THE OUTCOMES AND IMPACTS OF SMALL/MEDIUM-SIZED CONSERVATION PROJECTS

3:3

INTRODUCTION &KEY CONCEPTS

DESIGNING & IMPLEMENTING
THE EVALUATION

PRISMMODULESMETHOD FACTSHEETS& ANNEXESTABLE 2: POSSIBLE INDICATORS AND METHODS FOR MEASURING GOVERNANCE OUTCOMES/IMPACTS CONTINUEDATTRIBUTE OF GOOD GOVERNANCEPOSSIBLE INDICATORSMEASURES / SOURCES OF INFORMATION /METHODS

TRANSPARENT

Change in the availability, frequency, adequacy and accessibility (easily
understandable forms and media, appropriate language) of information
to those who will be affected by decisions and their enforcement, where
aplicable.

Documentary evidence: Subjective assessmentof available information on decisions and thedecision-making process (language, media,
accuracy).

LIVELIHOODS &GOVERNANCE
OUTCOME 3.
GOVERNANCE
ARRANGEMENTS
GIVE INCREASEDRECOGNITION/RESPECT

Questionnaires & Focus groups (e.g. to assess
who received information, who understoodinformation, who felt information was
adequate).
RESPONSIVE

Change in the time taken for institutions to respond or act (e.g. onrequests for information; requests for intervention).

Documentary evidence.
Questionnaires.

EFFECTIVE AND EFFICIENT

Change in the level of resources (e.g. time, funding, materials) needed to
produce desired results (resources are used sustainably; equipment iswell maintained and lasts longer; materials are re-used and recycled).

Documentary evidence: Where records
are kept, use these (e.g. financial records,
procurement records, maintenance andservicing records for equipment). Otherwisebase on interviews with key informants.

EQUITABLE AND INCLUSIVE

Change in the extent to which all members of society feel that they havea stake in the community, and feel included.

Questionnaires
Focus groups

RULE OF LAW

Change in the extent to which laws are enforced impartially.

Focus groups.
Questionnaires.
Participatory Impact Assessment.

Good roles and practices in decision-making process, e.g. adoption of
key principles for decision-making process.
Change in the extent to which full human rights, particularly those ofminorities, are protected.

131

PRISM - TOOLKIT FOR EVALUATING THE OUTCOMES AND IMPACTS OF SMALL/MEDIUM-SIZED CONSERVATION PROJECTS

3:3

INTRODUCTION &KEY CONCEPTS

Outcome 4.

DESIGNING & IMPLEMENTING
THE EVALUATION

Community institutions have increased capacity

Where communally-shared property or resources are concerned,
the capacity of community institutions for regulating, managing
and policing access is key to achieving positive outcomes andimpacts.

LIVELIHOODS &GOVERNANCE
OUTCOME 4.
COMMUNITYINSTITUTIONSHAVE INCREASEDCAPACITY132

Evaluation of the capacity of community institutions involves
either the use of a capacity assessment tool or evaluation of anorganisation in relation to specific dimensions of capacity thathave been the focus of the project (e.g. financial management
procedures or legal status). See the factsheet: Community-based
Organisational Capacity Assessment for an example of a relatively
simple tool that can be used to evaluate changes in the capacity of
a community-based organisation.

PRISM
MODULESMETHOD FACTSHEETS& ANNEXES

Main issues•
Community-based Organisational Capacity
Assessment largely depends on self-assessment by
the organisation. However, many of the dimensionsof organisational capacity can (and ideally should) be
verified through documentary evidence – e.g. minutesof meetings, membership lists, bank accounts, audits,
legal registrations, business plans and organisational
strategies, observed equipment and office space, etc.

Use of the tool could also be combined with a Participatory Impact
Assessment (PIA), e.g. at a group meeting, to score an institution’s
capacity before and after the project and explore the reasons forany changes, and whether they can be attributed to the project.

PRISM - TOOLKIT FOR EVALUATING THE OUTCOMES AND IMPACTS OF SMALL/MEDIUM-SIZED CONSERVATION PROJECTS

3:3

INTRODUCTION &KEY CONCEPTS

Outcome 5.

LIVELIHOODS &GOVERNANCE
OUTCOME 5.
WELL-BEINGGOALS MET

DESIGNING & IMPLEMENTING
THE EVALUATION

The dimensions of well-being that are relevant will vary from place
to place, and so the aspects that need to be assessed for a given
project will depend on the context. Relevant well-being dimensions
may be suggested by the project’s Theory of Change.
The main purpose of a project that addresses people’s livelihoods is
to bring about changes in the well-being of individuals, households
or the wider community, in such a way that they cease or reduce
activities that are damaging the environment (or adopt activities
that benefit the environment). Well-being has been described ashaving three main dimensions (see Conservation and impacts onhuman well-being in Section 1), and within each dimension there
may be many contributing factors, often specific to a particularcommunity or household.
The design stages of your project should have explored this with thecommunity, in order to identify:

133

whose activities are damaging the environment/

METHOD FACTSHEETS& ANNEXES

biodiversity (so that the project can be effectively
targeted);

Well-being goals met

The concept of livelihoods is concerned with how people makea living, including the assets that they have available (e.g. land,
equipment, education, social networks and employment). Theirdecisions are affected by their well-being objectives – their vision ofwhat constitutes a “good life”. For some, this may be a high income,for some it may be the safety and security of their children, and forothers it may be about belonging to social and cultural networks.
People’s well-being objectives will affect their decisions about how
they use natural resources, and this in turn may affect the outcomeof projects that are concerned with livelihoods.

•

PRISM
MODULES•

which well-being dimensions need to be addressed(through livelihoods initiatives) to allow (or encourage)
the damaging activities to be stopped.

This will help you to identify appropriate indicators, and to selectsuitable methods for assessing impact. More than one method may
be required, depending on the specific indicator(s) chosen. Potential
methods include Participatory Impact Assessment (PIA), Basic
Necessities Survey (BNS), questionnaires, focus groups, direct
observation and documentary evidence e.g. (e.g. clinic records;
school attendance; stores and supplies sales).
Main issues:•
Focusing on certain target villages or specific livelihoodgroups within a community may be the most direct andobvious approach for maximising conservation value
from the project (and for impact assessment); but it isalso important to assess any indirect impact on the wider
community. For example, if the distribution of well-being
support and benefit is perceived to be inequitable, thismay lead to a breakdown in community relationships,
which could in-turn have an impact on governance of
natural resources. Increasing the value and marketabilityof natural resources may lead to appropriation of benefits
by a minority (“elite capture”) and marginalisation or
exclusion of traditional users. Efforts should be made toassess these less tangible and more indirect impacts oflivelihood and well-being projects.

PRISM - TOOLKIT FOR EVALUATING THE OUTCOMES AND IMPACTS OF SMALL/MEDIUM-SIZED CONSERVATION PROJECTS

3:3

INTRODUCTION &KEY CONCEPTS

Outcome 6.

LIVELIHOODS &GOVERNANCE
OUTCOME 6.
DAMAGING
OCCUPATION/LIVELIHOODABANDONED OR
REDUCED134

DESIGNING & IMPLEMENTING
THE EVALUATION

Damaging occupation/livelihood abandoned or
reducedEvaluation at this outcome stage involves assessing and comparing
the levels of damaging activity before and after the projectintervention. Where feasible, direct observation surveys may
be the most reliable method as they overcome problems such as
people’s unwillingness to reveal participation in illegal activity. The
module on species & habitat management describes methods forassessing levels of some of the most common damaging activities,
including hunting (by snares, firearms and poisons), live trapping,
agricultural encroachment, overharvesting of non-timber forest
products, disturbance, pollution, and timber and fuelwood
extraction.
Aside from direct observation, you can use questionnaires toassess levels of damaging or illegal use by individuals. Participatory
Impact Assessment (PIA) methods can also be used to investigate
people’s livelihood strategies, and the contribution being made by
the damaging activity, at community level.

PRISM
MODULESMETHOD FACTSHEETS& ANNEXES

Main issues:•
Although you may have identified a particular target
group for your livelihoods intervention (e.g. households
clearing forest for farmland, a sector of the community
involved in hunting or trapping, etc.) your survey may
need to sample from the wider community. This is toassess whether those who cease hunting are not justbeing replaced by “‘new” farmers/hunters etc. with nonet impact on the damaging activity.
•

Information on illegal or anti-social activities may bevery sensitive, and people may not give honest answers
about these issues in responses to questionnaires or in
focus group discussions, even if they are interviewed in
private and if questionnaires are anonymised. Where
this difficulty is likely or is suspected, direct observation
methods, expert opinion and people’s perceptions may
give more accurate results.

These methods can only be used to assess impact on damaging
and/or illegal activities if you are confident that people arewilling to discuss matters freely and honestly. Ideally, in order toassess whether your project has been the cause of any changed
behaviour, you need to include questions about uptake of theintroduced livelihood activity, changed governance and/or changes
in well-being. These questions could all be included in the samequestionnaire (in different sections).

PRISM - TOOLKIT FOR EVALUATING THE OUTCOMES AND IMPACTS OF SMALL/MEDIUM-SIZED CONSERVATION PROJECTS

3:4

INTRODUCTION &KEY CONCEPTS

DESIGNING & IMPLEMENTING
THE EVALUATION

PRISMMODULESMETHOD FACTSHEETS& ANNEXES

PolicySummaryThis chapter addresses methods for evaluating the outcomes andimpacts of conservation projects attempting to influence policy, inorder to contribute to improvements in the conservation status ofbiodiversity/ecosystem/habitat targets. Areas covered include:

POLICY
SPECIFICALLY THIS MODULE COVERS:

FACTSHEETS & ANNEXES

What do we mean by policy outcomes & impacts?

•

Media tracking

What should I consider when evaluating policy
outcomes & impacts?

•

Media scorecards•

Observation checklist for documenting meetings•

Policymaker ratings•

Bellwether methodology•

Civil society tracking tool

Understanding how your project actions aim to change
policy to benefit conservation targets.
Data collection methods for evaluating policy
outcomes & impacts:
1. Improved policy influence7 in government or
multilateral institutions;
2. Improved policy influence in private sector;
3. Environmental movement strengthened;
4. Evaluating implementation of policy and associatedpractice.
Further reading

135

7Influence can include: developing, changing, adopting, blocking, defending or
implementing policy. These objective categories build on those used in WWF (2009a):“Resources for Implementing the WWF Project and Programme Standards: Monitoring
Advocacy Work”, January 2009.

PRISM - TOOLKIT FOR EVALUATING THE OUTCOMES AND IMPACTS OF SMALL/MEDIUM-SIZED CONSERVATION PROJECTS

3:4

INTRODUCTION &KEY CONCEPTS

DESIGNING & IMPLEMENTING
THE EVALUATION

PRISMMODULESMETHOD FACTSHEETS& ANNEXES

What do we mean by policy outcomes & impacts?
The term “policy” is often used to capture a diverse range of
instruments (e.g. laws, regulations, strategies, plans, programmes,
budgets, declarations, procedures, multilateral environmental
agreements).

POLICY
WHAT DO WE
MEAN BY POLICYOUTCOMES &IMPACTS?

When evaluating policy outcomes & impacts it is crucial first to
clarify the policy/ advocacy objective(s) which will outline who
exactly (i.e. individuals, groups, organisations, policy fora) the
project is trying to have impact on.
In doing so, it is important to remember that policy/decision
makers are not a homogenous group, and that policy/decisionmaking happens at multiple levels (e.g. international, regional,
national and/or local). Without clarifying this beforehand, anevaluation that allows you accurately to describe progress towards
the desired impact(s) will be impossible.

Photo credit: SPNL136

PRISM - TOOLKIT FOR EVALUATING THE OUTCOMES AND IMPACTS OF SMALL/MEDIUM-SIZED CONSERVATION PROJECTS

3:4

INTRODUCTION &KEY CONCEPTS

DESIGNING & IMPLEMENTING
THE EVALUATION

What should I consider when evaluating policy outcomes &impacts?
Main challenges
Attribution - It can be very difficult to determine links between
policy influencing actions, changes in policy and any subsequentchanges relating to conservation targets.

POLICY
WHAT SHOULD ICONSIDER WHENEVALUATINGPOLICYOUTCOMES &IMPACTS?

•

•

137

o

Multiple factors may have an influence on policy.

oLack of a credible counterfactual means thattraditional matching evaluation designs will seldom
be applicable (although key informant interviews
or wider stakeholder questionnaires/surveys canprovide this in some form).

o

Policymakers may be unwilling to admit that theirdecisions were influenced by one group.

Policy influence is highly complex, non-linear and canbe irrational - with policies shaped by a multitude ofinteracting forces and actors. This makes it very difficult
to predict with confidence the likely consequences(and the timeframe) of a set of activities on policy
development and implementation.

PRISM
MODULES•METHOD FACTSHEETS& ANNEXES

It may also be contentious to attempt overtly to affirm
attribution/influence over a particular policy process or
outcome. Even the terminology itself can put advocacy/
policy partners and targets off. It may be best to use
alternative terms or approaches, such as “contribution
assessment”, “partnership” or “effectiveness”.

Timeframes can be unpredictable - Changes in policies, laws,
regulations and their enforcement can take years (and the timerequired for those policies themselves to have impact on theconservation targets concerned may be much longer still), but
evaluation data are often required before goals are achieved.
Similarly strategies and milestones may shift with the dynamic
external policy context.
Contribution is more likely than attribution - This is partly
because of the complexity of the environments that policy
outcomes occur in, but also because policy influence is often mosteffectively achieved through alliances, coalitions and networks.
Hidden decision-making processes may be used by bureaucracies
and politicians - This creates disincentives for sharing goodpractice: if one organisation/community of practice found the“magic bullet” to influence and shared it, then they would be facedwith greater competition for influence.

Evaluation designs using control groups are usually
unsuitable for policy influencing work, as it is difficult
to establish a plausible counterfactual because of thespecificity of policy systems (and also the difficulty toseparate a ‘control group’ geographically from the policy
impact or ‘sphere of influence’).

PRISM - TOOLKIT FOR EVALUATING THE OUTCOMES AND IMPACTS OF SMALL/MEDIUM-SIZED CONSERVATION PROJECTS

3:4

INTRODUCTION &KEY CONCEPTS

POLICY
WHAT CAN SMALL/
MEDIUM-SIZEDPROJECTS DO TO
EVALUATE POLICYOUTCOMES
AND IMPACTS
EFFECTIVELY?

138

DESIGNING & IMPLEMENTING
THE EVALUATION

What can small/medium-sized projects do to evaluate policy
outcomes and impacts effectively?
Use a theory-based approach - The specific challenges associatedwith attribution mean that theory-based evaluation designs areoften the most feasible for projects with policy components.
Developing a robust Theory of Change (or equivalent) can beparticularly useful, as this forms a road-map of how the policy
influencing activities are envisaged to result in the desired changes
in policy, which then allows you to identify the points where dataneed to be collected, and acts as a framework to explain how
observed changes may or may not result in subsequent policy
and/or conservation outcomes and impacts. These approaches
can be complemented by information from Participatory Impact
Assessments and/or interviewing key informants.

PRISM
MODULESMETHOD FACTSHEETS& ANNEXES

Focus more on qualitative indicators, rather than quantitative These may need to be “proxy” indicators (i.e. providing an indirect
measure or sign of change), since results of policy actions are often
less tangible than others.
It is important to consider what the project’s main contribution
is intended to be, and to focus assessment on that. Therefore,
standard methods that identify an independent variable and a
specific set of dependent variables likely to affect change are notoften well suited for measurement of advocacy and policy work.
Be flexible - As policy work is highly complex and is influenced bymultiple interacting forces/actors, policy influencing strategies
need to evolve over time, and therefore data collection andinterpretation needs to be flexible to adapt to this.

Focus on the process, not just the intended result - Policy
evaluation typically focuses on the journey rather than just thedestination. In addition to demonstrating progress, this approach
reduces the risk that the evaluation will conclude that the wholeadvocacy effort was a failure if advocacy goals are not achieved
within the project’s timeframe. It is also likely that unintended
outcomes may be identified, particularly if the external policy
environment has altered significantly since the start of the
project. Sometimes original policy/advocacy goals may need to bereviewed in light of changing contexts, as they may no longer beconsidered viable or to be the most effective approach.

PRISM - TOOLKIT FOR EVALUATING THE OUTCOMES AND IMPACTS OF SMALL/MEDIUM-SIZED CONSERVATION PROJECTS

3:4

INTRODUCTION &KEY CONCEPTS

DESIGNING & IMPLEMENTING
THE EVALUATION

Understanding how your project actions aim to change policy tobenefit conservation targets

PRISM
MODULESMETHOD FACTSHEETS& ANNEXES

The framework below illustrates how actions focused on policy
may lead to a positive change in the conservation target.

FIGURE 1: POLICY OUTCOMES & IMPACTS
FRAMEWORKPOLICYUNDERSTANDING
HOW YOURPROJECTACTIONS AIM TO
CHANGE POLICYTO BENEFITCONSERVATIONTARGETS

1. Improved policy
inﬂuence ingovernment or
multilateral
institutions
POLICYACTIONSe.g. Providing evidence topolicymakers, Lobbying& negotiating withstakeholders, Publiccampaigns &advocacy

2. Improved policy
inﬂuence inprivate sector

4. New/improved
policy and/or
associated practice
implemented

3. Environmental
movement
strengthened

Actions

139

Outputs

Intermediateoutcomes

Threat reduction
outcomes

Impact on
Biodiversity

PRISM - TOOLKIT FOR EVALUATING THE OUTCOMES AND IMPACTS OF SMALL/MEDIUM-SIZED CONSERVATION PROJECTS

Threat(s) reduced

Improvement inbiodiversity status

3:4

INTRODUCTION &KEY CONCEPTS

DESIGNING & IMPLEMENTING
THE EVALUATION

PRISMMODULES

Framework notes
Outcome 1 refers to actions where a group (or groups) work to
influence laws, strategies, programmes or budgets at nationalor local government levels, and/or in intergovernmental
processes that influence such national/local level policy (throughdeclarations, statements/communiqués, decisions, resolutions,etc.).

POLICYFRAMEWORKNOTESOutcome 2 refers to actions that focus on influencing the policies
and the institutional behaviour of the private sector. This may
involve work to encourage environmentally friendly practices aspart of sustainable development.
Specific actions for both outcomes 1 and 2 are likely to involve a
combination of providing evidence and advice to key influencers,
carrying out public campaigns and advocacy, and lobbying andnegotiating.
Outcome 3 relates to actions to increase the capacity of theenvironmental movement to influence policy processes, to holddecision-makers to account, to present options to policy-makers
and to keep the public informed of shortcomings. Actions include
increasing capacity to work together effectively in networks/
alliances/coalitions at the local, national and international levels.
Outcome 4 relates to evaluating the implementation of policy andassociated practice that has been adopted/changed/blocked as aresult (or partly as a result) of the project. In many cases, policies
are already in place but have not yet been fully implemented andevaluated. Actions aimed at supporting the evaluation of policies
or at building capacity for implementation would be included inthis category.

140

PRISM - TOOLKIT FOR EVALUATING THE OUTCOMES AND IMPACTS OF SMALL/MEDIUM-SIZED CONSERVATION PROJECTSMETHOD FACTSHEETS& ANNEXES

3:4

INTRODUCTION &KEY CONCEPTS

DESIGNING & IMPLEMENTING
THE EVALUATION

PRISMMODULESMETHOD FACTSHEETS& ANNEXESTABLE 1: SUMMARY OF METHODS FOR ASSESSING THE OUTCOMES AND IMPACTS OF PROJECT ACTIVITIES RELATED TO
POLICY INFLUENCEOUTCOMES/ IMPACT

EXAMPLE INDICATORSMETHODS

1.

Improved policy
influence ingovernment
or multilateral
institutions

•
•
•
•
•
•
•

Engagement of key influencers in development of action plan
Changes in rhetoric
Changes in wording of policies or conventions
Ratification of conventions
Changes in legislation (or legislation maintained, or negative changes prevented)
Changes in budget allocations
Monitoring procedures in place for the policy

Media tracking
Media scorecard
Documentation of meetings
Policymaker ratings
Key informant interviews
Focus groups2.

Improved policy
influence in private
sector•Changes in particular company practices/ policies or engagement/ rhetoric on theissue
Changes in sector-wide consensus and Codes of PracticeFinance sector investment in the sector/company change
Introduction of labelling or certification systems
Environmental and social impacts of the above, e.g. market share of sustainableproducts and resulting impact
Transparency and impact of change (e.g. triple bottom line introduced in company
accounts, natural capital accounting, public auditing and reporting)

Bellwether methodology
Questionnaires
Key informant interviews

Increase in NGO advocacy and policy skills, capacity, knowledge andeffectivenessGreater synergy of aims/activities in environmental networks/ movements
Increase in collaboration, trust, unity and sustainability between movement
members
Increase in size of movement (number of organisations, size of organisations)

Civil Society Tracking Tool
Network health scorecard

Change in decision-makers’ awareness of pros and cons of existing policies andpractices
Change in capacity of government officials to implement policies
Extent to which policies are implemented
Extent to which implemented policies achieve the desired effectEnvironmental and social impacts of implemented policies

Key informant interviews
Policy tracking

POLICYMETHODS FOREVALUATINGPOLICYOUTCOMES &IMPACTS

•
•
•
•
•

3.

Informed by: WWF (2009a): “Resources
for Implementing the WWF Project
and Programme Standards: Monitoring
Advocacy Work”, January 2009.

141

Environmental
movement
strengthened

•
•
•
•

4.

New/improved policy
and/or associatedpractice implemented

•
•
•
•
•

PRISM - TOOLKIT FOR EVALUATING THE OUTCOMES AND IMPACTS OF SMALL/MEDIUM-SIZED CONSERVATION PROJECTS

3:4

INTRODUCTION &KEY CONCEPTS

DESIGNING & IMPLEMENTING
THE EVALUATIONThe numbers and corresponding notes below relate to theoutcomes in Table 1 above. See the PRISM Species/Habitat
Management chapter for methods for evaluating the change in thetarget biodiversity outcome.

Outcome 1.

Improved policy influence in government or
multilateral institutions

POLICYOUTCOME 1.
IMPROVED POLICYINFLUENCE IN
GOVERNMENT OR
MULTILATERALINSTITUTIONS

142

This involves looking at the processes by which your project isworking to influence public decision-making, either by targetinggovernments (local, regional or national) or multilateral institutions(organisations formed between several nations to work on issues
that relate to all the countries involved, for example the EU, OECD
or UN agencies). These processes could include laws, the creationof programmes, allocation of resources and implementation.

PRISM
MODULESMETHOD FACTSHEETS& ANNEXESMedia tracking, media scorecards and observation checklists are
all useful data collection methods that can be used to track change
and inform the judgements you make in relation to this outcome.
Periodic key informant interviews or focus groups with relevant
experts, decision-makers and other key stakeholders could also beused.
At a more advanced stage, policymaker ratings can be used togauge political will on a particular policy issue or in relation to
proposals under discussion among a defined group of policymakers(e.g. legislature, council).

Potential data collection methods
The most basic data collection method is simply to keep a record(for example a written log, email inbox or database) wherecomments, anecdotes and examples of “uptake” or influence arerecorded. For example the log could track: public messaging;monitoring; parliamentary committee meetings; key individuals’
participation; and tracking of key phrases/rhetoric throughout.
This would be essentially a collection of largely informal andanecdotal evidence about the use of research or advice, which
can provide useful ongoing monitoring and contribute to deeper
analyses once a number of instances or patterns are accumulated.

PRISM - TOOLKIT FOR EVALUATING THE OUTCOMES AND IMPACTS OF SMALL/MEDIUM-SIZED CONSERVATION PROJECTS

3:4

INTRODUCTION &KEY CONCEPTS

Outcome 2.

DESIGNING & IMPLEMENTING
THE EVALUATION

Improved policy influence in the private sector

The focus here is on looking at the effect of the project on thepolicies, practices and institutional behaviour of the private
sector. It may involve work to change particular bad practice,
work to encourage the introduction of environmental monitoringand policies, or “rewards” or incentives for good practice such aslabelling/certification schemes.

POLICYOUTCOME 2.
IMPROVEDPOLICY INFLUENCE
IN THE PRIVATESECTOROUTCOME 3.
ENVIRONMENTAL
MOVEMENT
STRENGTHENED

143

Possible data sources include: reviews of company reports/
data; tracking public rhetoric (e.g. public and company media);participation, rhetoric and commitments in partner meetings(e.g. roundtables), as well as reviewing secondary literature (e.g.
company, sector or public records; consumer groups/reports,such as FSC (Forest Stewardship Council); industry association
reports and on-the-ground evidence of implementation (e.g.
Environmental Impact Assessments).

PRISM
MODULES

Outcome 3.

METHOD FACTSHEETS& ANNEXES

Environmental movement strengthened

Here evaluation is focused on increasing the capacity of theenvironmental movement: to influence the policy process; tomonitor government enforcement of existing laws and policies;
to keep the public and policy makers informed of shortcomings;
and to demand transparency and accountability. It also includes
increasing capacity to work together effectively in networks/
alliances/coalitions at the local, national and international levels.
Sources of data include: records of action taken (e.g. description/
minutes of meetings or engagement in policy processes); numbersof proposals made to power holders, and numbers of approaches
solicited from power holders and the media.
The Civil Society Tracking Tool and network health scorecards
are relevant tools that can be used to collect data to inform
evaluations.

Key informant interviews, questionnaires and bellwether
methodology are all relevant methods for monitoring andreviewing/evaluating this type of outcome.

PRISM - TOOLKIT FOR EVALUATING THE OUTCOMES AND IMPACTS OF SMALL/MEDIUM-SIZED CONSERVATION PROJECTS

3:4

INTRODUCTION &KEY CONCEPTS

Outcome 4.

POLICYOUTCOME 4.
NEW/IMPROVED
POLICY AND/OR ASSOCIATED
PRACTICEIMPLEMENTED

144

DESIGNING & IMPLEMENTING
THE EVALUATION

PRISMMODULES

New/improved policy and/or associated practice
implemented

The focus here is on evaluating the effect of the project’s
policy action(s) on the adoption, modification and subsequent
implementation of targeted policies. Evaluating outcomes andimpacts at this stage is a significant challenge for small/mediumsized conservation projects due to the length of time that it takes
for these outcomes & impacts to become measurable.
Some countries, however, have online searchable databases thatcan be used to track the adoption and implementation of policies.
Key informant interviews with relevant decision makers can also
sometimes be used to track the process of changes in policy,
although be aware that many decision makers may be unwillingto admit that a specific factor influenced a particular decision.
References to evidence provided by the project or by related
networks (in speeches, meetings or documents related to thepolicy) can provide a proxy indicator of contribution.

PRISM - TOOLKIT FOR EVALUATING THE OUTCOMES AND IMPACTS OF SMALL/MEDIUM-SIZED CONSERVATION PROJECTSMETHOD FACTSHEETS& ANNEXES

3:4

INTRODUCTION &KEY CONCEPTS

DESIGNING & IMPLEMENTING
THE EVALUATION

Further readingThis chapter has been informed by the following resources:

POLICY•ODI - Background Note - A guide to measuring policy& advocacy influence http://www.odi.org/sites/odi.org.
uk/files/odi-assets/publications-opinion-files/6453.pdf .

•UNICEF Advocacy Toolkit - http://www.unicef.org/
evaluation/files/Advocacy_Toolkit.pdf .

•BOND - Assessing effectiveness in influencing power
holders https://my.bond.org.uk/sites/default/files/
impact-builder/Advocacy.pdf .

FURTHER READING

145

PRISM
MODULESMETHOD FACTSHEETS& ANNEXES

Additional resources and reading to consider:o

http://www.betterevaluation.org/ - good,accessible materials and webinars on designingevaluations.

ohttps://www.intrac.org/what-we-do/monitoringevaluation-learning/ - accessible materials andresources on monitoring and evaluation. The
Theory of Change materials are particularly
helpful.

o•Annie E Casey Foundation – A guide to measuringadvocacy and policy http://www.aecf.org/m/
resourcedoc/aecf-aguidetomeasuringpolicyandadvoca
cy-2007.pdf .

http://www.evaluationinnovation.org/sites/
default/files/Pathways%20for%20Change.pdf - areally useful paper outlining ten models or typesof Theories of Change which could apply to andbe adapted for various projects/programmes.
The Evaluation Innovation site http://www.

•

WWF (2009a). “Resources for Implementing the
WWF Project and Programme Standards: Monitoring
Advocacy Work”, January 2009.

evaluationinnovation.org/ is also very useful initself.

•WWF (2009b). “Resources for Implementing the
WWF Project and Programme Standards: Monitoring
Advocacy Work – Appendices”, January 2009.

PRISM - TOOLKIT FOR EVALUATING THE OUTCOMES AND IMPACTS OF SMALL/MEDIUM-SIZED CONSERVATION PROJECTS

3:5

INTRODUCTION &KEY CONCEPTS

DESIGNING & IMPLEMENTING
THE EVALUATION

PRISMMODULESMETHOD FACTSHEETS& ANNEXES

Species & Habitat Management
SummaryThis chapter contains methods for evaluating the outcomes
and impacts of conservation projects attempting to facilitate or
implement direct management of species or habitats.

SPECIES & HABITATMANAGEMENTMETHOD FACTSHEETS•

Scorecard for evaluating changes in knowledge gaps•

Scorecard for evaluating action plan completion•

Scorecard for evaluating action plan adequacy•

Threat reduction scoring•

Scorecard for evaluating changes in species’ status•

Scorecard for evaluating changes in habitat status

What should I consider when evaluating species andhabitat management outcomes/impacts?

•

Using field assessment methods for evaluatingchanges in species’ status

Understanding how project actions contribute toimproving the status of a species or habitat•

Using remote assessment methods evaluatingchanges in habitat status

Methods for evaluating species and habitat
management outcomes and impacts:•

Using field assessment methods for evaluatingchanges in habitat status

SPECIFICALLY THIS MODULE COVERS:

1. Knowledge improved
2. Conservation action plan developed anddisseminated
3. Threats to species/habitats reduced

ANNEXES•

Field methods for species monitoring•

Sampling approaches for monitoring different species
to assess their status•

Analytical issues when estimating abundance,
distribution, survival and reproduction•

Field methods for habitat monitoring•

Sampling approaches for monitoring habitat status

4. Recovery of species/habitats promoted
5. Species status improved (population size, trends,
range size, extinction risk)
6. Habitat status improved (extent, condition,
fragmentation)

146

PRISM - TOOLKIT FOR EVALUATING THE OUTCOMES AND IMPACTS OF SMALL/MEDIUM-SIZED CONSERVATION PROJECTS

3:5

INTRODUCTION &KEY CONCEPTS

DESIGNING & IMPLEMENTING
THE EVALUATION

What should I consider when evaluating species and habitat
management outcomes/impacts?

SPECIES & HABITATMANAGEMENT
WHAT SHOULD ICONSIDER WHENEVALUATINGSPECIES AND
HABITATMANAGEMENTOUTCOMES/IMPACTS?

147

Time-lags. It often takes time for species populations and habitats
to recover once threats have been reduced and/or barriers torecovery have been removed/reduced. Small projects usually
have a short duration, and direct improvements to species or
habitat status may not be detectable within the timeframe of the
project. In many cases, monitoring and evaluation should focus
only on the outcomes that will occur during the lifetime of the
project, unless post-project evaluation is possible, in which casemeasuring improvements to species or habitat status may be
worth considering.

PRISM
MODULESMETHOD FACTSHEETS& ANNEXES

Project scale. The scale of the evaluation should be in line with thesize of the project. Sometimes the most robust approaches are not
the most appropriate ones to use, especially if you have limited
resources in terms of time or people. Good evaluation design willtake into account the size of the project in determining the amountof monitoring effort that is appropriate.
Baseline data. In many cases, baseline data may be available
in the form of existing species/habitat monitoring data. Check
beforehand to see what information is available. The potential forusing certain methods will be affected by whether baseline data
(e.g. population estimates) are available. If not, techniques thatinvolve retrospectively assessing the project should be used.

Evaluation focus. It may not be appropriate to attempt to assess
the impact of a single project’s actions directly on species or
habitat status because of time-lags and the complexities ofidentifying cause and effect. Instead, consider more targetedmeasurement and monitoring of the outcomes and impacts ofparticular interventions that contribute to species or habitat
outcomes, for example by filling a key knowledge gap or reducing a
specific threat.

PRISM - TOOLKIT FOR EVALUATING THE OUTCOMES AND IMPACTS OF SMALL/MEDIUM-SIZED CONSERVATION PROJECTS

3:5

INTRODUCTION &KEY CONCEPTS

DESIGNING & IMPLEMENTING
THE EVALUATION

Attributing outcomes/impacts to species and habitat
management actions

Detailed guidance on evaluation designs is provided in the section
on choosing an evaluation design. Here we consider the designs
likely to be of greatest relevance to evaluating the outcomes andimpact of project actions on species and habitat management.

SPECIES & HABITATMANAGEMENTATTRIBUTINGOUTCOMES/IMPACTS TO
SPECIES AND
HABITATMANAGEMENTACTIONS

148

Interviewing key informants - Use the opinion of relevant
individuals/experts to score the relevant variable (e.g. level ofknowledge, magnitude of threat, category of extinction risk, etc)
at the beginning of the project, at the end of the project, and ineither an actual or a hypothetical situation in which the project
had not taken place (a “counterfactual scenario”). The difference
between the actual change measured, and the situation in the“no-project” scenario, represents the outcome or impact of the
project. This may be the only feasible approach for many small
and medium-sized conservation projects. Judgements should bemade as objectively as possible, and notes provided to justify thescores selected. Where multiple informants provide scores, use
the mean. The scorecard methods outlined in this chapter are alldesigned to be used in this way. Informants can also be asked torule out alternative explanations as a way of investigating further
the reasons behind any observed changes and the extent to whichthese can be attributed to the project.

PRISM
MODULESMETHOD FACTSHEETS& ANNEXES

Matching designs - Compare the outcomes (e.g. habitat condition)at the project location with those at a control location. The
compared sites must have similar environmental and socioeconomic conditions (e.g. habitat type, human influence, etc.)
allowing for differences in outcomes/impacts to be attributed
to project actions. This requires monitoring of both project andcomparison locations in the same way, and will require additionaltime and resources. The power of this approach is improved if
populations are also monitored before the start of the project. In
practice, small projects will often struggle to identify control sites
and to devote the necessary resources needed to monitor them,so before using a matching design to evaluate species and habitat
management outcomes/impacts, you will need to ensure that you
have the necessary resources to do so.
Before-after designs - Monitor the parameter of interest (e.g.
population size) over time to compare its position before and after
project interventions. This requires an evaluation of the situation
before project actions are implemented and a further evaluation
once they have been completed. As biological systems are often
extremely complex and subject to many external factors, thisapproach should generally be limited to short-term effects where itis easy to rule out alternative explanations.

PRISM - TOOLKIT FOR EVALUATING THE OUTCOMES AND IMPACTS OF SMALL/MEDIUM-SIZED CONSERVATION PROJECTS

3:5

INTRODUCTION &KEY CONCEPTS

DESIGNING & IMPLEMENTING
THE EVALUATION

PRISMMODULESMETHOD FACTSHEETS& ANNEXES

FIGURE 1: SPECIES AND HABITAT MANAGEMENTOUTCOME AND IMPACT FRAMEWORK8SPECIES & HABITATMANAGEMENTUNDERSTANDING
HOW PROJECTACTIONSCONTRIBUTE TO
IMPROVING THE
STATUS OF A SPECIESOR HABITATRESEARCH &MONITORINGTO IMPROVEKNOWLEDGE
e.g. survey population,
study ecology, identify
threats, assess
extinction risk

1. Knowledge
improved toprioritise action
for species/habitat

DEVELOPINGSPECIES/HABITAT/SITE ACTION PLAN

e.g. convene stakeholders,
prioritise interventions,agree timelines,
disseminate plan

DIRECTLY
REDUCING THREATSTO SPECIES/HABITATS
e.g. enforcing regulations,removing snares,
controlling invasive
species, reducing
habitat loss/
degradation

3. Threats tospecies/habitat
reduced

PROMOTING
SPECIES/HABITATRECOVERY
2. Species/site/
habitat action
plan(s) developed

4. Recovery of
species/habitats
promotede.g. increased habitat
extent, improved habitat
condition, enhanced
habitatconnectivity

Actions

Outputs

Intermediateoutcomes

5. Improved species
status e.g. increased
population size, slower
population declines, increased
reproductive success,increased distributionsize

6. Improved habitat
status e.g. increased habitat
extent, improved habitat
condition, enhanced
habitat connectivity

Threat reduction
outcomes

Impact on
Biodiversity

Colours, shapes and terms for the boxes match those used by the Conservation Measures
Partnership and Miradi Conservation Planning software www.miradishare.org/actions.

8149

PRISM - TOOLKIT FOR EVALUATING THE OUTCOMES AND IMPACTS OF SMALL/MEDIUM-SIZED CONSERVATION PROJECTS

3:5

INTRODUCTION &KEY CONCEPTS

DESIGNING & IMPLEMENTING
THE EVALUATION

Framework notes
The framework is designed to help you to understand how theactions implemented by a conservation project may benefit
one or more species or habitats. It is useful to clarify this before
considering which outcomes or impacts to measure.

SPECIES & HABITATMANAGEMENTFRAMEWORK
NOTES

150

Conservation projects may attempt to improve the status of
species or habitats in a number of ways, either indirectly byincreasing knowledge (for example of the distribution, threats, or
biological requirements relating to a species), developing actionplans, or directly by tackling threats or promoting the recovery
of a species. Figure 1 illustrates these different types of actions
and shows how they link together in a logical pathway to lead
ultimately to an improvement in the status of species or habitats.
Small- and medium-sized projects typically involve actions
addressing just one or two of these steps. The numbers for eachoutcome correspond to the categories of information in Table 1
and in the summaries below. These will direct you to appropriate
methods for assessing outcomes and impacts.

PRISM
MODULESMETHOD FACTSHEETS& ANNEXES

Outcomes 1 and 2 are focused on creating the conditions forfurther conservation actions to take place, while outcomes 3 and4 relate to actions addressing the threats and barriers to species/
habitat recovery. Outcome 3 results from directly addressing
the scope or severity of threats (e.g. searching for and removing
snares, erecting signs and fences to discourage encroachment,
protecting nests from trappers, eradicating invasive species, etc.),
whereas outcome 4 results from actions that reduce barriers torecovery by overcoming the ecological limitations imposed byexisting threats (e.g. providing nest boxes for species that utilise
tree-holes if availability of holes limits population size, planting
trees to promote habitat restoration, etc.). Often projects involve a
combination of these two types of action. Outcomes 5 and 6 relate
to changes in the status of species and habitats. It is usually the aimof a conservation project to have an effect at this species/habitat
status level.

PRISM - TOOLKIT FOR EVALUATING THE OUTCOMES AND IMPACTS OF SMALL/MEDIUM-SIZED CONSERVATION PROJECTS

3:5

INTRODUCTION &KEY CONCEPTS

DESIGNING & IMPLEMENTING
THE EVALUATION

PRISMMODULESMETHOD FACTSHEETS& ANNEXES

EXAMPLE

SPECIES & HABITATMANAGEMENTUNDERSTANDING
HOW PROJECTACTIONSCONTRIBUTE
TO IMPROVINGTHE STATUS OF
A SPECIES OR
HABITAT

Target site: Three projects are all working to support thesubsequently put into action, something the team decided

conservation of a coral reef, areas of which have been severely

was not measureable in the six-month timeline of the project.

degraded by the activities of dive operators. On the reef it

Therefore they decided to focus evaluation on how effectively

takes around ten years for coral to recolonise an area that has

the project had addressed the identified knowledge gap, and to

been degraded.

assess their contribution to the development of an action planfor the site (outcomes 1 & 2).

Project A was a six-month project surveying a previously
unstudied section of the reef in order to discover the extent

Project B was aiming to mitigate or reduce the intensity ofand diversity of coral present and to make recommendations

threat. While this should lead directly to improvements infor future conservation efforts.

habitat status, the team decided, given the time it takes forcoral to recolonise a degraded area, that such outcomes were

Project B was a two-year project aiming to reduce the impact

unlikely to be measurable within the two-year timeline of theof dive boats visiting a well-studied section of the reef, byproject. Therefore the team decided to focus on evaluatingpromoting the implementation of a set of good-practicewhether the project had been successful in reducing any of thestandards for dive operators to follow.

threats (outcome 3) targeted by the good practice standards

Project C was a three-month effort to carry out follow-up
surveys of an area that had previously been surveyed at regular(e.g. ensuring that dive operators were tethering boats tobuoys rather than directly on to the reef).

intervals over a ten-year period following the establishment of

Project C also could have focused on threats (outcome 3) ora marine reserve.

recovery (outcome 4), but given that the project length covered

While the eventual goal of all three projects was to see an
improvement in the status of the reef, there is a difference inwhat they can expect to measure with the time and resources

the amount of time it takes for degraded areas of reef to show
signs of recovery, the team decided to focus on measuring
whether or not any recovery had taken place (outcome 6).

available.
Project A focused on gathering and communicating
the information needed to make future conservation
decisions. A successful outcome required that the project’s
recommendations were taken up by policymakers and151

PRISM - TOOLKIT FOR EVALUATING THE OUTCOMES AND IMPACTS OF SMALL/MEDIUM-SIZED CONSERVATION PROJECTS

3:5

INTRODUCTION &KEY CONCEPTS

DESIGNING & IMPLEMENTING
THE EVALUATION

PRISMMODULESMETHOD FACTSHEETS& ANNEXESTABLE 1. SUMMARY OF THE METHODS COVERED IN THIS CHAPTEROUTCOMES/ IMPACTMETHOD

EXAMPLES OF INDICATORS PRODUCEDHOW ATTRIBUTED TO PROJECTEVALUATION DESIGNREQUIRES BASELINE?

EFFORT

1. Knowledge
improved

Knowledge gaps scorecard
Score 1-5 for level of knowledge for 6 types ofknowledge

Score for knowledge of threats

Score at beginning, at end andfor counterfactual

Interviewing key
informants

No

Low

2. Action plan(s)developed &disseminatedAction plan completion scorecard
Score 1-10 for completion

Score for action plan completion

Score at beginning, at end andfor counterfactual

Interviewing key
informants

No

Low

Action plan adequacy scorecard
Score 1-3 for adequacy

Score for action plan adequacy

Score at beginning, at end andfor counterfactual

Interviewing key
informants

No

Low

3. Threats
reducedThreat reduction scoring
Score 0-3 for each of timing, scope and severity

Magnitude of threat to species or
habitatScore at beginning, at end andfor counterfactual

Interviewing key
informants

No

Low

4. Recoveryof species/
habitats
promoted

Assess productivity, survival, population size,
habitat restoration, etc.

Survival rate; population size; area of
habitat restored

Estimate % change owing toproject

Matching design or
before-after design

Yes

Medhigh5. Species
statusimproved

Species status scorecard
Scorecard for change in species abundance

Score for change in species
abundance

Score at beginning, at end andfor counterfactual

Interviewing key
informants

No

Low

Field assessment methods for species
Assess changes in species status (population
size, trends, range size, etc.)

Population size, number of locations

Estimate % change owing toproject

Matching design or
before-after design

Yes

Medhigh

Habitat status scorecard
Scorecard for change in habitat quality

Score for change in habitat quality

Score at beginning, at end andfor counterfactual

Interviewing key
informants

No

Low

Remote assessment methods
Assess changes in habitat status using remotesensing data

Forest cover; deforestation rate

Estimate % change owing toproject

Matching design or
before-after design

Yes

Med

Field assessment methods for habitats
Assess changes in habitat status using fieldsurveys

Mean tree diameter at breast height;
population presence/ abundance

Estimate % change owing toproject

Matching design or
before-after design

Yes

Medhigh

6. Habitat
statusimproved

152

PRISM - TOOLKIT FOR EVALUATING THE OUTCOMES AND IMPACTS OF SMALL/MEDIUM-SIZED CONSERVATION PROJECTS

3:5

INTRODUCTION &KEY CONCEPTS

DESIGNING & IMPLEMENTING
THE EVALUATION

PRISMMODULESMETHOD FACTSHEETS& ANNEXESFirst choose which outcome/impact in the framework (Figure 1)
you judge to be most appropriate to evaluate. Then use Table 1 andthe correspondingly numbered sections below to choose the mostappropriate monitoring and evaluation method (considering thetime and resources available, among other factors), and refer to thespecific Methods factsheets and associated annexes.

SPECIES & HABITATMANAGEMENTMETHODS FOREVALUATINGSPECIES AND
HABITATMANAGEMENTOUTCOMES AND
IMPACTSFor several types of species/habitat management outcomes andimpacts, evaluation is challenging and requires time and resourcesthat may exceed the capacities of many small and medium-sized
conservation projects. Therefore, in addition to more robust
methods, we also provide simpler, quicker “scorecard” methods.
These can be applied using observed data collected during the
project or as part of an Interviewing key informants evaluation
design where the experience and knowledge of key individuals isused to assess the project’s outcomes and impacts. Respondents
should provide explanatory notes and justifications for each score
selected.

Photo credit: By Scdnr via Wikimedia Commons

153

PRISM - TOOLKIT FOR EVALUATING THE OUTCOMES AND IMPACTS OF SMALL/MEDIUM-SIZED CONSERVATION PROJECTS

3:5

INTRODUCTION &KEY CONCEPTS

Outcome 1.

SPECIES & HABITATMANAGEMENTOUTCOME 1.
KNOWLEDGEIMPROVEDDESIGNING & IMPLEMENTING
THE EVALUATION

Knowledge improved

Many conservation projects involve actions that attempt toimprove knowledge of the status of a species or habitat (in order,
ultimately, to identify and prioritise appropriate conservation
actions). Lack of knowledge may mean that project actions are
ineffective or even damaging, so it may be necessary for projects
to focus on (or to contain an element of) improving knowledge inorder to inform subsequent conservation actions.
Types of knowledge gaps that a project might address include:•

distribution;•

population size;•

trends (e.g. in population size, distribution, habitat
extent);•threats;•

ecology;•extinction risk;•

solutions (i.e. which responses are most appropriate/
effective).

Use the factsheet Scorecard for evaluating knowledge gaps toevaluate the degree to which project actions filled knowledge gaps.
This method uses a scoring system that can be conducted before
and after the action and for a hypothetical scenario in which the
project did not take place.

154

PRISM
MODULESMETHOD FACTSHEETS& ANNEXES

It is worth noting that projects may generate important knowledge
about one or more of the aspects listed above as an unintended
consequence of project actions (for example surveys to assess
the population size of one species may discover additional speciesof conservation importance not previously known to occur inan area). This can still be reported as an outcome/impact of the
project, and the level of knowledge for the aspect concerned canbe scored using the method above.
It is important to appreciate that projects often present
good opportunities for testing the effectiveness of particularinterventions, but these opportunities are frequently missed,
meaning that important information is not generated or sharedwith the wider conservation community. For example, a smallscale conservation project may provide nest boxes for a criticallyendangered bird species, but there is some uncertainty about
the most appropriate height at which the nest boxes should be
deployed. To test this, a project could simply place half of theboxes at 2 m height and the other half at 6 m, and then monitor
the proportion of each set of boxes that is occupied by the target
species (and ideally also the success rate of the nests). Reporting
the results of such a test (in the peer-reviewed literature, or at
http://www.conservationevidence.com/) would add to the body ofevidence about the effectiveness of nest boxes and their utility forthe particular species of concern, thus facilitating more effective
conservation in future.
See the Scorecard for evaluating knowledge gaps for an example.

PRISM - TOOLKIT FOR EVALUATING THE OUTCOMES AND IMPACTS OF SMALL/MEDIUM-SIZED CONSERVATION PROJECTS

3:5

INTRODUCTION &KEY CONCEPTS

Outcome 2.

SPECIES & HABITATMANAGEMENTOUTCOME 2.
SPECIES/
SITE/HABITATACTION PLAN(S)DEVELOPEDDESIGNING & IMPLEMENTING
THE EVALUATION

Species/site/habitat action plan(s) developed

This section addresses projects working towards the developmentof species action plans or site management plans. It presents
simple methods for scoring the degree of completion of a plan
(on a 1-10 scale) and the adequacy of a plan (on a 1-9 scale). Thechange in these scores during the course of a project is compared
to a hypothetical scenario in which the project did not take place, inorder to estimate the outcome and impact of project actions.

PRISM
MODULES

9
Action plans typically compile information on the status of the species or habitat (historic andcurrent distribution, population size/abundance, habitat extent/condition), the threats thatare relevant, and the conservation actions required, locations for these, the implementers
involved and the timeframe for completion. For more information see:•

IUCN’s Strategic Planning for Species Conservation: A Handbook: http://cmsdata.
iucn.org/downloads/scshandbook_2_12_08_compressed.pdf,•

BirdLife International’s Species Action Plan Development Manual: http://www.
birdlife.org/sites/default/files/species_action_plan_manual.pdf•

TNC’s Conservation Action Planning (CAP) Handbook: https://www.
conservationgateway.org/Documents/Cap%20Handbook_June2007.pdf.

A key outcome in conserving species and habitats is often thecompilation of an Action Plan. Terms used for such plans include
Species Action Plan, Species Recovery Plan, Site Management
Plan, Habitat Recovery Plan, and Habitat Conservation Plan. Theseare essentially blueprints for conserving one or more populations,species or habitats, and the process of compiling them can beeffective in bringing all stakeholders to agree and prioritise actions
required9.
Small and medium-sized conservation projects may have as an
objective the compilation of such an action plan (and often theimplementation of at least part of it), or they may aim to contributeto the compilation of a plan which is being developed by others.
Such projects can evaluate their outcomes and impacts according
to the degree of completion of the action plan and/or the adequacyof the action plan. The factsheets below give guidance andexamples for evaluating the impact of project actions on both ofthese aspects.
Factsheet: Scorecard for degree of completion of an action plan
Factsheet: Scorecard for adequacy of an action plan

155

METHOD FACTSHEETS& ANNEXESPRISM - TOOLKIT FOR EVALUATING THE OUTCOMES AND IMPACTS OF SMALL/MEDIUM-SIZED CONSERVATION PROJECTS

3:5

INTRODUCTION &KEY CONCEPTS

Outcome 3.

DESIGNING & IMPLEMENTING
THE EVALUATION

Threats to species/habitat reducedThis section describes a simple method for assessing changes in
the impacts of individual threats to one or more species or habitats
brought about by project actions.

SPECIES & HABITATMANAGEMENTOUTCOME 3.
THREATS TO
SPECIES/HABITATREDUCED

Many conservation projects attempt to reduce the threats
impacting on a biodiversity target, for example by establishing or
enhancing anti-poaching patrols, guarding against illegal habitat
destruction, eradicating or managing invasive alien species, etc.
Guidelines exist from various sources for implementing detailed
monitoring of particular threats. Approaches commonly used ininternational conservation such as SMART or HCV10 are usually
too demanding on time and resources for smaller projects, instead,
we recommend using an evaluation design that involves scoring of
the type and magnitude of threats affecting the species or habitat.
This is based on the “threats” component of BirdLife International’s
Important Bird and Biodiversity Area site monitoring protocol11.

PRISM
MODULESThe Spatial Monitoring and Reporting Tool (SMART) is a data collection tool for protectedarea rangers and provides a systematic approach for documenting incidents of threats atsites: http://www.smartconservationtools.org/
10

The HCV Threat Monitoring Protocol is designed to standardise the monitoring ofanthropogenic threats to High Conservation Value areas within oil palm landscapes, andcontains detailed advice on how establish and run patrol teams to detect and monitor
different threats.
11(BirdLife International 2008): http://datazone.birdlife.org/userfiles/file/IBAs/
MonitoringPDFs/IBA_Monitoring_Framework.pdf .

Use the factsheet Threat reduction scoring to evaluate changes
in the magnitude of threats to species or habitats. The change
in the level of threats during the course of a project is compared
to a hypothetical scenario in which the project did not takeplace, in order to estimate the outcome and impact of project
actions. Justifications for scores selected should be provided, andquantitative data on threat scope, severity (intensity) and impact
should be used if available.

156

METHOD FACTSHEETS& ANNEXESPRISM - TOOLKIT FOR EVALUATING THE OUTCOMES AND IMPACTS OF SMALL/MEDIUM-SIZED CONSERVATION PROJECTS

3:5

INTRODUCTION &KEY CONCEPTS

Outcome 4.

SPECIES & HABITATMANAGEMENTOUTCOME 4.
RECOVERY OF
SPECIES/HABITATSPROMOTED157

DESIGNING & IMPLEMENTING
THE EVALUATION

Recovery of species/habitats promoted

Conservation actions may aim to promote the recovery of species(for example by supplementary feeding or manual pollination)or habitats (for example by habitat restoration or creation), or
both (for example by planting native seedlings). This section gives
examples of suitable indicators for monitoring different types ofproject actions, but these should be tailored as appropriate for any
particular project action. Guidance under outcomes 6 and 7 can beused for species and habitats respectively to identify appropriate
methods for monitoring these indicators at project location(s) andat comparable location(s) in order to evaluate the outcomes andimpacts of project actions.
Conservation project actions may attempt to promote therecovery of species or habitats directly in a number of ways (rather
than via efforts to reduce threats, which is the subject of section 3).
These can be broadly categorised into six groups (Table 2):•

Directly increasing reproductive success;•Directly increasing survival;•

Establishing or strengthening ex-situ conservation
programmes;•

Attempting reintroduction, benign introduction, ortranslocation;•

Increasing genetic diversity;•

Accelerating habitat recovery (e.g. through seedlingplanting) to create or restore habitat.

PRISM
MODULESMETHOD FACTSHEETS& ANNEXES

Such actions aim to remove or overcome barriers to species/
habitat recovery, such as limits on breeding sites, food supplies,
available habitat for species, or unsuitable conditions for habitat
recovery. All may be relevant to projects aimed at the conservationof particular species, while the last may also be relevant to projects
aimed at conserving particular habitats.
It is important to appreciate the distinction between removing
barriers to recovery (the focus of this section) and reducing
threats (section 3). In some cases, there may be long time-lags
between removing or reducing a threat and the species or habitat
recovering. Separate actions to promote recovery may reducethese time-lags. In other cases, it may not be feasible to remove
or reduce threats sufficiently, and promoting recovery may be aneffective short-term strategy.

PRISM - TOOLKIT FOR EVALUATING THE OUTCOMES AND IMPACTS OF SMALL/MEDIUM-SIZED CONSERVATION PROJECTS

3:5

INTRODUCTION &KEY CONCEPTS

DESIGNING & IMPLEMENTING
THE EVALUATION

PRISMMODULESMETHOD FACTSHEETS& ANNEXESTABLE 2: POSSIBLE INDICATORS AND METHODS FOR MEASURING PROJECT ACTIONS AIMED AT REMOVING OR
OVERCOMING BARRIERS TO SPECIES/HABITAT RECOVERY

SPECIES & HABITATMANAGEMENTPOSSIBLE
INDICATORSAND METHODSFOR MEASURING
PROJECT ACTIONS

158

BIODIVERSITYTARGET

AIM OF PROJECTACTIONS

EXAMPLES OF PROJECT ACTIONSEXAMPLE INDICATORS TO MONITORMETHODS

Species

Increase
reproductive
success

Nest box provision, clutch
manipulation, protection frompredators, supplementary feeding ofbreeders, manual pollination

Number of occupied nest boxes, clutch/
brood/litter survival, number of viable seeds,
or indirect measures such as changes in
population abundance

Field surveys for evaluatingchanges in species status

Species

Increase survival

Supplementary feeding, disease/pathogen/parasite management(e.g. vaccination), seedling aftercare(watering, weeding)

Adult mortality, longevity (for short-lived
species), survival rates or indirect measures
such as changes in abundance

Field surveys for evaluatingchanges in species status

SpeciesEstablish/
strengthen ex-situ
conservation

Removal of individuals (including seeds)
from the wild for captive breedingor artificial propagation, increasing
productivity of captive breeding or
artificial propagation

Number of (mature) individuals in ex-situ
conservation, programmes, number of
seedlings, breeding success/ productivity of
captive population

Collate data from all
relevant ex-situ programme
institutions/ agencies/
locations

Species

Reintroduction,
benign introduction,translocation

Reinforcement of wild plant & animal
populations, assisted movement of
individuals between locations or
subpopulations

Population estimates, number of locations
occupied, area of habitat occupied, number of
sub-populations, extent of distribution

Field surveys for evaluatingchanges in species status

Species

Increase genetic
diversity within
population

Sourcing genes from multiple
populations, collecting pollen frommultiple trees, connecting habitat
fragments

Number of new breeding stock introduced

Collate data on the originof introduced individuals/
breeding stock. See also
Field surveys for evaluatingchanges in species status

Species/
habitats

Habitat restoration,
habitat creation

Planting seedlings, removal ofcompetitors, management of
succession, water management (e.g.
flooding, re-flooding)Area of habitat restored/created, abundanceof target or indicator species

Field surveys for evaluatingchanges in species status andField assessment methods
for evaluating changes inhabitat status

PRISM - TOOLKIT FOR EVALUATING THE OUTCOMES AND IMPACTS OF SMALL/MEDIUM-SIZED CONSERVATION PROJECTS

3:5

SPECIES & HABITATMANAGEMENTPOSSIBLE
INDICATORSAND METHODSFOR MEASURING
PROJECT ACTIONSINTRODUCTION &KEY CONCEPTS

DESIGNING & IMPLEMENTING
THE EVALUATION

Determining the outcome and impact of project actions aimed
at removing or overcoming barriers to species/habitat recovery
requires an assessment of the relevant indicators at the beginningand end of a project. For some parameters, such as numbers ofindividuals in ex-situ conservation programmes, or numbers oflocations occupied, it may be possible to assess retrospectively
their value at the beginning of the project period. For others, suchas population abundance or adult survival rate, this may not be
possible, and establishing a baseline at the beginning of a projectis critical. Choosing which indicator to measure will depend on thespecies or habitat that is targeted, the size of the site, the resources
available, the project duration, and the likely magnitude of change
resulting from project activities relative to background variation.

PRISM
MODULESMETHOD FACTSHEETS& ANNEXES

EXAMPLEA project aimed to promote recovery of a threatened species ofcavity-nesting songbird by providing nest boxes. A total of 50
wooden nest boxes were erected at a height of 3–4 m on trees
within ten test plots (five boxes per plot).
Test plots and ten similar control plots (which were notprovided with nest boxes) were subsequently visited four times
during the breeding season. The proportion of boxes occupied
and the population density of the target species within eachplot were measured.
The team found that 32% of boxes were used by the target
species for nesting. A further 28% had nesting material inthem, but no active nest. The breeding population densities of
the target species were significantly higher in the test plots,
compared to the ten control plots (4.2 songbirds/ha vs. 1.4/hafor control plots).
The team interpreted results as showing that the introductionof nest boxes had a potential positive outcome on breeding
population densities.

159

PRISM - TOOLKIT FOR EVALUATING THE OUTCOMES AND IMPACTS OF SMALL/MEDIUM-SIZED CONSERVATION PROJECTS

3:5

INTRODUCTION &KEY CONCEPTS

Outcome 5.

SPECIES & HABITATMANAGEMENTOUTCOME 5.
SPECIES STATUS
IMPROVEDDESIGNING & IMPLEMENTING
THE EVALUATION

Species status improved

This section provides a simple scorecard approach using theinformed opinion of certain individuals (key informants) toassess the impact of project actions on species status. Where
project resources allow for more detailed and robust evaluation,
we summarise the most relevant field methods for monitoring
different groups of species, and give guidance on sampling
approaches.
The ultimate aim of most conservation projects is to improve thestatus of one or more species or habitats. However, attempting
to detect and/or quantify such impacts will not be appropriate
for many small- and medium-sized projects because of time-lags,
the resources required to measure such impacts directly, and thechallenge of attributing changes to project actions.
A relatively low-cost measure is to use the Factsheet: Scorecard
for evaluating changes in species status, either based on datacollected during the project or used as part of an interviewing key
informants evaluation design, to estimate the impact of the projecton species status.
To help inform such scores, or to evaluate changes in species status
more directly, robust and repeatable field surveys are typicallyrequired. See the Factsheet: Field surveys methods for evaluatingchanges in species status for more information.
Deciding on an appropriate field survey method will depend on
the taxonomic group being surveyed, for example surveying birds
requires a very different approach compared with monitoring
aquatic insects. The Annex: Field methods for species monitoring

160

PRISM
MODULESMETHOD FACTSHEETS& ANNEXES

summarises the approaches that are most commonly used for
different taxonomic groups, and provides links to further sourcesof information12.
As well as the method by which individuals are detected andcounted, it is also important to consider sampling strategies.
Although it may be possible to undertake a complete survey inwhich all individuals in the area of interest are counted, changes
in population size or range extent will typically be estimated fromsample surveys in which only a proportion of the population isrecorded. The Annex: Sampling approaches for evaluating changes
in species status provides further details on both full and sample
surveys, and lists sources of more detailed methods.
Species status may also be assessed in terms of extinction risk
using the IUCN Red List Categories and Criteria, which involves
applying data on parameters such as population and range size andtrend to a set of criteria with quantitative thresholds. As the Red
List categories are relatively broad measures of extinction risk,
actions implemented by small- and medium-sized conservation
projects will rarely lead to a change in the status of a target species
that is of sufficient magnitude to cross the thresholds for a lower
Red List category, so this approach is less useful for measuringoutcomes and impacts. Population Viability Analysis is analternative and more quantitative approach, but typically requires
input data that are too detailed to be feasible to mobilise in a smallto medium-sized project.

12
More detailed information on the methods outlined here can be found in the Cambridge
Handbook of Biodiversity Methods: https://sunsetridgemsbiology.wikispaces.com/file/view/
Biodiversity+Handbook.pdf .

PRISM - TOOLKIT FOR EVALUATING THE OUTCOMES AND IMPACTS OF SMALL/MEDIUM-SIZED CONSERVATION PROJECTS

3:5

INTRODUCTION &KEY CONCEPTS

Outcome 6.

SPECIES & HABITATMANAGEMENTOUTCOME 6.
HABITAT STATUS
IMPROVEDDESIGNING & IMPLEMENTING
THE EVALUATION

Habitat status improved

This section provides a simple scorecard approach using theinformed opinion of certain individuals (key informants) to assess
the impact of project actions on habitat status. Where resources
allow more detailed and robust evaluation, we provide guidanceon how to measure the impact of project actions on habitat statususing remote sensing (e.g. for habitat extent and fragmentation)and field methods (e.g. for habitat condition via indicator species orvegetation structure).
Note that measuring changes in habitat status and attributingchanges to project actions is challenging, and attempting todetect and/or quantify such impacts will not be appropriate formany small- and medium-sized projects because of time-lags,
the resources required to measure such impacts directly, and thechallenge of attributing changes to project actions.

PRISM
MODULESMETHOD FACTSHEETS& ANNEXESThe Factsheet: Field assessment methods for evaluating changes
in habitat status involves gathering relatively simple data fromthe field on stressors (e.g. roads, pollutants) or condition (e.g.
connectivity, indicator species abundance, vegetation structure)13.
This section is most relevant to project actions that attempt toimprove habitat extent, condition and connectivity at a local
scale. Project actions aimed at improving habitat status at theregional scale tend to involve policy interventions (such as theestablishment of protected areas), for which the guidance in the
PRISM Policy module is relevant; or they tend to be beyond thesmall- and medium-scale project budgets and durations consideredhere.

A relatively low-cost measure is to use the Factsheet: Scorecard
for evaluating changes in habitat status, either based on datacollected during the project or used as part of an interviewing key
informants evaluation design to estimate the impact of the projecton habitat status.
To underpin such scores, or to evaluate habitat status more
directly, two approaches can be used:
The Factsheet: Remote assessment methods for evaluatingchanges in habitat status uses remote sensing data (images
obtained through satellite imagery or aerial photography), typicallyin the form of maps of land cover or land use.

161

More detailed information on the methods outlined here can be found in the Cambridge
Handbook of Biodiversity Methods: https://sunsetridgemsbiology.wikispaces.com/file/view/
Biodiversity+Handbook.pdf .
13

PRISM - TOOLKIT FOR EVALUATING THE OUTCOMES AND IMPACTS OF SMALL/MEDIUM-SIZED CONSERVATION PROJECTS

3:5

INTRODUCTION &KEY CONCEPTS

DESIGNING & IMPLEMENTING
THE EVALUATION

PRISMMODULES

Further Reading
A number of publications cover these issues in more detail. In
particular, we recommend the following.

SPECIES & HABITATMANAGEMENTFURTHER READING

Cambridge Handbook of Biodiversity Survey Methods - Survey,
Evaluation & Monitoring: provides an overview of methods for
all taxa. . https://sunsetridgemsbiology.wikispaces.com/file/view/
Biodiversity+Handbook.pdf
Sutherland WJ (2006). Ecological census techniques, 2nd Edition.
Cambridge University Press, Cambridge.
Bibby CJ, Burgess ND, Hill DA and Mustoe SH (2000). Bird Census
Techniques, 2nd Edition. Academic Press, London.
Sutherland W, Newton I and Green RE (eds) (2004). Bird Ecology
and Conservation: A Handbook of Techniques. Oxford University
Press, Oxford.
Chartered Institute of Ecology and Environmental Management:
Sources of survey methods. http://www.cieem.net/sources-ofsurvey-methods-sosm- .
Gill D and Daltry J (2014). Brief 3: How to make a monitoringplan for threatened tree species. Global Trees Campaign http://
globaltrees.org/wp-content/uploads/2015/03/GTC-Brief-3monitoring-plan-lo-res.pdf .

162

PRISM - TOOLKIT FOR EVALUATING THE OUTCOMES AND IMPACTS OF SMALL/MEDIUM-SIZED CONSERVATION PROJECTSMETHOD FACTSHEETS& ANNEXES

4:0

INTRODUCTION &KEY CONCEPTS

DESIGNING & IMPLEMENTING
THE EVALUATION

PRISMMODULESSECTION 4
METHOD FACTSHEETSAND ANNEXES163

PRISM - TOOLKIT FOR EVALUATING THE OUTCOMES &ANDIMPACTS
IMPACTSOF OF
SMALL/MEDIUM-SIZEDSMALL/MEDIUM-SIZEDCONSERVATIONCONSERVATIONPROJECTSPROJECTSMETHOD FACTSHEETS& ANNEXES

4:0

INTRODUCTION &KEY CONCEPTS

DESIGNING & IMPLEMENTING
THE EVALUATION

PRISMMODULESMETHOD FACTSHEETS& ANNEXES

General Method Factsheets

METHOD
FACTSHEETSAND ANNEXES

General Method Factsheets
Completing a Theory of Change
Identifying why you want to evaluate
Evaluation feasibility checklist
Completing a prioritisation matrix
Developing ethical protocols for an evaluation
Developing gender protocols for an evaluation

Evaluating outcomes and impacts of projects aiming tomaintain or increase the values of ecosystem services
Questionnaires
Key informant interviews
Focus groups
Direct observation
Documentary evidenceEntering, organising and cleaning data
Analysing Quantitative data
Analysing qualitative data
Evaluation report template

164

PRISM - TOOLKIT FOR EVALUATING THE OUTCOMES AND IMPACTS OF SMALL/MEDIUM-SIZED CONSERVATION PROJECTS

4:0

INTRODUCTION &KEY CONCEPTS

DESIGNING & IMPLEMENTING
THE EVALUATION

Factsheet: Completing a Theory of Change

GENERAL
FACTSHEETSCOMPLETING
A THEORY OF
CHANGE

Summary:This factsheet outlines the steps required to construct a simple“Theory of Change” for a conservation action or project. A
Theory of Change is an articulation of the change you expect yourproject to achieve within the context you are working, showing
each logical step along the way. A well thought-out Theory of
Change provides a framework that will enable you to map out
how the project will bring about change, identify the steps wherean evaluation can and should take place and to develop the key
questions the evaluation will address.
If your project is already underway and you did not developthese aspects at the outset you can use a Theory of Change (orequivalent) retrospectively or mid-way through the project. While
completing a full, detailed Theory of Change can be quite a large
undertaking involving consultation with multiple stakeholders,for the purposes of evaluating a small- medium sized project it isusually enough to take a simplified approach.

165

PRISM
MODULESMETHOD FACTSHEETS& ANNEXES

Introduction
Theory of Change is a tool for:•well considered project planning, underpinned byrobust and credible project design;•

ongoing project monitoring, providing a framework for
effective evaluation;•communicating and fundraising for your projects.

What does a Theory of Change do?
•
Identifies the long-term change you want to achieve andworks backwards to show how you will get there.
•

Asks how and why change will happen, helping to clarify
why you are doing what you plan to do.

•

Looks at the larger context within which your projectwill operate.

•

Articulates the individual logic steps between project
elements, clearly showing cause and effect.

•

Clarifies known assumptions necessary for success.

PRISM - TOOLKIT FOR EVALUATING THE OUTCOMES AND IMPACTS OF SMALL/MEDIUM-SIZED CONSERVATION PROJECTS

4:0

INTRODUCTION &KEY CONCEPTS

GENERAL
FACTSHEETSCOMPLETING
A THEORY OF
CHANGEDESIGNING & IMPLEMENTING
THE EVALUATION

When to develop a Theory of Change
Spending time on a Theory of Change when designing a project
helps to ensure that the project has a clear logic behind it. If one
is not developed when a project is planned, it can still be done ata later stage; and it provides a useful framework for an interim
review, re-design or evaluation of project progress and impact.

What does a Theory of Change look like?
A Theory of Change can look however you would like it to look,depending on what is most useful for your project team. Most
Theories of Change (including the examples given in PRISM) areflow diagrams. These can be single strands or complex webs,
depending on the size and complexity of the project and the levelof detail you have chosen to present.
There are several methods for thinking through a Theory of
Change process. Some of the terms you may know are situation
analysis, problem trees and objective trees. All of these end up
with a flow diagram type of output. Focusing on the changes you
are aiming for or have achieved, as opposed to the problems you
are trying to tackle (such as in a situation analysis or problem tree),is what distinguishes the Theory of Change from other projectplanning approaches.

Definitions of project stages used in PRISM
Actions – What the project does to bring about change. For
example conducting biodiversity surveys, training workshops,
education/outreach campaigns.

166

PRISM
MODULESMETHOD FACTSHEETS& ANNEXES

Outputs – What is produced by the project’s actions. These areusually measured in terms of quantity and quality of delivery, forexample the number of reports produced from survey data or thenumber of individuals trained.
Outcomes – The changes brought about by the project’s actions.
For a conservation project these can be further divided into:•

intermediate outcomes which show progress towards
subsequent outcomes/impacts; and•threat reduction outcomes which represent the final
change(s) that need to happen in order to impact on aconservation target.

Impacts – The long-term, lasting change(s) brought about by the
project’s actions. For conservation projects these typically relate
to changes in species and/or habitat status.
Assumptions – The links between each of the stages above areunderpinned by assumptions. For example a project focused on
training park rangers to carry out anti-poaching patrols makes
the assumption that training park guards will result in more
effective patrols. A project will also make assumptions about
external factors that the project does not have control over, butthat may influence the project’s results if they do not remain true.
For example, in order for training to be successful, park rangers
need to have sufficient resources to carry out their work (e.g.
equipment, salary). These kind of assumptions are particularly
important for a small/medium-sized project to consider, as there
will often be a large number of factors that could influence theproject’s outcomes/impacts and they will form an integral part ofthe story you will use to explain and contextualise your results.

PRISM - TOOLKIT FOR EVALUATING THE OUTCOMES AND IMPACTS OF SMALL/MEDIUM-SIZED CONSERVATION PROJECTS

4:0

INTRODUCTION &KEY CONCEPTS

DESIGNING & IMPLEMENTING
THE EVALUATION

PRISMMODULESMETHOD FACTSHEETS& ANNEXES

FIGURE 1: DEFINITION OF PROJECT STAGES USED IN
PRISM

GENERAL
FACTSHEETSCOMPLETING
A THEORY OF
CHANGE

ActionOutputs

IntermediateoutcomeThreat reduction
outcome

Impact on
biodoversity

ExamplesExamplesExamplesExamplesExamplesBiodiversity surveys
Training workshops
Education/Outreachcampaigns
Advocating forchanges in policy# of reports produced
from survey data

Information in reports
used to inform decision making

Decrease in poaching due toeducation campaign

Species population
increases# number of individuals
trainedIndividuals trained apply
new skills to conservation
targetPredation of native species
eliminated after eradication ofinvasive predators

Habitat status of targetedsites improves

Assumptions
ExamplesPolitical events do not affect fieldwork
Weather conditions remain constant
Local commodity prices remain stable

167

PRISM - TOOLKIT FOR EVALUATING THE OUTCOMES AND IMPACTS OF SMALL/MEDIUM-SIZED CONSERVATION PROJECTS

4:0

INTRODUCTION &KEY CONCEPTS

DESIGNING & IMPLEMENTING
THE EVALUATION

PRISMMODULESMETHOD FACTSHEETS& ANNEXES

Required steps
Step 1.

GENERAL
FACTSHEETSCOMPLETING
A THEORY OF
CHANGE

Identify the long term change the project istrying to bring about.

EXAMPLEA project aims to reverse
a decline in a population ofStart by thinking about the problem that the project is trying
to address, then identify the long-term change your projectwill contribute towards to address this problem. Usually in aconservation project the intention will be to have an impact on thestatus of a particular species and/or habitat.

chimpanzees, therefore theultimate impact the projectis trying to bring about is anincrease in the number ofchimpanzees.

STEP 1168

PRISM - TOOLKIT FOR EVALUATING THE OUTCOMES AND IMPACTS OF SMALL/MEDIUM-SIZED CONSERVATION PROJECTS

Chimpanzee
population
increases

4:0

INTRODUCTION &KEY CONCEPTS

Step 2.

GENERAL
FACTSHEETSCOMPLETING
A THEORY OF
CHANGEDESIGNING & IMPLEMENTING
THE EVALUATION

PRISMMODULESMETHOD FACTSHEETS& ANNEXES

Identify and connect outcomes.

Work backwards to identify the outcomes that need to happen
during your project in order to achieve the desired impact. In a
conservation project a good place to start is to look at your desired
impact (see above) and then use the IUCN threat classifications(http://www.iucnredlist.org/technical-documents/classificationschemes/threats-classification-scheme) to work out which threatsneed to be addressed in order for the impact to be achieved.

Once the primary threat(s) have been identified you can thencontinue to work backwards to identify the intermediate outcomesthat need to be achieved in order to reduce/mitigate these threats.
Another approach to this is to re-write the project’s objectives(as outlined in the project proposal) to turn them into outcome
statements (statements that reflect the change they are trying tobring about).

EXAMPLE

STEP 2The project identified several
factors that could potentiallyaffect the target population,
however in this case theprimary threat that needed
to be addressed in order forthe chimpanzee population

Park guards
conduct more
effective patrolsMoreeffective patrols
deter hunters

to increase was poaching.
Working backwards, the team
identified the intermediateoutcomes that needed tohappen in order to bring about
a decrease in poaching.

169

PRISM - TOOLKIT FOR EVALUATING THE OUTCOMES AND IMPACTS OF SMALL/MEDIUM-SIZED CONSERVATION PROJECTSPoaching
decreases

Chimpanzee
population
increases

4:0

INTRODUCTION &KEY CONCEPTS

Step 3.

GENERAL
FACTSHEETSCOMPLETING
A THEORY OF
CHANGEDESIGNING & IMPLEMENTING
THE EVALUATION

PRISMMODULESMETHOD FACTSHEETS& ANNEXES

Identify outputs & actions.

Based on the outcomes and impacts you have identified, the next
step is to identify the action (or group of actions) that you will carryout in order to make these outcomes and impacts happen.

outcomes and impacts. Two particularly useful resources areConservation Evidence and the CMP/IUCN conservation actionsclassification.

If you are at the development stage of the project there are several
resources you can use to help to identify the most appropriate
actions and approaches for achieving particular conservation

If your project is already underway or completed you should insert
the action(s) you have carried out during the project.

EXAMPLE

STEP 3

Based on the outcomes defined, the project team identified
training to park guards as a crucial action that the project could
undertake in order to bring about the necessary outcomes/
impacts.

Training park
guards to conduct
ranger patrols

170

Training
provided to park
guardsPark guards
conduct more
effective patrolsMoreeffective patrols
deter hunters

PRISM - TOOLKIT FOR EVALUATING THE OUTCOMES AND IMPACTS OF SMALL/MEDIUM-SIZED CONSERVATION PROJECTSPoaching
decreases

Chimpanzee
population
increases

4:0

INTRODUCTION &KEY CONCEPTS

Step 4.

GENERAL
FACTSHEETSCOMPLETING
A THEORY OF
CHANGE
STEP 4

171

DESIGNING & IMPLEMENTING
THE EVALUATION

PRISMMODULES

Test the logic.

Test the logic linking the project stages from actions – outputs– outcomes – impacts. At each stage, ask: why will this change
lead to the desired change at the end of the project, or this action
lead to that change? i.e. “If X happens, does Y happen? Is anything
missing?” This process also helps to start discussions about
assumptions (step 5).
Many projects involve multiple actions and results chains whichshould be linked together to show how they interact, and how
they contribute to the achievement of the project’s outcomes andimpacts. Refer to the end of this factsheet for some examples ofTheories of Change for projects involving multiple actions.

PRISM - TOOLKIT FOR EVALUATING THE OUTCOMES AND IMPACTS OF SMALL/MEDIUM-SIZED CONSERVATION PROJECTSMETHOD FACTSHEETS& ANNEXES

4:0

INTRODUCTION &KEY CONCEPTS

Step 5.

GENERAL
FACTSHEETSCOMPLETING
A THEORY OF
CHANGE
STEP 5

DESIGNING & IMPLEMENTING
THE EVALUATION

PRISMMODULESMETHOD FACTSHEETS& ANNEXES

Identify assumptions.

Assess whether any assumptions are being made about things
that have to be in place for you to achieve the desired outcomes
and impacts, but which you will not work on. For example a criticalassumption for a livelihoods project involving the growing of crops

might be that weather conditions remain stable enough forcrops to grow. This is not something that the project will address,
but the project may need to be adjusted if this assumption is nolonger true.

EXAMPLEASSUMPTIONS

The team identified (a) the resources and access required by park•

Park Guards have adequate resources to carry out their work

guards, and (b) the absence of other factors that might lead to a•

Park Guards are able to access areas needed to patrol

decline in the chimpanzee population, as the assumptions which•

need to remain in place in order for the project’s actions to bring

Hunted species populations do not decrease due to disease orother natural factors

about the necessary change.

Training park
guards to conduct
ranger patrols

172

Training
provided to park
guardsPark guards
conduct more
effective patrolsMoreeffective patrols
deter hunters

PRISM - TOOLKIT FOR EVALUATING THE OUTCOMES AND IMPACTS OF SMALL/MEDIUM-SIZED CONSERVATION PROJECTSPoaching
decreases

Chimpanzee
population
increases

4:0

INTRODUCTION &KEY CONCEPTS

Step 6.

GENERAL
FACTSHEETSCOMPLETING
A THEORY OF
CHANGEDESIGNING & IMPLEMENTING
THE EVALUATION

Review Theory of Change with project team andkey stakeholders.

Show the Theory of Change to the project’s key stakeholders,invite feedback on the following questions and make any necessarychanges:•

Does the logic make sense (i.e. will one stage lead toanother)?

•Have any critical assumptions been missed?

PRISM
MODULESMETHOD FACTSHEETS& ANNEXES

Further reading
Please note that this factsheet outlines how to construct a very
simple Theory of Change to help guide and focus the developmentof an evaluation. For further reading on how to conduct a
more detailed Theory of Change you can refer to the following
resources:
Conservation International: Constructing Theories of Change forecosystem based adaptation projects http://www.conservation.
org/publications/Documents/CI_IKI-ToC-Guidance-Document.pdf
Margoulis et al. (2013). Results Chains: a tool for conservation
action design, management and evaluation: http://www.
ecologyandsociety.org/vol18/iss3/art22/ .

STEPS 6 & 7
Step 7.

Use the Theory of Change to design and plan
your evaluation.

USAID TIPS factsheet: Building a results framework. http://pdf.
usaid.gov/pdf_docs/Pnadw113.pdf .
Center for Theory of Change: http://www.theoryofchange.org/

Use your Theory of Change to identify and prioritise what you
will measure at each stage of your project to determine progress
towards your desired impact and outcomes, and to assess theperformance of your actions. Refer to PRISM step 1 for more
information on how to do this.

what-is-theory-of-change/ .
http://www.evaluationinnovation.org/sites/default/files/
Pathways%20for%20Change.pdf - a really useful paper outlining
ten models or types of Theories of Change which could apply toand be adapted for various projects/programmes.
The Evaluation Innovation site http://www.evaluationinnovation.
org/ is also very useful in itself.

173

PRISM - TOOLKIT FOR EVALUATING THE OUTCOMES AND IMPACTS OF SMALL/MEDIUM-SIZED CONSERVATION PROJECTS

4:0

INTRODUCTION &KEY CONCEPTS

DESIGNING & IMPLEMENTING
THE EVALUATION

PRISMMODULESMETHOD FACTSHEETS& ANNEXES

EXAMPLE THEORIES OF CHANGE FROM PROJECTSINVOLVING MULTIPLE ACTIONS:GENERAL
FACTSHEETSCOMPLETING
A THEORY OF
CHANGE
EXAMPLE
THEORIES OF
CHANGE FROMPROJECTSINVOLVING
MULTIPLE
ACTIONS

Informing
stakeholders
about sustainable
forestmanagement
practices

Stakeholders
are informedabout
sustainable
practices

Local people havean improved
understanding ofsustainable
practices

Local people
adopt moresustainable
practices

Training
stakeholders onsustainable
extractionof NTFP

Individuals
trainedTrained individuals
have new skills forsustainable
extraction

Trainedindividuals apply
new skills

Local people
manage forest
resources moresustainably
Forest loss
slowed/reversed

Provide direct
investment for
rattan enrichment

Households
receiveinvestment for
rattanenrichment

Forest protection
agreements in
exchange for NTFP
extraction rights

Agreements in
placeHouseholdlivelihood/
wellbeing
improves

Rattan provides
additional
income tohouseholds

Local people
have right toutilise forest
resources

Status of native forest
species (including
Edward’s Pheasant)improves

Local people areincentivised to
manage forest
more sustainably

Critical Assumptions
Price of NTFPs remain stable or increase
Weather conditions remain stable enoughfor rattan to grow
Populations of native forest species do notdecrease due to disease or other natural
factors

Actions174

Outputs

Intermediateoutcomes

Threat reduction
outcomes

Impact on
Biodiversity

PRISM - TOOLKIT FOR EVALUATING THE OUTCOMES AND IMPACTS OF SMALL/MEDIUM-SIZED CONSERVATION PROJECTS

4:0

INTRODUCTION &KEY CONCEPTS

DESIGNING & IMPLEMENTING
THE EVALUATION

PRISMMODULESMETHOD FACTSHEETS& ANNEXES

Critical Assumptions

Sustainability ofMbeliling Productive
Landscape reached dueto the improvement oflocal livelihood

Impact on thearea/habitat

Unsupported program/project coming into thearea will be controlled
by the cooperative,refers to the agreed
management plan ofMbeliling Landscape

GENERAL
FACTSHEETS
Threat reduction
Outcome(s)Opening new
farm landdecreased

COMPLETING
A THEORY OF
CHANGE
EXAMPLE
THEORIES OF
CHANGE FROMPROJECTSINVOLVING
MULTIPLE
ACTIONS

Cooperative become
representative
organisation inpromotingconservation area

Income ofcooperative
member / farmer
increased
Intermediate
Outcome(s)The price ofagriculture productespecially the candle
nut getting increase

Actions

Member ofcooperative
continuously
increaded

Cooperative apply
an accountable,transparent, credible
and professional
management

Joint marketingbusiness plan
operated

Regular shipmentof candle nut toJava carried out

Local partnershipdeveloped
Agreement withinter-island
buyer reachedSupport of
stakeholder increased
Total of member
increased
Total of capitalincreased

Capacity fororganizational
development increased
Capacity foradministration
management
increased

Number of internal
agreements
reachedCash injection
as capital support
for cooperative

Implementationof joint marketing

Series of meetingand study to develop
partnershipSocialization andpromotion ofcooperative andthe activities

Series oftraining andaccompaniment tobuild the capacity ofcooperative

Series ofmeeting withcooperative
memberOutputs

Actions

175

Collecting of forest
resources to support
income decreased

Outputs

Intermediateoutcomes

Threat reduction
outcomes

Impact on
Biodiversity

PRISM - TOOLKIT FOR EVALUATING THE OUTCOMES AND IMPACTS OF SMALL/MEDIUM-SIZED CONSERVATION PROJECTS

4:0

INTRODUCTION &KEY CONCEPTS

DESIGNING & IMPLEMENTING
THE EVALUATION

PRISMMODULES

Factsheet: Identifying why you want to evaluate

GENERAL
FACTSHEETSIDENTIFYING WHY
YOU WANT TOEVALUATE

176

Summary
There are many reasons to evaluate and many potential ways inwhich the results of an evaluation could be used. For example you
might want to demonstrate results externally, use results to learninternally or a combination of both. You may also want to clarify
whether you are most interested in what change has happened,or whether you are more interested in how and why change has
happened.
One of the most common mistakes in an impact evaluation is not
identifying why you want to evaluate and what the evaluation
results will be used for. This is critical because being clear about the
purpose and the stakeholders who will ultimately use the results
helps to determine the questions, evaluation design and methods
you will use to carry out the evaluation.

PRISM - TOOLKIT FOR EVALUATING THE OUTCOMES AND IMPACTS OF SMALL/MEDIUM-SIZED CONSERVATION PROJECTSMETHOD FACTSHEETS& ANNEXES

4:0

INTRODUCTION &KEY CONCEPTS

DESIGNING & IMPLEMENTING
THE EVALUATION

Step 2Required steps
Step 1

GENERAL
FACTSHEETSIDENTIFYING WHY
YOU WANT TOEVALUATE

METHOD FACTSHEETS& ANNEXES

Identify what the stakeholders want to know, andhow they will use the results of the evaluation

Identify evaluation stakeholders

As a project team, ask the following questions:
1. Who are the main evaluation stakeholders? - Typicallyyour evaluation stakeholders will involve the projectteam/implementing organisation, the project donor(s)s and other key stakeholders whose involvement is
crucial to the success of the project. However you
should also consider the following:•

Which groups/individuals are most likely to beaffected by the evaluation?

•

Which groups/individuals are most likely to makedecisions about the project?

•

Whose actions and/or decisions will be influenced
by their engagement with the evaluation process
and/or the evaluation findings?

STEPS 1 & 2

177

PRISM
MODULES2.

How can the main stakeholders of the evaluation be
involved?

3.

What challenges/barriers might prevent/restrict the
involvement of stakeholders?

4.

Who are the target audiences for the evaluation(i.e. those who are interested in knowing about theevaluation findings, but are not among the primaryevaluation users)?

Ask the following questions both as the project team and (ifpossible) to all other identified evaluation stakeholders:1.

How could the evaluation results contribute to theimprovement of the project?

2.

How could the evaluation results contribute to making
decisions about the project?

3.

What outcomes do you expect from the evaluation
process (what would success look like)?

4.

What could be done differently as a result ofanswering these questions?

PRISM - TOOLKIT FOR EVALUATING THE OUTCOMES AND IMPACTS OF SMALL/MEDIUM-SIZED CONSERVATION PROJECTS

4:0

INTRODUCTION &KEY CONCEPTS

DESIGNING & IMPLEMENTING
THE EVALUATION

PRISMMODULESSTEP 3 - COMPILE INFORMATION IN THE FOLLOWING TABLE:STAKEHOLDER GROUP

GENERAL
FACTSHEETSIDENTIFYING WHY
YOU WANT TOEVALUATE

WHAT DO THEY WANT TO KNOW?

HOW WILL THEY USE THE INFORMATION?

STAKEHOLDER GROUPWHAT DO THEY WANT TO KNOW?

HOW WILL THEY USE THE INFORMATION?

Project team/
implementing NGO

Was the action successful?

Modifying the project
Demonstrating results to donors

Project Team
DonorsOther key stakeholders(e.g. local authorities,
politicians, law
enforcement agencies,
local community)
Add others as necessary

STEP 3EXAMPLE

Why/how was the action successful?

178

DonorsWas the project a success?

External communications

Local authorities

What was the impact on local people?

Informing management/policy
decisions

Wider conservation
community

What lessons were learnt?

Distributing results
Using results to inform conservation
practice

PRISM - TOOLKIT FOR EVALUATING THE OUTCOMES AND IMPACTS OF SMALL/MEDIUM-SIZED CONSERVATION PROJECTSMETHOD FACTSHEETS& ANNEXES

4:0

INTRODUCTION &KEY CONCEPTS

DESIGNING & IMPLEMENTING
THE EVALUATIONKEY CONSIDERATION

Available time:

EVALUATIONFEASIBILITYCHECKLIST
Instructions
Use the following criteria toassess whether a particularevaluation question is
feasible to answer.
If you answer “no” toany of the following key
considerations, these willneed to be addressed
before the evaluation cango ahead.

179

Availability of data:

METHOD FACTSHEETS& ANNEXESDETAILS

If the outcome/impact of interest is not measurable within the time available you will needto look at your Theory of Change (or equivalent) and identify intermediate outcomes thatcan be used to provide a reliable measurement of future impact.

If necessary data are not already available then you need to be able to collect these data over the course of the project.

Are baseline data available or obtainable?

If baseline data are not yet available, are there specific plans for when baseline data would
be collected and/or for the use of an evaluation design that does not require baseline data?

Are existing documents available?

For example previous reports, national statistics data, monitoring data. Can authors(individuals or organisations) be contacted if more information or clarification is needed?

Are any current monitoring data available?

What information is already being captured as part of project/institutional monitoring
frameworks, and will this be sufficient for carrying out the evaluation?

Available resources:

YES/NO

Many long-term outcomes (such as changes in biodiversity status) can take a long time for results to show, so
you need to ensure that the outcome you are evaluating will be measurable within the time available.

Is the outcome/impact measurable within
the time available?

GENERAL
FACTSHEETSPRISMMODULES

Ensure that your plans are realistic in light of the time, skills and resources available to the project.

Does the team have the necessary skills tocarry out the evaluation?

Is additional capacity/training needed? Are there any potential conflicts of interest?

Is there sufficient funding to carry out theevaluation?

What resources are needed, and is there enough funding to cover these?

Will the team have enough time to collectthe necessary data?

For example have time considerations relating to other responsibilities and local contexts
been factored into the project plan?

Are roles and responsibilities clear within
the team?

Is it clear who will be responsible for carrying out each part of the evaluation?

Are there any major external events that
could influence the plan?

For example elections or seasonal changes during the planned evaluation that may affectits feasibility.

Are there any ethical concerns?

Are there any ethical concerns that may affect the feasibility of the evaluation?

Have potential risks been identified?

Have major risks to the evaluation been identified and discussed (e.g. physical security
risks, weather constraints)?

Will key stakeholders be available toparticipate?

Will key stakeholders be present at the required times, taking into account e.g. farming
seasons, leave, etc?

PRISM - TOOLKIT FOR EVALUATING THE OUTCOMES AND IMPACTS OF SMALL/MEDIUM-SIZED CONSERVATION PROJECTS

4:0

INTRODUCTION &KEY CONCEPTS

DESIGNING & IMPLEMENTING
THE EVALUATION

Factsheet: Completing a prioritisation matrix

GENERAL
FACTSHEETSCOMPLETING A
PRIORITISATION
MATRIX

SummaryThis factsheet outlines a simple method which provides a template
for the organisation, prioritisation and selection of evaluationquestions. A prioritisation matrix allows you to organise theproject’s outcomes & impacts, develop questions relating to eachoutcome and apply a set of criteria to guide you in prioritising thequestions to address in the evaluation.

PRISM
MODULESMETHOD FACTSHEETS& ANNEXES

Introduction
The guidance in this factsheet is meant to complement the
information given in PRISM Step 1 -– What do you need to know?.
The criteria applied in the matrix shown on the next page below
are as follows:•

Importance to project users;•

Importance to project success;•How well known is the action being evaluated?

•

Time required (for outcome/impact to become feasible
to measure);•

Ease of data collection;•

Resources required.

Ideally the matrix should be completed by the project team, thenreviewed in conjunction with the project’s stakeholders andrevised if necessary to ensure that all are happy with the questions
the evaluation will address.

180

PRISM - TOOLKIT FOR EVALUATING THE OUTCOMES AND IMPACTS OF SMALL/MEDIUM-SIZED CONSERVATION PROJECTS

4:0

INTRODUCTION &KEY CONCEPTS

DESIGNING & IMPLEMENTING
THE EVALUATION

PRISMMODULESMETHOD FACTSHEETS& ANNEXES

Required steps
Step 1.

GENERAL
FACTSHEETSStart with the project’s outcomes & impacts.

Take the outcomes & impacts from your Theory of Change (orequivalent) and insert them into the following matrix. Put the
project’s longer-term outcomes and impacts at the top and work
your way down. See example below.

COMPLETING A
PRIORITISATION
MATRIXSTEP 1

Training
stakeholders onsustainable
extractionof forest
products

Individuals
trainedPayinghouseholds
10 year forest
protection fee andpersuading them toinvest part of it
on rattan
enrichment

Households
receiveinvestment for
rattanenrichment

Forest protection
agreements in
exchange for forest
product extraction
rightsAgreements inplace

Trainedindividuals apply
new skills

Local people
manage forest
resources moresustainably

Householdlivelihood/
wellbeing
improves

Local people areincentivised to
manage forest
more sustainably

Rattan provides
additional
income tohouseholds

Local people
have right toutilise forest
resources

Forest loss
slowed/reversed

Status of native forest
species (including
Edward’s Pheasant)improves

Critical Assumptions
Prices for forest products remain stable or increase
Weather conditions remain stable enough for
rattan to grow
Populations of native forest species do not

Actions

181

Outputs

Intermediateoutcomes

Threat reduction
outcomes

Impact on
Biodiversity

PRISM - TOOLKIT FOR EVALUATING THE OUTCOMES AND IMPACTS OF SMALL/MEDIUM-SIZED CONSERVATION PROJECTS

decrease due to disease or other natural factors

4:0

INTRODUCTION &KEY CONCEPTSOUTCOME/
IMPACT

GENERAL
FACTSHEETSEVALUATIONQUESTION(S)DESIGNING & IMPLEMENTING
THE EVALUATION

WHICH STAKEHOLDERS
CARE ABOUT THIS?

PRISMMODULESIMPORTANCETO USERSIMPORTANCETO PROJECTSUCCESSHOW WELLKNOWN ISTHE ACTIONBEINGEVALUATED?

EASE OF DATACOLLECTIONCAN THIS BEANSWEREDGIVEN YOURTIME &RESOURCESPRIORITY

Status of native(High, Med,(High, Med,(High, Med,(High, Med,

(Yes, No,

(High, Medium, Low,

forest species
improves

Low)

Low)

Low)

Low)

Partially)

Eliminate)
Provide justification
for the answers)

Forest loss
slowed/reversed

COMPLETING A
PRIORITISATION
MATRIXSTEP 1

Local people
manage forest
resources moresustainably
Local people areincentivised to
manage forest
more sustainably
Local people haveright to utilise
forest resources
Householdlivelihood/
wellbeing
improves
Rattan provides
additional income
to households

Trainedindividuals apply
new skills

182

METHOD FACTSHEETS& ANNEXESPRISM - TOOLKIT FOR EVALUATING THE OUTCOMES AND IMPACTS OF SMALL/MEDIUM-SIZED CONSERVATION PROJECTS

4:0

INTRODUCTION &KEY CONCEPTS

Step 2.

GENERAL
FACTSHEETSCOMPLETING A
PRIORITISATION
MATRIXSTEP 2

183

DESIGNING & IMPLEMENTING
THE EVALUATION

Develop potential evaluation questions for eachoutcome/impact and identify who will use the results.

PRISM
MODULESMETHOD FACTSHEETS& ANNEXES

Refer to step 1 in section 2 of the Toolkit for more information
on how to develop evaluation questions and how to identifying
evaluation stakeholders.

OUTCOME/
IMPACTEVALUATIONQUESTION(S)

WHICH STAKEHOLDERS
CARE ABOUT THIS?

IMPORTANCETO USERSIMPORTANCETO PROJECTSUCCESSHOW WELLKNOWN ISTHE ACTIONBEINGEVALUATED?

EASE OF DATACOLLECTIONCAN THIS BEANSWEREDGIVEN YOURTIME &RESOURCESPRIORITY

Status of native

To what extent have populations ofParticipating households(High, Med,(High, Med,native forest species changed since theproject began and if so to what extent

Local authorities;
Project team

Low)

Low)

(High, Med,
Low)

(High, Med,

forest species
improves(Yes, No,
Partially)(High, Medium, Low,
Eliminate)
Provide justification

can this be attributed to the project?

Project funder

Forest loss
slowed/reversed

Has there been a change in the rate offorest loss since the project began and if
so can this change be expected to persist

Participating households
Local authorities;
Project teamProject funder

Local people
manage forest
resources moresustainably

Has there been a change in the way local
people manage forest resources and if
so to what extent can this be attributed
to the project?

Participating households
Local authorities;
Project teamProject funder

Local people areincentivised to
manage forest
more sustainably

Have the project’s actions given local
people more of an incentive to manage
forest resources sustainably and if so
can this be expected to persist?

Participating households;
Local authorities

Local people haveright to utilise
forest resources

Have agreements provided local people
with the right to utilise forest resources?

Participating households
Local authorities

Householdlivelihood/
wellbeing
improves

Have household livelihoods and/or
wellbeing status improved as a result of
the project’s actions

Participating households
Local community
Project Funder

Rattan provides
additional income
to households

Has the rattan planted with support
from the project provided (or will
provide) households with additional
income?

Participating households
Local community

Trainedindividuals apply
new skills

Are trained individuals applying any ofProject teamParticipating households

the skills learnt?

PRISM - TOOLKIT FOR EVALUATING THE OUTCOMES AND IMPACTS OF SMALL/MEDIUM-SIZED CONSERVATION PROJECTS

Low)

for the answers)

4:0

INTRODUCTION &KEY CONCEPTS

Step 3.

DESIGNING & IMPLEMENTING
THE EVALUATION

Populate the matrix based on the following criteria:

Importance to users, importance to project success, how well

GENERAL
FACTSHEETSCOMPLETING A
PRIORITISATION
MATRIXSTEP 3184

PRISM
MODULESMETHOD FACTSHEETS& ANNEXES

known the action is and the level of time, data and resources
required. Refer to section 2 of PRISM for more information on how
to apply these criteria.

OUTCOME/
IMPACTEVALUATIONQUESTION(S)

WHICH STAKEHOLDERS
CARE ABOUT THIS?

IMPORTANCETO USERSIMPORTANCETO PROJECTSUCCESSHOW WELLKNOWN ISTHE ACTIONBEINGEVALUATED?

EASE OF DATACOLLECTIONCAN THIS BEANSWEREDGIVEN YOURTIME &RESOURCESPRIORITY

Status of native

To what extent have populations ofParticipating households

High

High

Low

Low

No

forest species
improves

native forest species changed since theproject began and if so to what extent

Local authorities;
Project team(High, Medium, Low,
Eliminate)
Provide justification

can this be attributed to the project?

Project funder

for the answers)

Forest loss
slowed/reversed

Has there been a change in the rate offorest loss since the project began and if
so can this change be expected to persist

Participating households
Local authorities;
Project teamProject funder

Med

High

Low

Low

No

Local people
manage forest
resources moresustainably

Has there been a change in the way local
people manage forest resources and if
so to what extent can this be attributed
to the project?

Participating households
Local authorities;
Project teamProject funder

High

High

Low

Med

Yes

Local people areincentivised to
manage forest
more sustainably

Have the project’s actions given local
people more of an incentive to manage
forest resources sustainably and if so
can this be expected to persist?

Participating households;
Local authorities

Med

High

Low

Med

Yes

Local people haveright to utilise
forest resources

Have agreements provided local people
with the right to utilise forest resources?

Participating households
Local authorities

High

High

Med

High

Yes

Householdlivelihood/
wellbeing
improves

Have household livelihoods and/or
wellbeing status improved as a result of
the project’s actions

Participating households
Local community
Project Funder

High

High

Med

Low

Partially

Rattan provides
additional income
to households

Has the rattan planted with support
from the project provided (or will
provide) households with additional
income?

Participating households
Local community

Med

High

Med

Med

Partially

Trainedindividuals apply
new skills

Are trained individuals applying any ofProject teamParticipating households

Med

Med

Med

Med

Yes

the skills learnt?

PRISM - TOOLKIT FOR EVALUATING THE OUTCOMES AND IMPACTS OF SMALL/MEDIUM-SIZED CONSERVATION PROJECTS

4:0

INTRODUCTION &KEY CONCEPTS

Step 4.

DESIGNING & IMPLEMENTING
THE EVALUATIONCOMPLETING A
PRIORITISATION
MATRIXSTEP 4OUTCOME/
IMPACTEVALUATIONQUESTION(S)

WHICH STAKEHOLDERS
CARE ABOUT THIS?

IMPORTANCETO USERSIMPORTANCETO PROJECTSUCCESSHOW WELLKNOWN ISTHE ACTIONBEINGEVALUATED?

EASE OF DATACOLLECTIONCAN THIS BEANSWEREDGIVEN YOURTIME &RESOURCESPRIORITY

Status of native

To what extent have populations ofParticipating households

High

High

Low

Low

No

Eliminate – Not

forest species
improves

native forest species changed since theproject began and if so to what extent

Local authorities;
Project teampossible to measure
within project

can this be attributed to the project?

Project fundertimeline

Forest loss
slowed/reversed

Has there been a change in the rate offorest loss since the project began and if
so can this change be expected to persist

Participating households
Local authorities;
Project teamProject funder

Med

High

Low

Low

No

Eliminate – Not
possible to measure
within project
timeline

Local people
manage forest
resources moresustainably

Has there been a change in the way local
people manage forest resources and if
so to what extent can this be attributed
to the project?

Participating households
Local authorities;
Project teamProject funder

High

High

Low

Med

Yes

High - Crucial toproject success andmeasurable with available time / resources

Local people areincentivised to
manage forest
more sustainably

Have the project’s actions given local
people more of an incentive to manage
forest resources sustainably and if so
can this be expected to persist?

Participating households;
Local authorities

Med

High

Low

Med

Yes

High - Crucial toproject success andmeasurable with available time / resources

Local people haveright to utilise
forest resources

Have agreements provided local people
with the right to utilise forest resources?

Participating households
Local authorities

High

High

Med

High

Yes

High - Crucial toproject success andmeasurable with available time / resources

Householdlivelihood/
wellbeing
improves

Have household livelihoods and/or
wellbeing status improved as a result of
the project’s actions

Participating households
Local community
Project Funder

High

High

Med

Low

Partially

High - Crucial toproject success andmeasurable with available time / resources

Rattan provides
additional income
to households

Has the rattan planted with support
from the project provided (or will
provide) households with additional
income?

Participating households
Local community

Med

High

Med

Med

Partially

Med – Rattan yet tobe harvested which

Trainedindividuals apply

Are trained individuals applying any of
the skills learnt?

Project teamParticipating households

means income canonly be estimated.
Med

Med

new skills

185

METHOD FACTSHEETS& ANNEXES

note explaining why you ranked each question as high, medium
orlow. Once complete you should review this list with the projectstakeholders, re-visit the Theory of Change to ensure that nothingimportant has been missed and revise accordingly.

Prioritise the questions to be answered.

Review the information in the matrix and prioritise the questions
based on the information entered. If possible you should make a

GENERAL
FACTSHEETSPRISMMODULESPRISM - TOOLKIT FOR EVALUATING THE OUTCOMES AND IMPACTS OF SMALL/MEDIUM-SIZED CONSERVATION PROJECTS

Med

Med

Yes

Low – Not asimportant as otherquestions

4:0

INTRODUCTION &KEY CONCEPTS

Step 5.

GENERAL
FACTSHEETSCOMPLETING A
PRIORITISATION
MATRIXSTEP 5

DESIGNING & IMPLEMENTING
THE EVALUATION

Plan the evaluation around the prioritised evaluationquestions.

Plan the evaluation around the questions identified, starting withthe highest priority questions and working your way down. Aim toanswer no more than five main questions during the evaluation.
For the example given above, the team selected the following
three questions to answer during the evaluation:•

Has there been a change in the way local people manage
forest resources and if so to what extent can this be
attributed to the project?

•

Have the project’s actions given local people more of an
incentive to manage forest resources sustainably and if
so can this be expected to persist?

•

Have agreements provided local people with the rightto utilise forest resources?

•

Have household livelihoods and/or wellbeing status
improved as a result of the project’s actions

PRISM
MODULES

Acknowledgements
Guidance in this section is based on Sharp L and Frechtling J (1997).
Overview of the design process for mixed method evaluation. In L
Sharp and J Frechtling (eds.), User-Friendly Handbook for Mixed
Method Evaluations.
Retrieved from http://www.nsf.gov/pubs/1997/nsf97153/start.
htm .

Once the questions have been identified you can then develop
indicators for each of them, insert this information into the PRISM
evaluation and select appropriate methods and evaluation designs
that will enable you to answer the identified questions. Refer to
PRISM step 2 for more information on how to do this.

186

METHOD FACTSHEETS& ANNEXESPRISM - TOOLKIT FOR EVALUATING THE OUTCOMES AND IMPACTS OF SMALL/MEDIUM-SIZED CONSERVATION PROJECTS

4:0

INTRODUCTION &KEY CONCEPTS

DESIGNING & IMPLEMENTING
THE EVALUATION

Once you have identified your evaluation questions and methods it
is important to plan and map out how data will be collected, when
it will be collected, who will collect it and estimate how much it will
cost.

GENERAL
FACTSHEETSCOMPLETING ANEVALUATION PLAN

Below is a simple template that you can use, modify or adapt toyour particular project to map out and plan your data collection.

PRISM
MODULESMETHOD FACTSHEETS& ANNEXES

See Step 2. What data do you need to collect? and the Module
Chapters for information on completing columns 5-9
For column 10 (key assumptions) see factsheet: Completing a
Theory of Change for more information on identifying and dealing
with assumptions

See Step 1. What do you need to know for guidance on completingcolumns 1-4

COMPLETING AN EVALUATION PLAN
OUTCOME/
IMPACT (FROM
THEORY OF
CHANGE OR
EQUIVALENT)187

EVALUATIONQUESTIONSUBQUESTION(S)INDICATOR(S)DATA
COLLECTIONMETHOD(S)EVALUATIONDESIGNPRISM - TOOLKIT FOR EVALUATING THE OUTCOMES AND IMPACTS OF SMALL/MEDIUM-SIZED CONSERVATION PROJECTSPERSON(S)RESPONSIBLE
FOR DATACOLLECTION

WHEN WILLDATA BE
COLLECTED?

ESTIMATED
COSTS

KEY
ASSUMPTIONS(FACTORS THAT
MAY AFFECT
THE PROJECT’SRESULTS IF NO
LONGER TRUE)

4:0

INTRODUCTION &KEY CONCEPTS

DESIGNING & IMPLEMENTING
THE EVALUATION

PRISMMODULESMETHOD FACTSHEETS& ANNEXES

EXAMPLE: COMPLETED EVALUATION PLAN
OUTCOME/
IMPACT (FROM
THEORY OF
CHANGE OR
EQUIVALENT)EVALUATIONQUESTION

GENERAL
FACTSHEETSEXAMPLE:COMPLETEDEVALUATION
PLAN

Small group of
local people are
actively engaged
in conservation at
the siteIs there a small
group of local
people who are
actively engaged
in conservation
at the site?

SUB-QUESTION(S)INDICATOR(S)DATA
COLLECTIONMETHOD(S)EVALUATIONDESIGNPERSON(S)RESPONSIBLE
FOR DATACOLLECTION

WHEN WILL DATA
BE COLLECTED?

Have local people
been informed about
conservation at the
project site?

Number of local people
attending a meeting

Focus group

Before - After

Social surveyor

March 2017 & May
2018

Local people’s
work schedules
allow them toattend focus
groups

Are local people moreinterested in engaging inconservation at the site
and if so, is this due to theproject?

Establishment of a local
conservation group

Directobservation

Before - After

Social surveyor

March 2017 & May
2018

Local people’s level ofinterest in engaging inconservation at the site

Focus group

Local people’s
work schedules
allow them toattend focus
groupsLocal people’s level ofknowledge regarding
conservation at projectsite after meetings/
presentations

Social surveyor

Has the project affected
local perceptions towards
birdwatchers

Local communities
appreciate theimportance of birds
at project site arepositive about
birdwatching at
the site.

Habitat at managed
part of project site
is improved forshorebirds

188

How hasthe project
affected local
communities
perceptions ofbirdwatching
and theimportanceof birds at the
project site?

Has the habitat
being managed
by the project
been improvedfor shorebirds,
compared toneighbouringsalt pans?

Questionnaire Baseline
Is there a perceived
economic benefit to local
communities (e.g. bird
tourism revenues)

ESTIMATEDCOSTS

Community perceptions
before and after the
project

Has the project madepeople more positive
about birds and theimportance of the site ingeneral

Focus group Baseline
Participatory
ImpactAssessment(PIA)

Surveystakeholders
before - after Social surveyor

Baseline surveys March 2017
Follow-up survey
and PIA - May 2018

Social surveyor

Participatory
ImpactAssessment

Salary of social
surveyor ($30/
day for 14 days
data collection+ 7 days
analysis andreporting)

Project timeline
is long enoughfor economic
benefits tobecome visible tolocal people

Baseline surveys March 2017
Follow-up survey
and PIA - May 2018

Has the frequency of
SBS and the diversityof shorebird species in
BCST managed salt pansincreased, compared toother salt pans?

Frequency and diversityof shorebird populations
at both managed salt
pans and neighbouringsalt pans

Shorebird Survey

Have other indicatorsof habitat quality (e.g.
salinity, invertebrate
density) at project site
improved

Indicators of habitat
quality at managed
salt pans

Habitat surveys(salinity,
invertebrate
diversity etc)

Matching
design-Measure at
project site &neighbouringsalt pans

Local bird
surveyor

Measurebefore-after at
project site

Project
Coordinator

Baseline survey June 2017

Researcher

Follow-up survey June 2018

PRISM - TOOLKIT FOR EVALUATING THE OUTCOMES AND IMPACTS OF SMALL/MEDIUM-SIZED CONSERVATION PROJECTS

Weekly - During
winter months
Baseline data to becollected March2017

Local people’s
work schedules
allow them toattend focus
groupsLocal people arewilling to provide
honest answers
regarding theirperceptions

Baseline surveys March 2017
Follow-up survey
and PIA - May 2018

Social surveyor

KEY
ASSUMPTIONS(FACTORS THAT
MAY AFFECT
THE PROJECT’SRESULTS IF NO
LONGER TRUE)

Salary of local
surveyor ($20/
day for 30
days)

Other thanthose managed
by the project,environmental
conditions do
not significantly
affect shorebird
numbers or
behaviour

Stipend forresearcher($20/day for
10 days datacollection + 5days analysis
and reporting)

Other thanthose managed
by the project,environmental
conditions are
similar at survey
times

4:0

INTRODUCTION &KEY CONCEPTS

DESIGNING & IMPLEMENTING
THE EVALUATION

Factsheet: Developing ethical protocols for an evaluation

GENERAL
FACTSHEETSDEVELOPINGETHICAL
PROTOCOLS
FOR ANEVALUATION

Many organisations that conduct conservation projects,
and organisations that fund projects, have their own ethicalreview process that any activities will need to adhere to. Suchethical standards will apply to evaluation methods as well.
Some conservation projects may have no formal ethical review
requirements but it is still essential that work conducted, bothin the project itself and in the assessment of impacts, is ethicallysound. This factsheet provides some guidance on the things thatshould be considered when conducting activities involving people.

PRISM
MODULES

Considerations for the design phase
Minimise harm - There is an obligation to minimise any potential
psychological or physical risk to participants from taking part in
the study. As well as direct risks there may be indirect ones, forexample by asking people to take part in the research you may be
reducing the time they have to spend with family and friends or atwork, or you may be affecting their status in the local community.
It may not be possible to avoid all harm, but risks need to bebalanced against expected benefits.

There are also important ethical considerations for undertaking
any evaluation involving species populations, habitats or
stakeholders involved outside the main part of the project (e.g.
as part of a control group). For example a particular method may
damage a particular habitat or a method can cost the participants
time that could otherwise be spent working, earning money, being
active within their community or with family/friends. Poorlyplanned evaluations can be unethical, can waste people’s time andcan damage both the project and the perception of conservation
work in the local community and beyond.
There are ethical considerations that apply to activity design,during implementation and after the activity has been completed.
These considerations may require preparation on behalf of theteam before the project starts (for example, finding a securestorage unit for data).

189

METHOD FACTSHEETS& ANNEXESPRISM - TOOLKIT FOR EVALUATING THE OUTCOMES AND IMPACTS OF SMALL/MEDIUM-SIZED CONSERVATION PROJECTS

4:0

INTRODUCTION &KEY CONCEPTS

GENERAL
FACTSHEETS

Considerations when collecting data
Consent - Participation needs to be voluntary, with theparticipants being informed that they are contributing to a
research project and given details of what they are required to do.
They must give their consent to this, either verbally or in writing.
Projects are increasingly being encouraged to adopt the principleof Free, Prior and Informed Consent (FPIC)14.

DEVELOPINGETHICAL
PROTOCOLS
FOR ANEVALUATIONoFree - Consent for involvement in the evaluation should
be given voluntarily and without coercion, intimidation,
manipulation or bribery.

o

Prior - Consent should be sought sufficiently far inadvance of the beginning of the evaluation, i.e. in theearly stages of planning the evaluation, and should
respect the time requirements of the participants.

o

14

190

DESIGNING & IMPLEMENTING
THE EVALUATION

Informed - Information on the evaluation should bedelivered in appropriate language and in a culturally
appropriate way, in sufficient time to be understood.
Information must be provided in a way that is clear,consistent, accurate, constant and transparent.
Information must reach all relevant participating
communities and any marginalised groups, and be
provided on an ongoing basis throughout the process.

PRISM
MODULESo

METHOD FACTSHEETS& ANNEXES

Consent - Communities and individuals have thefreedom to grant or withhold their consent at any stageof the evaluation. It must be accepted that the projectwill not proceed and/or that engagement must be
stopped if participants decide that they do not want tobegin or to continue, or if they decide to withhold theirconsent at any time.

Right to withdraw - During the survey the participants need to
be informed that they can choose not to partake in part of thestudy or can stop participation at any time, or can ask for theirinformation to be deleted.
Confidentiality - The participants need to be informed that theirdata will be collected, stored and used in confidence; and their data
should be anonymised so that individuals cannot be identified inany analysis or written output.
Transparency - The participants need to be informed that they cancontact the study leaders to ask questions or express concerns
regarding the study; and they should be provided with contact
details of the study leader to enable this.
Debrief - Participants should formally agree with the informationthat is being attributed to them and should be able to edit or delete
their data.

Adapted from: UN-REDD Programme (2013): Guidelines on Free,Prior and Informed Consent.

PRISM - TOOLKIT FOR EVALUATING THE OUTCOMES AND IMPACTS OF SMALL/MEDIUM-SIZED CONSERVATION PROJECTS

4:0

INTRODUCTION &KEY CONCEPTS

GENERAL
FACTSHEETSDEVELOPINGETHICAL
PROTOCOLS
FOR ANEVALUATIONDESIGNING & IMPLEMENTING
THE EVALUATION

Considerations following data collection•
Data storage – Hard copies should be kept in a locked
cupboard and electronic data must be stored in a secureway, being accessible only by named project members,
to protect the participants.
•

Returning study findings – The participants shouldbe thanked for their participation and be provided
with a suitable debrief on the study’s findings and its
implications at the time of completion.

•

Anonymity – Data used in reporting should beanonymised so that the participants cannot be
identified.

PRISM
MODULESMETHOD FACTSHEETS& ANNEXES

Important questions to consider
What is being asked?
Is the evaluation activity possibly going to cause any physicalor psychological harm? Harm may occur in many ways, either
directly (for example if a participant is stressed by questioning) orindirectly (for example through repercussions in the community if aparticipant is blamed for negative reports). Potential consequences
are obviously a consideration for the person and whether theyagree to participate, but they may also be balancing this with thedesire to co-operate. It is, therefore, a responsibility of the project
team to minimise negative consequences for the participants.
Does the evaluation activity aim to deceive participants about the
information they are providing? In some situations the questions
being asked may be getting information out of people in an indirect
or less than honest way. Deceiving people may be unethicaland unjustified, particularly if the information being obtained issensitive, personal or if it potentially implicates people in negative
or illegal behaviours.
Does the evaluation activity involve actions or questions
concerned with sensitive or illegal behaviour? Although in somecircumstances a project may be trying to gather information on
sensitive/illegal behaviour, are you collecting any identifying or
personal information that links any individuals to this behaviour?
Do you have an obligation to report illegal behaviour and do theparticipants know this?

191

PRISM - TOOLKIT FOR EVALUATING THE OUTCOMES AND IMPACTS OF SMALL/MEDIUM-SIZED CONSERVATION PROJECTS

4:0

INTRODUCTION &KEY CONCEPTS

GENERAL
FACTSHEETSDEVELOPINGETHICAL
PROTOCOLS
FOR ANEVALUATION192

DESIGNING & IMPLEMENTING
THE EVALUATION

How it is being asked?
Does the evaluation activity collect data using a third party ororganisation (e.g. a consultancy or the internet)and therefore
potentially share personal information with other companies?
Using third party survey tools may require the participant to reveal
their identity to the service provider, either directly, throughbeing requested to provide their details, or indirectly throughautomatic collection of home addresses by visiting or IP addressesby the computer programme. The team should make sure thatparticipants understand any privacy or data collection agreements
they are entering into.
Is the evaluation activity offering rewards for participation? This
will affect involvement, but consideration also needs to be given towhether rewards are equally available to all community members
or are just benefitting a subset of the population. Providing
rewards to some members of the community and not to others
may cause conflict. This may be averted if equal rewards areavailable to everyone for different actions or at different times, butthere should be transparency, consultation and agreement about
any such rewards.

PRISM
MODULESMETHOD FACTSHEETS& ANNEXES

Who is being asked?
Does the evaluation activity involve children or vulnerable people?
Special care needs to be taken when dealing with these groups sothat they are not exposed to harm. For example, do the project
team members have the relevant training and any requisite formal
clearances for dealing with children/vulnerable groups? Children
and other vulnerable people may not fully understand what they
are agreeing to take part in, and the consent and presence of
responsible guardians should be a requirement for work with suchpeople.
More information on ethical considerations and standards canbe found at http://www.esrc.ac.uk/files/funding/guidance-forapplicants/esrc-framework-for-research-ethics-2015/ and http://
the-sra.org.uk/wp-content/uploads/ethics03.pdf .

PRISM - TOOLKIT FOR EVALUATING THE OUTCOMES AND IMPACTS OF SMALL/MEDIUM-SIZED CONSERVATION PROJECTS

4:0

INTRODUCTION &KEY CONCEPTS

DESIGNING & IMPLEMENTING
THE EVALUATION

Factsheet: Developing gender protocols for an evaluation

GENERAL
FACTSHEETSDEVELOPINGGENDER
PROTOCOLS
FOR ANEVALUATION

193

Many conservation actions will have outcomes and impacts
that differ according to the gender of the project’s stakeholders.
Mainstreaming (or otherwise addressing) gender in yourevaluation can therefore provide extremely useful informationand insights that would not be captured if participants are viewed
as a gender-homogenous group. Reference to gender is also arequirement for many donors in the development sector.

What is gender?
Gender is a social construct that refers to relations between
and among women and men, based on their relative roles. It
encompasses the economic, political and socio-cultural attributes,
constraints and opportunities associated with being male orfemale. As a social construct, gender varies across cultures, isdynamic and open to change over time. Because of the variationin gender roles across cultures and over time, these roles should
not be assumed, but investigated. Gender is also a rights issue,
so for example women’s rights are laid down in the Convention
on the Elimination of all Forms of Discrimination Against Women(CEDAW). In the development community there is often arequirement for a Human Rights Based Approach.

PRISM
MODULESMETHOD FACTSHEETS& ANNEXES

Tips for incorporating gender into an evaluation
Ideally a project will have carried out a gender analysis at the
beginning, or a social survey which clearly distinguishes data
only applicable to women from data only applicable to men. The
evaluation can then measure impacts and changes affecting eachgroup separately if necessary. In many projects however, suchdata do not exist or they are inadequate. In order to focus theevaluation it is recommended to:•

identify whether gender has been addressed in all of the
project’s outcomes & impacts related to communities
and local empowerment, and to what extent theoutcomes & impacts affect particularly either women ormen;•(if there are no baseline data) develop questions andformulate indicators that are sex-disaggregated andgender-sensitive, such as numbers and percentagesof men and women (not just “community members”).
Questions and indicators could refer to the roles thatwomen and men play in natural resource management,or to and attitudes and practices concerning female
participation and leadership (e.g. how womenparticipate in community decision making).

PRISM - TOOLKIT FOR EVALUATING THE OUTCOMES AND IMPACTS OF SMALL/MEDIUM-SIZED CONSERVATION PROJECTS

4:0

INTRODUCTION &KEY CONCEPTS

DESIGNING & IMPLEMENTING
THE EVALUATION

Designing the evaluation and collecting data•
Apply gender sensitivity to methods, e.g. disaggregate
questionnaire responses by gender.

GENERAL
FACTSHEETSDEVELOPINGGENDER
PROTOCOLS
FOR ANEVALUATION•

When appropriate, try to ensure gender balance among
evaluation participants.

•

Use both male and female data collectors, and ensure
they have received gender-sensitive training.

•

Ensure confidentiality and sensitivity, e.g. participants
may provide different answers depending on the gendermakeup of the group. It is often useful to organise focus
group meetings solely with women, to ensure that their
voice is heard.

Analysing and interpreting results•

During analysis, disaggregate data by gender.

•

Use the data to evaluate whether and how the project’s
results affected men and women differently.

Questions to ask•
How did the project respond to the identified specificinterests and needs of women and men?

194•

How did the project address the rights of men andwomen?

•Is the purpose and outcome of the intervention in
keeping with the goals of gender equity and beneficial
outcomes for women and men?

PRISM
MODULESMETHOD FACTSHEETS& ANNEXESKey Resources
Conservation and Gender. Fauna & Flora International: http://
www.fauna-flora.org/initiatives/conservation-and-gender/
Guidelines for integrating gender into conservation.
Conservation International:
English: https://docs.google.com/file/d/0B23PUHp4FJPZb3AzNUZWNENkLVk/edit
French: https://docs.google.com/file/d/0B23PUHp4FJPZNHR0N2VlMDNqY1k/edit
Spanish: https://docs.google.com/file/d/0B23PUHp4FJPZVnFYb1BTTVd5UUU/edit .
Conservation Leadership Programme - Tips for Integrating
Genderhttp://www.conservationleadershipprogramme.org/
media/2014/09/CLP-Gender-Integration-Tips_Final.pdf .
Addressing gender in impact evaluation
https://www.odi.org/sites/odi.org.uk/files/odi-assets/publicationsopinion-files/9934.pdf .

PRISM - TOOLKIT FOR EVALUATING THE OUTCOMES AND IMPACTS OF SMALL/MEDIUM-SIZED CONSERVATION PROJECTS

4:0

INTRODUCTION &KEY CONCEPTS

DESIGNING & IMPLEMENTING
THE EVALUATION

Factsheet: Entering, organising and cleaning data
Step 1.

GENERAL
FACTSHEETSENTERING,
ORGANISINGAND CLEANINGDATASTEP 1

195

Enter and organise data

With both quantitative and qualitative data, it is important thatdata are entered in a logical format that can be easily understoodand analysed.
•

For quantitative data, use Microsoft Excel or anotherspreadsheet software package to enter your data into
an electronic format.

•

For qualitative data, type up all the data into a word
processing program such as Microsoft Word.

PRISM
MODULESYou would use a similar organisational approach for qualitative
data.
•

Create a file for each interview, observation site, focus
group, etc.

•

Within each file, organise the data by question, timeinterval, and/or topic (depending on which method
makes most sense).

Before you being entering, develop a system to organise your data.
•

For example, if you administered a survey toschoolchildren and students, it would probably be best
to create one data spreadsheet for the parent survey
and another spreadsheet for the student survey.

•

Make sure that each participant’s responses are
assigned to a unique participant identifier and that
responses are organised by survey item/question.

METHOD FACTSHEETS& ANNEXESPRISM - TOOLKIT FOR EVALUATING THE OUTCOMES AND IMPACTS OF SMALL/MEDIUM-SIZED CONSERVATION PROJECTS

4:0

INTRODUCTION &KEY CONCEPTS

Step 2.

DESIGNING & IMPLEMENTING
THE EVALUATION

Clean data

No matter what type of data or method you use, be sure always togo back and review for errors.

GENERAL
FACTSHEETSENTERING,
ORGANISINGAND CLEANINGDATASTEP 2Checking data for errors is commonly called “cleaning”. Cleaningdata is critical because “dirty” data can severely influence yourresults.
Three commonly used cleaning methods are:Spot-checking - Comparing the raw data with theelectronically-entered data to check for data-entry andcoding errors. To spot-check quantitative survey data, you
would for example randomly select several participants’completed paper questionnaires and compare them withthe data in the electronic spreadsheet. For qualitative data
you would use this approach to check whether participants’words were recorded accurately and attributed to the rightindividuals.
If you find an error in your first round of spot-checking you
should randomly check another round of the raw data. If you
continue to find errors, and it is clear that they are more thanjust isolated incidents, you will need to go over all of the raw
data to ensure that each record has been entered correctly.

PRISM
MODULES

while “yes” responses are assigned a value of 1. Therefore,
any number other than a 0 or 1 in the associated column on
the spreadsheet would be an obvious error.
Logic checks - Carefully reviewing the electronicallyentered data to make sure that the answers to the questions“make sense.” For example, if participant 1 in a spreadsheet
indicated that they did not attend a project workshop,
it would be illogical for this participant to have provided
a satisfaction rating for the workshop. The only logicalresponse for this participant would be something equivalent
to “not applicable”.
If you find any errors using the techniques above, you will need togo back to the original raw data for the participant(s) concernedand enter the correct data.

Acknowledgement
Guidance in this factsheet is based on the Pell Institute’s Evaluation
Toolkit http://toolkit.pellinstitute.org/evaluation-guide/analyze/
enter-organize-clean-data/ .

Eye-balling - Reviewing the data for errors that may haveresulted from a data-entry or coding mistake. For example,
a survey question might read: “Did you attend one of the
project’s workshops?” Participants can only respond to thisquestion with a “no” or “yes.” “No” is assigned a value of 0,

196

METHOD FACTSHEETS& ANNEXESPRISM - TOOLKIT FOR EVALUATING THE OUTCOMES AND IMPACTS OF SMALL/MEDIUM-SIZED CONSERVATION PROJECTS

4:0

INTRODUCTION &KEY CONCEPTS

GENERAL
FACTSHEETSANALYSINGQUANTITATIVEDATASTEPS 1 & 2

PRISM
MODULESMETHOD FACTSHEETS& ANNEXES

Factsheet: Analysing quantitative data

Step 2.

Analysing quantitative data involves examining numeric data
(data collected in the form of numbers) to look for patterns andtrends that can help you to answer your evaluation questions.
This factsheet outlines some simple analyses that you can use todescribe your data along with some links to resources that can help
you to perform more complex analyses.

The first and most simple analysis you can undertake is to organise
your data according to the different variables in your dataset,for example by gender, species, site or income bracket. Adding apercentage allows you to see what proportion of your total sample
are represented in each category.

Step 1.

GenderFemale = 16 (53%)
Male = 14 (47%)

Target community

Village A = 15 (50%)
Village B = 15 (50%)

Primary source of householdincomeCoffee = 10 (33%)
Livestock = 10 (33%)
Plantain = 5 (17%)
Tomatoes = 3 (10%)
Fruit = 2 (7%)

How important is the longterm sustainability of local
areas of forest to yourhousehold?

Very important = 15 (50%)
Quite important = 2 (7%)
Neutral = 3 (10%)
Not very important = 8 (26%)
Not important at all = 2 (7%)

Identify what kind of data you have
Examples of quantitative data include:

DATA TYPE

CATEGORICAL/CONTINUOUS

DESCRIPTION

EXAMPLE

Nominal data

CategoricalBasicclassification
data. Data has nological order

Male/Female
Species
Sites

Data has alogical order,
but differences
between values
are not constantJuvenile - Adult

Ordinal data

197

DESIGNING & IMPLEMENTING
THE EVALUATION

CategoricalInterval data

Continuous

Data has alogical orderand differences
between values
are standardised.
There is nonatural zero

Likert scale:(1=strongly
agree, 2= agreeslightly, 3=neither
agree or disagree,
4=slightly
disagree,5=Strongly
disagree)Ratio data

Continuous

Data is ordered,
has standardised
differences
between values
and a natural zero

Speciespopulation size
HouseholdincomeOrganise your data

Sample size = 30

PRISM - TOOLKIT FOR EVALUATING THE OUTCOMES AND IMPACTS OF SMALL/MEDIUM-SIZED CONSERVATION PROJECTS

4:0

INTRODUCTION &KEY CONCEPTS

DESIGNING & IMPLEMENTING
THE EVALUATION

Frequency tables are a good visual way of summarising nominal or
numeric data by displaying the count of observations (number oftimes a value of a variable occurred) in a table.

GENERAL
FACTSHEETSANALYSINGQUANTITATIVE
DATA

PRISM
MODULES

Step 3.

METHOD FACTSHEETS& ANNEXES

Describe your data

Use simple calculations to describe your data. The most commonlyused calculations are:Mean – Average of scores for a particular variable

NUMBER OF
CHILDREN PER
HOUSEHOLDTALLYFREQUENCY

1

II

2

2

IIII

4

3

I

1

4+

II

2

Minimum & Maximum values – Highest & lowest value for aparticular variable
Median – The middle point in the dataset.
Mode – The most common number score
So the following dataset could be described in the following way:

STEP 3

198

More information: http://www.betterevaluation.org/en/
resources/guides/frequency_tables

NUMBER OF
CHILDREN PER
HOUSEHOLD

TALLY

1

II

2

IIII

3

I

4

II

PRISM - TOOLKIT FOR EVALUATING THE OUTCOMES AND IMPACTS OF SMALL/MEDIUM-SIZED CONSERVATION PROJECTSHouseholds = 9
Total Children = 21
Mean = 2.3
Max = 4
Min = 1Mode = 2
Median = 2

4:0

INTRODUCTION &KEY CONCEPTS

Step 4.

DESIGNING & IMPLEMENTING
THE EVALUATION

PRISMMODULES

More information: Emery Evaluation - How to analyse pivot tables
in excel https://youtu.be/rybcOdIbH6g

Disaggregate your data

Further explore your data by separating out and disaggregating
the data across your different variables.

GENERAL
FACTSHEETSANALYSINGQUANTITATIVEDATA

Cross-tabulations are a simple method of showing how frequently
two variables (e.g. gender and bustmeat consumption) occur
together.

HOW OFTEN DO YOU EAT BUSHMEAT?

MALES

FEMALESAt least once a week6

3

At least once a month

21

10

At least once a year

9

9

Never

1

12

STEP 4

Cross tabulations can then be used to make simple graphic tovisualise your results:
How often do you eat bushmeat?

Never

At least once a year

At least once a month

At least once a week

0

199

5

METHOD FACTSHEETS& ANNEXES

10

15

20

25

PRISM - TOOLKIT FOR EVALUATING THE OUTCOMES AND IMPACTS OF SMALL/MEDIUM-SIZED CONSERVATION PROJECTS

4:0

INTRODUCTION &KEY CONCEPTS

GENERAL
FACTSHEETSANALYSINGQUANTITATIVE
DATA

200

DESIGNING & IMPLEMENTING
THE EVALUATION

PRISMMODULESMETHOD FACTSHEETS& ANNEXES

More advanced data analysis methods
There are a wide range of more advanced analysis methods thatyou can perform on your dataset. A selection of these are outlinedin the table below along with some links to relevant resources.
Most of these analyses require the use of a statistical software
package (e.g. R or SPSS)METHODDESCRIPTIONMORE INFORMATION

Correlation

Determining how strongly two or more variables are related to each other. A positive
correlation (+1.0 to 0) indicates that the variables will either increase or decrease together,while a negative correlation (0 to -1.0) indicates that as one variable increases, the other(s)will decrease.

http://betterevaluation.org/evaluation-options/

Multivariate
regression

Identifying what combinations of variables best predict another variable of interest. For
example, what combinations of household enterprises best predict their overall income
levels.

http://betterevaluation.org/resources/guide/

Decision orclassification
tree

Identifying what combinations of attributes best predict another attribute of interest.
Useful where only binary data are available (e.g. male versus female, group member or not).
For example identifying the simplest predictors of a household’s poverty status.

EvalC3 – an Excel tool for developing decision

Clustering

Identifying any natural groupings in the cases being examined, e.g. households or villages
based on data about their multiple attributes.

http://www.betterevaluation.org/evaluation-

Anomaly / outlier
detection

Finding exceptions to common patterns (such as the main types of households).

http://www.betterevaluation.org/evaluation-T-test

Test for comparing the means (averages) of two samples.

correlation

descriptive_multivariate_statistics

tree models https://evalc3.net/

options/multivariatedescriptive

options/multivariatedescriptive

PRISM - TOOLKIT FOR EVALUATING THE OUTCOMES AND IMPACTS OF SMALL/MEDIUM-SIZED CONSERVATION PROJECTS

http://wise.cgu.edu/wise-tutorials/tutorial-t-test/

4:0

INTRODUCTION &KEY CONCEPTS

DESIGNING & IMPLEMENTING
THE EVALUATION

Acknowledgements
This factsheet is based on the following resources:
Pell Institute Evaluation Toolkit: http://toolkit.pellinstitute.org/
evaluation-guide/analyze/analyze-quantitative-data/

GENERAL
FACTSHEETSANALYSINGQUANTITATIVE
DATA

201

Better Evaluation page: http://www.betterevaluation.org/en/
plan/describe/look_for_patterns

PRISM
MODULESMETHOD FACTSHEETS& ANNEXES

Useful resources for quantitative data analysis
WISE: Web Interface for Statistics Education – Website thatorganises and hosts a large amount of statistics resources
including tutorials, demonstrations and excel templates http://
wise.cgu.edu/
Analysing data in Excel – Useful guidance on how to conduct
simple data entry and analyses using t-tests http://www.centenary.
edu/academics/religion/dotto/rel308/Measure.pdf
Emery Evaluation – Short videos explaining how to perform basic
functions in Excel for interpreting and analysing evaluation data
http://emeryevaluation.com/excel-for-evaluation/

PRISM - TOOLKIT FOR EVALUATING THE OUTCOMES AND IMPACTS OF SMALL/MEDIUM-SIZED CONSERVATION PROJECTS

4:0

INTRODUCTION &KEY CONCEPTS

DESIGNING & IMPLEMENTING
THE EVALUATION

PRISMMODULES

Factsheet: Analysing qualitative data

GENERAL
FACTSHEETSANALYSINGQUALITATIVEDATASTEP 1

Summary
Qualitative data is text or narrative based, not numerical. It isgathered through methods such as questionnaires, interviews,
focus groups and observations. Analysing and reporting qualitative
data requires approaches that are different from those used fornumerical data. The type of analysis to be carried out should beplanned during the design phase of the project, to make sure
you have collected all the correct data you need. This factsheet
describes how to code and analyse qualitative evaluation data.

--

Prior to fieldwork: preparing your metadata
Create a database (also known as an archive) to record
details of the types of data you are collecting, forexample datasheets, audio files or digital records. It
is essential to have such a database to link the resultsof analyses to the raw data and identify where data isstored.
The database should include:
•

202

•

data source details, for example the source (e.g.
interviews, focus groups, social media, books,
websites), date, location, project members whocollected the data.

•

the stages of data processing, to keep trackof analysis phases. For example, record thedates when the following stages happen: “datacollected”, “transcription started”, “transcription
completed” (transcription is when paper/audio/
visual data are converted to an electronic format,
so you will need to include “file names” and “filelocation”), “analysed” (again you may need new “filenames” and “file location”), “reported” (record thereport name and location).

Required steps

Step 1METHOD FACTSHEETS& ANNEXES-

Ensure that data storage meets data protection
requirements. This relates to original data collection
notes and electronic copies of transcribed information.

-

Prepare data collection equipment, e.g. label sheets,
memos, notepads.

-

Code and label any tapes or digital files (if you are goingto make audio and/or visual recordings) with theirunique identification number, and record this in the
database,

data identification details e.g. unique identifying
codes, the format (audio, video, paper etc.), themethod of collection.

PRISM - TOOLKIT FOR EVALUATING THE OUTCOMES AND IMPACTS OF SMALL/MEDIUM-SIZED CONSERVATION PROJECTS

4:0

INTRODUCTION &KEY CONCEPTS

Step 2.

STEPS 2 & 3

After each event or interview, make general notes
about any details you recall that could be relevant orinteresting but may not have been captured by yourdata collection method, for example comments made or
behaviour observed.

−

Prepare memos (short notes, about two lines long,that capture the essence of what you learned) from anactivity based on an initial analysis by the data collector.

−

Gather any materials developed during participatory
activities and make sure they are correctly labelled.

−

Update the database with any changes and record thatthe data have been collected.

−

Make data copies as soon as possible in case anythingshould happen to your raw data (your master data).
This means copying up field notes and data sheets, anddownloading audio/video/picture files. Ensure all copies
are stored in a suitably secure manner.
o

203

PRISM
MODULESo

Immediately after an event/interview−

GENERAL
FACTSHEETSANALYSINGQUALITATIVEDATADESIGNING & IMPLEMENTING
THE EVALUATION

If no access to computers is available, paper files
should be copied into another notebook or folder.
This is especially important if the same notebook
is used for several events before information isrecorded on a computer. In such situations, if theoriginal notebook gets lost, the copied notes willremain and the entire season’s work will not be
lost.

−

METHOD FACTSHEETS& ANNEXES

If intending to delete original files after
downloading them (e.g. to reuse memory cards),
make sure the files have transferred correctly (e.g.
listen to the middle, the end and a couple of otherrandom points within an audio file). Also createa second backup copy of the digital file and store
each of them on different memory drives, so if one
drive becomes corrupted or deleted, there will bea copy of the raw data.

After copying paper files they should be stored in a
separate, secure (dry/safe/locked) location.

Step 3.

Transcribe material−

Audio/video files and paper records should (if notalready done) be fully and completely transcribed into a
digital format to make analysis easier.

−This may done by an assistant or third party not
involved in the research, so that they do not emphasise
particular points or select sections of data fortranscription It is on the other hand also a good way forthe researcher to get to know their data. In some cases
an exact transcription will be made, where all captured
information is included in the electronic version, as close
to the original conversation as possible.

PRISM - TOOLKIT FOR EVALUATING THE OUTCOMES AND IMPACTS OF SMALL/MEDIUM-SIZED CONSERVATION PROJECTS

4:0

INTRODUCTION &KEY CONCEPTS

Step 4.

GENERAL
FACTSHEETSANALYSINGQUALITATIVEDATADESIGNING & IMPLEMENTING
THE EVALUATION

Double-check data−

This is an essential part of the process for both
qualitative and quantitative data so that errors intranscription can be minimised.

−Transcripts of audio files should be checked by listening
to the file while reading the transcript.

−

This can be done by a person who did not enter thedata and, if it is the researcher who is carrying out
the analysis, this is a good opportunity to note any
interesting quotations that can be used later to illustrate
key findings in your Monitoring & Evaluation report.

STEPS 4, 5 & 6
Step 5.

PRISM
MODULES

Step 6.

Analyse contextual and demographic data

It is important to have a good understanding of the participantsfrom whom the data were collected, the tools used to collect thedata, and the local context. This information will help your analysis
and interpretation of the data, and it is particularly important
if your data are collected by different researchers in various
locations.
Demographic data about research participants can be put into atable such as the example shown in Table 1, and then statistics canbe prepared on the age, gender, occupation, education level andother relevant details relating to your research participants. This
information can be related to qualitative analyses, for example it
may reveal that people from particular age groups or occupations
had particular responses to the project intervention.

Review your evaluation questions

Before analysing your data, you should always start by reviewing
your main evaluation questions, i.e. the reason you undertook theevaluation in the first place. This will help you to organise your data
and focus your analysis.

TABLE 1: TEMPLATE FOR COLLECTING DEMOGRAPHIC DATAPARTICIPANT NUMBER

1When reviewing your evaluation questions you should remind
yourself of the theoretical basis for why you thought theintervention was suitable (the details behind your Theory of
Change framework) and what you thought it would achieve (theoutcomes).

204

METHOD FACTSHEETS& ANNEXES

2
3PRISM - TOOLKIT FOR EVALUATING THE OUTCOMES AND IMPACTS OF SMALL/MEDIUM-SIZED CONSERVATION PROJECTSLOCATION

AGE

GENDER

OCCUPATIONEDUCATION

4:0

INTRODUCTION &KEY CONCEPTS

Step 7.

GENERAL
FACTSHEETSANALYSINGQUALITATIVEDATASTEP 7DESIGNING & IMPLEMENTING
THE EVALUATION

Code the data

Coding data is the process of looking for recurring ideas, concepts,
themes, topics, phrases, comments or even single words which
emerge from the data. When the analyst identifies these items acode is given to classify a particular group of ideas. This could bewithin the contributions of a single respondent, for example in an
unstructured interview a person may repeatedly comment on thesame topic, or it could be derived from the contributions of several
respondents. Patterns of codes and emergent themes can then be
identified.
Coding can either be done by stipulating codes in advance (apriori) or letting codes be suggested by the data (groundedtheory). A priori codes can be suggested by the theoretical basis
for the intervention and by what is expected to be affected or tohave changed. For example, if you want to know whether yourintervention has affected hunting rates and whether people havechanged their behaviour, you might code for terms you deem to be
relevant such as “food”, “hunting”, “income”, “traditions” or “habits”.
Alternatively you can be data-led, and code recurring themes inthe data, but for this it is important to set aside any pre-conceived
expectations. It is important to ask yourself what is being
described, what is happening or being said, what is being assumed
by the interviewee and how their demographic status may affecttheir statements.
You are likely to use many codes - some general, and some more
specific. For instance, a general code might be “income” and you
could use it to identify data that are relevant to the subjects’ level

205

PRISM
MODULESMETHOD FACTSHEETS& ANNEXESof education. A more specific code might be “income from hunting”
which you might choose because the data refer to understanding
levels of hunting. You will need to go through your data in detail,coding it according to the types of themes and issues that emerge.
By coding data as “income” or “awareness” you are marking it in a
way that means you can find it and return to it later, knowing that
this particular piece of data is about income or awareness (it could
be about both, in which case you will have applied both codes to it).
In this way the code will help you to identify relevant items of datathat you can pull together later to say something about income
and/or awareness.
When developing a coding scheme, either a priori or particularly
for data defined codes, it is essential to pilot-test the coding
scheme. Test the codes on a subset of the responses, and adjust
the coding categories if necessary. Develop clear definitions
for what does and does not fall into a particular code. This will
ensure that the analysis process is clear to other users. Multiple
analysts should be involved in coding, to reduce the subjectivityof the process, and to ensure that the codes are being appliedconsistently. When the final set of coding categories has been
defined, the full data set is then coded. The first coded data files
should then be re-coded, as a way of testing whether the coding
categories were being used in the same way at the end of thecoding as they were at the beginning. If the re-coding leads todifferent results from the first pass, then a review of the processneeds to be conducted, as this implies that the coding definitions
have changed during the coding process.

PRISM - TOOLKIT FOR EVALUATING THE OUTCOMES AND IMPACTS OF SMALL/MEDIUM-SIZED CONSERVATION PROJECTS

4:0

INTRODUCTION &KEY CONCEPTS

GENERAL
FACTSHEETSANALYSINGQUALITATIVEDATASTEP 8206

DESIGNING & IMPLEMENTING
THE EVALUATION

Coding helps to work out what the data are telling you andthe relationships and patterns in your data. As your researchdevelops you may define many codes, building up an increasingly
detailed understanding of the data. If you are working as a teamon the process of coding, it is important to develop a sharedunderstanding and agreement on the names of the codes and what
they mean. The codes you develop are likely to change as yourresearch proceeds and as you develop new understandings of the
topics you are researching.
−

Carefully read through your records and makecomments about the key patterns, themes and issues inthe data.

−

Highlight similar data in the same colour and give eachcolour-group a descriptive code that suggests what it isreferring to.

−

As you read more records, different codes may suggest
themselves. It will then be necessary to re-read earlier
records to compare things to see whether relevant
statements have been missed or previous codes should
be renamed. Coding data requires detailed analysis ofdata and constant comparisons of text, so each record
may need to be read several times.

−

Keep a list of your codes and their definitions. During
the coding process, codes may be added/deleted andtheir definitions refined. Notes should be kept on why
codes are being changed and on your thoughts andideas about the coding.

PRISM
MODULES

Step 8.

METHOD FACTSHEETS& ANNEXES

Identify and summarise codes

Once you have an initial list of codes, and agreed definitions ofthese codes, you can begin organising your data into hierarchies,
with similar codes grouped under a broad title. For example,
negative and positive comments about the project could comeunder “feedback on project”, while people’s ideas about how the
project may change their lives in the future could come under“perceived project impacts”, and suggestions for changes toproject logistics, financing, resources and scope etc. could comeunder “suggested improvements”. These headings can be usedlater when you prepare a report on your findings. Next, you cansummarise the main themes, drawing on contextual data and otherinformation that can help you to better understand your findings.
Note how frequently each pattern or theme occurs and who talked
about this theme, for example people of a particular gender, agegroup or occupation. It is usually best to look for the range ofviews expressed by participants rather than trying to quantify the
responses. However, if your data clearly show which participants
made particular comments, some quantification of responses may
be appropriate.

PRISM - TOOLKIT FOR EVALUATING THE OUTCOMES AND IMPACTS OF SMALL/MEDIUM-SIZED CONSERVATION PROJECTS

4:0

INTRODUCTION &KEY CONCEPTS

Step 9.

GENERAL
FACTSHEETSANALYSINGQUALITATIVEDATASTEP 9DESIGNING & IMPLEMENTING
THE EVALUATIONInterpret findings, assess contributions tooutcomes and impacts, and summarise

This involves putting your data into a wider perspective. Thatmeans comparing your results with what you expected, andwith your original project objectives, indicators and researchquestions. You would then summarise your main findings, underbroad headings such as “Listening Patterns” and “Project Impacts”.
Interpretation goes beyond description. It means attachingsignificance to what your data are telling you, making sense of yourfindings, offering explanations, drawing conclusions and lessons
learnt, and imposing order onto a complex and messy picture. Yourfindings should:
1.

be led by the data, not by preconceived ideas or
assumptions;2.

get rid of any misconceptions, if this is possible;

3.

illuminate important things that you didn’t know but
should know.

You also need to consider the extent to which your project may
have contributed to the impacts identified, based on the evidence
that you have collected and analysed. This means understanding
how various aspects of your project work in combination withother activities to produce certain outcomes and impacts.

PRISM
MODULESfindings are limited to these people in this place. Gradually, overtime, you can increase the scope of your research, talk to more
people from different backgrounds and in different locations. The
important point is to be aware of the limitations of the research
you have done, and to make this clear to readers of your reports.
The next step is to investigate issues of interest further.
It is always best to take a cautious approach to the interpretationof your data. This means avoiding leaping to conclusions or making
assumptions about why something happened. It is always useful
to discuss your interpretations at length with others. Learning
how to produce a good interpretation of qualitative data can bechallenging and is likely to take some time. However, it is worth
persevering with, as an effective and rigorous interpretation canreally improve your project reports. Seeking feedback and support
from others with more experience in this area should help you toimprove more quickly.

Acknowledgements
This factsheet is based on the following resources:
Equal Access Participatory M&E toolkit: http://www.
betterevaluation.org/toolkits/equal_access_participatory_
monitoring .

You may start with very limited data, which only tell you abouta certain group of people in a certain place. This is fine as abeginning, as long as you make it clear in your reports that the207

METHOD FACTSHEETS& ANNEXESPRISM - TOOLKIT FOR EVALUATING THE OUTCOMES AND IMPACTS OF SMALL/MEDIUM-SIZED CONSERVATION PROJECTS

4:0

INTRODUCTION &KEY CONCEPTS

DESIGNING & IMPLEMENTING
THE EVALUATION

Factsheet: Evaluation report template

GENERAL
FACTSHEETSEVALUATIONREPORTTEMPLATE

After you have decided who you report to and in what format, you
should create a detailed evaluation report that addresses all of
your evaluation questions. You can then take the detailed reportand extract summary information for relevant audiences andadapt the report into the most appropriate format(s).
Your detailed report should include the following section headings
and content:
Cover page and title
Your title should be descriptive of your project, and you may want
to relate it to your project’s overall goals.
An appropriate image will add visual appeal to your report.
Executive summary
This is summary of the main findings, lessons andrecommendations from your evaluation. Some people, depending
on how busy they are, will only read the executive summary. Itshould generally not be longer than two pages.
Introduction

208•

Overview of the project and its goals.

•

Key stakeholders and target audience.

•

Intervention logic.

PRISM
MODULESMETHOD FACTSHEETS& ANNEXES

This should include an overview of the project that is being
evaluated, including the timeframe, main stakeholders, and project
goals. It is good to provide an intervention logic that outlines what
you sought to achieve and what you did along the way. You may
want to describe in greater detail particular activities that werecritical in delivering outcomes.
Evaluation framework•

Purpose of the evaluation.

•

Key evaluation questions.

•

Evaluation team.

•Evaluation method (including limitations).

You should also outline the purpose of the evaluation, including
the evaluation audience and what they want to know. This includes
highlighting the key evaluation questions. You may want to include
the full monitoring and evaluation (M&E) plan as an appendix. It is
important to note who made up the evaluation team. You should
also provide an overview of the evaluation method (you can link
this back to the M&E plan in the appendix) and any limitations
applying to the methodology. You may want to use a table thathighlights the quantitative and qualitative methods used in theevaluation.

PRISM - TOOLKIT FOR EVALUATING THE OUTCOMES AND IMPACTS OF SMALL/MEDIUM-SIZED CONSERVATION PROJECTS

4:0

INTRODUCTION &KEY CONCEPTS

DESIGNING & IMPLEMENTING
THE EVALUATION

Evaluation findings

GENERAL
FACTSHEETSEVALUATIONREPORTTEMPLATE•

Evaluation question 1.

•

Evaluation question 2.

•

Evaluation question 3 etc.

A good way to present your evaluation findings is to use yourkey evaluation questions as the main sub-headings (e.g. “was thedelivery model effective in changing participants’ behaviours?”).
You would then use the information collected through your projectto make a judgement and answer the key question. Rememberthat you do not want simply to present information, but rather
to interpret the information and make a value judgement. Usegraphics where appropriate, and remember that you do not haveto present all the information you have collected. You may present
some of the information from your monitoring in an appendix.

PRISM
MODULES

References
Provide details of any other publications or sources of informationthat you have used in your report.
AppendicesThis is where you provide detailed information that some of youraudience members may want to refer to. This includes your full
M&E plan, questionnaires that were used, detailed results andinformation, statistical analyses, etc.
Acknowledgement
Adapted from Evaluation Toolbox: Detailed Evaluation Report:
http://evaluationtoolbox.net.au/index.php?option=com_content&
view=article&id=48&Itemid=60.

Conclusion and recommendations
This is where you may want to do a high-level summary of thesuccesses and lessons of your project based on your evaluation
findings. You may also want to communicate how the evaluation
findings will be used (in terms of information for future projects,
changes in policy, etc.). You should also make a list of key
recommendations (which are also presented in the executive
summary).

209

METHOD FACTSHEETS& ANNEXESPRISM - TOOLKIT FOR EVALUATING THE OUTCOMES AND IMPACTS OF SMALL/MEDIUM-SIZED CONSERVATION PROJECTS

4:0

INTRODUCTION &KEY CONCEPTS

DESIGNING & IMPLEMENTING
THE EVALUATION

PRISMMODULESMETHOD FACTSHEETS& ANNEXES

Factsheet: Evaluating outcomes and impacts of projects aiming
to maintain or increase the values of ecosystem services
Include costs ofintervention

GENERAL
FACTSHEETSEVALUATINGOUTCOMES
AND IMPACTS
OF PROJECTSAIMING TO
MAINTAIN OR
INCREASE THE
VALUES OFECOSYSTEMSERVICES

210

Some conservation projects include actions that aim to maintainor improve the delivery of ecosystem services (the benefits thatpeople derive from nature) as part of a strategy to conserve
species and/or habitats more effectively. Evaluation of theoutcomes and impacts of conservation projects on ecosystemservices is not covered by this manual because an entire toolkit
exists with guidance on how changes in ecosystem services at asite can be measured.
The Toolkit for Ecosystem Service Site-based Assessment (TESSA):
http://www.birdlife.org/worldwide/science/assessing-ecosystemservices-tessa can be used to compare the values of ecosystemservices (in either biophysical or economic units) for a site under
two scenarios: (a) with the project and (b) without the project (seeFigure right).

‘withproject’

Quantify andvalue ecosystem
sevices

CONSERVATIONINTERVENTION

NET ECONOMIC
VALUE

Subtract
e.g. value of homes
and land protectedfrom ﬂooding

e.g. restorationof wetland
habitat‘without
project’

Quantify andvalue ecosystem
sevices

For example, a project may carry out activities that aim to restore
a wetland site for migratory birds. As part of this project, therestoration may also protect people’s homes and farmland fromflooding. The impact of the intervention on flood protection
value can be demonstrated by measuring the actual impact of
the created wetland at the end of the project, compared with thevalue without the project (i.e. if the intervention had not takenplace). The value of that impact can be demonstrated in monetary($) or non-monetary terms (e.g. number of houses protected fromflooding; area of land protected from flooding).

PRISM - TOOLKIT FOR EVALUATING THE OUTCOMES AND IMPACTS OF SMALL/MEDIUM-SIZED CONSERVATION PROJECTS

4:0

INTRODUCTION &KEY CONCEPTS

GENERAL
FACTSHEETSEVALUATINGOUTCOMES
AND IMPACTS
OF PROJECTSAIMING TO
MAINTAIN OR
INCREASE THE
VALUES OFECOSYSTEMSERVICES211

DESIGNING & IMPLEMENTING
THE EVALUATION

PRISMMODULESThis requires identification of the “without project” comparison.
In TESSA this is done by using a counterfactual approach whereby
data on ecosystem services are collected from a similar site (interms of environmental and social context) and extrapolated to theproject site to provide alternative values from those obtained as aresult of the project. It may also be applicable to use a before andafter comparison if the state before the project is representativeof what would happen without the project. Other approaches
outlined in Step 2 could be used for the “without project”
comparison provided the assumptions are made explicit.

Livelihoods & Governance module: This module in PRISM
could be used to measure changes in some ecosystem services –specifically the benefits obtained from harvested wild goods (e.g.
fuelwood, medicinal plants, fish) or cultivated goods (e.g. rattan,
coffee) in relation to livelihoods. For example, see Participatory
Impact Assessment Method 1 - Before and after scoring for
guidance on how changes in income might be measured.

PRISM - TOOLKIT FOR EVALUATING THE OUTCOMES AND IMPACTS OF SMALL/MEDIUM-SIZED CONSERVATION PROJECTSMETHOD FACTSHEETS& ANNEXES

4:0

INTRODUCTION &KEY CONCEPTS

DESIGNING & IMPLEMENTING
THE EVALUATION

Factsheet: Questionnaires

GENERALEVALUATIONMETHOD
FACTSHEETSQUESTIONNAIRES

Summary
Questionnaires are one of the most commonly applied methods
for gathering data from project participants. They can be more
complex and time consuming to produce than is anticipated.
However, thorough planning and piloting, ensuring questions
relate to the aims of the evaluation, having a clearly identified
strategy for securing responses and good preparation for data
analysis can ensure that questionnaires are extremely valuable
tools for action evaluation.
Questionnaire surveys can be carried out by telephone, post,or in person, and they have the potential to record informationfrom a large sample of respondents. Interviews can be i) fully
structured, using set questions that the interviewer sticks to, or ii)
semi-structured, using open-ended questions with the interviewer
following up interesting responses with supplementary questions,or iii) respondent-led conversations, with the interviewer having
prepared some areas for discussion but allowing the conversation
to unfold as the respondent wishes. The choice of which of thesevariants to use depends on the type and aim of the intervention
and the resources available. For example if an intervention is
focused on influencing the attitudes of a small number of herders
in the mountains, it is not worthwhile to use a postal surveyof the entire village to evaluate effectiveness. Conversely if
the intervention is trying to raise awareness of a conservation
threat throughout a community, in-depth and time-consuminginterviewing of a very few select people is unlikely to indicate
whether the message has been received across the community.

212

PRISM
MODULESMETHOD FACTSHEETS& ANNEXES

Some sub-sampling will often be necessary because of time andmoney restrictions, and there is the possibility that key community
members may be aware of attitudes in the wider community.
Main questionnaire survey types
Pre–post questionnaires: Surveys that require participants tocomplete two questionnaires, one before the project and one after
the project.
Retrospective pre-post questionnaires: Surveys that requireparticipants to complete a single questionnaire after the project
action which asks them to compare their answers with thesituation before the project.
Post activity questionnaires: Questionnaires comprising a setof questions asking participants to rate the effectiveness of aparticular activity (e.g. a training workshop, a presentation, acommunity engagement event).
Post project questionnaires: Questionnaires that ask participants
to self-report on changes that have occurred as a result of the
project. This is similar to a retrospective pre-post questionnaire
except that it does not ask participants to rate their knowledge/
attitudes/behaviours before the project.

PRISM - TOOLKIT FOR EVALUATING THE OUTCOMES AND IMPACTS OF SMALL/MEDIUM-SIZED CONSERVATION PROJECTS

4:0

INTRODUCTION &KEY CONCEPTS

DESIGNING & IMPLEMENTING
THE EVALUATION

Required steps
Step 1.

GENERALEVALUATIONMETHOD
FACTSHEETSQUESTIONNAIRES

Determine the purpose

Before creating a questionnaire, start by answering a few
important questions:•

What do I need to know?

•

Why do I need to know it?

•

What will happen as a result of this questionnaire?

•

Can I get the information from existing sources insteadof conducting a survey?

STEPS 1 & 2••

Are there any ethical issues associated with the datacollection?
Is the topic covering sensitive issues e.g illegal wildlife
trade, culturally sensitive topics, etc? If so, this will affectthe methods used.

It is a good idea to start with your evaluation objectives andquestions. Here are some examples:
Evaluation objectives:
•

•

213

To determine the level of bushmeat consumption inhouseholds in the project area and in control area(s).
To determine the average number of patrols carried out
by rangers before and after a training programme.

PRISM
MODULES•METHOD FACTSHEETS& ANNEXESTo determine the average cost per meal in thehouseholds of project participants.

Evaluation questions:•

How many households in the community hunt on a
weekly basis?

•

What new skills do project participants possess after a
training course?

•

What is the monthly income of households involved in
the project?

Step 2.

Decide what you are measuring

As with determining the purpose, this should be based on theobjectives of your action and the evaluation of its outcomes andimpact. Consider which of the following you are aiming to measure:•

Attitude•

Knowledge•

Skills•Goals, intentions, aspirations•

Behaviours and practices•

Perceptions of knowledge, skills, or behaviour.

PRISM - TOOLKIT FOR EVALUATING THE OUTCOMES AND IMPACTS OF SMALL/MEDIUM-SIZED CONSERVATION PROJECTS

4:0

INTRODUCTION &KEY CONCEPTS

Step 3.

GENERALEVALUATIONMETHOD
FACTSHEETS•QUESTIONNAIRES

214

Decide who should be asked

The population you sample from, as well as the sampling method
you use, affects how your research findings can be generalised. In
other words, for whom do the results apply? This is an indication of
the external validity of the study.

•

STEP 3

DESIGNING & IMPLEMENTING
THE EVALUATION•

What is the appropriate population (group of people/
subjects) to be studied or questioned?
A population is the complete set of subjects that can bestudied: people, objects, animals, plants, etc.
A sample is a sub-set of subjects that can be studied
to make the evaluation/research project moremanageable.

PRISM
MODULESMETHOD FACTSHEETS& ANNEXES

Once a target audience is defined, the questionnaire development
may be influenced by features of that audience, for example:•

the age of respondents, particularly if working withchildren;•familiarity with tests and questionnaires;•

potential cultural or language barriers;•

ethical considerations for engaging with this group ofparticipants;•

gender, including restrictions and hierarchies. For
example, can you interview people of the opposite
sex, can you interview people that are not head of thehousehold (power dynamics), etc? How can responses
be gathered from disadvantaged voices (e.g. women)?

PRISM - TOOLKIT FOR EVALUATING THE OUTCOMES AND IMPACTS OF SMALL/MEDIUM-SIZED CONSERVATION PROJECTS

4:0

INTRODUCTION &KEY CONCEPTS

Step 4.

GENERALEVALUATIONMETHOD
FACTSHEETS

Understand the audience

It is important to know about the target audience for thequestionnaire. This will inform the decisions made at Step 5 about
how to collect responses. Factors which may help to define theaudience for the questionnaire include:•

QUESTIONNAIRES•STEPS 4 & 5

•

•

•

215

DESIGNING & IMPLEMENTING
THE EVALUATION

The likelihood of interaction with the people the projectis aiming to influence, e.g. do you want to question
people who attend a certain event, use a particular
habitat or location, or interact with a target species?
A representative public sample requires a broad rangeof respondents, and data from a census could be usefulin assessing whether the sample really represents the
population. For example, if all hunters are young males,
their attitudes may not represent those of the older or
female members of the population.
Employment may be important, for example selecting
people working in fisheries to monitor relevant
behavioural responses.
Hobbies – some questionnaires will target people whohave connections with a particular subject through theway they spend their free time, rather than their paid
employment.
Finding out when are people most easily engaged. For
example trying to sample parents with young children
in the evening may mean they are distracted by family
responsibilities; in which case weekends may be a better
time.

PRISM
MODULES

Step 5.

METHOD FACTSHEETS& ANNEXES

Choose an appropriate means of data collection

The sampling method can then be decided, depending on the
population sample selected:•

Should a census or sampling be used?

•

If a large enough random sample is taken, the results
can be statistically similar to taking a census of an entire
population, yet involving less effort and cost.

There is a variety of ways in which samples can be chosen from a
population:•

Simple random (e.g. pulling names from a hat).

•

Systematic random (e.g. every fifth name – but
be careful that the system you choose does notaccidentally match a pattern in the population being
sampled).

•Stratified random (separate samples for each subgroup).

•

Cluster sampling (treating intact groups that cannot
be broken up, such as classrooms, as subjects to besampled).

Whatever population is studied and whichever sampling method
is used, it is important to cover a sufficiently high percentage of
respondents to ensure that the sample is truly representativeof the population being studied. Non-response error affects thevalidity of the study, and a plan for recording non-responses thendealing with them, should be determined in advance.

PRISM - TOOLKIT FOR EVALUATING THE OUTCOMES AND IMPACTS OF SMALL/MEDIUM-SIZED CONSERVATION PROJECTS

4:0

INTRODUCTION &KEY CONCEPTS

GENERALEVALUATIONMETHOD
FACTSHEETSQUESTIONNAIRESSTEPS 5 & 6

216

DESIGNING & IMPLEMENTING
THE EVALUATIONOther possible errors can be avoided with simple procedures:“sampling error” is reduced by using a large, random sample orconducting a census; “frame error” is minimised by making surethat the list of potential subjects is current and accurate; and“selection error” is avoided by eliminating duplication from theselists.
Two further elements of data collection need to be considered:i) what format the questionnaire will take and ii) how it will
be distributed. Formats can include personal (face-to-face)interviews, online questionnaires, telephone interviews and mailed
questionnaires. Consider the literacy level of the respondents
when choosing a format for data collection, and ensure that
intended respondents are not excluded by the format you
choose. Distributing a questionnaire is a vital link in ensuring thatdata is collected. The channels through which a questionnaire
or interview is made available could skew the types of people
surveyed.

PRISM
MODULES

Step 6.

METHOD FACTSHEETS& ANNEXES

Establish Free, Prior and Informed Consent (FPIC)This lets the respondents know they are participating in astudy and how their information will be treated. “Passive”
consent is assumed if the respondent completes and returns thequestionnaire, but contact details should be provided to give therespondent the option of withdrawing their data after submission.
“Active” consent requires a participant to give formal approval (e.g.
by signing a form declaring that the respondent explicitly agrees
to participate, or allows his or her minor child to take part). See
Factsheet: Developing ethical protocols for an evaluation for more
information on FPIC.

PRISM - TOOLKIT FOR EVALUATING THE OUTCOMES AND IMPACTS OF SMALL/MEDIUM-SIZED CONSERVATION PROJECTS

4:0

INTRODUCTION &KEY CONCEPTS

Step 7.

DESIGNING & IMPLEMENTING
THE EVALUATION

Choose a collection procedure: confidential vs.
anonymous

Confidential

GENERALEVALUATIONMETHOD
FACTSHEETS217

METHOD FACTSHEETS& ANNEXES

Anonymous•

Names and other identifiers are not asked of
respondents.

•

As no other identifying codes are used, the researcher
is unable to follow up with non-respondents or to matchdata from pre-action with data from post-action. This
may not be a problem when doing random interviews(such as exit surveys).

•

Names or other identifiers collected in thequestionnaire or interview can be used for follow-up
data collection or to match data from pre-action withdata from post-action.

•

Individual data are not shared with anyone. Information
is not used for any purpose other than that stated in the
project.

•

Collecting basic descriptive information about
respondents is still useful for comparing respondents
with the population.

•

Confidentiality is never breached. This pledge is crucial
for data protection purposes and for obtaining consent
and ethical approval; in addition to obtaining honest,
complete answers from respondents.

••

Identifying information is anonymised for storage andanalysis purposes (essential for anything that could in
any way negatively impact the interviewee). Information
should be destroyed after the survey is completed or, if
the information needs to be stored for publication, thisshould be held on an encrypted hard drive and deleted
once the publication is completed.

One possible way of maintaining anonymity while also
keeping track of non-respondents is to send a separate
post-card along with the questionnaire. The respondent
can return this card separately, enabling him or her
to declare that “John/Mary Doe has returned thequestionnaire”.

QUESTIONNAIRESSTEP 7PRISM
MODULESPRISM - TOOLKIT FOR EVALUATING THE OUTCOMES AND IMPACTS OF SMALL/MEDIUM-SIZED CONSERVATION PROJECTS

4:0

INTRODUCTION &KEY CONCEPTS

Step 8.

DESIGNING & IMPLEMENTING
THE EVALUATION

Choose measurement scale and scoring

Use scales that provide the information needed and areappropriate for respondents. Some options include:

GENERALEVALUATIONMETHOD
FACTSHEETSQUESTIONNAIRESSTEP 8PRISM
MODULES

Feeling about a topic – e.g. Did you find the training workshop:
Boring (1-5), Interesting (1-5), Unlikely (1-5), Likely (1-5), Risky (1-5),
Safe (1-5), Important (1-5), Unimportant (1-5), Difficult (1-5), Easy
(1-5).
Open questions – qualitative data

Closed questions – quantitative data
A yes/no answer – e.g. Did you attend the workshop? Yes/No.
Agree/disagree with a statement – e.g. Do you agree or disagreewith the following statement? Seabird populations on island X are
healthy. Agree/Disagree.
Choose from a list of options – e.g. Which of the following
resources is most important to your household? Rattan(cultivated), rattan (wild harvested), banana, rice, pineapple,
cassava, timber, other forest products (e.g. honey, medicinal plants,
mushrooms).

A statement – e.g. How has your relationship with park authorities
changed since the project began?
List – e.g. Please list the main reasons why you visit the forest.
See the following factsheets for some further examples ofquestions and scales that can be used as part of a questionnaire
survey:Sample questions and scales for evaluating attitudes andawareness actions
Training evaluation form: example questions and scales

Rank order – e.g. From the following list choose the THREE which
are the most important to your household: 1 = most important, 2= second most often important 3 = third most important. Rattan(cultivated), rattan (wild harvested), banana, rice, pineapple,
cassava, timber, other forest products (e.g. honey, medicinal plants,
mushrooms).
Degree of agreement and disagreement: the Likert Scale – e.g.
The training provided skills that I did not have before. Strongly
agree, Agree, Neutral, Disagree, Strongly disagree, neither agree
or disagree.
Rate Items – How useful would you rate the training element of theworkshop? Not useful at all – Very useful: 1-7.
218

METHOD FACTSHEETS& ANNEXESPRISM - TOOLKIT FOR EVALUATING THE OUTCOMES AND IMPACTS OF SMALL/MEDIUM-SIZED CONSERVATION PROJECTS

4:0

INTRODUCTION &KEY CONCEPTS

Step 9.
•GENERALEVALUATIONMETHOD
FACTSHEETS

A cover letter or verbal introduction to the project
should include:

To ensure that the survey instrument you develop is appropriate
for your audience, “field test” your questionnaire with other people
similar to your respondents before administering the final version.
This will allow you to improve any unclear questions or procedures
and detect any errors. Following recommendations in this Toolkit
concerning questionnaire design and wording of questions will
reduce systematic measurement error, and this will improve thereliability of your study.

oa brief statement of the purpose of the study andits usefulness (one sentence or phrase);othe identity of any sponsor of the study;oa promise of confidentiality and an explanationof identification methods (e.g. code number onquestionnaire);
free, prior and informed consent - the cover lettershould include all of the information needed togain FPIC, which is required for most research
involving human subjects.

oa deadline for returning the survey;oadvice on what to do if questions arise or if the
respondent wishes to withdraw after submitting
their response;o219

METHOD FACTSHEETS& ANNEXES

Step 10. Pilot test

o

•

PRISM
MODULES

Provide a cover letter and instructions

QUESTIONNAIRESSTEPS 9 & 10

DESIGNING & IMPLEMENTING
THE EVALUATION

thanks to the respondents.

Include simple instructions on how to complete eachsection (e.g. how to mark answers: pen/pencil, circle,
check, etc.).

Tips for designing and implementing a good questionnaire
Use plain language•
Be direct and unambiguous.
•

Use the simplest language necessary.

•

Avoid jargon and acronyms.

•

Include definitions if needed.

Be brief•

Keep the questionnaire as short as possible (without
jeopardising reliability).

•

Focus on “need to know” questions and minimise “niceto know” information.

•

Put the most important questions up front (respondents
may get fatigued or hurried by later questions).

•

Include questions about demographic information at
the end, so the questionnaire is focused on the topic at
hand.

PRISM - TOOLKIT FOR EVALUATING THE OUTCOMES AND IMPACTS OF SMALL/MEDIUM-SIZED CONSERVATION PROJECTS

4:0

INTRODUCTION &KEY CONCEPTS

DESIGNING & IMPLEMENTING
THE EVALUATION

Make sure questions match the measurement scale selected,
and that the answer categories are precise•
Make sure answer choices correspond to the questions,both in substance and syntax.

GENERALEVALUATIONMETHOD
FACTSHEETSQUESTIONNAIRES••STEP 10•220

Be consistent in arranging the answers. While it isconventional to read English from left to right, and togo from “low” to “high,” the most important rule is toexplain clearly the “rule” being used in this respect, andto apply it consistently throughout the questionnaire.
Define timeframes when necessary. Instead of“recently,” specify e.g. “during August 2015”. When
doing this it is helpful to give the respondent something
to relate to (e.g. last monsoon).
Make sure answer categories do not overlap e.g. having
age categories for “20-30” and “30-40” means thatsomeone who is 30 could tick either box.

•

If you are using a continuum scale with numbers to
represent concepts, make sure to “anchor” at least the
top and bottom of the scale with terms that describe themeanings of the numbers (e.g. 1 = low, 10 = high).

•

Balance the “negative” or “low” answer choices (both innumber and degree) with “positive” or “high” choices on
the scale. For example, don’t give only positive answer
choices or five degrees of “positive” (e.g. great, excellent,
super, fantastic, awesome) and only one extreme“negative” response choice (e.g. terrible). Be aware, if
0 is used in the middle of the scale people will centre
there, so a scale of -2 to +2 is different from 1 to 5.

PRISM
MODULESMETHOD FACTSHEETS& ANNEXES•

An even number of answer choices doesn’t give therespondent an easy, “middle” choice. If you want to offera “neutral” or “no opinion” choice, then do it by design,
not by accident.

•

Determine in advance how questions will be scored,
what to do with missing data, incomplete or unclear
responses, etc.

Ask only one question at a time
Do not use “double-barrelled” questions that may confuse therespondent into not knowing how to answer.
Consider the confusion created by these examples:•
Do you hunt pigs and monkeys?
•

Do you grow coffee and rice?

•

Have you seen any logging and hunting in the nationalpark?

PRISM - TOOLKIT FOR EVALUATING THE OUTCOMES AND IMPACTS OF SMALL/MEDIUM-SIZED CONSERVATION PROJECTS

4:0

INTRODUCTION &KEY CONCEPTS

DESIGNING & IMPLEMENTING
THE EVALUATION

Avoid loaded or leading questions
Minimise bias or misinterpretation in questions by consulting
experts and pilot-testing the questions with a sample ofparticipants.

GENERALEVALUATIONMETHOD
FACTSHEETS

Examples of loaded questions:o
Do you treat your environment with kindness likea good person should?

QUESTIONNAIRES

Examples of leading questions:o
Do you agree that illegal forest clearance is a badthing?

STEP 10o

Do you feel you get enough support from theuseless government?

Arrange questions in a logical order•Group similar questions together (e.g. by topic orscoring method).

•

Number each question.

PRISM
MODULES

Other resources:
Survey Monkey: https://www.surveymonkey.co.uk/ – online toolfor developing, distributing and analysing questionnaire surveys.
Acknowledgements:
Guidance in this factsheet is based on the following resources:
RCRE factsheet: A step-by-step guide to developing effective
questionnaires and survey procedures for program evaluation& research - http://cahnrs.wsu.edu/fs/wp-content/uploads/
sites/4/2015/09/A-Step-By-Step-Guide-to-Developing-EffectiveQuestionnaires.pdf .
Community Sustainability Engagement Evaluation Toolbox:
Questionnaires - http://evaluationtoolbox.net.au/index.
php?option=com_content&view=article&id=58&Itemid=154 .

Provide space to tell more•
Give respondents room to comment on individualquestions or on the survey as a whole.
•

Ask for additional comments or suggestions.

Check readability

221•

Proof-read text before distributing it to participants.

•

Test readability in the pilot-testing step.

METHOD FACTSHEETS& ANNEXESPRISM - TOOLKIT FOR EVALUATING THE OUTCOMES AND IMPACTS OF SMALL/MEDIUM-SIZED CONSERVATION PROJECTS

4:0

INTRODUCTION &KEY CONCEPTS

DESIGNING & IMPLEMENTING
THE EVALUATION

Factsheet: Key informant interviews

GENERALEVALUATIONMETHOD
FACTSHEETSKEY INFORMANTINTERVIEWS

Summary
A key informant interview is a conversation between aninterviewer and a person who can provide a well-informed pictureof the issue being evaluated - often a leader in his or her field orcommunity. Key informant interviews follow a script or guide,
which may include prompts, called probing questions, that delve
more deeply into an issue or ensure that informants answer
questions in a way that is useful to the evaluation. Questions are
open-ended and often require that the informants talk about theirperceptions, experiences, and beliefs.

PRISM
MODULESMETHOD FACTSHEETS& ANNEXES

What are key informant interviews useful for?
Key informant interviews are an affordable way to understand acommunity or an issue in a deep, nuanced way. They are flexible
and allow new ideas and issues to emerge. They are also often
relatively quick and affordable. On the other hand, there is thepotential for an interviewer to influence informants’ responses.
Because key informant interviews can generate a large amount ofqualitative data - data that must be reviewed systematically so as
to maintain the findings’ credibility - key informant interviews canbe time-consuming and costly to analyse. Finally, key informants
must be selected carefully so that they accurately represent thedifferent viewpoints that may exist in relation to a project.

Interviews may be recorded (with participants’ permission) andtranscribed for later analysis. Where recording and transcription
are not practical or acceptable, someone can take detailed notes.
Key informant-interview data are usually qualitative. However,some quantitative data may be collected, as well; for example, thekey informant’s gender, age, and location or their estimate of thesize of a community population.

Photo credit: David Thomas

222

PRISM - TOOLKIT FOR EVALUATING THE OUTCOMES AND IMPACTS OF SMALL/MEDIUM-SIZED CONSERVATION PROJECTS

4:0

INTRODUCTION &KEY CONCEPTS

GENERALEVALUATIONMETHOD
FACTSHEETSThis method is useful in all phases of the project cycle identification, planning, implementation, and evaluation. For
example, it can provide information on the setting for a planned
activity that might influence project design; or it could reveal why
intended beneficiaries aren’t using services offered by a project.
Specifically, it is useful in the following situations:•when qualitative, descriptive information is sufficientfor decision-making;•when there is a need to understand motivation,
behaviour, and perspectives of customers andpartners. In-depth interviews of programme
planners and managers, service providers, hostgovernment officials and beneficiaries concerning theirattitudes and behaviours about an activity can help
to explain its successes and shortcomings.
•
when a main purpose is to generate recommendations.
Key informants can help to formulate recommendations
that can improve a programme’s performance.

KEY INFORMANTINTERVIEWS

223

DESIGNING & IMPLEMENTING
THE EVALUATION•

when quantitative data collected through othermethods need to be interpreted. Key informant
interviews can illuminate the “how” and “why” of what
happened: for example if a sample survey showed that
farmers were failing to make loan repayments, key
informant interviews could uncover the reasons for this.

•when preliminary information is needed to design
a comprehensive quantitative study. Key informant
interviews can help to frame the issues before thesurvey is undertaken.

PRISM
MODULESMETHOD FACTSHEETS& ANNEXES

Advantages:•
Provides information directly from knowledgeable
people.
•Provides flexibility to explore new ideas and issues not
anticipated during planning.

•

Relatively inexpensive and simple to conduct.

Limitations:•Not appropriate if quantitative data are needed.
•

May be biased if informants are not carefully selected.

•

Can be susceptible to interviewer biases.

•

May be difficult to prove validity of findings.

PRISM - TOOLKIT FOR EVALUATING THE OUTCOMES AND IMPACTS OF SMALL/MEDIUM-SIZED CONSERVATION PROJECTS

4:0

INTRODUCTION &KEY CONCEPTS

DESIGNING & IMPLEMENTING
THE EVALUATION

Required steps
Step 1.

GENERALEVALUATIONMETHOD
FACTSHEETSFormulate study questions

These relate to specific concerns of the study. Study questions
generally should be limited to five or fewer.

Step 2.

PRISM
MODULESbeneficiaries. It is best to include all major stakeholders so that adiversity of interests and perceptions can be captured. Second,
select a few people from each category after consulting withpeople familiar with the groups under consideration. Snowball
sampling can also be used, where interviewed individuals are
asked at the end of the interview for their suggestions for the next
person you can interview about the same subject.

Prepare a short interview guide
Step 4.

KEY INFORMANTINTERVIEWSSTEPS 1 TO 4

Key informant interviews do not use rigid questionnaires, sincethese inhibit free discussion. However, interviewers must have an
idea of what questions to ask. The guide should list major topics
and issues to be covered under each study question. Because
the purpose is to explore a few issues in depth, guides are usuallylimited to 12 items. Different guides may be necessary forinterviewing different groups of informants.

Step 3.

Establish Free, Prior & Informed Consent (FPIC)This lets the informants know they are participating in a study
and how their information will be treated. Establishing FPIC
requires a participant to give formal approval (e.g. by signing a form
declaring that the informant explicitly agrees to participate). See
Factsheet: Developing ethical protocols for an evaluation for more
information on FPIC.

Select key informants

The number should not be too large. It is preferable to start with asmall number, since often more people end up being interviewed
than is initially planned. Key informants should be selected
for their specialised knowledge and insightful perspectives
on a topic. Planners should take care to select informants that
represent range of different points of view. Selection consistsof two tasks. First, identify the groups and organisations fromwhich key informants should be drawn - for example, hostgovernment agencies, project implementing agencies, contractors,

224

METHOD FACTSHEETS& ANNEXESPRISM - TOOLKIT FOR EVALUATING THE OUTCOMES AND IMPACTS OF SMALL/MEDIUM-SIZED CONSERVATION PROJECTS

4:0

INTRODUCTION &KEY CONCEPTS

Step 5.

GENERALEVALUATIONMETHOD
FACTSHEETSKEY INFORMANTINTERVIEWSSTEPS 5, 6 & 7

DESIGNING & IMPLEMENTING
THE EVALUATION

Conduct interviews

Establish rapport. Begin with an explanation of the purpose of theinterview, the intended uses of the information and assurancesof confidentiality. Often informants will want assurances that theinterview has been approved by relevant officials. Except wheninterviewing technical experts, questioners should avoid jargon.
Sequence the questions. Start with factual questions. Questions
requiring opinions and judgments should follow. In general, begin
with the present and move to questions about the past or future.
Phrase questions carefully to elicit detailed information. Avoid
questions that can be answered by a simple yes or no. For example,
questions such as “Please tell me about the outreach campaign” arebetter than “Do you know about the outreach campaign?”
Use probing techniques. Encourage informants to detail the basis
for their conclusions and recommendations. For example, aninformant’s comment such as “The water programme has reallychanged things around here” can be probed for more details,
such as “What changes have you noticed?” “Who seems to havebenefited most?” “Can you give me some specific examples?”.
Maintain a neutral attitude. Interviewers should be sympathetic
listeners and avoid giving the impression of having strong viewson the subject under discussion. Neutrality is essential becausesome informants, trying to be polite, will say what they think theinterviewer wants to hear.
Minimise translation difficulties. Sometimes it is necessary
to use a translator, which can change the dynamics and add

225

PRISM
MODULESMETHOD FACTSHEETS& ANNEXES

difficulties. For example, differences in status between thetranslator and informant may inhibit the conversation. Often
information is lost during translation. Difficulties can be minimisedby using translators who are not known to the informants, andbriefing translators on the purposes of the study to reducemisunderstandings, and having translators translate the interview
word for word.

Step 6.

Take adequate notes

Interviewers should take notes and develop them in detail
immediately after each interview, to ensure accuracy. Use a set ofcommon sub-headings for interview texts, selected with an eye tothe major issues being explored. Common sub-headings ease data
analysis. Consider using a dictaphone as a back-up, but be mindfulthat recordings can take a long time to transcribe afterwards.

Step 7.

Analyse interview data

At the end of each interview, prepare an interview summary sheet
which provides an overview of the gathered information organised
into manageable themes, issues, and recommendations. Eachsummary should provide information about the key informant’s
position, the reason for their inclusion in the list of informants, themain points made, the implications of these observations, and any
insights or ideas the interviewer had during the interview.
Refer to the factsheet Analysing qualitative data for more information.

PRISM - TOOLKIT FOR EVALUATING THE OUTCOMES AND IMPACTS OF SMALL/MEDIUM-SIZED CONSERVATION PROJECTS

4:0

INTRODUCTION &KEY CONCEPTS

Step 8.

GENERALEVALUATIONMETHOD
FACTSHEETSKEY INFORMANTINTERVIEWSSTEP 8DESIGNING & IMPLEMENTING
THE EVALUATION

Check for reliability and validity

Key informant interviews are susceptible to error, bias andmisinterpretation, which can lead to flawed findings orrecommendations.
Check representativeness of the key informants. Take a second
look at the key informant list to ensure that no significant groups
were overlooked.
Assess reliability of the key informants. Assess the informants’knowledge, credibility, impartiality, willingness to respond, and thepresence of any outsiders who may have inhibited or otherwise
influenced their responses. Greater weight can be given to
information provided by more reliable informants.

PRISM
MODULES

Acknowledgements
Guidance in this factsheet is based on the following resources:
PACT - Field Guide for Evaluation (2014) - http://betterevaluation.
org/sites/default/files/Field%20Guide%20for%20Evaluation_
Final.pdf .
USAID Center for Development Information and Evaluation(1996). Conducting yey informant interviews. Performancemonitoring & evaluation tips. Washington DC, USAID - http://pdf.
usaid.gov/pdf_docs/PNABS541.pdf .

Check for interviewer or investigator bias. One’s own biases
as an investigator should be examined, including tendencies toconcentrate on information that confirms preconceived notions
and hypotheses, to seek consistency too early, to overlook
evidence that is inconsistent with earlier findings and to be partial
to the opinions of elite key informants.
Check for negative evidence. Make a conscious effort to look forevidence that questions preliminary findings. This brings out issues
that may have been overlooked.
Get feedback from informants. Ask the key informants for
feedback on major findings. A summary report of the findings
might be shared with them, along with a request for writtencomments. Often a more practical approach is to invite them toa meeting where key findings are presented, and to ask for their
feedback.

226

METHOD FACTSHEETS& ANNEXESPRISM - TOOLKIT FOR EVALUATING THE OUTCOMES AND IMPACTS OF SMALL/MEDIUM-SIZED CONSERVATION PROJECTS

4:0

INTRODUCTION &KEY CONCEPTS

DESIGNING & IMPLEMENTING
THE EVALUATION

Factsheet: Focus groups

GENERALEVALUATIONMETHOD
FACTSHEETS

Summary
In a focus group, a facilitator guides a group of people in a
discussion of their experiences, feelings, and preferences inrelation to the project or project action. The facilitator raises issues
identified in a discussion guide and uses probing techniques tosolicit views, ideas, and other information.

PRISM
MODULESMETHOD FACTSHEETS& ANNEXES

What are focus groups useful for?
Focus group can be used to solicit views, insights, andrecommendations of programme staff, customers, stakeholders,technical experts, or other groups.
Advantages:•

Can be low cost (but not in all situations).

•

Provides results in a short time period (once arranged).

•

Flexible format can allow the facilitator to explore
unanticipated issues and encourages interaction among
participants.

•

In a group setting participants may be able to provide
checks and balances on each other, reducing false or
extreme views.

FOCUS GROUPS

Limitations:

227•

Flexible format makes it susceptible to facilitator bias,
which can undermine the validity and reliability offindings.

•

Discussions can be side-tracked or dominated by a few
vocal individuals.

•

Generates relevant qualitative information, but noquantitative data for making generalisations for thewhole population.

PRISM - TOOLKIT FOR EVALUATING THE OUTCOMES AND IMPACTS OF SMALL/MEDIUM-SIZED CONSERVATION PROJECTS

4:0

INTRODUCTION &KEY CONCEPTS

DESIGNING & IMPLEMENTING
THE EVALUATION

Required steps
Step 1.

GENERALEVALUATIONMETHOD
FACTSHEETS
FOCUS GROUPSSTEPS 1 TO 4

Select the team

Decide who will facilitate the workshop and who will take notes.
Ideally the facilitator should be a native speaker who can put
people at ease. The team should have substantive knowledge ofthe topic under discussion.
If the discussions are to be conducted by someone without
previous experience in focus group techniques, training issuggested. This training can take the form of role playing,formalised instruction on topic sequencing and probing for
generating and managing group discussions, as well as pre-testing
discussion guides in pilot groups.

Step 2.

Select the participantsFirst, identify the types of groups and institutions that should be
represented (such as local people who are involved/affected by
the project, project staff, partners, technical experts, government
officials) in the focus groups. Often separate focus groups areheld for each type of audience. Second, identify the most suitablepeople in each group. One of the best approaches is to consult
key informants who know about local conditions ahead of time toselect suitable focus groups.

228

PRISM
MODULESMETHOD FACTSHEETS& ANNEXES

Participants in each group should be reasonably homogenous,
from similar socioeconomic and cultural backgrounds. They should
also share common traits related to the discussion topic. For
example, in a discussion on hunting, participants from households
who engage in hunting and park rangers responsible for enforcing
regulations should participate in separate focus groups. Household
members may be reluctant to discuss hunting in front of rangers,
as they may be concerned about potential repercussions. Ideally,
people should not know each other. Anonymity lowers inhibitionand prevents the formation of cliques.

Step 3.

Establish Free, Prior & Informed Consent (FPIC)This lets the informants know they are participating in a study
and how their information will be used. Establishing FPIC requiresa participant to give formal approval (e.g. by signing a form
declaring that the informant explicitly agrees to participate). See
Factsheet: Developing ethical protocols for an evaluation for more
information on FPIC.

Step 4.

Decide on timing and location

Discussions should last around one to two hours and should be
conducted in a location that is easy for participants to reach, andwith some degree of privacy.

PRISM - TOOLKIT FOR EVALUATING THE OUTCOMES AND IMPACTS OF SMALL/MEDIUM-SIZED CONSERVATION PROJECTS

4:0

INTRODUCTION &KEY CONCEPTS

Step 5.

GENERALEVALUATIONMETHOD
FACTSHEETSDESIGNING & IMPLEMENTING
THE EVALUATION

Prepare a list of discussion topics

In advance, prepare a brief outline that covers the topics to be
discussed. It should contain a limited number of topics, allowing
some time and flexibility to pursue unanticipated but relevant
issues. The outline provides a framework for the facilitator to use
in exploring, probing and asking questions. Use carefully worded
questions to introduce each topic and to keep the discussion
focused. Comments raised can still be explored when needed.

FOCUS GROUPS
Step 6.

Establish rapport. Often participants do not know what to expectfrom focus group discussions. It is helpful for the facilitator tooutline the purpose and format of the discussion at the beginningof the session, and to set the group at ease. Participants should be
told that the discussion is informal, that everyone is expected toparticipate and that divergent views are welcome.
Phrase questions carefully. Certain types of questions impedegroup discussions. For example, yes-or-no questions are onedimensional and do not stimulate discussion. “Why” questions put
people on the defensive and can cause them to take “politicallycorrect” sides on controversial issues.
Use open-ended questions where possible. These are more useful
because they allow participants to tell their story in their own
words and to add details that can result in unanticipated findings.

METHOD FACTSHEETS& ANNEXESFor example:-

How do you feel the project has affected thecommunity?

-

What do you think about the national park?

If the discussion is too broad, the facilitator can narrow responses
by asking such questions as:-

How do you feel the project’s education programme has
affected the community?

-

What do you think about the number of tourists visiting
the national park?

Conduct the discussion

STEPS 5 & 6

229

PRISM
MODULESUse probing techniques. When participants give incomplete orirrelevant answers, the facilitator can use the following techniques
to try to gain fuller, clearer responses:•

Repeat the question.

•

Say you have limited understanding of the issue, and ask
for specific details.

•

Pause for the answer.

•

Repeat the reply - hearing it again sometimes stimulates
conversation. Ask “when”, “what”, “where”, “which”, and“how” questions to provoke more detailed information.

•

Use neutral comments - “Anything else?” “Why do you
feel this way?”.

PRISM - TOOLKIT FOR EVALUATING THE OUTCOMES AND IMPACTS OF SMALL/MEDIUM-SIZED CONSERVATION PROJECTS

4:0

INTRODUCTION &KEY CONCEPTS

DESIGNING & IMPLEMENTING
THE EVALUATION

Try to control the discussion. In most groups, a few individuals
tend to dominate the discussion. To encourage more balanced
participation:

GENERALEVALUATIONMETHOD
FACTSHEETS-

Address questions to individuals who are reluctant to
talk.

-

Give non-verbal cues (look in another direction or stoptaking notes when an individual talks for an extended
period).

-Intervene, politely summarise the point, then re-focus
the discussion.

-

Take advantage of a pause and say “Thank you for thatinteresting idea, perhaps we can discuss it in a separate
session. Meanwhile with your consent, I would like tomove on to another item”.

FOCUS GROUPSSTEPS 6 & 7

PRISM
MODULES

Step 7.

Record the discussion

Tape recordings and written notes are useful. Notes should be
extensive and reflect the content of the discussion as well as nonverbal behaviour (facial expressions, hand movements).
Shortly after each group interview, the team should summarise
the information, the team’s impressions, and implications of
the information for the study. Ideally, the discussion should bereported in participants’ language, retaining their phrases andgrammatical usage.

Minimise group pressure. When an idea is being adopted without
any general discussion or disagreement, more than likely grouppressure is occurring. To minimise group pressure the facilitator
can probe for alternate views. For example, the facilitator can raise
another issue, or say “We had an interesting discussion but let’s
explore other alternatives.”

230

METHOD FACTSHEETS& ANNEXESPRISM - TOOLKIT FOR EVALUATING THE OUTCOMES AND IMPACTS OF SMALL/MEDIUM-SIZED CONSERVATION PROJECTS

4:0

INTRODUCTION &KEY CONCEPTS

Step 8.

GENERALEVALUATIONMETHOD
FACTSHEETS

231

Analyse results

After each session, the team should assemble the interview notes(transcripts of each focus group interview), the summaries, and anyother relevant data to analyse trends and patterns. Refer to thefactsheet Analysing qualitative data for more information.

PRISM
MODULESMETHOD FACTSHEETS& ANNEXES

Purpose of the report. Consider the objectives of the study andthe information needed for decision making. The type and scopeof reporting will guide the analytical process. For example, focus
group reports typically are: (1) brief oral reports that highlight key
findings; (2) descriptive reports that summarise the discussion; or(3) analytical reports that reveal trends, patterns.

When analysing the results, the team should consider:-

Words. Weigh the meaning of words participants used.
Can a variety of words and phrases categorise similar
responses?

Acknowledgement
Guidance in this factsheet is based on USAID Center for
Development Information and Evaluation (2011): Conducting
focus group interviews.

-

Framework. Consider the circumstances in which acomment was made (context of previous discussions,tone and intensity of the comment).

Performance monitoring & evaluation tips. Washington DC,
USAID - http://pdf.usaid.gov/pdf_docs/Pnadw110.pdf .

-Internal agreement. Consider whether shifts in opinions
during the discussion were caused by group pressure.

-

Precision of responses. Decide which responses
were based on personal experience and give themgreater weight than those based on vague impersonal
impressions.

-

The big picture. Pinpoint major ideas. Allocate time tostep back and reflect on major findings.

FOCUS GROUPSSTEP 8DESIGNING & IMPLEMENTING
THE EVALUATIONPRISM - TOOLKIT FOR EVALUATING THE OUTCOMES AND IMPACTS OF SMALL/MEDIUM-SIZED CONSERVATION PROJECTS

4:0

INTRODUCTION &KEY CONCEPTS

GENERALEVALUATIONMETHOD
FACTSHEETSDIRECTOBSERVATIONSTEP 1DESIGNING & IMPLEMENTING
THE EVALUATION

PRISMMODULES

Factsheet: Direct observation

Required steps

Summary
Most evaluation teams conduct some fieldwork, observing what’s
actually going on at activity sites. Often this is done informally,
without much thought to the quality of data collection. Direct
observation techniques allow for a more structured process.

Step 1.

What is direct observation useful for?
Direct observation techniques allow for a systematic, structured
process, using well-designed observation record forms.
Advantages:•
An event, institution, facility, or process can be studied
in its natural setting, thereby providing a richer
understanding of the subject.
Limitations:•Susceptible to observer bias. The very act ofobservation also can affect the behaviour being studied.

METHOD FACTSHEETS& ANNEXES

Determine the focus

Because of time and resource constraints, direct observation has
to be selective, looking at a few actions, events, or assumptions
that are central to the evaluation questions. For example, suppose
you want to evaluate a project which is working to promote
alternative sources of fuelwood. Here, evaluation could look at avariety of areas - e.g. community knowledge, fuelwood harvesting
rates, whether these groups are working more effectively or levelsof awareness among members of the public. The team should
narrow the focus to one or two of the areas that are likely togenerate the most useful information and insights.
Next, break down each action, event, or assumption into subcomponents. For example, if the team decides to look at capacityof local conservation groups, prepare a list of the tasks to observe,such as preparation of vaccine, consultation with mothers, andvaccine administration.
Each task may be further divided into sub-tasks; for example,
local conservation groups might carry out habitat restoration,awareness raising or advocacy work.
If you also want to assess physical facilities and surroundings,
prepare an inventory of items to be observed.

232

PRISM - TOOLKIT FOR EVALUATING THE OUTCOMES AND IMPACTS OF SMALL/MEDIUM-SIZED CONSERVATION PROJECTS

4:0

INTRODUCTION &KEY CONCEPTS

Step 2.

GENERALEVALUATIONMETHOD
FACTSHEETSDIRECTOBSERVATION

Develop direct observation forms

The observation record form should list the items to be observed
and should provide spaces to record observations. These forms
are similar to survey questionnaires, but investigators record theirown observations, not respondents’ answers. Observation record
forms help to standardise the observation process and to ensurethat all important items are covered. They also facilitate better
aggregation of data gathered from various sites or by various
investigators.
When preparing direct observation forms, consider the following:
1.

STEPS 2 & 3

Identify in advance the possible response categories foreach item, so that the observer can answer with a simple
yes or no, or by checking the appropriate answer. Closed
response categories help to minimise observer variability, andtherefore improve the quality of data.

2.

Limit the number of items in a form. Forms should normallynot exceed 40–50 items. If necessary, it is better to use two
or more smaller forms than a single large one that runs toseveral pages.

3.

Provide adequate space for recording additional observations
for which response categories were not determined.

4.

233

DESIGNING & IMPLEMENTING
THE EVALUATION

Use of computer software designed to create forms can bevery helpful. It can help to create a neat, unconfusing form
that is easily completed.

PRISM
MODULES

Step 3.

METHOD FACTSHEETS& ANNEXES

Select the site(s)Observations at a single site may be justified if the site can betreated as a typical case or if it is unique. Consider a situation inwhich all five local conservation groups established by a project havenot been performing well. Here, observation of a single group may
be justified as a typical case. Observations at a single group may
also be justified when the case is unique; for example, if only one offive groups have been having major problems, and the purpose of
the evaluation is to discover why. However, single site observations
should generally be avoided, because cases which the team assumes
to be typical or unique may not be. As a rule, several sites arenecessary to obtain a reasonable understanding of a situation.
In most cases, teams select sites based on expert advice. The
investigator develops criteria for selecting sites, then relies on
the judgment of knowledgeable people. For example, if a teamevaluating a project decides to observe three local conservationgroups - one highly successful, one moderately successful, and one
struggling - you may request project staff, local experts or otherinformants to suggest a few groups for each category. You will thenchoose three after examining their recommendations. Using morethan one expert reduces individual bias in selection.
Alternatively, sites can be selected based on data from performancemonitoring. For example, activity sites can be ranked from best to
worst based on performance measures, and then a sample can be
drawn from them.

PRISM - TOOLKIT FOR EVALUATING THE OUTCOMES AND IMPACTS OF SMALL/MEDIUM-SIZED CONSERVATION PROJECTS

4:0

INTRODUCTION &KEY CONCEPTS

Step 4.

GENERALEVALUATIONMETHOD
FACTSHEETSDIRECTOBSERVATIONSTEPS 4, 5 & 6

DESIGNING & IMPLEMENTING
THE EVALUATION

Decide on the best timing

Timing is critical in direct observation, especially when events
are to be observed as they occur. Poor timing can distort findings.
For example, participation in local conservation groups can varydepending on the time of year (e.g. when people are needed forplanting/harvesting crops or a time of year when many people goon holiday). If Local Conservation Group activities are observed
during these times then an inaccurate picture of its activities may
result.
Similarly people and organisations often follow daily routines
associated with set times. For example, credit institutions may
accept loan applications in the morning; farmers in tropicalclimates may go to their fields early in the morning and returnhome by noon. Observation periods should reflect these rhythms.

Step 5.

Establish Free, Prior & Informed Consent (FPIC)This lets the informants know that they are participating in astudy and how their information will be used. Establishing FPIC
requires a participant to give formal approval (e.g. by signing a form
declaring that the informant explicitly agrees to participate). See
Factsheet: Developing ethical protocols for an evaluation for more
information on FPIC.

234

PRISM
MODULES

Step 6.

METHOD FACTSHEETS& ANNEXES

Conduct the field observation

Establish rapport. Before embarking on direct observation, acertain level of rapport should be established with the people,
community, or organisation to be studied. The presence of outside
observers, especially officials or experts, may generate someanxiety among those being observed. Often informal, friendly
conversations can reduce anxiety levels.
Also, let them know the purpose of the observation is not to reporton individuals’ performance, but to find out what kind of problems
in general are being encountered.
Allow sufficient time for direct observation. Brief visits can bedeceptive, partly because people tend to behave differently in thepresence of observers. It is not uncommon, for example, for healthworkers to become more caring or for extension workers to be
more persuasive when being watched. However, if observers stayfor relatively longer periods, people become less self-conscious
and gradually start behaving more naturally. It is essential to stayat least two or three days on a site to gather valid, reliable data.
Use a team approach. If possible, two observers should observe
together. A team can develop more comprehensive, higher quality
data, and reduce individual bias.
Train observers. If many sites are to be observed, non-experts canbe trained as observers, especially if observation forms are clear,straightforward, and mostly closed-ended.

PRISM - TOOLKIT FOR EVALUATING THE OUTCOMES AND IMPACTS OF SMALL/MEDIUM-SIZED CONSERVATION PROJECTS

4:0

INTRODUCTION &KEY CONCEPTS

Step 7.

GENERALEVALUATIONMETHOD
FACTSHEETSSTEPS 7, 8 & 9Complete forms

Take notes as inconspicuously as possible. The best time forrecording is during observation. However, this is not always
feasible because it may make some people self-conscious or may
disturb the situation. In these cases, recording should take place assoon as possible after observation.

Step 8.

DIRECTOBSERVATIONDESIGNING & IMPLEMENTING
THE EVALUATION

Step 9.

METHOD FACTSHEETS& ANNEXES

Check for reliability and validity

Direct observation techniques are susceptible to error and biasthat can affect reliability and validity. This can be minimised byfollowing some of the procedures suggested, such as checking therepresentativeness of the sample of sites selected; using closedended, unambiguous response categories on the observation
forms; recording observations promptly; and using teams ofobservers at each site

Analyse the data

Data from closed-ended questions from the observation form canbe analysed using basic procedures such as frequency counts andcross-tabulations. Statistical software packages such as SAS or
SPSS facilitate such statistical analysis and data display.
Analysis of any open-ended interview questions can also provide
extra richness of understanding and insights. Here, use of databasemanagement software with text storage capabilities, such as
dBase, can be useful.

235

PRISM
MODULES

Acknowledgement
Guidance in this factsheet is based on USAID (2011): Using direct
observation techniques.
Performance monitoring & evaluation tips factsheet. Washington
DC, USAID - http://pdf.usaid.gov/pdf_docs/pnadw104.pdf .

PRISM - TOOLKIT FOR EVALUATING THE OUTCOMES AND IMPACTS OF SMALL/MEDIUM-SIZED CONSERVATION PROJECTS

4:0

INTRODUCTION &KEY CONCEPTS

DESIGNING & IMPLEMENTING
THE EVALUATION

Factsheet: Documentary evidence

GENERALEVALUATIONMETHOD
FACTSHEETSDOCUMENTARYEVIDENCE

Summary
Documentary evidence refers to the use of anything constituting a“document” which provides a record of the variables used to assesschange in indicators.
Various types of document may be used as sources of evidence,
including public records (from schools, clinics); publications(magazines, newspapers); private papers (diaries, letters); minutes
and records of meetings (NGOs, CBOs, government); legal
documents; financial records, budgets and records of trade andmarketing; websites and webpages (including social media); andvisual documents (e.g. videos, films and photographs). Whenplanning to use documentary methods to assess impacts, it isimportant to be aware of what documents are available, andwhether you can have access to them.
Documentary evidence can be used in three main ways:
Direct use of documentary data/information: In many cases thecontent of documents (or the documents themselves) provide a
direct indication of change relevant to a project’s objectives. For
example, legal documents which show changes in tenure or rights
to use land and resources or attendance records of meetings whichshow increased attendance by women.

PRISM
MODULES

example, a project that seeks to change awareness of the need toprevent bushfires may analyse the content of newspapers over a
12-month period to see how often key words such as “bushfire”,“erosion”, and “climate change” appear.
Analytical reading: This involves thinking carefully, critically anddeeply when reading documents – even for example quantitative
records. As noted below, any author of a document has abackground, experiences and views which may affect the way
they present information – what is included and what is left out.
Reading documents analytically and critically aims to make senseof this, and to interpret documents in the context of other issues,previously gained knowledge and information.
Documentary data need to be interpreted with caution – it isimportant to be aware of how documents have been used anddeveloped – by whom and for what purpose. Is the information
contained in the document credible and representative; is it
trustworthy? For example, can attendance records at a school betrusted if the teacher who compiles them is paid according to thenumber of pupils?

Content analysis / quantitative analysis: Content analysis is amethod for summarising any form of content by counting various
aspects of the content. This enables objective evaluation. Content
analysis, although it often analyses written words, is a quantitative
method, and the results are numbers and percentages. For

236

METHOD FACTSHEETS& ANNEXESPRISM - TOOLKIT FOR EVALUATING THE OUTCOMES AND IMPACTS OF SMALL/MEDIUM-SIZED CONSERVATION PROJECTS

4:0

INTRODUCTION &KEY CONCEPTS

DESIGNING & IMPLEMENTING
THE EVALUATION

PRISMMODULES

Evaluation Design Factsheets

METHOD
FACTSHEETSAND ANNEXES

Participatory Impact Assessment
Interviewing key informants
Theory-based designs
Matching designs
Before and after designs
Decision tree for selecting an evaluation design

237

PRISM - TOOLKIT FOR EVALUATING THE OUTCOMES AND IMPACTS OF SMALL/MEDIUM-SIZED CONSERVATION PROJECTSMETHOD FACTSHEETS& ANNEXES

4:0

INTRODUCTION &KEY CONCEPTS

DESIGNING & IMPLEMENTING
THE EVALUATION

Evaluation design: Participatory Impact Assessment

EVALUATIONDESIGN
FACTSHEETSEVALUATIONDESIGN:PARTICIPATORYIMPACTASSESSMENT

SummaryThis is type of design that focuses on asking project participants
about their perceptions of what changes have occurred. It resultsin both quantitative and qualitative data, with participants
being asked direct questions about the kinds of outcomes andimpacts the project has brought about. It is particularly useful forevaluating the ways in which the project has affected stakeholders.
For many small/medium-sized conservation projects that involve
people, this is likely to be the most feasible way of attributingchange to the project. This type of design is attractive in that it canbe relatively cheap, data are relatively easy to obtain and a baseline
study does not need to have been carried out. However care must
be taken in order to generate results that are useful and valid.

PRISM
MODULES

Process
Participatory Impact Assessment (PIA) is based on the recognition
that “local people are capable of identifying and measuring theirown indicators of change” (Catley et al. 2013: 5). PIA aims toanswer three key questions:•

What changes have there been in the community since thestart of the project?

•

Which of these changes are attributable to the project?

•

What difference have these changes made to people’s lives?

For further reading refer to Catley et al. (2013): p45–50: http://fic.
tufts.edu/assets/PIA-guide_revised-2014-3.pdf which provides anumber of examples of three different ways in which participatory
methods can be used to evaluate social outcomes and impacts.

238

METHOD FACTSHEETS& ANNEXESPRISM - TOOLKIT FOR EVALUATING THE OUTCOMES AND IMPACTS OF SMALL/MEDIUM-SIZED CONSERVATION PROJECTS

4:0

INTRODUCTION &KEY CONCEPTS

DESIGNING & IMPLEMENTING
THE EVALUATION

Eight main steps are proposed in the PIA approach:
Step 1.

EVALUATIONDESIGN
FACTSHEETSEVALUATIONDESIGN:PARTICIPATORYIMPACTASSESSMENTSTEPS 1 & 2

239

Define the questions to be answered - Many
impact assessments and evaluations try to answer
too many questions, leading to superficial orinconclusive results on a wide range of issues andhence uncertainty about how to use this information.
Although it is tempting to try and capture as muchinformation about a project as possible, it is usually
better to limit the impact assessment to a maximumof five key questions, and answer these questions
with confidence. If you have already worked with
communities to identify their impact indicators at the
beginning of the project, the assessment questions
will be framed according to these indicators. Morelikely, you will be using a retrospective approach, andworking with project participants jointly to define theassessment questions.

PRISM
MODULES

Step 2.

METHOD FACTSHEETS& ANNEXES

Define the geographical and time limits of the project- Defining the geographical (spatial) boundaries of a
project aims to ensure that everyone understands thephysical limits of the area in which impact is supposed
to take place. Defining the project’s time boundaries
aims to ensure that everyone is clear about the time
period being assessed. Participatory (or community)
mapping is a useful visualisation method to use
at the beginning of an assessment to define thegeographical boundary of the project area (seefactsheet: Community mapping). It also acts as a good“ice-breaker”, since many people can be involved.
A timeline is an interviewing method that captures
the important historical events in a community, as
perceived by the community itself, and it positions
the project start date and end date against these
events. This method helps to ensure that everyone
involved in the assessment understands which
project is being assessed as well as the project
timeframe, and it helps people to recall events andchanges that occurred during the project.

PRISM - TOOLKIT FOR EVALUATING THE OUTCOMES AND IMPACTS OF SMALL/MEDIUM-SIZED CONSERVATION PROJECTS

4:0

INTRODUCTION &KEY CONCEPTS

Step 3.

EVALUATIONDESIGN
FACTSHEETS

Identify and prioritise locally defined impact
indicators - PIA proposes the use of indicators
identified by community participants, since they havetheir own priorities for improving their lives, andtheir own ways of measuring change.
•

EVALUATIONDESIGN:PARTICIPATORYIMPACTASSESSMENT
STEP 3240

DESIGNING & IMPLEMENTING
THE EVALUATION

One way of collecting community indicatorsof impact is simply to ask project participants,
when the project starts, what changes in theirlives they expect to occur as a direct result of the
project. Alternatively, in cases where the project
has already been implemented, you can ask whatchanges have already occurred. This should bedone separately for each project activity that you
plan to assess. If the project has a technical focus,
for example natural resource management, theprovision of agricultural inputs or livestock, ask theparticipants how they benefit from the ownershipor use of the resources in question. These benefits
are impact indicators.

•

When collecting community indicators it isimportant to capture the views of different groupsof people within the community. Women willoften have different priorities and expectations ofproject impact compared with those of men.

•

If the community or participants produce many
impact indicators, ask them to prioritise theindicators using ranking. It is important not to havetoo many indicators: as with the key assessment

PRISM
MODULESMETHOD FACTSHEETS& ANNEXES

questions, it is better to have a few good indicators
than too many poor ones.
•

An understanding of the context is essential inderiving meaning from these indicators. Simply
measuring changes in livelihoods impact indicators
will not tell us much about impact unless you
understand the reasons behind those changes.
An understanding of livelihoods and context
is therefore an important part of any impact
assessment.

•

One of the strengths of participatory methods,
if used correctly, is that they can help to reveal
important contextual factors (the complex social,political and economic contexts that affect project
outcomes) and so improve understanding of themechanisms through which change has or has
not been achieved. Participatory selection ofindicators is an important part of this process.

PRISM - TOOLKIT FOR EVALUATING THE OUTCOMES AND IMPACTS OF SMALL/MEDIUM-SIZED CONSERVATION PROJECTS

4:0

INTRODUCTION &KEY CONCEPTS

Step 4.

EVALUATIONDESIGN
FACTSHEETSEVALUATIONDESIGN:PARTICIPATORYIMPACTASSESSMENTSTEPS 4, 5 & 6

241

DESIGNING & IMPLEMENTING
THE EVALUATION

Decide which methods to use, and test them - Thisrefers to the data collection methods used tomeasure the indicators. The following factsheets
outline methods that can be used as part of a PIA:•PIA Method 1: Before and after scoring•

PIA Method 2: Proportional piling•

PIA Method 3: Tally method•

PIA Method 4: Matrix scoring•

PIA Method 5: Before and after impact calendars

Each method has its strengths and weaknesses, andsome are more appropriate in certain cultures. Where
methods produce numerical measurements it isessential that they are conducted alongside a semistructured interview process, in order to understandthe reasons for any observed changes.

PRISM
MODULESMETHOD FACTSHEETS& ANNEXES

Step 5.

Decide which sampling method and sample size touse - The sampling method is likely to be purposive(e.g. “typical” villages) or random sampling. Thereis no simple answer to the question of what sample
size to use: this depends on the type and number ofquestions and the methods used. In most situations,the important thing is to capture the overall trend,
and this can usually be done with a reasonably small
sample size as long as the methods are appliedconsistently.

Step 6.

Assess project attribution - Participatory ranking andscoring methods can be used to assess attribution: ingeneral, the preferred approach is to try to separate
the project and non-project causative factors, andto find the relative importance of these factors in
the explanation of an identified positive or negative
outcome or impact. By identifying what changes(if any) can be specifically attributed to project
activities, PIA places the project activities within thewider economic, social and environmental context ofan area, alongside a range of non-project factors that
influence livelihood strategies and outcomes.

PRISM - TOOLKIT FOR EVALUATING THE OUTCOMES AND IMPACTS OF SMALL/MEDIUM-SIZED CONSERVATION PROJECTS

4:0

INTRODUCTION &KEY CONCEPTS

Step 7.

EVALUATIONDESIGN
FACTSHEETSEVALUATIONDESIGN:PARTICIPATORYIMPACTASSESSMENT

DESIGNING & IMPLEMENTING
THE EVALUATION

Triangulate - Triangulation (using multiple methods
and/or data sources to increase the credibility
and validity of results) is important for all datacollection methods, including participatory methods.
Information and numbers from PIA can be crosschecked by:•reading secondary reports, statistics and literature
related to the project area, and to similar past
projects;•

using different PIA methods to measure the same
indicator;•

asking the same question in different ways withina given PIA, e.g. combining ranking or scoring withan informal interview.

PRISM
MODULES

Step 8.

Feed back and verify the results with the community- It is essential to discuss the results of the analysis
with communities and other stakeholders. This is alast opportunity to “ground–truth” the results, andthe discussions usually reveal further insights into
project outcome and impact processes. Focus groups,
e.g. by gender, are advisable for getting the best
feedback quality.

With all participatory methods there is a danger that
respondents are more likely to cite project-related
factors if someone from the project is administering
the survey. It is better, therefore, if you can get an
independent third party to carry out the assessments.

STEPS 7 & 8

242

METHOD FACTSHEETS& ANNEXESPRISM - TOOLKIT FOR EVALUATING THE OUTCOMES AND IMPACTS OF SMALL/MEDIUM-SIZED CONSERVATION PROJECTS

4:0

INTRODUCTION &KEY CONCEPTS

DESIGNING & IMPLEMENTING
THE EVALUATION

PRISMMODULES

Evaluation design: Interviewing key informants

EVALUATIONDESIGN
FACTSHEETSEVALUATIONDESIGN:INTERVIEWINGKEY
INFORMANTS

243

This approach uses key informant interviews to ask certain key
individuals for information to gain a measure of what conditions
were like before the project, what changes have occurred, what
would have happened in the absence of the project, to identify
other possible explanations and, if feasible, to assess whether
these explanations can be ruled out. Examples of key informants
include experts with particular knowledge of the project’s target
site/species or the particular action that the project is carrying
out. Individual experts can be consulted, but it is usually preferable
to consult several so that a wide range of experience can be
drawn on. The scorecards in the species and habitat management
module are designed to be used in this way. See Factsheet: Key
informant interviews for more information on how to conduct a
key informant interview.

PRISM - TOOLKIT FOR EVALUATING THE OUTCOMES AND IMPACTS OF SMALL/MEDIUM-SIZED CONSERVATION PROJECTSMETHOD FACTSHEETS& ANNEXES

4:0

INTRODUCTION &KEY CONCEPTS

DESIGNING & IMPLEMENTING
THE EVALUATION

EXAMPLE

PRISM
MODULESMETHOD FACTSHEETS& ANNEXES

While the results of the household survey provided
quantitative data (such as proportions of people engaged

EVALUATIONDESIGN
FACTSHEETS

Example: Using key informant interviewing in socioeconomic

in fishing and marine harvesting, proportion of ratings

assessment of Ebiil MPA in Palaufor perceived changes of marine life and catch effort, andEstablished in 2000, Ebiil became the first legislated Marine
Protected Area of Palau. Located in Ngarchelong State, thearea is an important aggregation site for groupers and istraditionally a trochus sanctuary. The Ebiil Conservation Act

EVALUATIONDESIGN:INTERVIEWINGKEY
INFORMANTS

of 2000 closed this channel and surrounding reefs to all fishingactivities and prohibited unauthorised entry. Socioeconomic
assessments were conducted by the Palau International
Coral Reef Center in 2005 and 2010 to generate information
to improve the management of Ebiil and to ensure that the
local community would benefit from the MPA. In the 2010
assessment, key informant interviews were conductedin addition to a survey that covered 279 households ofNgarchelong resource users. A combination of purposiveand snowball sampling schemes was used to select 15 key
informants, consisting of community and traditional leaders,
related state governmental officers, and fishers and harvestersof marine life in the Ngarchelong reef areas. Purposive
sampling is commonly used in qualitative research. A purposive
sample includes people of interest to serve a very specific needor purpose of the study. Snowball sampling is where individuals
are interviewed and asked at the end of the interview for theirsuggestions for the next person to be interviewed about thesame subject.

244

perception of changes in and benefits for the community),
the interviews were used to collect more in-depth dataon changes in marine life and fishing patterns, benefitsof Ebiil to the community, the effectiveness of the MPA
management, and alternative ways of conserving the natural
resources of Ngarchelong and supporting the well-being of
the communities. The key informant interviews revealed
that many traditional fishing methods, such as spear and trap
fishing, are no longer practised. This seems to be the result of
a combination of new technologies, loss of interest in using
local methods, and a decline in fish numbers. The key informant
interviews also highlighted the need for better enforcement
and appreciation of the traditional “bul” (restrictions placedby traditional leaders and in this case a closed marine area) for
conservation, and political conflicts (power struggles) between
traditional leaders and state authorities. Despite the potential
for better conservation results if the bul and the MPA wereoperated in a mutually supportive way, there are existing
conflicts that would need to be addressed and resolved for thisto happen.
Source: https://www.conservationgateway.org/Documents/
Measures%20Case%20Study%20Ebiil%20MPA,%20Palau.pdf

PRISM - TOOLKIT FOR EVALUATING THE OUTCOMES AND IMPACTS OF SMALL/MEDIUM-SIZED CONSERVATION PROJECTS

4:0

INTRODUCTION &KEY CONCEPTS

DESIGNING & IMPLEMENTING
THE EVALUATION

Evaluation design: Theory-based designs

EVALUATIONDESIGN
FACTSHEETSEVALUATIONDESIGN:
THEORY-BASEDDESIGNS

245

A theory-based design uses the project’s Theory of Change to test
whether the project’s actual results match the results that aremapped out in the Theory of Change. First the project is mappedout in a Theory of Change and then data are collected at each stage
in the chain. If the results match what was expected then this canbe an indication that the change observed is due to the project(see Figure 1). Conversely if the results do not match the Theoryof Change, this casts doubt on the ability to attribute any changes
to the project (see Figure 2). In the case of Figure 2 you would thenlook to investigate the reasons why the outcomes laid out in the
Theory of Change did not occur as expected.
These approaches are popular because they are not do not require
a control/comparison group and they are often less resourceintensive than most designs that use a counterfactual. However,using this type of design effectively requires care, and above all athorough understanding of the mechanisms that drive change andthe systems being impacted.

PRISM
MODULESMETHOD FACTSHEETS& ANNEXES

An evaluation using a theory-based design might include thefollowing (Adapted from UNICEF Impact Evaluation Brief 10:
Strategies for Causal Attribution): https://www.unicef-irc.org/
publications/pdf/brief_6_overview_strategies_causal_attribution_
eng.pdf•

Achievement of intermediate outcomes – checking
whether all cases that achieved the final impacts alsoachieved the intermediate outcomes identified in the
Theory of Change.

•

Checking results against expert predictions – making
predictions based on the Theory of Change or an
emerging theory of wider contributors to outcomes, andthen following up on whether or not these predictions
actually materialise over time.

•

Checking timing of impacts – determining whether thetiming of impacts is consistent with a causal relationship,
again with reference to the Theory of Change. For
example, whether the impact occurs a reasonable timeafter the programme or policy has taken effect.

•

Checking consistency with existing literature –checking results against what is known from reviewing
the literature in the area of interest, to identify
consistencies/inconsistencies. This must be done
cautiously and with explicit mention of any limitations in
the existing literature.

PRISM - TOOLKIT FOR EVALUATING THE OUTCOMES AND IMPACTS OF SMALL/MEDIUM-SIZED CONSERVATION PROJECTS

4:0

INTRODUCTION &KEY CONCEPTS

DESIGNING & IMPLEMENTING
THE EVALUATION

PRISMMODULESMETHOD FACTSHEETS& ANNEXES

FIGURE 1: THEORY OF CHANGE WHERE RESULTS
SUPPORT CAUSAL ATTRIBUTION.

EVALUATIONDESIGN
FACTSHEETS

Training
park guards
to conduct
rangerpatrols

EVALUATIONDESIGN:
THEORY-BASEDDESIGNS

Training
provided topark guards

Park guards
apply new skills
during patrols

Park guards
conduct more
effective patrols

Training delivered

Evidence that new
skills are being
appliedEvidence that
more effective
patrols are being
carried out

Poaching
decreases

Hunted species
population
increases

Evidence fordecrease inpoaching

Evidence for species
population
increase

Poaching
decreases

Hunted species
population
increases

FIGURE 2: THEORY OF CHANGE WHERE RESULTS DO
NOT SUPPORT CAUSAL ATTRIBUTION.

Training
park guards
to conduct
rangerpatrols

246

Training
provided topark guards

Park guards
apply new skills
during patrols

Training delivered

If there is noevidence that new
skills are being
appliedPark guards
conduct more
effective patrols

Then further outcomes/impacts can’t be attributed to the action, even if thereis evidence for a decrease in poaching or an increase in species population

PRISM - TOOLKIT FOR EVALUATING THE OUTCOMES AND IMPACTS OF SMALL/MEDIUM-SIZED CONSERVATION PROJECTS

4:0

INTRODUCTION &KEY CONCEPTS•

EVALUATIONDESIGN
FACTSHEETSEVALUATIONDESIGN:
THEORY-BASEDDESIGNSDESIGNING & IMPLEMENTING
THE EVALUATIONInterviewing key informants – this is not about asking
key informants whether they believe the action has
produced the impacts (which can be affected by theirlevel of knowledge of causal processes and theirintentions regarding the continuation of the action).
It is instead about asking them to explain the causal
processes following their involvement. For example,
in a programme that aims to support the national
government to develop new evidence-based policies,
a key informant interview might ask about the processof developing the policies, and work backwards to theprogramme, avoiding leading questions that assume theprogramme has been the major cause. These interviews
can provide evidence to explain, for example, how
a training programme made a difference in terms ofthe government’s subsequent ability to develop andnegotiate particular policies.

A related but slightly different approach involves identifying andconsidering possible alternative explanations for the achievementof outcomes and impacts and then gathering data to see if thesecan be ruled out.

247

PRISM
MODULESMETHOD FACTSHEETS& ANNEXES

Interviewing key informants – Asking experts in the specific
type of programme, community members or other stakeholders
to identify other possible explanations and, if feasible, to assess
whether these explanations can be ruled out.

General elimination methodology – this is carried out in twostages:(1)

Identifying possible explanations (including that theobserved changes are indeed due to the action, plus
as many alternative explanations as possible) using
a combination of options such as those listed above(e.g. key informant interviews and brainstorming, andreviewing previous evaluations/research); and(2)Gathering and analysing data to see if the possible
alternative explanations can be ruled out. This option
for causal attribution is most effective when used in
combination with the other strategies.

PRISM - TOOLKIT FOR EVALUATING THE OUTCOMES AND IMPACTS OF SMALL/MEDIUM-SIZED CONSERVATION PROJECTS

4:0

INTRODUCTION &KEY CONCEPTS

DESIGNING & IMPLEMENTING
THE EVALUATION

Evaluation design: Matching designs

EVALUATIONDESIGN:
MATCHING
DESIGNS

ec
oj
Pr

t
ge
ar
tT

l
tro
Con

Time

Figure 1: Matching design.
This kind of evaluation design involves identifying controls comparable sites or groups of non-project participants withessential characteristics that are similar to those being targeted by
the project. Once a control has been selected, data are collectedfrom both the control and the project target, and the results arethen compared to give a measure of how much of the observedchange can be attributed to the project’s actions. In a true
experimental design the assignment of controls is done randomly;
but in a conservation project random allocation is extremely
challenging (and in many cases impossible), so instead a control
group can be selected based on a number of variables that are
relevant to the project action(s) being evaluated.
An evaluation using control groups/sites might include thefollowing:

248

METHOD FACTSHEETS& ANNEXES1.

A comparison of the site/stakeholder group targetedby the project against a site/stakeholder group thatwas not targeted.

2.

A comparison of project and non-project targets
within the same site/stakeholder group.

3.

A comparison of different actions in the same site/
stakeholder group.

Change due
to project
Total change

EVALUATIONDESIGN
FACTSHEETSPRISMMODULESIn practice the first of these approaches provides the most robust
measurement of attribution, but it can be both difficult and costly
to identify and then collect data on a suitable non-project group.
The second and third approaches are therefore more likely to be
applicable to small/medium-sized conservation projects.
Once the control group/site has been identified, there are several
options for collecting data. The following are arranged in orderof strength. There is a trade-off between the strength of each ofthese designs and the costs/resources required.
Before-After Control Impact (BACI) – Data are collected fromboth the control and target site/group before and after the project
action. If trends at project target and control match each otherbefore the project, but diverge afterwards, this provides strong
evidence for the effect of project actions.
No pre-test control group – Data are collected from the control
group/site after the project action only. Data are still collectedfrom the target site/group before and after the project action.
Post-test only – Data are collected from the target and control
group/site after the project action only.

PRISM - TOOLKIT FOR EVALUATING THE OUTCOMES AND IMPACTS OF SMALL/MEDIUM-SIZED CONSERVATION PROJECTS

4:0

INTRODUCTION &KEY CONCEPTS

DESIGNING & IMPLEMENTING
THE EVALUATION

EXAMPLES
Example 1 - Human-tiger conflict in Nepal (comparison of

EVALUATIONDESIGN
FACTSHEETSEVALUATIONDESIGN:
MATCHING
DESIGNS

stakeholder groups)

PRISM
MODULES

Example 2 - Human-elephant conflict in Assam, India(comparison of different actions)
The growing human population in Assam and increasing
demand for land has led to fragmented habitat, forcing

Recovering tiger populations in Nepal, along with a growing

elephants into conflict with people. The raiding of crops byhuman population, are leading to increased human-wildlife

elephants is one of the major components of human-elephantconflict, and undermining conservation efforts by threatening

conflict, causing loss of life and livelihoods, and retaliation

lives and livelihoods in poor communities. The intended

against elephants. Chester Zoo and Eco-Systems India as

outcome of a three-year Darwin Initiative funded project,

part of the Assam Haathi Project (AHP) conducted a rigorous

designed and implemented by Chester Zoo and Greenassessment of the efficacy of action methods (provided byGovernance Nepal, is to reduce human-tiger conflict by

AHP) in use by communities, by analysing the “probability of

improving livestock husbandry, alleviating poverty throughcrop damage” and “area of crops damaged” against each action.

alternative livelihood options, conducting actions focused on

The effectiveness of the actions was analysed in two parts:

changing behaviours, and building capacity in User Community

their ability to prevent damage, and their ability to minimiseGroups. The project is administering surveys pre-, mid-, anddamage (when elephants are already in the crop fields).

post-action to project sites, and at pre- and post-action stages

Spotlights, chili fences and electric fences were found to be

to control groups (i.e. communities that do not receive project

the most effective methods for preventing crop damage when

actions) to help in assessing whether changes seen in projectused in isolation, but when used in combination with noise,communities can be attributed to the project’s activities

their efficacy was compromised. No actions were found to

and/or to other factors. Comparison groups were selected

reduce the extent of damage caused. AHP concluded that theby matching relevant key variables (e.g. livelihoods, naturalcommunities should focus on using chili fences, electric fences

resource use, tiger incidents) to project treatment groups. The

and spotlights, reduce the use of kunkies (trained domesticatedresults of the project will be communicated to comparison

elephants used to round up wild elephants and drive themgroups to facilitate learning and possible uptake of successful

away from villages) and chili smoke, and minimise the use of

strategies.

noise to increase the effectiveness of spotlights and fences(Davies et al., 2011).

249

METHOD FACTSHEETS& ANNEXESPRISM - TOOLKIT FOR EVALUATING THE OUTCOMES AND IMPACTS OF SMALL/MEDIUM-SIZED CONSERVATION PROJECTS

4:0

INTRODUCTION &KEY CONCEPTS

DESIGNING & IMPLEMENTING
THE EVALUATIONEVALUATIONDESIGN:
BEFORE–AFTERDESIGNS

Total change

et
arg
ct T
e
j
Pro

Observed
ChangeBaseline

Time

Figure 1: Before-after design.
These designs rely on comparing the situation before the action
with the one after the action.

Time series design
This is a more robust form of before-after design where data arecollected prior to the action, shortly after the action and several
weeks after the action. This series of assessments records any
change that has taken place and the persistence of that change.
For example actions that involve people are likely to respond wellimmediately after the action but the change may erode over timeas the people revert to their usual social life and habits. Note thatpre-testing can sensitise people to the outcome you are interested
in and can affect the results of subsequent surveys. For example
if the subjects know you are interested in a particular species asa result of questions in the first survey, they may answer morepositively about this species in the second survey. Events apartfrom the action may affect people’s responses to a topic, forexample if the national media pick up a story about an issue thatis similar to the one being addressed by the local project, this may
influence people’s responses.

Single case before and after study
This is the simplest and most commonly used type of before andafter design. It involves simply collecting relevant data on the
project target group/site (e.g. stakeholder awareness, species
population size, resource use), carrying out the project action, thenmeasuring again. These designs are most useful if the outcome/
impact being measured is short-term, for example the immediate
effect of a presentation/workshop on participant attitudes.
However, these designs are poor at evaluating complex or longterm changes, such as changes in community behaviour or in

250

METHOD FACTSHEETS& ANNEXES

species populations. That said, it should be noted that using these
designs is better than not measuring anything at all.

Evaluation design: Before–after designs

EVALUATIONDESIGN
FACTSHEETSPRISMMODULESPRISM - TOOLKIT FOR EVALUATING THE OUTCOMES AND IMPACTS OF SMALL/MEDIUM-SIZED CONSERVATION PROJECTS

4:0

INTRODUCTION &KEY CONCEPTS

DESIGNING & IMPLEMENTING
THE EVALUATION

PRISMMODULES

EXAMPLE
Example: Human-elephant conflict in Assam, India

EVALUATIONDESIGN
FACTSHEETS

The growing human population in Assam and increasing
demand for land has led to fragmentation of habitat, forcing
elephants into conflict with people. This conflict has become
an annual occurrence resulting in loss of crops, destructionof houses, and loss of human lives, and in turn, retaliation

EVALUATIONDESIGN:
BEFORE–AFTERDESIGNSagainst elephants. The main aim of the three-year Assam
Haathi Project (developed and delivered by Chester Zoo and
EcoSystems-India) was to reduce this conflict in two of theworst-affected areas, and to stabilise local people’s toleranceof wildlife and their willingness to participate in conservation.
The short-term effects of the project are evident in the data
gathered (before, during and after conflict-mitigation measures
were introduced) on depredation, damage and attitudes. The
extent of crop-raiding and damage to property decreased
considerably in both study sites (e.g. crop damage in Area 1
reduced from 113 ha in 2006 to 25 ha in 2009), there were no
human injuries or deaths and no elephant deaths. People inproject villages feel that the conflict situation had “improved”,
and that they are now “more able” to deal with depredation
than before.

251

PRISM - TOOLKIT FOR EVALUATING THE OUTCOMES AND IMPACTS OF SMALL/MEDIUM-SIZED CONSERVATION PROJECTSMETHOD FACTSHEETS& ANNEXES

4:0

INTRODUCTION &KEY CONCEPTS

DESIGNING & IMPLEMENTING
THE EVALUATIONMETHOD FACTSHEETS& ANNEXES

Decision tree for selecting evaluation design (adapted from
Woodhouse et al. 2016).

DECISION TREE FOR
SELECTING ANEVALUATION DESIGN

Monitorproject progresswith no measureof attribution

EVALUATIONDESIGN
FACTSHEETSDECISION TREEFOR SELECTING
AN EVALUATIONDESIGN

PRISM
MODULESNO

DON’T KNOW?
Refer back to
PRISM Step 2

OR

Is an evaluationneeded?
Would an evaluationof outcome(s)/impact(s) at thisstage be usefuland feasible?

YES

Are you?
Most interested
in how/whychange has
happened?

YES

YES

Most
interested in
how change has
affected
participants?

YES

Participatory
DesignYES

Theory-based
Design

YES

Project
participantsNO

Are you?
Most interested
in what change
has happened?

Refer back to
PRISM Step 1

NO

Is a strong
Theory of
Changeavailable?

YESNO

Is a real-life
counterfactual
available? (e.g.
control group/comparison site)
and is this feasible
to measure?
YES

NO

Baseline
data available/
will beavailable?

NO

Is it possible
to construct
a baseline/
counterfactualusing knowledge
of:

YES

YES

Participatory
DesignYES

Interviewing
Key Informants

OR
NO

Matching
DesignIs the outcome/impact relatively
short term and unlikely to be inﬂuenced byexternal factors?

YES

Before – After
Design252

PRISM - TOOLKIT FOR EVALUATING THE OUTCOMES AND IMPACTS OF SMALL/MEDIUM-SIZED CONSERVATION PROJECTS

Key
stakeholders
familiar withthe project

4:1

INTRODUCTION &KEY CONCEPTS

DESIGNING & IMPLEMENTING
THE EVALUATION

PRISMMODULES

Awareness & Attitudes factsheets and annexes

Sample questions and scales for evaluating attitudes andawareness actions
Planning a successful attitudes/awareness action

AWARENESS &ATTITUDES
FACTSHEETSAND ANNEXES253

PRISM - TOOLKIT FOR EVALUATING THE OUTCOMES AND IMPACTS OF SMALL/MEDIUM-SIZED CONSERVATION PROJECTSMETHOD FACTSHEETS& ANNEXES

4:1

INTRODUCTION &KEY CONCEPTS

DESIGNING & IMPLEMENTING
THE EVALUATION

Factsheet: Sample questions and scales for evaluating attitudes
and awareness actions

ATTITUDES &AWARENESS
FACTSHEETSAMPLEQUESTIONS ANDSCALES FOREVALUATING
ATTITUDES AND
AWARENESSACTIONS

254

This factsheet provides guidance on developing questionnaires forevaluating actions targeting attitudes and awareness. It outlines
different types of questions and scales that can be considered
in designing a questionnaire. This guidance is to be used inconjunction with the information in the factsheet: Questionnaires
and the module chapter on Attitudes and awareness that gives
details on designing this aspect of evaluation.
Two main types of questions are included in questionnaires:•

Closed questions have a pre-defined set of possible
answers, and they generate quantitative data.
They include questions which give tick boxes or ask
respondents to mark a position on a scale. Closed
questions can be used to gather responses from large
numbers of respondents, and to provide data sets
which are analysed using standard descriptive or more
complex statistical methods.

•

Open questions ask participants to use their own words
to answer the question, and they generate qualitative
data. These questions give greater insight into why
respondents have particular attitudes, awarenessor behaviours, because they allow the researcher toexplore the subject in more detail. Closed questions
could constrain the respondent to a given set ofanswers which may not include all the possible answers,
therefore missing a valuable finding.

PRISM
MODULES•METHOD FACTSHEETS& ANNEXES

It is entirely possible to have a questionnaire which
includes both closed and open questions.

A good questionnaire should be reliable and valid. Reliability
refers to whether the questionnaire consistently measures theintended attitude, and validity refers to whether the questionnairemeasures the attitude it is designed for rather than something else.
Reliability can be assessed by calculating Chronbach’s alpha (whichis possible in any basic statistics programme). Chronbach’s alpha isa calculation that measures the correlation between items (usually
over 10 items) and gives a score between 0 and 1. Questionnaires
with a higher alpha score show that the questions are measuring
the same underlying concept. As a rule of thumb, an alpha score ofover 0.8 is generally considered good. The inter-item correlations,the correlation between pairs of items, should also be examined.
For the inter-item correlation, one rule of thumb is that the score
should be between 0.2 and 0.5 (it should be noted that there issome variation in advice on this, with researchers historically
recommending anything from 0.15 to 0.7). If items are highly
correlated this suggests that one of them may be redundant, since
they are probably both recording very similar things. If items arepoorly correlated they may be unrelated. If high or low correlationis revealed, then it is worth investigating how the overall
Chronbach’s alpha changes when one of the items is removed.
Validity is difficult to assess. It is usually done by examining thecorrelation between the results of the questionnaire and resultsfrom other questionnaires addressing the same concept. Highcorrelation between two different measures indicates that they
are both designed to measure the same construct. Validity can

PRISM - TOOLKIT FOR EVALUATING THE OUTCOMES AND IMPACTS OF SMALL/MEDIUM-SIZED CONSERVATION PROJECTS

4:1

INTRODUCTION &KEY CONCEPTS

DESIGNING & IMPLEMENTING
THE EVALUATION

also be assessed by looking at measures related to the concept
being investigated. For example if the questionnaire is related tobehaviour and is accurately detecting the attitude determining
behaviour, then the questionnaire results should have a strong
relationship with behavioural observations (e.g. in a questionnaire
about people’s attitude towards recycling, you would expect
more positive scores to correlate with people that are observed torecycle the most).

ATTITUDES &AWARENESS
FACTSHEETSAMPLEQUESTIONS ANDSCALES FOREVALUATING
ATTITUDES AND
AWARENESSACTIONS

255

Pilot-testing the questionnaire is crucial to help you understandhow your questions are interpreted and how reliable thequestionnaire is. Testing should be conducted in as similar a setting
as possible to the main survey, with participants who are as similar
as possible to the target audience. This will test whether theparticipant recruitment strategy is likely to succeed and will givean indication of response rate. A pilot test should reveal whether
any unnecessary questions have been included (if two questions
asking about the same subject are very highly correlated then oneof them could be removed since they may both be detecting thesame effect). Piloting will also show whether questions are being
interpreted as intended, or whether they need to be rewritten.
Pilot testing also provides a practice dataset (albeit small) whichshould be used for testing the analytical approaches. It may benecessary to conduct more than one pilot test of a questionnaire,depending on the number of changes made at each test. You may
find that you ask questions!

PRISM
MODULESMETHOD FACTSHEETS& ANNEXESA useful approach in a questionnaire is to ask a number ofquestions which are all assessing the same concept. For example,
assessing attitudes towards a species simply by asking “Do you likesnow leopards?” may only get the answer the respondent thinks
you want to hear. Further questions may clarify their attitude: Do
you think snow leopards are an important wildlife species to havein the local area? Do you think snow leopards cause problems forpeople? Would you like to see a snow leopard in the wild? Should
there be more snow leopards in the area?
Single item measures are common in research and obviously allow
you to cover more in one questionnaire, but their reliability may
be low. When an item is scored (for example using the 1-5 Likert
scale described in the next section) the score will reflect theattitude combined with a random error-risk component. When thequestionnaire has multiple items all addressing the same concept,
this error risk is reduced in the overall score, giving greater overall
reliability than the single item approach.

PRISM - TOOLKIT FOR EVALUATING THE OUTCOMES AND IMPACTS OF SMALL/MEDIUM-SIZED CONSERVATION PROJECTS

4:1

INTRODUCTION &KEY CONCEPTS

DESIGNING & IMPLEMENTING
THE EVALUATION

PRISMMODULESMETHOD FACTSHEETS& ANNEXES

Closed questions
Types of scale
There are several ways of asking questions and measuring the
responses. The type of scale you use will depend on the response
required.

ATTITUDES &AWARENESS
FACTSHEETSAMPLEQUESTIONS ANDSCALES FOREVALUATING
ATTITUDES AND
AWARENESSACTIONS

Dichotomous scale
One of the simplest scales is the dichotomous scale, where
respondents are only able to give one of two opposing answers.
This is obviously unsuitable for more complex situations where
there may be a range of attitudes, but it can be useful for
determining awareness or for some aspects of determining
attitudes towards a subject.

The example below is a question taken from Kellert SR (1984).
American attitudes toward and knowledge of animals: An update.
In MW Fox and LD Mickley (eds.), Advances in animal welfare
science 1984/85, pp. 177-213. Washington, DC: The Humane
Society of the United States.

Q. OF THE FOLLOWING REASONS FOR HUNTING, WHICH DO YOU APPROVE OF OR OPPOSE?
LOCATION

Traditional native hunting (e.g. by Eskimos and Indians)
Hunting game animals for recreation and sport
Hunting waterfowl for recreation and sport
Hunting for meat
Hunting for recreation and meat
Hunting for a trophy

256

PRISM - TOOLKIT FOR EVALUATING THE OUTCOMES AND IMPACTS OF SMALL/MEDIUM-SIZED CONSERVATION PROJECTSAPPROVEOPPOSE

4:1

INTRODUCTION &KEY CONCEPTS

DESIGNING & IMPLEMENTING
THE EVALUATION

Likert scales
Another common approach to scales is to provide a statement
and then ask participants to rate their dis/agreement on a scalefrom “Strongly Agree” to “Strongly Disagree”. The results can thenbe coded numerically, for example from 1-5 or even 1-7 if the end
points are “Very Strongly Agree” and “Very Strongly Disagree”.

ATTITUDES &AWARENESS
FACTSHEETSAMPLEQUESTIONS ANDSCALES FOREVALUATING
ATTITUDES AND
AWARENESSACTIONS

257

A five- or seven-point scale is useful because it allows the
respondent to give more detail than the dichotomous scale, andto add an element of relationship to each answer (for example in
the hunting question above, a respondent may have been more
opposed to hunting for a trophy than they would for meat, but thequestionnaire results only informed the project that they were
opposed to both). These scales give a score to each answer whichallows quantitative analysis of the results, and comparisons canbe represented graphically. It must be recognised that these areordinal data. Scoring 4 is not twice as high as 2 because “agree”
is not twice as good as “disagree”. The scores are ranks, so a lowscore indicates that the respondent disagrees with the statement
while a high score indicates agreement. When multiple statements
are given, it is important to vary the positive and negative framing
so that all the “agree” answers do not align with one attitude.
If the hunting questions above were asked on a 5-point Likert
scale, a statement “hunting of wild animals should be banned”
would have the “strongly disagree” response reflecting the samepro-hunting values of the “strongly agree” responses to the otherstatements. This helps to prevent the respondent disengaging
from the questions and automatically selecting the same answer
throughout the questionnaire. In the example cited, the scoring of
the responses must be reversed to ensure that the final analysis
aligns the same values with the higher scores.

PRISM
MODULESMETHOD FACTSHEETS& ANNEXESThe example below is the short-form Nature Relatedness Scaledeveloped by Nisbet and Zelenski (2013) in “The NR-6: a new brief
measure of nature relatedness” which was developed to measurepeople’s connection to nature. This examines attitudes towards
nature, and has been used in a number of studies, but it waspiloted and developed in Canada so it may not be equally reliablein other cultures. It is also a medium to long-term measure thatwill not detect changes before or after a short intervention – anindividual’s answer about an ideal vacation spot or to any of theother statements is unlikely to change between the time before
a workshop and the time immediately after it, but answers maychange over a year while volunteering on a project.

PRISM - TOOLKIT FOR EVALUATING THE OUTCOMES AND IMPACTS OF SMALL/MEDIUM-SIZED CONSERVATION PROJECTS

4:1

INTRODUCTION &KEY CONCEPTS

DESIGNING & IMPLEMENTING
THE EVALUATION

PRISMMODULESMETHOD FACTSHEETS& ANNEXES

Likert scales
ExampleQ1.

Please state the extent to which you agree or disagree with the following statements (tick one per statement).
STRONGLY
AGREE

ATTITUDES &AWARENESS
FACTSHEETSAMPLEQUESTIONS ANDSCALES FOREVALUATING
ATTITUDES AND
AWARENESSACTIONS

258

A.

My ideal vacation spot would be a remote, wilderness area

B.

I always think about how my actions affect the environmentC.

My connection to nature and the environment is a part of myspirituality

D.

I take notice of wildlife wherever I amE.

My relationship to nature is an important part of who I amF.

I feel very connected to all living things and the earth

PRISM - TOOLKIT FOR EVALUATING THE OUTCOMES AND IMPACTS OF SMALL/MEDIUM-SIZED CONSERVATION PROJECTSAGREENEITHERDISAGREE

STRONGLYDISAGREE

4:1

INTRODUCTION &KEY CONCEPTS

DESIGNING & IMPLEMENTING
THE EVALUATION

Semantic differential scale
Another method is the semantic differential. This type of scale
presents objects and asks the respondent to rate them on a 7-point
scale which has pairs of opposite words at either end. This cangive a further detail on the aspects of a particular subject that the
respondent favours or dislikes.

ATTITUDES &AWARENESS
FACTSHEETSAMPLEQUESTIONS ANDSCALES FOREVALUATING
ATTITUDES AND
AWARENESSACTIONS

259

The following example is taken from an assessment of photographsthat represent different landscapes. The research was carriedout in Japan by Natori and Chenoweth (2008) in “Differences inrural landscape perceptions and preferences between farmers
and naturalists” (Journal of Environmental Psychology 28(3)PRISM
MODULESMETHOD FACTSHEETS& ANNEXES

250-267). Students and farmers were asked to score ten semantic
pairs in relation to how they rated each landscape represented by
the photograph they were presented with. (As the research was
conducted in Japanese, the translations below may have changed
the meanings slightly). The pilot results showed that the individual
pairs (items) and relationships between items (correlation of one
pair with another) represented the variables: preference (pair1), naturalness (2, 3), openness (4), vastness (5), stewardship (6,7), peacefulness (8, 9) and biodiversity (10). Responses to thesequestions may change in the short term, for example following
the viewing of video or reading information about the landscapeconcerned.

Q.

Please imagine you are in the landscape represented in the photograph and circle the number you think describes thelandscape. Circle 4 if you find it difficult to rate or if you are indifferent to the landscape. Q1. Photograph of landscape

1

Like

1

2

3

4

5

6

7

Dislike2

Natural

1

2

3

4

5

6

7

Artificial

3

Following nature’s rule

1

2

3

4

5

6

7

Against nature’s rule

4

Open

1

2

3

4

5

6

7

Enclosed

5

Vast

1

2

3

4

5

6

7

Compact

6

Well cared for

1

2

3

4

5

6

7

Deserted

7

Tidy

1

2

3

4

5

6

7

Cluttered

8

Peaceful

1

2

3

4

5

6

7

Not peaceful

9

Calm

1

2

3

4

5

6

7

Rough

10

Seems rich in life

1

2

3

4

5

6

7

Seems poor in life

PRISM - TOOLKIT FOR EVALUATING THE OUTCOMES AND IMPACTS OF SMALL/MEDIUM-SIZED CONSERVATION PROJECTS

4:1

INTRODUCTION &KEY CONCEPTS

ATTITUDES &AWARENESS
FACTSHEETSAMPLEQUESTIONS ANDSCALES FOREVALUATING
ATTITUDES AND
AWARENESSACTIONS260

DESIGNING & IMPLEMENTING
THE EVALUATIONOpen questions
Open questions are less easy to quantify or compare than closed
questions, but they can generate more detailed information
which can explore why certain attitudes or awareness exist.
Open questions can be very useful for getting information about
perceptions and ideas that the project team may be unaware of.
As with choosing any question that you wish to use, think about
how you will interpret and analyse the answers you receive. Some
examples of open questions are given below, along with somekey points to consider in relation to their wording and structure.
(These points may also be relevant to the wording and structure ofclosed questions).

PRISM
MODULESQ1.

METHOD FACTSHEETS& ANNEXES

Since the training event, have there been occasions when
you have had conversations with others (e.g. friends orfamily) about anything you learned?

This question:•aims to understand how the information from training
may have spread through the community;•

gives a defined timeframe instead of using ambiguous
words that people may understand in different ways.
For example some people understand the word “week”
to mean since Monday, while others may take it to mean
the past seven days. This may also depend on which day
they are asked.

•

carefully tries to ask whether people have talked toothers. Some may think they were not meant to pass
information on, and therefore feel guilty, while others
may not want to divulge what they talk about with theirfriends and family.

•

does not lead the respondent to give an answer
by proposing possible situations where they may
have talked to people. Some questions may requiresuggestions, but this question is aiming to assessvoluntary behaviour, so it is better to avoid suggesting
possible answers.

PRISM - TOOLKIT FOR EVALUATING THE OUTCOMES AND IMPACTS OF SMALL/MEDIUM-SIZED CONSERVATION PROJECTS

4:1

INTRODUCTION &KEY CONCEPTS

Q2.

ATTITUDES &AWARENESS
FACTSHEETSAMPLEQUESTIONS ANDSCALES FOREVALUATING
ATTITUDES AND
AWARENESSACTIONS

261

DESIGNING & IMPLEMENTING
THE EVALUATION

PRISMMODULES

Please describe what improvements, if any, you would maketo the leaflet that has been handed out.

Question examples

This question:•
asks for improvements, to frame the question
in a positive light rather than asking about what
may be wrong and hence inviting criticisism of the
project. People may feel uncomfortable about being
judgemental and negative about anything to do withthe project or the project team, so it is better to frame aquestion positively.

Q3.

Do you think you will change your practice as a result of thistraining? Please tell us how (we may be able to use this tohelp others).

This question:•
tries to evaluate future behaviour by seeking an
estimate of intention;•frames the response in an impersonal way (it may help
others) in order to mitigate any reluctance to respondeither because reporting of personal behaviour isinvolved or because a positive response could otherwise
be interpreted as a commitment for action.

METHOD FACTSHEETS& ANNEXESQUESTIONS ABOUT ATTITUDES
Questions about attitudes investigate the individual’s feelings andemotions in relation to the subject.
1.

Do you like/dislike [the subject]?

2.

Do you approve/oppose ...?

3.

To what extent do you think [the subject] is beautiful/
ugly/useful/useless/ boring/enjoyable/easy/difficult...?

4.

Does this make you feel happy/sad/excited/calm/
restless/worried...?

5.

Which of these [places/pictures/options] is yourpreferred one, and why...?

QUESTIONS ABOUT AWARENESS
Questions about awareness investigate how much someone
knows about the subject.
1.

Please match these [species] names and [species]
pictures.

2.

Are any of these [species] found in the local area?

3.

Have you seen these [species]?

4.

What, if any, threats to [subject] are you aware of?

PRISM - TOOLKIT FOR EVALUATING THE OUTCOMES AND IMPACTS OF SMALL/MEDIUM-SIZED CONSERVATION PROJECTS

4:1

INTRODUCTION &KEY CONCEPTS

DESIGNING & IMPLEMENTING
THE EVALUATIONSAMPLEQUESTIONS ANDSCALES FOREVALUATING
ATTITUDES AND
AWARENESSACTIONSIn what ways do people in this community benefit
from...?

Evaluation design

6.

Does this activity affect you...?

Pre–post (before-after) questionnaire
A “before and after” questionnaire asks participants to rate (forexample) their level of knowledge before the action and again
afterwards.

Questions about behaviour try to get people honestly to reportwhat they do and how it may have changed.
1.

In the last seven days, have you...?

2.

Do you have a bird feeder/pond/wood fire in thegarden/house...?

3.

Since [the action] have you had the opportunity to do[requested behaviour]...?

4.

Trying to recall what you did in [month before project
started] did you ever [behaviour]...?

5.

In the last summer/winter/wet season did you...?

Where possible, it may be better to measure behaviour using
methods that observe either the behaviour or indicators of
the behaviour unobtrusively. For example, changes in hunting
practices may be observed by measuring changes in the typesof meat available at the local market. Monitoring the count offootprints on a path may help you find out how many people areusing the woodland, and recording the locations of domestic
livestock may reveal changes in herding practices.

262

METHOD FACTSHEETS& ANNEXES5.

QUESTIONS ABOUT BEHAVIOUR

ATTITUDES &AWARENESS
FACTSHEET

PRISM
MODULESIt is also possible to ask participants after the action to describe
how they felt prior to it, but this is dependent on the reliability ofthe participants’ memory.
The focus of the questions should reflect the aims of theaction that you have identified at the beginning of the project.
Questionnaires that are perceived as irrelevant (or as too long)
may reduce the response rate you achieve.
Post only (after) questionnaire
A questionnaire which is only delivered after an event, rather thanboth before and after the event, is a post only questionnaire. This
may provide a snapshot of participant awareness, attitudes or
behaviour. Again the length and relevance of the questionnaireneed to be considered carefully if you are to achieve the best rateof response.
Post questionnaires can be administered immediately after anattitudes or awareness action and then at intervals afterwards.
To get a good response rate, the best approach is to ask people tocomplete the questionnaire while they are still at the event venue.
If you wish to carry out a post questionnaire some time after theaction, then you should consider how you will encourage people tofill in their responses and return them.

PRISM - TOOLKIT FOR EVALUATING THE OUTCOMES AND IMPACTS OF SMALL/MEDIUM-SIZED CONSERVATION PROJECTS

4:1

INTRODUCTION &KEY CONCEPTS

DESIGNING & IMPLEMENTING
THE EVALUATION

Annex: Planning a successful attitudes/awareness action

ATTITUDES &AWARENESSANNEX
PLANNING A
SUCCESSFULATTITUDES/
AWARENESSACTION

Conservation projects targeting biodiversity outcomes may
include actions to raise people’s awareness of a subject, or change
public or stakeholder attitudes. This is often in an effort tocatalyse a behaviour change, which can be essential to achieving
biodiversity outcomes. Increasing awareness or changing
attitudes can lead to behaviour changes, but the relationships
between these elements are complex and behaviour change isdifficult to achieve. Behaviour change models attempt to explainthese relationships, and they can be used to guide attitude andawareness actions. Such models may include adaptations of the
Theory of Reasoned Action, Theory of Planned Behaviour, Social
Cognitive Theory, Social Norms Theory, the Stages of Change
Model and other integrative models. Three components of a
successful action are design, planning and communication. A usefulacronym for these actions is MAST: Message, Audience, Strategy,
Timing; and this can be useful for structuring your thinking about
each of the three components.
The message of the action will be defined by the desired outcome.
The message should be shaped to resonate with the target
audience, relating to their interests, values and culture.

PRISM
MODULESThe communication strategy will be affected by a number offactors including the method (for example spoken or visual), themessenger (for example credibility, personality, culture) andenvironmental factors that may have little to do with the message(for example indoor or outdoor setting, lighting, seating, noise,prior audience understanding of the topic).
Timing of events is also crucial both in terms of the event
itself, for example avoiding clashes with other activities, and interms of message timing, for example it may not be effective
to try to change a behaviour that occurs in winter by means ofcommunications delivered in summer.
Evaluating the effect of actions designed to influence attitudes
and awareness is a critical part of a project and should ideallybe planned in the design stage. The effect of the actions may
be seen both within the project and within the audience, and it
may occur over a variety of timescales. It is critical that a robust
Theory of Change is used to associate the action with the target
outcome. Although it may not be possible to evaluate the effectof the action on the desired biodiversity outcome, interim stages
can be evaluated and can provide useful intelligence for internal
effectiveness monitoring and for reporting.

It is crucial to identify the appropriate audience for the message,
which may be the individual people whose behaviour could
influence the success of the conservation goal, the influential
peer group around these people, or influential groups in the wider
community.

263

METHOD FACTSHEETS& ANNEXESPRISM - TOOLKIT FOR EVALUATING THE OUTCOMES AND IMPACTS OF SMALL/MEDIUM-SIZED CONSERVATION PROJECTS

4:1

INTRODUCTION &KEY CONCEPTS

DESIGNING & IMPLEMENTING
THE EVALUATION

Framework for attitudes and awareness action planning

ATTITUDES &AWARENESS
ANNEX

PRISM
MODULES

Step 3.

The framework (Figure 1, Attitudes & awareness module) provides
a structure for developing a Theory of Change for attitude andawareness actions in any project. As with any action planning
it is necessary to go through several steps to identify the best
approach. The planning steps below work backwards along theTheory of Change, using current understanding of behaviour andpreliminary research, to ensure each step is linked to the desired
final outcome.
Step 1.

PLANNING A
SUCCESSFULATTITUDES/
AWARENESSACTION

Identify the outcome of interest for the project (i.e.
the desired improvement in biodiversity status).
This is the final (green) box in the framework and theultimate goal of the project. With high-impact actions
and a rapidly responding biodiversity goal, there may
be effects seen at this level, however it is unlikely thatmost attitude and awareness actions in small- medium
sized projects will be able to detect ultimate effects on
biodiversity.

Step 2.

Identify the threat to biodiversity which needs to be
reduced, and any human behaviours which cause thethreat.

Identify the desired human behaviour outcome.

Point 4 in the framework requires you to identify the
behaviour that is causing the threat to biodiversity.
Change in this behaviour will then be the outcome
for the attitude/awareness action. This requires anunderstanding of the links between the behaviour
and the impact on biodiversity, and how a change inbehaviour can contribute to a change in this impact.

Step 4.

Identify target audience linked to behaviour.
In order to influence attitudes, awareness or knowledge(points 2 and 3 in the framework) successfully, it isnecessary to identify the audience. Most awareness andattitude interventions will need to focus on a particularaudience. Even seemingly ubiquitous behaviours needto have a target audience identified, e.g. plastic carrierbags are used by many people, but different messages
and catalysts are likely to be required to produce a
response in different groups. It is necessary to focus
the action on the correct audience to ensure theappropriate link to the desired behaviour change.

This refers to the pink box in the framework, identifying
the factors that need to be altered if the biodiversity
goal is to be achieved.

264

METHOD FACTSHEETS& ANNEXESPRISM - TOOLKIT FOR EVALUATING THE OUTCOMES AND IMPACTS OF SMALL/MEDIUM-SIZED CONSERVATION PROJECTS

4:1

INTRODUCTION &KEY CONCEPTS

Step 5.

ATTITUDES &AWARENESSANNEX
PLANNING A
SUCCESSFULATTITUDES/
AWARENESSACTION265

DESIGNING & IMPLEMENTING
THE EVALUATION

Investigate why the behaviour occurs, by means ofresearch, for example using interviews with people
who exhibit the behaviour.

PRISM
MODULES

Step 6.

Develop an appropriate behaviour model forachieving the desired outcome(s).
Building a theoretical model which describes thefactors driving the behaviour will enable the project
to be planned more effectively in complex situations,
helping it to clarify the objective of the action and to
be more relevant to other work in future. This model
should be developed using the information gathered in
the previous steps, and should be informed by existing
behaviour change models such as those mentioned
previously.

Research should investigate attitudes and awareness
relating to the target biodiversity issue, other related
behaviour and its impacts, and potential alternative
behaviours. This will aim to address the motivations forthe current behaviour, barriers to adopting a different
behaviour, and potential enablers for adopting a new
behaviour. Collecting appropriate observational
data can provide an additional (often more objective)assessment of current behaviour.
Points 2 and 3 in the framework also require anunderstanding of the change in attitude, awareness or
knowledge that is required to influence the behaviour
at issue. In many situations, unexpected factors may
drive behaviour in complex ways, and these will not
be discovered without discussions with the people
involved. This also requires an open mind on the partof the researchers who may be focused on biodiversity
conservation but who may encounter many other values
and motives in this phase of research. After gathering
ideas and data from several sources, similarities inperceptions and suggestions for the behaviour canhighlight issues that need to be addressed.

METHOD FACTSHEETS& ANNEXES

Step 7.

Develop action to address the target attitude/awareness issue.
This is point 1 in the framework. Once a specificbehaviour change and target audience have been
defined, it is necessary to design the best method ofcommunicating to the audience, so that the action has
the best chance of achieving the intended behaviourchange. This should increase the chance of successhappening in accordance with the defined pathway fromaction> target behaviour> desired behaviour change>
reduction in threat> improvement in biodiversity status.

PRISM - TOOLKIT FOR EVALUATING THE OUTCOMES AND IMPACTS OF SMALL/MEDIUM-SIZED CONSERVATION PROJECTS

4:2

INTRODUCTION &KEY CONCEPTS

DESIGNING & IMPLEMENTING
THE EVALUATION

PRISMMODULESCapacity Development factsheets and annexes

Example questions and scales for training evaluation
Organisational capacity assessment tool
Network health scorecard

CAPACITY
DEVELOPMENT
FACTSHEETS AND
ANNEXES266

PRISM - TOOLKIT FOR EVALUATING THE OUTCOMES AND IMPACTS OF SMALL/MEDIUM-SIZED CONSERVATION PROJECTSMETHOD FACTSHEETS& ANNEXES

4:2

INTRODUCTION &KEY CONCEPTS

DESIGNING & IMPLEMENTING
THE EVALUATION

Factsheet: Training evaluation form: example questions andscalesCAPACITY
DEVELOPMENT
FACTSHEETTRAINING
EVALUATION
FORM: EXAMPLEQUESTIONS ANDSCALES

267

Training evaluation surveys tend to be short - a good guide is tomake sure that it takes no more than five minutes to complete. Try
it out yourself first! There are two types of questions: quantitative
questions are closed-ended (tick the box, or scales), while openended questions ask for written feedback. Quantitative questions
are extremely useful because they provide numbers that you cananalyse as part of your evaluation, with scope to compare acrossyears or sites, for example. Qualitative questions provide moredetail – they can help you judge why participants thought thetraining was useful/not useful; but they are less easy to analyse. It
is best to have a combination of the two types of questions.
The template below provides examples of different types ofquestions and scales that can be considered in designing a training
evaluation form. You can modify the questions to suit yourparticular training programme or event. You may not want to
ask all the questions in each section -choose those that are mostappropriate to your aims. See the factsheet Questionnaires forgeneral guidance on designing and implementing a questionnaire
survey.

PRISM
MODULESMETHOD FACTSHEETS& ANNEXES

Post-only surveys (Feedback forms)
A post-training evaluation form asks participants to rate theeffectiveness of various aspects of the training. The focus of thequestions should reflect the aims of the training that you haveidentified at the beginning of the project. Asking questions that do
not relate to the evaluation of your capacity development actions
is a waste of time and effort. Questionnaires that are too long (orare perceived as irrelevant) may also affect response rates you
achieve.
It is recommended that you keep to one type of scale so as not to
confuse the respondents.
You can cut and paste the question formatting into a new
document and modify as needed.

Five-point scale
The example below uses a five-point scale from “strongly agree”(SA) to “strongly disagree” (SD).
A five-point scale is useful because it allows people to choose ascore that is likely to match their true position. Using numbers
allows you to carry out analyses and to present them graphically.

PRISM - TOOLKIT FOR EVALUATING THE OUTCOMES AND IMPACTS OF SMALL/MEDIUM-SIZED CONSERVATION PROJECTS

4:2

INTRODUCTION &KEY CONCEPTS

DESIGNING & IMPLEMENTING
THE EVALUATION

PRISMMODULESMETHOD FACTSHEETS& ANNEXESQUESTIONS ABOUT THE PRESENTER AND MATERIALS
Please state the extent to which you agree or disagree with the following statements, where 1 is Strongly Agree and 5 is Strongly Disagree(tick one choice per statement).
STRONGLY
AGREE (1)CAPACITY
DEVELOPMENT
FACTSHEETTRAINING
EVALUATION
FORM: EXAMPLEQUESTIONS ANDSCALESA.

The presenter communicated the information clearly.

B.

The presenter engaged the audience.

C.

The presenter made the subject matter compelling.

D.

The presented was able to answer questions.

E.

The content was presented in a well-structured manner.

F.

The pace of the presentation was right for me.

G.

The handouts/supporting material were useful.

NEITHER(3)

DISAGREE(4)

STRONGLYDISAGREE (5)QUESTIONS ABOUT THE EFFECTIVENESS OF THE TRAINING
Please state the extent to which you agree or disagree with the following statements, where 1 is Strongly Agree and 5 is Strongly Disagree(tick one choice per statement).
STRONGLY
AGREE (1)

268

AGREE
(2)

A.

The training was relevant to my work.

B.

The training content was meaningful to my work.

C.

The training met my purpose in attending.

D.

The training content was related to the skills and knowledge I needed.

E.

The training made me think about my own work.

F.

The training motivated me to change how I deliver my work.

G.

I want to tell others about the useful things that were presented.

H.

I have the confidence to use the knowledge gained from thetraining in my work.

AGREE
(2)

PRISM - TOOLKIT FOR EVALUATING THE OUTCOMES AND IMPACTS OF SMALL/MEDIUM-SIZED CONSERVATION PROJECTSNEITHER(3)

DISAGREE(4)

STRONGLYDISAGREE (5)4:2

INTRODUCTION &KEY CONCEPTS

DESIGNING & IMPLEMENTING
THE EVALUATIONOpen Questions
Open questions are less easy to quantify or compare than closed
ones, but they can give you more depth of useful information.
Think about how you will interpret and analyse the answers when
you are choosing the open questions you are going to use.

CAPACITY
DEVELOPMENT
FACTSHEETTRAINING
EVALUATION
FORM: EXAMPLEQUESTIONS ANDSCALES

Q1
Choose one of these:
What was the most useful aspect of the training to you?
What was the best aspect of the training?
What did you learn that you did not know before?

Q2
Choose one of these:

PRISM
MODULES

Before and after questionnaire
A before and after questionnaire asks participants to rate theirlevel of knowledge before the training and again afterwards. The
focus of the questions should reflect the aims of the training thatyou have identified at the beginning of the evaluation. Asking
questions that do not relate to the evaluation of your capacity
development actions is a waste of time and effort. Questionnairesthat are too long (or are perceived as irrelevant) may also affect the
response rates you achieve.
These questionnaires are best completed anonymously, which
requires a level of organisation and planning that people do notalways have. For example, you could give everyone a numberedquestionnaire and ask them to remember the number allocatedto them (this is done in some school exams, for example).
Alternatively you could use an on-line method such as Survey
Monkey.

What aspect of the training would you like to see improved?
What could we have done differently?
What else would you like to see included in the training?

Q3
How do you hope to change your practice as a result of thistraining?

269

METHOD FACTSHEETS& ANNEXESPRISM - TOOLKIT FOR EVALUATING THE OUTCOMES AND IMPACTS OF SMALL/MEDIUM-SIZED CONSERVATION PROJECTS

4:2

INTRODUCTION &KEY CONCEPTS

DESIGNING & IMPLEMENTING
THE EVALUATION

Post-only questionnaire
Post-training questionnaires can be administered straight after the
event and then at intervals afterwards. The best way to get people
to fill in a questionnaire immediately after the training is to ask
them to do it while they are still at the venue. If you wish to carryout a post-training questionnaire some time after the training, thenit is useful to consider how you will encourage people to fill in andreturn the forms.

CAPACITY
DEVELOPMENT
FACTSHEETTRAINING
EVALUATION
FORM: EXAMPLEQUESTIONS ANDSCALES270

Try to keep the questionnaire as short as possible - a good
guide is to make sure that it takes no more than five minutes
to complete. Try it out yourself first! There are two types ofquestions: quantitative questions are closed-ended (tick the box,
or scales), while open-ended questions ask for written feedback.
Quantitative questions are extremely useful because they provide
numbers that you can analyse as part of your evaluation, withscope to compare across years or sites, for example. Qualitative
questions provide more detail – they can help you judge why
participants thought the training was useful/not useful; but they
are less easy to analyse. It is best to have a combination of the two
types of questions.

PRISM
MODULESMETHOD FACTSHEETS& ANNEXES

You can cut and paste the question formatting into a new
document and modify as needed.

Scales
The examples below show a series of scales and different options
for presenting questions in a before and after questionnaire.
When you are designing your own questionnaire it isrecommended that you keep to one type of scale so as not to
confuse the respondents.

PRISM - TOOLKIT FOR EVALUATING THE OUTCOMES AND IMPACTS OF SMALL/MEDIUM-SIZED CONSERVATION PROJECTS

4:2

INTRODUCTION &KEY CONCEPTS

DESIGNING & IMPLEMENTING
THE EVALUATION

PRISMMODULESMETHOD FACTSHEETS& ANNEXESQUESTIONS ABOUT KNOWLEDGE AND SKILLS
Using a scale of 1 to 5 where 1 = none at all and 5 = a lot, please rate the following:
QUESTION

BEFORE THE TRAINING

AFTER THE TRAINING

I rate my knowledge in (fill in topic here but make it specific)

CAPACITY
DEVELOPMENT
FACTSHEETTRAINING
EVALUATION
FORM: EXAMPLEQUESTIONS ANDSCALES

I rate my skills in (fill in topic here but make it specific)

QUESTIONS ABOUT CONFIDENCE AND MOTIVATIONThis questionnaire needs to be distributed before the training and then again after the training.
BEFORE TRAININGQUESTIONSTRONGLYDISAGREEDISAGREE

AGREE

AFTER TRAINING
STRONGLY
AGREE

I feel motivated to apply
skill X in my work
I have the confidence tomake the right decisions
about my conservation
work
…….

271

PRISM - TOOLKIT FOR EVALUATING THE OUTCOMES AND IMPACTS OF SMALL/MEDIUM-SIZED CONSERVATION PROJECTSSTRONGLYDISAGREEDISAGREE

AGREE

STRONGLY
AGREE

4:2

INTRODUCTION &KEY CONCEPTS

DESIGNING & IMPLEMENTING
THE EVALUATION

PRISMMODULESMETHOD FACTSHEETS& ANNEXESQUESTIONS ON THE APPLICATION OF SKILLS AND THE IMPACT OF THIS IN YOUR WORK
Question 1a) What conservation contribution have you made since returning to work?

CAPACITY
DEVELOPMENT
FACTSHEETb) How influential was [insert specific training provision] in you making this contribution?
Extremely influential, very influential, neutral, not influential.
Question 2

TRAINING
EVALUATION
FORM: EXAMPLEQUESTIONS ANDSCALESHow important has the training from us been in your career?
Extremely important, very important, neutral, not important.
Question 3
Compared to the time before the project, have you used any of the skills/knowledge acquired during the training? (Yes/No).
If yes, please provide at least one example. If no, please explain why not.

Question 4
Compared to the time before the project, do you see a change in your work as a result of applying your new skills?
If yes, please provide at least one example. If no, please explain why not.

272

PRISM - TOOLKIT FOR EVALUATING THE OUTCOMES AND IMPACTS OF SMALL/MEDIUM-SIZED CONSERVATION PROJECTS

4:2

INTRODUCTION &KEY CONCEPTS

DESIGNING & IMPLEMENTING
THE EVALUATION

Example of a before & after questionnaire used to evaluatechanges in skills and knowledge:

CAPACITY
DEVELOPMENT
FACTSHEETTRAINING
EVALUATION
FORM: EXAMPLEQUESTIONS ANDSCALES

A. Kanagavel, S. Parvathy & N. Divakar,
Education workshops improve the ability ofForest Departments to identify amphibiansin Western Ghats, India Conservation
Evidence (2017) 14, 21

273

Action: A project in Munnar, Kerala conducted two one-day
amphibian identification workshops aiming to improve theamphibian identification skills of Forest Department employees.
Workshops consisting of classroom and field sessions, wereconducted in Munnar, Kerala with 26 participants from three
regional Forest Departments. Classroom sessions consisted ofvisual presentations detailing the importance of amphibians andtheir diversity in the Western Ghats, specifically focusing onfive amphibians in Munnar: W. phrynoderma and Rhacophorus
pseudomalabaricus (both critically endangered), Micrixalus
adonis (not evaluated), Ghatixalus asterops (data-deficient) andRaorchestes beddomii (near threatened).
Evaluation: A survey was conducted before the workshop
to record participants’ prior knowledge of the five species.
Information on whether they recognized the species, its common
name in Malayalam and the habitat it was found in were requested
after showing them photographs of the species. An identicalsurvey was repeated four months later with 14 respondents tounderstand whether the workshop had improved their skills. The
remaining 12 workshop participants were not available after beingtransferred to work within other forest departments.
Results and interpretation: Four months after the workshop,
Forest Department employees had improved their recognition
skills for W. phrynoderma, R. pseudomalabaricus, M. adonis and R.
beddomii and the habitats they occurred in (Table 1).

PRISM
MODULESMETHOD FACTSHEETS& ANNEXES

The team concluded that these species were relatively easy
to distinguish and should act as focal species during official
monitoring of amphibian populations. Participants also had
substantial previous knowledge of R. pseudomalabaricus and M.
adonis. There was less improvement in identifying G. asterops
as it was probably not sufficiently distinct or characteristic.
The team found that the common name of all species was notretained by most respondents (Table 1), possibly because
common names were difficult to memorize and folk taxonomy
for most amphibians is generic and not specific to the species
level. This could be improved through a greater encouragementfrom the higher officials of the Forest Department for their staff
to equip themselves with such knowledge. Overall, workshops
improved the knowledge of Forest Department employees and,if undertaken annually, could consistently improve their skills in
identifying amphibians and eventually conserving them.

TABLE 1.
Number of Forest Department employees without prior knowledge thatsuccessfully identified amphibians and related information four months
after attending an education workshop (n=14, those with prior knowledge
excluded).
SPECIESRECOGNITIONCOMMON NAMEHABITAT

YES

NO

YES

NO

YES

NO

W. phrynoderma

14

0

2

12

11

3

R. pseudomalabaricus

2

1

2

12

10

4

M. adonis

5

3

3

11

9

2

G. asterops

3

8

0

14

6

8

R. beddomii

5

7

2

12

6

8

PRISM - TOOLKIT FOR EVALUATING THE OUTCOMES AND IMPACTS OF SMALL/MEDIUM-SIZED CONSERVATION PROJECTS

4:2

INTRODUCTION &KEY CONCEPTS

CAPACITY
DEVELOPMENT
FACTSHEETORGANISATIONALCAPACITYASSESSMENT
TOOL
STEP 1DESIGNING & IMPLEMENTING
THE EVALUATION

PRISMMODULES

Factsheet: Organisational Capacity Assessment tool

Required steps

Summary and introduction
Organisational capacity assessment tools (OCAs) can be used to
produce a baseline against which an organisation may be tracked,
and the impact of projects that aim to build its capacity can beevaluated. Used at the start of a project, an OCA also helps to
identify areas for improvement, and it can be used to informthe development of an action plan to address key institutional
weaknesses.

Step 1.

This factsheet explains how to use the in-built Organisational
Capacity Assessment tool on the Capacity for Conservation(CfC) website to evaluate actions that aim to develop and improveorganisational capacity.
CfC is a growing resource, developed and managed by several of
the world’s foremost international conservation organisations
and designed to support NGOs to develop their organisational
capacity. As well as the Organisational Capacity Assessment
tool, CfC also provides a template and guide for developing anorganisational plan, an extensive toolkit of capacity development
resources and a list of indicators that can be used to track changes
in organisational capacity.

Familiarise yourself with the tool.

The OCA tool can be accessed on the CfC website http://
capacityforconservation.org .
If you are not already signed up then you can register theorganisation and create a username and password for eachindividual who completes the assessment.
Signing up also allows you to access the CfC development plan,
toolkit and draft indicators.
The tool can be completed online or can be downloaded as an
Excel file and uploaded to the site at a later date. The tool storesassessment results automatically and aggregates responses acrossall organisation staff who complete the assessment.

Here we specifically describe how this resource can be used as partof a before and after evaluation design which uses scores assigned
to key areas of organisational capacity, to show change. The scoresshould be supplemented with justifications based on discussions
with those who completed the assessment and, if necessary, otherrelevant stakeholders.

274

METHOD FACTSHEETS& ANNEXESPRISM - TOOLKIT FOR EVALUATING THE OUTCOMES AND IMPACTS OF SMALL/MEDIUM-SIZED CONSERVATION PROJECTS

4:2

INTRODUCTION &KEY CONCEPTS

Step 2.

DESIGNING & IMPLEMENTING
THE EVALUATION

Identify the key areas of organisational capacity to beevaluated.

PRISM
MODULESCfC tool section on internal communications.

This will typically address the areas that were targeted by thecapacity development action. The CfC tool covers the following
areas of organisational capacity development:•
Organisational planning and management

CAPACITY
DEVELOPMENT
FACTSHEET•

Governance•

Human resources and staff management•

Leadership

ORGANISATIONALCAPACITYASSESSMENT
TOOL•Internal communications•

Fundraising•

Finance management•

Monitoring, evaluation and learning•

Project Planning and Management•

Constituency•

External communications•

Partnerships and networking.

STEP 2

Under each of these headings the tool contains a series ofquestions which each relate to a different aspect of organisational
capacity. It allows users to score each question as “not there”,“getting there”, “good” or “NA” (see below).

275

METHOD FACTSHEETS& ANNEXESPRISM - TOOLKIT FOR EVALUATING THE OUTCOMES AND IMPACTS OF SMALL/MEDIUM-SIZED CONSERVATION PROJECTS

4:2

INTRODUCTION &KEY CONCEPTS

Step 3.

DESIGNING & IMPLEMENTING
THE EVALUATION

Establish a baseline.

PRISM
MODULESMETHOD FACTSHEETS& ANNEXES

You can then click on the link “view detailed breakdown” to see adisaggregation of the overall score for each area.

Ideally this should be based on an assessment carried out before
the project begins. Such an assessment can also be vital in
identifying the aspects of capacity to target during the project andin focusing effort where it is most needed.

CAPACITY
DEVELOPMENT
FACTSHEETORGANISATIONALCAPACITYASSESSMENT
TOOL
STEP 3276

To complete the online tool, individuals can complete the full
assessment, or can focus on a limited number of the areas that are
applicable to the organisation and/or the capacity development
action concerned (fundraising, internal communications, etc.).
If possible the assessment should be completed by multiple
individuals from across the organisation. Care should be taken to
ensure that the scoring is completed as honestly and as objectively
as possible.
Once completed, the tool aggregates scores from across theorganisation and displays the results as percentage scores for eacharea of organisational capacity assessed.

If this is the first time that someone in the organisation has
completed the tool then this will act as a baseline for theevaluation. If scores for the organisation have previously beenentered into the site then you should use the scores recorded
immediately before the capacity development action begins as
your baseline.
If an assessment was not carried out before the capacity
development action, the scoring can still be carried out
retrospectively (after the project), although greater care must
be taken to ensure the accuracy of the results. Users can also
contact the site’s administrators who can download and provide
a breakdown of individual scores given at a specific time and by
specific individuals.

PRISM - TOOLKIT FOR EVALUATING THE OUTCOMES AND IMPACTS OF SMALL/MEDIUM-SIZED CONSERVATION PROJECTS

4:2

INTRODUCTION &KEY CONCEPTS

Step 4.

DESIGNING & IMPLEMENTING
THE EVALUATION

Carry out the post-project assessment.

Once the capacity development action is complete, fill in the tool
again to describe the situation after the action. Ideally the same
individuals who carried out the baseline assessment should also
complete the scoring after the project, so as to reduce bias andenhance the validity of the results.

CAPACITY
DEVELOPMENT
FACTSHEETORGANISATIONALCAPACITYASSESSMENTTOOLSTEPS 4, 5 & 6

PRISM
MODULESThis information can be summarised in the table below.

COMPARE AND DISCUSS SCORING
This information can be summarised in the table below.
CAPACITY AREA

% SCORE BEFORE% SCORE AFTERCHANGEJUSTIFICATION (REASONS FOR CHANGE)

Rather than completing the entire tool again you can complete
the sections that are most relevant to the evaluation. Note thatchanges in capacity in one area will often have knock-on effects inother areas, so you should bear this in mind when deciding which
sections to complete in the post-project assessment.
Step 6.
Step 5.

Complete follow-up assessments to track changes
over time.

Compare and discuss scoring.

Compare and contrast the scores given before and after thecapacity development project to see whether or not there has
been a change in capacity since the project began. The change inthe scores (if any) indicates whether the project had a positive,
negative or neutral effect on organisational capacity.

It is recommended that individuals in the organisation should plan
to review and complete the assessment at agreed points in the
future, so as to track long term changes in capacity and to continue
to identify areas for development.

Scores should be discussed with the individuals who completed theassessment, and a justification should be provided for any changes(or lack of change). The justifications are just as important as thescoring, as they helps to clarify how any change has come about,whether it can be attributed to the project and what lessons can belearnt.

277

METHOD FACTSHEETS& ANNEXESPRISM - TOOLKIT FOR EVALUATING THE OUTCOMES AND IMPACTS OF SMALL/MEDIUM-SIZED CONSERVATION PROJECTS

4:2

INTRODUCTION &KEY CONCEPTS

DESIGNING & IMPLEMENTING
THE EVALUATION

ExampleProject: The Forest Protection League (FPL) is an organisationthat works to support small NGOs which campaign against theconversion of tropical forest to timber plantations in South East
Asia. FPL carried out a project to support another NGO, Jambi
Forest Alliance (JFA), to develop their organisational capacity.

CAPACITY
DEVELOPMENT
FACTSHEETORGANISATIONALCAPACITYASSESSMENT
TOOL

PRISM
MODULES

Post–project assessment: After the FPL project had taken place,the same individuals who completed the organisational assessment
before the project were asked to complete the assessment again,
this time to reflect the situation after the project.

Jambi Forest Alliance organisational fundraising capacity
scores - after the project.

Establishing the baseline: An initial organisational capacity
assessment completed by individuals across JFA identified a
specific lack of capacity in relation to fundraising. This thenhelped FPL to design and implement appropriate actions to help
build JFA’s fundraising capacity, including training in proposal
development, identifying and engaging with donors anddeveloping a fundraising plan for the organisation. This initialassessment also provided a baseline for the evaluation.

Jambi Forest Alliance organisational fundraising capacity
scores - before the project.

278

METHOD FACTSHEETS& ANNEXESPRISM - TOOLKIT FOR EVALUATING THE OUTCOMES AND IMPACTS OF SMALL/MEDIUM-SIZED CONSERVATION PROJECTS

4:2

INTRODUCTION &KEY CONCEPTS

DESIGNING & IMPLEMENTING
THE EVALUATION

Comparing and discussing scoring: Discussion with the individuals
who completed the before and after assessment provided
justification for the scores. The main reasons provided for thechanges in scores all related to the capacity development actions
implemented by the FPL project.

CAPACITY
DEVELOPMENT
FACTSHEETORGANISATIONALCAPACITYASSESSMENT
TOOL

279

PRISM
MODULESMETHOD FACTSHEETS& ANNEXES

Acknowledgements
Capacity for Conservation: http://capacityforconservation.org

COMPARE AND DISCUSS SCORING
The changes in scores all related to the capacity development actions implemented by the FPL project
CAPACITY AREA

% SCORE BEFORE% SCORE AFTERCHANGEJUSTIFICATION (REASONS FOR CHANGE)

Fundraising - Overall

14

68

+54

Improvement in percentage scores for all aspects offundraising capacity by 50+

Fundraising – Policy on donors

0

50

+50

Policy on donors now exists, awaiting approval fromboard

Fundraising – Fundraising plan

33

100

+67

Comprehensive fundraising plan now in place and fully
agreed by the organisation

Fundraising – Proposal
development

0

50

+50

Relevant staff given training in proposal development

Fundraising – Donor
diversification

17

67

+50

Fundraising plan now targets a wider range of donors,
both national and international

Fundraising – Unrestrictedfundraising plan

0

50

+50

Fundraising plan now includes strategy for unrestrictedfunding

Fundraising – Donor reporting

25

75

+50

Fundraising plan assigns roles and responsibilities formeeting reporting requirements agreed across theorganisation

PRISM - TOOLKIT FOR EVALUATING THE OUTCOMES AND IMPACTS OF SMALL/MEDIUM-SIZED CONSERVATION PROJECTS

4:2

INTRODUCTION &KEY CONCEPTS

DESIGNING & IMPLEMENTING
THE EVALUATION

PRISMMODULESCAPACITY
DEVELOPMENTANNEX
NETWORKHEALTHSCORECARD

Available from: http://networkimpact.
org/downloads/NH_Scorecard.pdf

280

PRISM - TOOLKIT FOR EVALUATING THE OUTCOMES AND IMPACTS OF SMALL/MEDIUM-SIZED CONSERVATION PROJECTSMETHOD FACTSHEETS& ANNEXES

4:3

INTRODUCTION &KEY CONCEPTS

DESIGNING & IMPLEMENTING
THE EVALUATION

PRISMMODULES

Livelihoods and Governance factsheets and annexes

Participatory Impact Assessment (PIA) methods
PIA Method 1: Before and after scoring
PIA Method 2: Proportional piling

LIVELIHOODS AND
GOVERNANCE
FACTSHEETS AND
ANNEXESPIA Method 3: Tally method
PIA Method 4: Matrix scoring
PIA Method 5: Before and after impact calendars
Community Based Organisation Capacity Assessment
Community Mapping
Basic Necessities Survey (BNS)
Participatory Governance Assessment
Participatory photo evaluation

281

PRISM - TOOLKIT FOR EVALUATING THE OUTCOMES AND IMPACTS OF SMALL/MEDIUM-SIZED CONSERVATION PROJECTSMETHOD FACTSHEETS& ANNEXES

4:3

INTRODUCTION &KEY CONCEPTS

LIVELIHOODS &GOVERNANCE
FACTSHEETPIA METHOD 1:
BEFORE AND
AFTER SCORINGOPTION 1
STEPS 1, 2 & 3

DESIGNING & IMPLEMENTING
THE EVALUATION

PRISMMODULESMETHOD FACTSHEETS& ANNEXESPIA Method 1: Before and after scoring

Required steps

Introduction
This method is designed to be used as part of a Participatory
Impact Assessment (PIA) evaluation design. Refer to the factsheet
Evaluation Design: Participatory Impact Assessment for more
information on how to use this method as part of a PIA.

The choice of scoring method used for before and after scoring
has a considerable effect on the interpretation of the results. In
general, two main approaches are used:

Simple scoring requires informants to use counters such as seeds,stones, nuts or beans to assign a specific score to each item orindicator in a list. For example, ten counters per item might bemade available and people asked to assign scores of between 0 and
10 depending on the importance of the item.
Before and after scoring adapts and expands simple scoring tocompare impacts or items at two points in time, typically before a
project and then during or after the project. Definitions of “before”,“after” or “during” are derived from project timelines. This method
is particularly useful for measuring impact where project baseline
data are weak or non-existent.
If the method is standardised and repeated with a sufficientnumber of informants it is possible to present results frombefore and after scoring using conventional statistics, with meanscores and 95% confidence intervals for statistically significant
differences.

Option 1: using the same number of counters “before” and“after”.
Step 1.

Give the informant a number of counters, e.g. 20,
and ask them to assign these counters to show theimportance of different sources of income (or otherindicator of project impact) before the project.

Step 2.

When they have finished, give the informant another20 counters and ask them to distribute them to show
the situation after the project. Note that because thismethod uses the same number of counters before
and after, it cannot therefore indicate whether
the total income increased, decreased, or
stayed the same during the project.

Step 3.

Discuss with the informant thereasons for any changes
between “before”
and “after”.

Photo credit: Iain Dickson

282

PRISM - TOOLKIT FOR EVALUATING THE OUTCOMES AND IMPACTS OF SMALL/MEDIUM-SIZED CONSERVATION PROJECTS

4:3

INTRODUCTION &KEY CONCEPTS

DESIGNING & IMPLEMENTING
THE EVALUATION

Option 2: allowing informants to select the number of counters
for scoring “after”.
Step 1.

LIVELIHOODS &GOVERNANCE
FACTSHEETPIA METHOD 1:
BEFORE AND
AFTER SCORING

Give the informant 20 counters, and asked them toassign these counters to show the importance ofdifferent sources of income (or other indicator of
project impact) before the project.

Step 2.

When they have finished, ask the informant to choosea total number of counters to show the situation after
the project – they can select 20 counters again, or opt
for more or fewer counters in total.

Step 3.

Ask them to distribute the counters to show thesituation after the project. This scoring system allows
changes in total income to be assessed, as well as
the relative importance of the different sources ofincome. The 20 counters at the start of the project
represent a nominal baseline.

Step 4.

Discuss with the informant the reasons for any
changes between “before” and “after”.

OPTION 2
STEPS 1 TO 4

PRISM
MODULES

numbers or crop yields. People may be unwilling or uncomfortablediscussing exact amounts in these cases, but with this scoring
method, sensitive questions such as “How much money did you
make?” or, “How many cattle do you own?” are not necessary.

EXAMPLEA forest in Kabale District of Uganda was heavily usedby local communities, who depended on sales of forest
products for cash income, especially to pay school fees andfor medicines. However, extraction of timber for the saleof fuelwood and charcoal in the District’s capital was at
levels that were unsustainable and which were degrading
the forest, reducing its value for biodiversity (including asmall population of chimpanzees) and ecosystem services(watershed protection, with Kabale being a highland areawith steep hill slopes). A project implemented by a local
NGO sought to introduce alternatives to charcoal andfuelwood as sources of income. The alternatives included
beekeeping, ecotourism (focused on the chimps), craftmaking and processing of mushrooms harvested from theforest under licence from the Forest Department.

Scoring against a nominal baseline can be useful for measuringchanges in sensitive impact indicators such as income, livestock

283

METHOD FACTSHEETS& ANNEXESPRISM - TOOLKIT FOR EVALUATING THE OUTCOMES AND IMPACTS OF SMALL/MEDIUM-SIZED CONSERVATION PROJECTS

4:3

INTRODUCTION &KEY CONCEPTS

DESIGNING & IMPLEMENTING
THE EVALUATION

Scoring
Having used a timeline to define “before” and “after”, a scoring
exercise assessed the impact of the project on off-farm income,using a nominal baseline of 20 counters. Results for an individual
household are shown in the table below.

LIVELIHOODS &GOVERNANCE
FACTSHEETPIA METHOD 1:
BEFORE AND
AFTER SCORING

SOURCE OF OFF-FARM INCOME

Honey from hives

Tour guiding (ecotourism)

CraftsFuelwood sales

Dried mushroom sales

SCORE

Before***After********

Before

**

After

*****

Before

*

After

*

Before********

After

***

Before

*****

After

**

Before

*

After

*****

METHOD FACTSHEETS& ANNEXES

Total Before (Nominal baseline): 20
Total After: 24 (20% increase from baseline).

SCORING

Charcoal making

284

PRISM
MODULESPRISM - TOOLKIT FOR EVALUATING THE OUTCOMES AND IMPACTS OF SMALL/MEDIUM-SIZED CONSERVATION PROJECTS

4:3

LIVELIHOODS &GOVERNANCE
FACTSHEETPIA METHOD 1:
BEFORE AND
AFTER SCORINGINTRODUCTION &KEY CONCEPTS

DESIGNING & IMPLEMENTING
THE EVALUATION

Discussion
Discussions based on the responses were revealing, and were asimportant as the scores themselves. For example, respondents
explained that they had been unable to secure additional markets
for their crafts (mainly baskets made to local designs). Tourists,
mainly back-packers, were not interested in purchasing suchbulky items. Dried mushrooms had been especially popular andsuccessful – a five-fold increase compared to “before”. Households
were exploring ways of increasing production (through cultivation)in order to meet demand and to reduce reliance on wild-harvested
mushrooms.

PRISM
MODULESMETHOD FACTSHEETS& ANNEXES

FIGURE. 1:

The exercise was repeated for 80 households, which allowed
the calculation of means and 95% confidence levels as shown in
Figure 1.

Acknowledgements
Catley A, Burns J, Abebe D and Suji O (2013). Participatory Impact
Assessment: A Design Guide. Feinstein International Center, Tufts
University, Somerville.
Available from http://fic.tufts.edu/assets/PIA-guide_
revised-2014-3.pdf

285

PRISM - TOOLKIT FOR EVALUATING THE OUTCOMES AND IMPACTS OF SMALL/MEDIUM-SIZED CONSERVATION PROJECTS

4:3

INTRODUCTION &KEY CONCEPTS

DESIGNING & IMPLEMENTING
THE EVALUATIONPIA Method 2 - Proportional piling

PRISM
MODULES

Step 2.

Introduction
This method is designed to be used as part of a Participatory
Impact Assessment (PIA) evaluation design. Refer to the factsheet
Evaluation Design: Participatory Impact Assessment for more
information on how to use this method as part of a PIA.

LIVELIHOODS &GOVERNANCE
FACTSHEETPIA METHOD 2:PROPORTIONALPILINGSTEPS 1 & 2

286

Community participants are asked to score the importance ofdifferent possible causal factors for a given impact or outcome byplacing counters (e.g. seeds or stones) on representations of eachpotential causal factor provided by picture cards or other visual
aids, ideally made with local materials.

METHOD FACTSHEETS& ANNEXES

Community participants should then be divided
into different focus groups, e.g. women and men.
Individual members of each group can then distribute20, 50 or 100 stones or seeds among the potential
explanatory factors (as represented by the visual
images). It should be noted that the greater thenumber of counters, the longer the exercise takes– fewer counters can be used if there are fewer
variables. The results can be aggregated acrossthe focus groups, or analysed separately to revealsocially-differentiated impacts.

ExampleRequired steps

Scoring

Step 1.

An example of proportional piling is presented in Table 1 whichshows the scoring (using 100 counters) of project and non-project
factors that were judged to have contributed to positive change
in incomes and sustainability of a local fishery, including throughabandonment of fine-mesh nylon nets and implementation of aclosed season whilst fish were spawning.

Before embarking on proportional piling of the
project and non-project factors, it is importantto have an informed discussion about the rangeof possible explanatory factors or “independent
variables”. This discussion could be summarised in theform of a “causal diagram” showing all the potential
project and non-project factors. Some kind of clear
visual image is then needed for each of these factors– it may be helpful to have a local artist produce this.

The focus group was asked to list and rank the factors (project andnon-project) that had contributed to the changes, and then scoredthese factors using the counters. The conclusion was that the
project-related factors made a 34% (22+8+4) relative contribution
to sustainability of the fishery.

PRISM - TOOLKIT FOR EVALUATING THE OUTCOMES AND IMPACTS OF SMALL/MEDIUM-SIZED CONSERVATION PROJECTS

4:3

INTRODUCTION &KEY CONCEPTS

DESIGNING & IMPLEMENTING
THE EVALUATIONTABLE 1: ATTRIBUTION BY RANKING AND SCORING
FOR A FOOD SECURITY PROJECT.

LIVELIHOODS &GOVERNANCE
FACTSHEETPIA METHOD 2:PROPORTIONALPILINGQUESTION

PROJECT OR
NON-PROJECT FACTOR

RANK

SCORE

Enforcement by
Government

Non-project1

1

Secure tenure overfishing grounds

Non-project2

2

Organisation of afishing cooperative

Project

3

3

Government
extension service

Non-project4

4

Provision of larger
mesh nets

Project

5

5Provision of improved
fish smoking ovens

Project

6

6

PRISM
MODULESMETHOD FACTSHEETS& ANNEXES

Discussion of scores
As with any PIA method, the discussion that takes place inrelation to the scoring is an important part of the method, as itprovides explanations for changes and can help to reveal sociallydifferentiated impacts. For example, the discussion in this example
revealed that whilst secure tenure over the fishery had benefited
the community, giving them more control and an incentive tomanage resources for the long-term, there were also some
negative social and environmental impacts. A group of migrants
who had settled in the village several years previously were notbeing given the same rights and had been excluded. As a result
they had turned to alternative sources of income, and there wereunconfirmed reports that the incidence of snare-setting in a
nearby forest reserve had increased.
In this example the scoring was done as a group. However, theexercise could be done at individual or household level, which
would allow the calculation of means and confidence levels.

Acknowledgements
Catley A, Burns J, Abebe D and Suji O (2013). Participatory Impact
Assessment: A Design Guide. Feinstein International Center, Tufts
University, Somerville.
Available from http://fic.tufts.edu/assets/PIA-guide_
revised-2014-3.pdf.
Richards M (2011). Social and Biodiversity Impact Assessment(SBIA) Manual for REDD+ Projects: Part 2 – Social Impact
Assessment Toolbox. Climate, Community & Biodiversity Allianceand Forest Trends, with Rainforest Alliance and Fauna & FloraInternational. Washington, DC.
287

PRISM - TOOLKIT FOR EVALUATING THE OUTCOMES AND IMPACTS OF SMALL/MEDIUM-SIZED CONSERVATION PROJECTS

4:3

INTRODUCTION &KEY CONCEPTS

DESIGNING & IMPLEMENTING
THE EVALUATIONPIA Method 3: Tally method

Introduction
This method is designed to be used as part of a Participatory
Impact Assessment (PIA) evaluation design. Refer to the factsheet
Evaluation Design: Participatory Impact Assessment for more
information on how to use this method as part of a PIA.

LIVELIHOODS &GOVERNANCE
FACTSHEETPIA METHOD 3:
TALLY METHODSTEPS 1 TO 4

288

The tally method involves asking participants to identify thefactors that have caused changes in e.g. specified aspects of theirlivelihoods or well-being. One advantage of this method is thatby not pre-defining the potential factors that have contributed tochange in the community, there is less risk of influencing people’s
responses. On the other hand there may be a bias towards
mentioning project-related factors and omitting important nonproject factors, especially if the participants know the study is
being carried out to analyse project impacts. Note that a large
sample is needed to be confident when using the tally method.

PRISM
MODULESMETHOD FACTSHEETS& ANNEXES

Required steps
Step 1.

Select a sample of participants to take part in thesurvey.

Step 2.

Ask individual respondents in the sample to list all
the factors they think have contributed to a specificproject outcome or impact. Do not prompt them – itis important that they are allowed to generate the listof factors. Interview each respondent on his/her own(it is important that they haven’t heard how others
have responded).

Step 3.

When everyone in your sample has been asked, add
up the number of times each potential cause wasmentioned.

Step 4.

Identify which causes are linked to the project andwhich are not, and tabulate the results.

PRISM - TOOLKIT FOR EVALUATING THE OUTCOMES AND IMPACTS OF SMALL/MEDIUM-SIZED CONSERVATION PROJECTS

4:3

INTRODUCTION &KEY CONCEPTS

DESIGNING & IMPLEMENTING
THE EVALUATION

Example74 project participants were asked the open-ended question“What has contributed to reduced encroachment into the forest
by farmers?”. Based on their responses, a number was added every
time each factor was mentioned. The results showed that five ofthe top six most frequently mentioned factors relating to reduced
forest encroachment were directly related to the project.

LIVELIHOODS &GOVERNANCE
FACTSHEETPIA METHOD 3:
TALLY METHOD

289

Acknowledgement
Catley A, Burns J, Abebe D and Suji O. (2013). Participatory Impact
Assessment: A Design Guide. Feinstein International Center, Tufts
University, Somerville.
Available from http://fic.tufts.edu/assets/PIA-guide_
revised-2014-3.pdf.

PRISM
MODULESMETHOD FACTSHEETS& ANNEXESTABLE 1: REASONS FOR REDUCED FOREST
ENCROACHMENT FOR AGRICULTURE.
QUESTION

PROJECT OR
NON-PROJECT FACTOR

Introduction of water
harvesting methods

Project

68

Introduction ofagroforestry

Project

59

Local cooperative forvillage processing offarm-produce

Project

50

Increased penalties
for forest clearance

Non-project46

Training in vegetableproduction

Project

38

Clear demarcation offorest boundary

Project30

New road improving
access to markets

Non-project10

Government advice on
preventing crop pests
and diseases

Non-project8

Better than average
rainfall in last 3 years

Non-project

5

PRISM - TOOLKIT FOR EVALUATING THE OUTCOMES AND IMPACTS OF SMALL/MEDIUM-SIZED CONSERVATION PROJECTSNO. OF RESPONSES(N = 74)

4:3

INTRODUCTION &KEY CONCEPTS

LIVELIHOODS &GOVERNANCE
FACTSHEETPIA METHOD 4:
MATRIX SCORINGSTEPS 1 & 2

DESIGNING & IMPLEMENTING
THE EVALUATION

PRISMMODULESPIA Method 4 - Matrix scoring

Required steps

Introduction
This method is designed to be used as part of a Participatory
Impact Assessment (PIA) evaluation design. Refer to the factsheet
Evaluation Design: Participatory Impact Assessment for more
information on how to use this method as part of a PIA.

Designing the matrix - identifying the items to compare, and theindicators (features for comparison)

Matrix scoring involves the comparison of different items, project
activities or services using a list of indicators (characteristics orqualities of the items). In cases where project and non-project
items, impacts, activities or services are compared, the comparison
can often be a powerful way of understanding project impacts inthe context of pre-existing services or activities.
Like other scoring methods, matrix scoring can be standardised
and repeated with different individual informants or groups ofinformants. Also in common with other methods, matrix scoringuses semi-structured interviews to understand the reasoning
behind people’s scores.

Step 1.

Identify the items to be compared (relevant to yourproject’s objectives). They might be different types offood, different service providers, different crops, ordifferent types of income-generating activity. Where
possible, include some items or activities that are notpart of the project or are not related to it.

Step 2.

Use pair-wise comparisons to identify the indicators,
as follows:•
Select two items and ask people which item
is more important and why. They will state apreference and give reasons why one item is more
important than another. These reasons are yourindicators, showing differences between items.
•

290

METHOD FACTSHEETS& ANNEXESPRISM - TOOLKIT FOR EVALUATING THE OUTCOMES AND IMPACTS OF SMALL/MEDIUM-SIZED CONSERVATION PROJECTS

Repeat the comparison using different pairs of
items, until the informants are no longer offering
new indicators (reasons for differences), but arereferring back to the indicators/reasons they havealready identified. You will now have a full list of
indicators to use.

4:3

INTRODUCTION &KEY CONCEPTS

DESIGNING & IMPLEMENTING
THE EVALUATION

Conducting the scoring
Step 3.

Step 4.

LIVELIHOODS &GOVERNANCE
FACTSHEETSTEPS 3 TO 9

Select one of the indicator pictures and verify its
meaning with the informant(s); place the picture
adjacent to the item pictures.

Step 6.

Using a pile of around 30 stones or seeds, ask
informants to score the items against the indicator,
using all of the stones. The scores are then checkedand questions are asked to reveal the reasons behindthe scoring.

Step 7.

291

Place the pictures of the items in a row on the
ground. Verify the meaning of each picture with theinformant(s).

Select the second indicator and place this below
the first, and repeat the scoring process with thisindicator (with the same number of stones). Again,ask questions to check the scores and show thereasons for the scores.

METHOD FACTSHEETS& ANNEXES

Step 8.

Taking each indicator in turn, repeat the scoring andgradually add more rows to the matrix until all of theindicators have been scored.

Step 9.

Ask further questions to clarify, probe and explore
the scores, so that the reasons for each set of scores
are explained fully.

Prepare pictures (or find objects) that depict theitems to be scored and the indicators.

Step 5.

PIA METHOD 4:
MATRIX SCORING

PRISM
MODULES

ExampleA project implemented by a local NGO sought to introduce
alternatives to charcoal and fuelwood sales as sources of income.
The alternatives included beekeeping, ecotourism (focusedon chimpanzees), craft-making and processing of mushrooms
harvested from the forest under licence from the Forest
Department.
Table 1 shows the reasons given for preferences among different
income-generating activities in a project which sought to provide
alternatives to fuelwood production and charcoal-making (seeexample in L&G Method 1).
Table 2 shows how matrix scoring was then used to assess thecharacteristics of the income-generating activities – both thosethat were introduced by the project and those already existing in
the community.

PRISM - TOOLKIT FOR EVALUATING THE OUTCOMES AND IMPACTS OF SMALL/MEDIUM-SIZED CONSERVATION PROJECTS

4:3

INTRODUCTION &KEY CONCEPTS

DESIGNING & IMPLEMENTING
THE EVALUATIONTABLE 1: REASONS GIVEN FOR INCOME-GENERATING ACTIVITY PREFERENCES.

Honey vscraftsWe prefer honey, as markets are local – people even in the village arewilling to pay a good price.

Honey vsecotourism

We prefer honey because it is more reliable – we can’t predict when
tourists will come to the forest.

Honey vscharcoal

We prefer honey because it is legal – with tourism we are at risk of being
fined or imprisoned if we are caught.

Fuelwood vscharcoal

We prefer fuelwood because the benefits are immediate and don’t require
investment; for charcoal it can take many days to create the kiln, etc.

Dried mushrooms
vs honey

We prefer dried mushrooms because the product can be managed and it
gives income year-round; honey can only be harvested in the dry season.

TABLE 2: MATRIX SCORING OF DIFFERENT INCOME-GENERATING ACTIVITIES.
PARTICIPANTS WERE GIVEN 15 COUNTERS TO SCORE FOR EACH INDICATOR.
THE NUMBER OF DOTS INDICATES THE DEGREE TO WHICH EACH AN INCOME
GENERATING ACTIVITY SHOWS THE CHARACTERISTIC CONCERNED.

292

INDICATORHONEYECOTOURISM

CRAFTSCHARCOAL

FUELWOODDRIED MUSHROOMS

Locality ofmarkets

*****

*

*

**

***

***

Reliability ofincome

***

**

*

***

***

***

Legality –reduced risk

****

****

***

Immediacy of
income

****

**

*

*

*****

**

Little investment
needed

**

**

***

*

*****

**

Year-roundincome

**

*

*

***

****

****

****

PRISM
MODULESMETHOD FACTSHEETS& ANNEXES

Discussion of scores
It is clear from this exercise that fuelwood meets many of thedesirable characteristics of a useful source of income – localityof markets, immediacy of income, little investment required,etc. However, discussions revealed that its illegality and the risk
associated with being caught is a key factor encouraging people
to switch to alternatives (especially honey and dried mushrooms)
when these are offered, and when technical and financial support
is made available to help in overcoming any barriers to adoption.
Note that this method does not assess impact (in terms of levelof uptake of the alternative livelihood) but it can help improveunderstanding of the issues underpinning people’s choices andbehaviours.

Acknowledgements
Catley A, Burns J, Abebe D and Suji O. (2013). Participatory Impact
Assessment: A Design Guide. Feinstein International Center, Tufts
University, Somerville.
Available from http://fic.tufts.edu/assets/PIA-guide_
revised-2014-3.pdf.
Richards M (2011). Social and Biodiversity Impact Assessment(SBIA) Manual for REDD+ Projects: Part 2 – Social Impact
Assessment Toolbox. Climate, Community & Biodiversity Allianceand Forest Trends, with Rainforest Alliance and Fauna & FloraInternational. Washington, DC.

PRISM - TOOLKIT FOR EVALUATING THE OUTCOMES AND IMPACTS OF SMALL/MEDIUM-SIZED CONSERVATION PROJECTS

4:3

INTRODUCTION &KEY CONCEPTS

LIVELIHOODS &GOVERNANCE
FACTSHEETDESIGNING & IMPLEMENTING
THE EVALUATION

PRISMMODULESPIA Method 5 - Before and after impact calendars

Required steps

Introduction
This method is designed to be used as part of a Participatory
Impact Assessment (PIA) evaluation design. Refer to the factsheet
Evaluation Design: Participatory Impact Assessment for more
information on how to use this method as part of a PIA.

Step 1.

Give project participants a set number of counters(e.g. 30 stones).

Step 2.

Draw a 12-month calendar on the ground (or use
cards, one for each month of the year).

Step 3.

Ask participants to distribute their counters between
the months to show the annual distribution of theindicator before the project, for example:

Impact calendars can be useful for measuring impacts against
seasonally variable indicators. A comparison of the “before” and“after” project situation, when combined with semi-structured
interviews on the reasoning behind people’s scores, can help todemonstrate and understand project impact.

PIA METHOD 5:
BEFORE AND
AFTER IMPACTCALENDARSSTEPS 1 TO 5

Step 4.

293

METHOD FACTSHEETS& ANNEXES•

use of their harvested crop;•use of cash savings;•

incidence of sickness in the village/household;•attendance of children at school;•

absence of community members on migration;•

availability of water for drinking/livestock.

If there are differences in the timing of fluctuations/
events between women and men, each variable
should be considered separately for women and formen (and clearly marked). Alternatively, the group
may decide to prepare separate calendars for womenand for men.

PRISM - TOOLKIT FOR EVALUATING THE OUTCOMES AND IMPACTS OF SMALL/MEDIUM-SIZED CONSERVATION PROJECTS

4:3

INTRODUCTION &KEY CONCEPTS

Step 5.

LIVELIHOODS &GOVERNANCE
FACTSHEETPIA METHOD 5:
BEFORE AND
AFTER IMPACTCALENDARSSTEP 6

Step 6.

Leave the stones in place, and ask the participants torepeat the exercise, this time to show the situation
after the project. Start by providing them with thesame number of stones (30 – your nominal baseline)
but before they begin distributing the counters tellthem that they can add or take away stones to show
any net change.

Record the results and discuss the reasons for theway the counters have been distributed, and thedifferences before and after the project. Look forany patterns that can be seen, and cross-check andprobe for possible inconsistencies. It is important tonote different perspectives being represented among
different participants. If there are several different
groups, ask each group to present its calendar
to the others for their reactions and comments.
The following questions can be used to guide thediscussion, but these should be adapted according to
the focus of the exercise.
•

294

DESIGNING & IMPLEMENTING
THE EVALUATION

Are there significant differences between
the “before” and “after” calendars? Are thesedifferences attributed to the project, or is there
some other explanation?

•

How do women’s calendars compare with men’s?

•

What are the key linkages or “cause and effect”
relationships? For example income and food

PRISM
MODULESMETHOD FACTSHEETS& ANNEXES

supply, or rainfall and labour, or weather anddisease?
•Are the indicators fairly stable or do they show
great seasonal variations?

•

Which are the most difficult or high-risk times
in the year? How do people plan for these andrespond to any risks?

•

Are there any differences compared with thesituation 10/20/30 years ago?

•

What are the possible implications of thesefindings for any future intervention?

ExampleA project aimed to improve the ecotourism facilities at a
community in an area adjacent to a newly designated forest
reserve popular with birdwatchers, in order to extend the season
for tourists. Discussions had shown that income was concentrated
in just three months (June-August) and that the community was
generally unhappy with the restrictions which the reserve placedon their access to resources, threatening their long-term support
for its protection.
Participants were given 20 counters representing the total
income received from ecotourism in a given year. Using 12 cards
to represent each month of the year, participants were asked to
distribute the counters along a 12-month calendar to show how
the monthly income from ecotourism was spread over the courseof the year.

PRISM - TOOLKIT FOR EVALUATING THE OUTCOMES AND IMPACTS OF SMALL/MEDIUM-SIZED CONSERVATION PROJECTS

4:3

INTRODUCTION &KEY CONCEPTS

DESIGNING & IMPLEMENTING
THE EVALUATION

PRISMMODULESMETHOD FACTSHEETS& ANNEXES

EXAMPLEINDICATORJANUARYFEBRUARY

MARCH

APRIL

MAY

JUNE

JULY

AUGUSTSEPTEMBER

**

*****

*****

*****

***

***

***

***

***

**

*

*****

*************

Year before
project
Year after
project

LIVELIHOODS &GOVERNANCE
FACTSHEETPIA METHOD 5:
BEFORE AND
AFTER IMPACTCALENDARS****

Year after project(non-participating
community)

Before the project, tourists visited the area mostly in high season(June-August). After the project the community was receiving
income from tourists over a longer period. This was not the casefor communities that were not part of the project, suggestingthat the project had the outcome of attracting tourists to thecommunity outside the traditional high season.

OCTOBERNOVEMBER

DECEMBER**

Other resources
Catley A, Burns J, Abebe D and Suji O. (2013). Participatory
Impact Assessment: A Design Guide. Feinstein International
Center, Tufts University, Somerville.
Available from http://fic.tufts.edu/assets/PIA-guide_
revised-2014-3.pdf.
Richards M (2011). Social and Biodiversity Impact Assessment(SBIA) Manual for REDD+ Projects: Part 2 – Social Impact
Assessment Toolbox. Climate, Community & Biodiversity Allianceand Forest Trends, with Rainforest Alliance and Fauna & FloraInternational. Washington, DC.
Fauna & Flora International (2013). Seasonal Calendar.
Conservation, Livelihoods and Governance Programme, tools forparticipatory approaches.
PDF download: http://www.fauna-flora.org/wp-content/uploads/
Seasonal-Calendar.pdf .

295

PRISM - TOOLKIT FOR EVALUATING THE OUTCOMES AND IMPACTS OF SMALL/MEDIUM-SIZED CONSERVATION PROJECTS

4:3

INTRODUCTION &KEY CONCEPTS

DESIGNING & IMPLEMENTING
THE EVALUATION

Factsheet: Community-based Organisational Capacity
Assessment

LIVELIHOODS &GOVERNANCE
FACTSHEET

296

Introduction
Organisational capacity assessment (OCA) can be used to produce
a baseline against which performance of a community-basedorganisation (CBO) may be tracked and the impact of actions
that aim to build their capacity can be assessed. In addition toevaluations, an OCA carried out at the start of a project also helps
to identify areas for improvement, and can be used to inform
development of an action plan to address key weaknesses in the
CBO.

COMMUNITYBASEDORGANISATIONALCAPACITYASSESSMENT

Many different tools for OCA exist and NGOs and CBOs vary
widely in purpose, type, location, external context and otherfactors. It is important, therefore, to target aspects of anorganisation for assessment that are relevant. See the Table 1:

STEPS 1 TO 4

The evaluation design described below uses a before and afterscorecard-based assessment of the capacity of the CBO being
targeted by the project to show change. This is combined withfocus group discussions to identify reasons for change and any
attribution to the project.

PRISM
MODULESMETHOD FACTSHEETS& ANNEXES

Required steps

Step 1.

Familiarise yourself with the tool, the aspects oforganisational capacity covered and the scoring
system outlined in Table 1: Community Based
Organisation Assessment tool

Step 2.

Meet with the CBO and explain the process at least
two weeks before the actual assessment date.
Provide the questionnaire (Table 1) and give an
overview of the purpose of the exercise, explaining
any terms and identifying whether the CBO willcomplete the whole questionnaire or focus on
specific sections that are relevant to the actions
being implemented by the project. It is recommended
that one or two of the CBO office-holders should bewell versed in the process and should be the ones tolead the group’s assessment together with the main
(external) assessor.

Step 3.

Encourage the group to organise a consultative
meeting with a few selected members of the group to
go through the questionnaire in advance.

Step 4.

Convene a meeting to conduct the assessment. A
participatory approach is encouraged in which notjust the members of the CBO are involved but also
representatives from other stakeholders such as
relevant government personnel, other groups, NGOs,
and private sector companies working in the area.

CBO Assessment Questionnaire for a simple questionnaire-based
tool, which can be adapted to evaluate specific aspects of CBO
capacity.

PRISM - TOOLKIT FOR EVALUATING THE OUTCOMES AND IMPACTS OF SMALL/MEDIUM-SIZED CONSERVATION PROJECTS

4:3

INTRODUCTION &KEY CONCEPTS

Step 5.

LIVELIHOODS &GOVERNANCE
FACTSHEETCOMMUNITYBASEDORGANISATIONALCAPACITYASSESSMENT

DESIGNING & IMPLEMENTING
THE EVALUATION

Go through the questions one at a time, allowing
plenty of time for clarification and discussion.
When attaching a score to any of the questions,encourage the participants to write down reasonsfor their scores – this is very helpful for the analysis
and discussions that follow. A space is provided forentering this information. Explain to participants thatthey should mark the box that is closest to describing
the situation at hand: descriptions do not need
to be perfect. Consensus must be reached by theparticipants before the score is recorded.

PRISM
MODULES

Step 6.

When all relevant aspects have been scored, workout the weakest points in the CBO where capacity
needs to be built, as well as the strongest aspects
which can sustain the CBO. Confirm with themembers that the results are a true reflection of thesituation in the organisation, and where necessary
work out any changes to the assessment throughgroup discussion.
Take the averages of the scores for each of the capacity
areas and enter them into the following table in the“scores before column”

CBO ASSESSMENT SCORECARDCAPACITY ASPECTSCORE BEFORESCORE AFTERCHANGEREASONS FOR CHANGESTEPS 5 & 6

297

METHOD FACTSHEETS& ANNEXESPRISM - TOOLKIT FOR EVALUATING THE OUTCOMES AND IMPACTS OF SMALL/MEDIUM-SIZED CONSERVATION PROJECTSEVIDENCE

4:3

INTRODUCTION &KEY CONCEPTS

Step 7.

LIVELIHOODS &GOVERNANCE
FACTSHEETCOMMUNITYBASEDORGANISATIONALCAPACITYASSESSMENTSTEPS 7 TO 10

298

DESIGNING & IMPLEMENTING
THE EVALUATION

If the assessment is being done at the start ofthe project (to establish a baseline), discuss withparticipants how they think capacity will be built,
what resources will be needed, where they thinkthey could get the resources, and what the role of the
project should be.

Step 8.

Compile all this information (assessment results,
explanations, plan of action) into a report and share itwith the CBO and other stakeholders.

Step 9.

At the end of the project carry out the assessmentin the same way and enter the average of the scoresin the “scores after” column in the CBO assessmentscorecard, also complete the remaining columns toshow the change in scores, the reasons for change
and the evidence to support the scoring. These lasttwo columns are particularly important for assessing
attribution (determining whether any change canbe attributed to the project). See the example on
the following pages for an example of a completed
scorecard.

PRISM
MODULESMETHOD FACTSHEETS& ANNEXES

ExampleSouth Mara Water Users Association is a Local Conservation
Group (LCG) working in the Masurura Swamp, Mara, in Tanzania,
one of the critical wetlands in the Lake Victoria Basin.
A project aiming to develop the capacity of the LCG held an
initial Participatory Meeting with North Mara Water Users
Association representatives. Under coordination of 2 project teammembers 17 participants (4 women and 12 men from the LCG
and 1 participant from the Ward Extension Office) completed
Organizational Capacity Assessment tool questionnaires.
Capacity areas were rated from 0-4 where: Highly satisfactory = 4(High level of capacity in place); Satisfactory = 3 (Moderate level ofcapacity in place); Unsatisfactory with positive signs = 2 (Basic levelof capacity in place); Unsatisfactory = 1 (Clear need for increased
capacity); Nothing positive = 0 (No capacity, completely urgent
action required).
After the project the same individuals were asked to complete thequestionnaire again. Scores from before and after the project werecompared to show changes in the capacity of the group. The tablebelow shows a summary of the scores from the full assessment
based on the different capacity areas.

PRISM - TOOLKIT FOR EVALUATING THE OUTCOMES AND IMPACTS OF SMALL/MEDIUM-SIZED CONSERVATION PROJECTS

4:3

INTRODUCTION &KEY CONCEPTS

DESIGNING & IMPLEMENTING
THE EVALUATION

PRISMMODULESMETHOD FACTSHEETS& ANNEXESTABLE 1. SUMMARY OF ORGANISATIONAL CAPACITY ASSESSMENT RESULTS.
CAPACITY ASPECTSCORE BEFORESCORE AFTERCHANGEREASONS FOR CHANGE

EVIDENCE

Aspirations andstrategy330Organisation
skills2

3+1

The project designed and ran a programme of training
on organisational management, including budgeting,good governance and leadership. It also led a process,with Board members and senior staff, to put in placea clear strategy with vision and goals. However, as aresult of staff turnover, capacity gaps still remain.

• Organisational strategy• Meetings of the Board• Records of staff attendance attraining courses

Humanresources

3

4

+1

With funding from a local charitable foundation theorganisation was able to recruit a communications
officer to fill a critical staffing gap. This change cannot
be attributed to the project.

• Staff list, job description andorganigram

Systems andinfrastructure

2

4

+2

The project’s focus was on improving systems
for financial management (which were poor) andaccounting. These are now fully in place.

• Financial management handbook• Democratically elected treasurer
on the Board• Audited accounts available

Organisational
structure220Mean score

2.6

3.2

Acknowledgements
BirdLife International (2008). Capacity Assessment Tool for Site
Support Groups. Nairobi, Kenya.
http://www.birdlife.org/sites/default/files/attachments/BirdLife_
Africa_SSG_Capacity_Assessemen_%20Tool_v4_1.pdf

299

Bond (undated) Assessing effectiveness in building the capacity oforganisations and institutions. Bond, UK.
https://my.bond.org.uk/sites/default/files/impact-builder/
CapacityDevelopment.pdf

PRISM - TOOLKIT FOR EVALUATING THE OUTCOMES AND IMPACTS OF SMALL/MEDIUM-SIZED CONSERVATION PROJECTS

4:3

INTRODUCTION &KEY CONCEPTS

DESIGNING & IMPLEMENTING
THE EVALUATION

PRISMMODULESMETHOD FACTSHEETS& ANNEXESTABLE 3: COMMUNITY BASED ORGANISATION CAPACITY ASSESSMENT QUESTIONNAIRE

LIVELIHOODS &GOVERNANCE
FACTSHEETCOMMUNITYBASEDORGANISATIONALCAPACITYASSESSMENT

Name of CBO:

Location:

Date of assessment:

Name of assessor:

Number of staff
members of CBO:

CBO budget for last
12 months:

Number of CBO
members (nonstaff):(If applicable)
Total score from any
previous assessment

Date of previous
assessment:

Names of participants in assessment meeting and where they are from (external stakeholder participation is encouraged)

300

1.

2.

3.

4.

5.

6.

7.

8.

9.

10.

PRISM - TOOLKIT FOR EVALUATING THE OUTCOMES AND IMPACTS OF SMALL/MEDIUM-SIZED CONSERVATION PROJECTS

4:3

INTRODUCTION &KEY CONCEPTS

MAIN CAPACITY
AREAS

DESIGNING & IMPLEMENTING
THE EVALUATION

PRISMMODULESSCORE(0-4)*ASPECTS BEING ASSESSEDASPIRATIONS & STRATEGY
1. Values and Purpose: the CBO has a clear understanding of the purpose of their mission andstrategic goals, and what it aspires to achieve, and adheres to the values of the group.
2. Risk assessment: the CBO understands and responds to the risks to which the group is
exposed, including likelihood, impacts and potential solutions/mitigation measure.

LIVELIHOODS &GOVERNANCE
FACTSHEET

Mission, Vision andStrategic Goals

COMMUNITYBASEDORGANISATIONALCAPACITYASSESSMENT

3. Information about CBO: The entire membership is well informed about the group’s
programmes, to the level that they can contribute effectively in the group’s objectives and plans.
4. Overarching Goals: CBO’s vision translated into clear, bold set of (up to three goals) that thegroup aims to achieve specified by a well-defined time frame for attaining goals.

Conservation Goals

5. Involvement in Conservation: the CBO has a demonstrable interest in conservation, has
projects and activities that relate to the conservation issues at their IBA site.

Overall Strategy

6. Strategic Focus: the CBO has a strategic action plan complete with clear objectives,
achievable targets and clear focus on environmental conservation.

Organizational
Review

7. Group Review: the CBO leadership periodically reviews the structure and programmes of thegroup to ensure effectiveness and best use of resource.

ProgramDevelopment

8. Projects Development: the CBO understands the project development process,
implementation procedures, including work planning, how donors work, reporting andaccounting.

Fundraising
and Revenue
Generation

9. Sustainability Measures: the CBO has reliable sources of income, and is able to mobilize
resources through proposal writing and/or has established (or has the potential of establishing)income generating nature-based enterprises.

Goals/performancetargets

10. Monitoring: The CBO together with the entire membership keeps track of theimplementation of plans against the group’s objective.

Total score(Aspirations &Strategy)
Average score(Aspirations &Strategy)301

METHOD FACTSHEETS& ANNEXESPRISM - TOOLKIT FOR EVALUATING THE OUTCOMES AND IMPACTS OF SMALL/MEDIUM-SIZED CONSERVATION PROJECTSBRIEFLY EXPLAIN THE
REASONS FOR YOUR SCORE

4:3

INTRODUCTION &KEY CONCEPTS

MAIN CAPACITY
AREAS

DESIGNING & IMPLEMENTING
THE EVALUATION

PRISMMODULESSCORE(0-4)*ASPECTS BEING ASSESSEDORGANISATION SKILLS
1. Planning Process: The CBO has a clearly documented and well understood process for how it
should plan and review its work.
Operational
Planning

LIVELIHOODS &GOVERNANCE
FACTSHEETCOMMUNITYBASEDORGANISATIONALCAPACITYASSESSMENT

Internal
Communication

External
Relationships

Performance
Analysis andprogramadjustments

2. Resources: planning within the CBO takes into account financial and human resources
available.
3. Communication within CBO: planning and all other activities within the group are properly
communicated. Key information is effectively and timely shared with all the officials and entire
CBO membership.
4. Meetings: meetings are convened regularly, involving the entire membership (always
meeting the quorum regulations). The agenda is circulated well in advance to ensure effective
contribution and deliberations are recorded in written minutes available to all members.
5. Learning from Others/Collaboration and Partnerships: the CBO takes time to learn and draw
lessons from other partners and stakeholders who have relevant knowledge and experience
before undertaking work. The CBO has agreed guidelines on collaboration.
6. External Communication: all communications from outside including letters, e-mails andphone calls are dealt with promptly by the officials.
7. Managing Change: where major changes takes place in the program adjustments group, i.e. inmanagement, projects, partners etc.—the organization’s leadership works together as a team toinstitute necessary management interventions.
8. Filing System: the CBO has a filing system that ensures all documents are well organizedmaking it easy to access and track information.

Knowledge/Data
Management

9. Sharing Knowledge: the CBO encourages openness and transparency in sharing of
knowledge/data amongst the group members.

Total score(Organisational
skills)Average score(Organisational
skills)302

METHOD FACTSHEETS& ANNEXESPRISM - TOOLKIT FOR EVALUATING THE OUTCOMES AND IMPACTS OF SMALL/MEDIUM-SIZED CONSERVATION PROJECTSBRIEFLY EXPLAIN THE
REASONS FOR YOUR SCORE

4:3

INTRODUCTION &KEY CONCEPTS

MAIN CAPACITY
AREAS

DESIGNING & IMPLEMENTING
THE EVALUATION

PRISMMODULESSCORE(0-4)*ASPECTS BEING ASSESSEDHUMAN RESOURCES
Membershipcomposition andcommitment

LIVELIHOODS &GOVERNANCE
FACTSHEETCOMMUNITYBASEDORGANISATIONALCAPACITYASSESSMENT

Membership/
Governance1. Commitment and Involvement: CBO membership commitment to conservation, to the needsof the entire membership and widely involves the membership in group’s activities, consistent
with individual’s skills, experience and availability.
2. Governance: responsibilities are shared amongst officials. Members assist the leader
in establishing and articulating the CBO’s objectives, plans and in reviewing the CBO’s
performance.
3. Leadership: The CBO’s leaders are democratically elected and are answerable to members.

4. Members’ Skills and Experiences: members of the CBO handling different aspects areproperly trained and experienced to deliver effectively and efficiently.

Professionalism andSkills5. Knowledge and Expertise on Conservation: the CBO has scientific and technical expertiseon biodiversity issues (e.g. botanists, ecologists etc.) and socio-economic issues (lawyers,
economists, sociologists etc.).
6. Learning Opportunities: the CBO provides opportunities for its membership to increaseknowledge, skills and experience.

Co-ordination
Team/Process

7. Delegation and Co-ordination: the CBO leadership delegates wisely, giving other members
an opportunity to take responsibility in the running of the group. The leadership ensures coordination of activities of the group.
8. Conflict Management: the CBO has laid down procedures for conflict management andresolution. Members can appeal in case when they feel they have been treated unfairly.

Motivation and
Rewards

9. Member’s Motivation: the CBO leadership understands its membership and takes timeto encourage, develop, congratulate, reward and recognize, etc., to ensure the members are
motivated at all times.

Decision Making
Framework

10. Decision Making: the CBO leadership takes into account the views of its membership andother stakeholders, the risks, financial situations, etc. before making any decisions on behalf ofthe group.

Total score(Human resources)
Average score(Human resources)

303

METHOD FACTSHEETS& ANNEXESPRISM - TOOLKIT FOR EVALUATING THE OUTCOMES AND IMPACTS OF SMALL/MEDIUM-SIZED CONSERVATION PROJECTSBRIEFLY EXPLAIN THE
REASONS FOR YOUR SCORE

4:3

INTRODUCTION &KEY CONCEPTS

MAIN CAPACITY
AREAS

DESIGNING & IMPLEMENTING
THE EVALUATION

PRISMMODULESSCORE(0-4)*ASPECTS BEING ASSESSEDSYSTEMS, INFRASTRUCTURE AND ORGANISATIONAL STRUCTURE
Monitoring Systems

1. Monitoring System: A basic monitoring system is in place where information is recorded
about projects, inputs and outputs against objectives and activities.
2. Financial Management: the CBO has a bank account, and a treasurer who is democraticallyelected, and is responsible for managing the group’s finances.

LIVELIHOODS &GOVERNANCE
FACTSHEET

Financial
Operations
Management

COMMUNITYBASEDORGANISATIONALCAPACITYASSESSMENT

3. Accounting Procedures: the CBO keeps books of accounts and has reliable procedures toensure that resources are managed properly and the records properly kept.
4. Financial Transparency: the CBO’s financial management and reporting is transparent andthe leadership is open to discuss financial matters with the membership as appropriate.
5. Budgeting: the CBO members are involved in developing and monitoring the budget annually.

Infrastructure

6. Building and Office Space: The CBO owns or has access to an office to conduct meetings,
carries out group operations and stores the group’s resources. The CBO also has access toreliable communication facilities- telephone, computers etc.

Total score(Systems andinfrastructure)
Average score(Systems andinfrastructure)
ORGANISATIONAL STRUCTURE

Organisational
Structure

Organisational
Design1. Participation of Members: the CBO’s leadership encourages members to express theiropinions about the group’s work and operation and is open to different points of view.
2. Constitution/Bye Laws: The CBO’s meets all legal requirements. A constitution is developedin a participatory manner, which clearly defines the roles and responsibilities of the officials.
3. Organizational Design: the CBO has a structure that ensures effective and best use ofresources.
4. CBO Operational Structure: the CBO structure chart shows the line of responsibility
between the different positions in the group.

Total score(Organisational
skills)Average score(Organisational
skills)304

METHOD FACTSHEETS& ANNEXESPRISM - TOOLKIT FOR EVALUATING THE OUTCOMES AND IMPACTS OF SMALL/MEDIUM-SIZED CONSERVATION PROJECTSBRIEFLY EXPLAIN THE
REASONS FOR YOUR SCORE

4:3

INTRODUCTION &KEY CONCEPTS

DESIGNING & IMPLEMENTING
THE EVALUATION

Factsheet: Community mapping

LIVELIHOODS &GOVERNANCE
FACTSHEETCOMMUNITY
MAPPINGSTEPS 1 TO 5

Introduction
In many countries, land rights are determined by complex andoften overlapping systems of formal law and customary norms.
The most accurate information regarding land use, access,resource rights and ownership is often held by local communities.
Participatory mapping is therefore an appropriate method forassessing impacts on rights to land and resources, as well as otherspatial changes. The method can be used to show information
regarding the occurrence, distribution, access to and use ofresources; topography; human settlements; land tenure; andlivelihoods activities. Cartographic precision is not important, as
this tool is not designed for demarcating boundaries or calculatingareas under a particular land use. It can however be used to
identify ownership of land, and the location, access and use of
key natural resources. By repeating the mapping exercise fordifferent time periods it can be used to monitor changes in tenure
and resource rights. This tool is most suitable for a geographicallylimited area. For larger areas it may be appropriate to produce
more than one map.

Required steps

Step 1.

305

Ask participants to select a suitable place in which to
draw the map and the medium they will use, which
could be on the ground using stones, seeds, sticks andcoloured powder; on the floor using chalk; or directly
onto a large sheet of paper using pencils and pens.

PRISM
MODULESMETHOD FACTSHEETS& ANNEXES

Step 2.

Agree with participants exactly what area the map
will cover, such as a village a river basin, etc.

Step 3.

Explain that the quality of the drawing is notimportant and it does not matter if the map is notentirely accurate or to scale. If any of the participants
are illiterate it is important to use symbols anddrawings, with a key to interpret the symbols into
the local language (and where necessary, into (a)relevant language(s) for other intended users of theinformation).

Step 4.

Ask participants to start by preparing the outline orboundary of the map and then identify the central
point or an important landmark within the area (suchas a mosque, school or market place).

Step 5.

Other important landmarks can now be drawn.
Participants should develop the content of the map
according to what they think is most important. The
map does not need to show every individual house,
shop, field, etc., but rather the area where they arelocated. Local landmarks and features might include:•
infrastructure and services (e.g. roads, houses,
bridges, schools, health clinics, bus stops, shops
and markets);•

special places (e.g. location of medicinal plants,
places of worship, sacred sites, cemeteries);•

water sites and sources;

PRISM - TOOLKIT FOR EVALUATING THE OUTCOMES AND IMPACTS OF SMALL/MEDIUM-SIZED CONSERVATION PROJECTS

4:3

INTRODUCTION &KEY CONCEPTS

LIVELIHOODS &GOVERNANCE
FACTSHEET

Step 6.

COMMUNITY
MAPPING
Step 7.

STEPS 6 TO 10

Step 8.

306

DESIGNING & IMPLEMENTING
THE EVALUATION•

agricultural land (e.g. crop varieties and locations,grazing areas), soils, slopes, elevations;•natural resources (e.g. areas of forest, rivers,
location of species of specific interest) and areas ofeg. predation or crop raiding;•hazards/areas prone to risk (e.g. floodplains,infrastructure vulnerable to floods, drought-prone
areas).

Although it might take some time to get going, theprocess should not be rushed. Once the mapping isunderway, sit back and observe, and only interrupt if
absolutely necessary in order to clarify something or
to help participants if they get stuck.
If the map is being drawn on the ground, once thebroad outline has been established, participants
can start making a copy on to paper (indicating
which direction is north). This process is importantbecause extra information and corrections can often
emerge as a result. Also it is important that a copy
or permanent record of the map is available if thecontributors want it.
If the focus of your assessment is on tenure and rights
to resources, ask relevant information to be enteredonto the map.

PRISM
MODULES

Step 9.

METHOD FACTSHEETS& ANNEXES

Discuss and analyse the map(s). Ask participants todescribe what they have produced and ask questions
about anything that is unclear. If there are several
different groups, each group should present anddescribe its map to the others for their reactions andcomments. Are there major differences? If so, notethese and whether a consensus is reached. Note thata consensus is not necessarily a desired outcome –differences in perceptions of land use can be a very
useful basis for further discussions

Step 10. To assess a project’s impact (e.g. land tenure or rights
to resources), complete the mapping exercise at thestart (baseline) and at the end of the project. If you
don’t have a baseline then ask participants to drawa series of maps to illustrate changes. Ask questions
about the map to help you understand the causesof any changes and about the contribution of yourproject.

Ackowledgement
Reproduced with permission (and with minor adaptations only)
from: Fauna & Flora International (2013). Guidance sheet on
Community Resource Mapping. Conservation, Livelihoods andGovernance Programme, Tools for participatory approaches.
Available at: http://www.fauna-flora.org/wp-content/uploads/
Community-Resource-Mapping.pdf .

PRISM - TOOLKIT FOR EVALUATING THE OUTCOMES AND IMPACTS OF SMALL/MEDIUM-SIZED CONSERVATION PROJECTS

4:3

INTRODUCTION &KEY CONCEPTS

DESIGNING & IMPLEMENTING
THE EVALUATION

Factsheet: Basic Necessities Survey (BNS)LIVELIHOODS &GOVERNANCE
FACTSHEETBASIC NECESSITIESSURVEY (BNS)
STEP 1Introduction
If poverty can be defined broadly as “the lack of basic necessities”,
a valid approach to poverty assessment is to check whether a
project has resulted in a change in the extent to which people’s“basic necessities” are being met. Unlike income-based approaches
to poverty assessment (e.g. numbers of people living on less than
US$2/day), there is no a priori definition of “basic necessities”,
partly since what can be considered as a basic necessity is likely tovary both with location (different communities and cultures) andover time.
The Basic Necessities Survey (BNS) method15 has been adapted
to a number of contexts, most comprehensively by the Wildlife
Conservation Society (WCS) for social impact monitoring inconservation landscapes16. The BNS method measures poverty
change over time according to whether community members thinkthey are getting more or less of their basic necessities than before
the project, or since the last time that the BNS was carried out.
The BNS method can be implemented in control and treatment(project) communities in order to assess attribution, and istherefore a very useful method when used in conjunction with aquasi-experimental approach, assuming that the project expects
to have an impact on the general poverty level of the projectcommunities. Alternatively, and in the absence of any control
communities, it could be used in conjunction with participatory
impact assessment methods or the Theory of Change approach(see Factsheet: Completing a Theory of Change) to assess thecausation of any changes in poverty levels.

307

PRISM
MODULESMETHOD FACTSHEETS& ANNEXES

The BNS is undertaken in three main steps:
Step 1.
Identification of possible basic necessities via focus
groups;Step 2.

Application of the survey;

Step 3.

Analysis of the data collected.

Required steps
Step 1.

Identification of possible basic necessities via focus
groupsUse focus group discussions to generate an initial list of
goods (e.g. TV, bicycle, radio, wheelbarrow, machete) andservices (e.g. all school age children attending school; walking
distance to a health clinic) that the participants may or may
not think are basic necessities. In focus group discussions
some individuals may dominate the discussion, e.g. women may
not talk much in front of men and minorities may not talk when in

The BNS method was developed in 1998 by Rick Davies (http://www.mande.co.uk/
welcome), an independent monitoring and evaluation expert working for ActionAid. This
description of the BNS method is adapted from a version in: Richards M (2011). Social andBiodiversity Impact Assessment (SBIA) Manual for REDD+ Projects: Part 2 – Social Impact
Assessment Toolbox. Climate, Community & Biodiversity Alliance and Forest Trends, withRainforest Alliance and Fauna & Flora International. Washington, DC (http://www.foresttrends.org/documents/files/doc_2981.pdf) which has itself been adapted from a version
licensed under the Creative Commons Attribution-Noncommercial-Share Alike 3.0 Licence.
To view a copy of this licence, visit: http://creativecommons.org/licenses/by-nc-sa/3.0/ .
15

Wilkie D, Wieland M and Detoeuf D (2015). A guide to the modified Basic Necessities
Survey: Why and how to conduct BNS in conservation landscapes. WCS, New York, USA(http://programs.wcs.org/carbon/Communities/WCS-Resources/Publications.aspx) .

16

PRISM - TOOLKIT FOR EVALUATING THE OUTCOMES AND IMPACTS OF SMALL/MEDIUM-SIZED CONSERVATION PROJECTS

4:3

INTRODUCTION &KEY CONCEPTS

DESIGNING & IMPLEMENTING
THE EVALUATION

PRISMMODULESa group composed mostly of the ethnic majority. Also women
may have different views from those of men about what the basic
necessities for life are, and young people may also have different ideas
about basic needs compared with those of older people. A mixed age
and mixed gender focus group may be adequate to capture thesedifferences, but more often you will need to conduct several groups
composed of different community members:

LIVELIHOODS &GOVERNANCE
FACTSHEETBASIC NECESSITIESSURVEY (BNS)
STEP 1•men-only and women-only groups;•

groups of young people and groups of older people;•groups of ethnic minorities.

It is important that the list includes items which almost everyone
would agree are basic necessities (e.g. enough food each day),and others where there are likely to be different views (e.g.
having a TV). The list should include between 20 and 30 villagedefined goods and services (about equal numbers of each).
This should include some items that only a few people in thegroup think are current necessities, but which could become
necessities in the future. At this point it is a list of possible basic
necessities, not a final list of agreed basic necessities.

METHOD FACTSHEETS& ANNEXES

yes or no answer (e.g. “a healthy family” or “well trainedteachers”), or that cannot be reliably observed by different
people. Keep in mind that for each element of your final list,
people will have to answer two simple questions: “Do you
have it?” “Is it really necessary (a basic necessity)?” It is very
important for a yes or no answer to be given easily andwith confidence.
•

If the description of a good or service is too vague, it willtake too much time for people to answer the question. For
example, asking whether someone has “access to a goodschool” can be very complicated: the school might be good,
but the household may not have the money to send theirchildren to this school. Be very specific about services, e.g.
by defining what type of school you are talking about, whatis “good”, and how you define having access to it, in orderto ensure that your question is understood in the way you
intend it to be.

WHAT IS ACCESS?
Access to a service is sometimes difficult to define, so it is

Ensuring your list is clear and easily measurable

important to think about what this means. Does it just mean
the service is available? Or does it mean that it is available and

•

•

308

Goods should be durable goods that can be used multiple
times. Consumable goods (e.g. food, beverages, petrol)
can run out and should not be included in the list. If one ofthese consumable items is very important, turn it into a
service, for example “meal three times a day”.
The list should not have services that can be easily
misinterpreted, those that are difficult to record with athe household has the ability to purchase/use the service? For
example a secondary school may be in a town nearby, but if ahousehold doesn’t have the money to pay for accommodation,
do they really have access to it? For the BNS, access means thata household is actually able to use the service, not that the service
simply exists in the village.

PRISM - TOOLKIT FOR EVALUATING THE OUTCOMES AND IMPACTS OF SMALL/MEDIUM-SIZED CONSERVATION PROJECTS

4:3

INTRODUCTION &KEY CONCEPTS

Step 2.

LIVELIHOODS &GOVERNANCE
FACTSHEETBASIC NECESSITIESSURVEY (BNS)
STEP 2

DESIGNING & IMPLEMENTING
THE EVALUATION

Application of the survey

Meet with village leaders
Before conducting a BNS survey, talk with the village elders
and residents to explain the purpose of the study and to
ask their permission to conduct surveys in the village. It
is extremely important to take the time to explain to theauthorities and to the participants why you are conducting
the BNS, detailing the process, showing how they canbenefit from the results of the survey and ensuring thatthey completely understand the objective of your presencein the village. For more guidance on conducting surveys,
including selecting a sample to survey, see the Factsheet:
Questionnaires.

Define “basic necessity”
If people don’t understand what a basic necessity is, they may
answer “yes” to the entire set of questions, making it very
difficult for you to analyse the answers. A common mistakeis to think that”necessity” is the same as “important”. It isnot. Here is the definition of a basic necessity: “Something all
families should be able to have and no family should have tolive without”.
The interviewee must understand that a basic need is morethan just something they would like to have to make theirlives better, and it is more than something that is importantto have. It is something that you can’t live without, something

309

PRISM
MODULESMETHOD FACTSHEETS& ANNEXES

essential to life. The problem is, in some languages, words like“necessary” or “essential” don’t exist. So before going into thefield, sit with your team to define “basic necessity” in the local
language.
After you finish your explanation, to check whether the
person you are interviewing understands the difference
between a basic need and something important, you can ask
two questions, using goods or services that are not in yourlist. For example, if in your final list you don’t have a pair ofshoes and sunglasses, you can ask: “Do you think a pair ofshoes is a necessity?” It most likely is, and most people would
answer “yes” if they understood the explanation you gavethem. If they don’t, then you will have to explain it again. Ifthey do, then you can ask another question: “Do you thinka pair of sunglasses is a necessity?” It is most likely not, andmost people, if they understood the explanation you gavethem, should answer “no”. If they don’t, you will have to
explain the definition of a basic necessity again. If they do, itmeans you can start your survey.
Conducting the survey - questions to ask
Ask each male or female household head (picked randomly) in
your survey sample two basic questions:•

Which items on the list do you consider are basic
necessities that everyone should have, and no-one
should have to do without?

•Which items on the list does your household possessnow?

PRISM - TOOLKIT FOR EVALUATING THE OUTCOMES AND IMPACTS OF SMALL/MEDIUM-SIZED CONSERVATION PROJECTS

4:3

INTRODUCTION &KEY CONCEPTS

DESIGNING & IMPLEMENTING
THE EVALUATION

PRISMMODULESINFORMATION IN THE DATA SHEETSurvey information

LIVELIHOODS &GOVERNANCE
FACTSHEETBASIC NECESSITIESSURVEY (BNS)•The date on which the survey was conducted.

•

The name of the village.

•

The name of the person in the household who
participated in the survey.

•

The name of the head of the household.

•

The unique ID number of the household
surveyed (this is vital to keep the surveys
separate from each other).

Household demography
For each member of the family, record information suchas their gender, the number of years they spent in school,
their year of birth and ethnicity.

310

PRISM - TOOLKIT FOR EVALUATING THE OUTCOMES AND IMPACTS OF SMALL/MEDIUM-SIZED CONSERVATION PROJECTSMETHOD FACTSHEETS& ANNEXES

4:3

INTRODUCTION &KEY CONCEPTS

DESIGNING & IMPLEMENTING
THE EVALUATION

PRISMMODULES

Step 3.

METHOD FACTSHEETS& ANNEXES

Analysis of the data collected

TABLE 1: EXAMPLE OF A COMPLETED HOUSEHOLD BNS FORM.

LIVELIHOODS &GOVERNANCE
FACTSHEETBASIC NECESSITIESSURVEY (BNS)
STEP 3ASSET ORSERVICEITEM

HAVE NOW?
YES = 1, NO = 0

ARE BASIC NECESSITIES?
YES = 1, NO = 0

Asset
Asset

1 hectare of land per person

01Electric light

1

1

Asset

Bicycle

1

0

Asset

Concrete rice drying yard

1

0

Asset

Wooden rice chest

1

1Service3 meals a day

1

1

Asset

Buffalo or cow

01

Service

All children studying to level 2

0

1

Asset

Well with well head

0

1

Asset

Stone built house

0

0

Asset

Thick cotton blanket

11Service

Doctor visiting house when sick

1

1

Asset

Electric fan00ServiceA new set of clothes each year11Service

Livestock vaccination00Service

Meat once a week

0

1

Asset

Pesticide pump

0

0

Asset

Watch

0

0

Service

Access to loans

0

1

Asset

Radio

0

0

Asset

Toilet - built of stone

0

1

Asset

Table made of good wood

1

1

Asset

Two-compartment wood wardrobe

0

0

Asset

TV

0

0

Asset

Bathroom

0

0

Asset

Motorbike00

The list of items can be read out to respondents or typed on cards
which are then sorted in response to the two questions above.
Table 1 presents an example of a completed household BNS form.

Data analysis involves the following steps (see Tables 2 and3):•Determine which items are “basic necessities”– these are defined as items which over 50%of the households interviewed think are basic
necessities;•

Calculate a weighting (fraction) for each item
based on the percentage of households who
think it is a basic necessity. For example if 350
out of 420 households interviewed think it is abasic necessity, the weighting is 350/420 = 0.833(Tables 2 and 3);•

Calculate a maximum possible score for ahousehold with all the basic necessities (this is thesum of all the weighting scores for the items in the“basket” of basic necessities – Column A in Table
1);•

For each household, calculate a poverty index(%) by adding up the weighted scores for thathousehold (Column B) and dividing this by the
maximum score (Column A), as shown in Table 2.

As well as basic survey information, it is also helpful to collect somedemographic information about each household that you interview.
This will help you to understand differences in impacts of yourproject within the communities where you are working (see box).
311

PRISM - TOOLKIT FOR EVALUATING THE OUTCOMES AND IMPACTS OF SMALL/MEDIUM-SIZED CONSERVATION PROJECTS

4:3

INTRODUCTION &KEY CONCEPTS

DESIGNING & IMPLEMENTING
THE EVALUATIONTABLE 2. AN EXAMPLE OF A BASIC NECESSITIES WEIGHTING TABLE
FROM A BNS DATASET. THOSE ITEMS NOT CONSIDERED TO BE A BASIC
NECESSITY BY MORE THAN 50% OF THE 420 PEOPLE SURVEYED ARE
HIGHLIGHTED IN RED.
ITEMLIVELIHOODS &GOVERNANCE
FACTSHEETBASIC NECESSITIESSURVEY (BNS)

312

PRISM
MODULESMETHOD FACTSHEETS& ANNEXESTABLE 3: EXAMPLE OF A HOUSEHOLD BNS POVERTY SCOREFOR BASIC NECESSITIESITEMHAVE NOW?
YES = 1, NO = 0

COLUMN A
WEIGHTING (FRACTION)

COLUMN B
POVERTY SCORE

1 hectare of land per person

0

0.995

0.000

Electric light

1

0.995

0.995

# WHO CONSIDER
NECESSITY% WHO CONSIDER
NECESSITY (WEIGHTING)

1 hectare of land per person

418

99.5%

Bicycle

1

0.995

0.995

Electric light

418

99.5%

Concrete rice drying yard

1

0.988

0.988

Bicycle

418

99.5%

Wooden rice chest

1

0.986

0.986

Concrete rice drying yard

415

98.8%

3 meals a day

1

0.983

0.983

Wooden rice chest

414

98.6%

Buffalo or cow

0

0.981

0.000

3 meals a day

413

98.3%

All children studying up to level 2

0

0.981

0.000

Buffalo or cow

412

98.1%

Well with well head

0

0.979

0.000

All children studying up to level 2

412

98.1%

Stone built house

0

0.976

0.000

Well with well head

411

97.9%

Thick cotton blanket

1

0.971

0.971

Stone built house

410

97.6%

Doctor visiting the house when sick

1

0.950

0.950

Thick cotton blanket

408

97.1%

Electric fan

0

0.931

0.000

Doctor visiting the house when sick

399

95.0%

A new set of clothes each year

1

0.924

0.924

Electric fan

391

93.1%

Livestock vaccination

0

0.919

0.000

A new set of clothes each year

388

92.4%

Meat once a week

0

0.833

0.000

Livestock vaccination

386

91.9%

Pesticide pump

0

0.800

0.000

Meat once a week

350

83.3%

Watch

0

0.774

0.000

Pesticide pump

336

80.0%

Access to loans

0

0.767

0.000

Watch

325

77.4%

Radio

0

0.743

0.000

Access to loans

322

76.7%

Total

18.471

7.793

Radio

312

74.3%

Toilet - built of stone

188

44.8%

Poverty score

7.793

Table and chairs made of good wood

175

41.7%

Maximum possible score

18.471

Two-compartment wooden wardrobe

135

32.1%

Poverty index

43.29%

TV

88

21.0%

Bathroom

78

18.6%

Motorbike

32

7.6%

PRISM - TOOLKIT FOR EVALUATING THE OUTCOMES AND IMPACTS OF SMALL/MEDIUM-SIZED CONSERVATION PROJECTS

4:3

INTRODUCTION &KEY CONCEPTS

DESIGNING & IMPLEMENTING
THE EVALUATIONTO LEAD A GOOD BNS INTERVIEW:LIVELIHOODS &GOVERNANCE
FACTSHEETBASIC NECESSITIESSURVEY (BNS)•

Take gender into account•Take 30 minutes maximum•

Use local language•

Interview in private•

Use women to survey women; minorities tosurvey minorities•

Provide a clear and concise introduction•

Define basic necessity•

Ensure BNS is NOT a wish list•

Don’t influence answers•

Look around you•

Estimate ages if they don’t know

PRISM
MODULESREPEAT SURVEYS AND BASIC NECESSITIESPerceptions of “basic necessities” can change overtime. When conducting a second BNS (with the same
households) if significant time has elapsed since theprevious survey (e.g. more than 2 years), the focus groupexercise should be repeated to see if any additional items
need to be added to the list or old ones deleted (sinceby now nearly all households may have an item on theoriginal list). Scores can be calculated for each household
both on the basis of a new extended list and, after
excluding the new items, on the basis of the old list.

Household demography
For each member of the family, record information suchas their gender, the number of years they spent in school,
their year of birth and ethnicity.

313

METHOD FACTSHEETS& ANNEXESPRISM - TOOLKIT FOR EVALUATING THE OUTCOMES AND IMPACTS OF SMALL/MEDIUM-SIZED CONSERVATION PROJECTS

4:3

INTRODUCTION &KEY CONCEPTS

DESIGNING & IMPLEMENTING
THE EVALUATION

PRISMMODULESThe poverty index can range from 0% when the family possesses
none of the basic necessities to 100% when it has all of them.

LIVELIHOODS &GOVERNANCE
FACTSHEETMETHOD FACTSHEETS& ANNEXES

Differentiation
By collecting basic social information about households thatyou interview it is possible to investigate how your project has
impacted on the poverty levels of different groups of stakeholders(based on e.g. female-headed households; ethnicity; age ofhousehold head; level of education; ownership of land, etc.).

If poverty scores are recalculated using all items (even those notconsidered basic necessities) and the poverty index recalculatedusing the maximum score from only the basic necessity items, thena score of greater than or equal to 100% denotes households living
at or above the poverty line (i.e. those that have all or more thanthe basic necessities)17.

TABLE 4. HOUSEHOLD BNS FORM – FOLLOW-UP SURVEY.

BASIC NECESSITIESSURVEY (BNS)

To rank order households by wealthand to differentiate between households
that own all goods and have access to allservices on the BNS list (i.e. those thatare at or above the poverty line), it isnecessary to calculate the total value of all
the goods they own. To do that, you mustrecord the number of each asset ownedby the household and the village price forall the goods in the list. Details on how to
do this are contained in the original source
document (Wilkie et al., 2015).
17

314

ASSET ORSERVICEITEM

HAVE NOW?
YES = 1, NO = 0

ARE BASIC NECESSITIES?
YES = 1, NO = 0

HAD IN PREVIOUS SURVEY?
YES = 1, NO = 0

Asset

1 hectare of land per person

0

1

1

Asset

Electric light

1

1

1

Asset

Bicycle

0

0

0

Asset

Concrete rice drying yard

1

0

0

Asset

Wooden rice chest

1

1

1

Service3 meals a day

1

1

1

Asset

Buffalo or cow

1

1

1

Service

All children studying to level 2

0

1

1

Asset

Well with well head

1

1

1

Asset

Stone built house

0

0

0

Asset

Thick cotton blanket

1

1

1

Service

Doctor visiting house when sick

1

1

1

Asset

Electric fan

1

0

0

ServiceA new set of clothes each year

1

1

1

PRISM - TOOLKIT FOR EVALUATING THE OUTCOMES AND IMPACTS OF SMALL/MEDIUM-SIZED CONSERVATION PROJECTSREASON GIVEN FOR CHANGE

Frame broke, can’t afford repairs

Project loan allowed purchase
Constructed by local government– not project

Purchased with earnings fromsale of milk

4:3

INTRODUCTION &KEY CONCEPTS

DESIGNING & IMPLEMENTING
THE EVALUATION

Impact and attribution
One approach to assessing the impact of a project, and determine
attribution, is to carry out the survey in “matched” control villages.
For small and medium sized projects this is unlikely to be possible
or cost effective. There are also likely to be problems in matching,as each community may have its own definition of its basic needs.

LIVELIHOODS &GOVERNANCE
FACTSHEETBASIC NECESSITIESSURVEY (BNS)315

An alternative is to add columns to the BNS form. This would ask
respondents if they think that any change in ownership of a basic
necessity was due to the project, and if yes, why they think this(Table 4).
Alternatively, results could be summarised at community level,
and any significant changes (e.g. in overall poverty levels, or in thenumber of households with access to particular assets and services
in the list of basic necessities) can be discussed at community
or focus group meetings, using PIA methods (see factsheet:
Participatory Impact Assessment).

PRISM
MODULESMETHOD FACTSHEETS& ANNEXES

Further resources
Richards M (2011). Social and Biodiversity Impact Assessment(SBIA) Manual for REDD+ Projects: Part 2 – Social Impact
Assessment Toolbox. Climate, Community & Biodiversity Allianceand Forest Trends, with Rainforest Alliance and Fauna & FloraInternational. Washington, DC (http://www.forest-trends.org/
documents/files/doc_2981.pdf) .
Wilkie D, Wieland M and Detoeuf D (2015). A guide to themodified Basic Necessities Survey: Why and how to conduct
BNS in conservation landscapes. WCS, New York, USA (http://
programs.wcs.org/carbon/Communities/WCS-Resources/
Publications.aspx) .
Wildlife Conservation Society (2007). Household Surveys - a toolfor conservation design, action and monitoring. Technical Manual
4. http://global.wcs.org/DesktopModules/Bring2mind/DMX/
Download.aspx?EntryId=5369&PortalId=0&DownloadMethod=attachment .

PRISM - TOOLKIT FOR EVALUATING THE OUTCOMES AND IMPACTS OF SMALL/MEDIUM-SIZED CONSERVATION PROJECTS

4:3

INTRODUCTION &KEY CONCEPTS

DESIGNING & IMPLEMENTING
THE EVALUATION

PRISMMODULESMETHOD FACTSHEETS& ANNEXES

Factsheet: Participatory Governance Assessment

LIVELIHOODS &GOVERNANCE
FACTSHEETPARTICIPATORY
GOVERNANCEASSESSMENT

DEFINITIONS OF THE THREE KEY ATTRIBUTES OF
EFFECTIVE GOVERNANCE

Introduction
Many factors can play a role in whether or not a group will be able
to govern access and use of natural resources sustainably, andthus effectively. The Natural Resource Governance Tool, NRGT18
focuses on three key attributes of governance: authority, capacity,and power. If a governance group lacks authority to govern (i.e.
people do not trust them to represent and protect their interests),
it will fail to be effective over the long term. If a governance group
has insufficient capacity to govern (i.e. to decide what to do andimplement those decisions), then, even if it is perceived to belegitimate in the eyes of key resource users and rights holders, it isunlikely to be able to govern access to and use of natural resources.
Lastly, even when a governance group is perceived as being thelegitimate authority, and even when it has the capacity to plan andto act, if it does not have the political, economic, or policing power
to exert its authority, it will be unable to govern effectively.

Authority is the perception by natural resource users
and rights holders that a governance group genuinely
represents their interests and has legal or customary
jurisdiction to govern “their” natural resources.
Capacity refers to the skills, abilities, or other resourcesthat allow a group to govern natural resources
effectively.
Power is the ability to influence behaviours or decisions.
Aspects of authority (such as legitimacy) and capacity(financial resources) can enhance a group’s power.

Participatory Governance Assessment (PGA) can be used to assessgood governance practices in local organisations, community
groups, etc. The method involves meeting with a group of selectedparticipants who evaluate the extent to which the decision making
and management practices in a community group conform to theattributes of good governance.

Wilkie D, Detoeuf D, Wieland M and Cowles P (2015). Guidelines for learning and applying
the Natural Resource Governance Tool (NRGT) in landscapes and seascapes. Page 55. USAID,
Washington, D.C. and WCS, Bronx NY. USA https://programs.wcs.org/DesktopModules/
Bring2mind/DMX/Download.aspx?EntryId=28641&PortalId=97&DownloadMethod=attac
hment .
18

316

PRISM - TOOLKIT FOR EVALUATING THE OUTCOMES AND IMPACTS OF SMALL/MEDIUM-SIZED CONSERVATION PROJECTS

4:3

INTRODUCTION &KEY CONCEPTS

DESIGNING & IMPLEMENTING
THE EVALUATION

PRISMMODULESMETHOD FACTSHEETS& ANNEXES

Required steps
Preparatory phase

Step 1.

LIVELIHOODS &GOVERNANCE
FACTSHEETPARTICIPATORY
GOVERNANCEASSESSMENT

317

Initiate preliminary discussions with the localorganisation to identify who will take part in theassessment. Ideally, at least 75% of the organisation’s
members should take part. If the organisation is very
diverse in its membership then different categoriesof member should be proportionally represented.
Make sure that women, marginalised groups anddisadvantaged caste and ethnic groups take part.

Step 2.

External stakeholders from government (e.g. District
Forest Officers), NGOs, local representatives frompolitical parties, and members of other community
groups could also be invited.

Step 3.

Agree a time and date for the assessment meetingand send out invitations.

Step 4.

Prepare the necessary documentation/forms.

THE IMPORTANCE OF LOCAL LANGUAGE
Governance involves some complex issues. Theinterviewer should speak in the local language to makeit easier for local people to understand the questions
and the concepts of governance. Before going into thefield, the team needs to work together to translate thequestionnaire into the local language and to ensure that
all interviewers present the questions in the same way.

PRISM - TOOLKIT FOR EVALUATING THE OUTCOMES AND IMPACTS OF SMALL/MEDIUM-SIZED CONSERVATION PROJECTS

4:3

INTRODUCTION &KEY CONCEPTS

DESIGNING & IMPLEMENTING
THE EVALUATION

The assessment phase

Step 5.

LIVELIHOODS &GOVERNANCE
FACTSHEET

Step 6.

Using the list of questions, (Table 1) facilitate a
discussion among the participants, leading to an
evaluation of the group’s governance performance.

PARTICIPATORY
GOVERNANCEASSESSMENT

Step 7.

Make detailed notes of the discussion and responses,
taking particular note of the views of women,
the poor and marginalised, and people fromdisadvantaged caste and ethnic groups. Final scoresshould be agreed by consensus.

Step 8.

318

At the start of the meeting, explain very carefully
the purpose of the Participatory Governance
Assessment, how the tool works and that it aims
to help them improve their natural resources
governance. Make sure that everyone understands.

It is extremely important that the scores aresupported by a narrative analysis, to help explain
the ratings for each governance attribute. This
allows a more nuanced evaluation of opportunities
for strengthening the ability of the group to govern
the conservation and sustainable use of natural
resources within the landscape or seascape, and lays
the foundation for evaluating whether such actions
resulted in strengthened governance.

PRISM
MODULES

Step 9.

METHOD FACTSHEETS& ANNEXES

Once all the questions are completed, transfer the
results to a spider-web diagram and display it for
all to see. Spider (or radar) diagrams help people
to visualise and interpret the results (See example
below).

Step 10. If the PGA is carried out at the start of the project,
this forms your baseline against which impacts on
governance can be assessed. To assess the impactof your project on an organisation’s governance,
repeat the steps above at the end of the project, andcompare the “before” and “after” spider diagrams. As
governance gets better, the triangles/cobwebs getbigger. Discuss the reasons for any changes in order
to identify those which can be attributed to yourproject.

Step 11. If you don’t have a “before project” baseline then you
could add extra columns to your Assessment Scoring
Sheet. Ask respondents to recall the “before project”
situation (and the reasons for any changes comparedwith the current (“after project”) situation. Ask
whether they can be attributed to the project. See
example in Table 1.

PRISM - TOOLKIT FOR EVALUATING THE OUTCOMES AND IMPACTS OF SMALL/MEDIUM-SIZED CONSERVATION PROJECTS

4:3

INTRODUCTION &KEY CONCEPTS

DESIGNING & IMPLEMENTING
THE EVALUATION

PRISMMODULESMETHOD FACTSHEETS& ANNEXESTABLE 1: ASSESSMENT SCORING SHEET ADAPTED FOR BEFORE AND AFTER IMPACT ASSESSMENT.
ATTRIBUTES

CRITERIA

Legitimacy

The group has no legal or

ASSESSMENTSCORE (CURRENT AFTER PROJECT)-2

traditional authority
The process of legal or

LIVELIHOODS &GOVERNANCE
FACTSHEET-1

traditional recognition of thegroup is underway
The group has a legal or

PARTICIPATORY
GOVERNANCEASSESSMENT

When the projectX

REASONS FOR CHANGE(BEFORE-AFTER PROJECT)

When the project

started the group wasstarted the group was

an informal assemblage

an informal assemblageof households using theof households using theforest. Their status wasforest. Their status was

not formally recognised

not formally recognisedby government althoughby government although

they had legitimacy inthey had legitimacy in

its role is not accepted by all

the community. The

the community. The

stakeholders

project has supported

project has supported

their legal registration

their legal registration

as a Forest User Group.

as a Forest User Group.

1

traditional recognition only todetermine what resource andland use is permitted
The group is officially2

recognised by a legal actof traditionally as having
jurisdiction
Responsibility

319

ASSESSMENTSCORE BEFORE
PROJECT

traditional recognition but

The group has legal or

0

COMMENTS/
EXPLANATION FOR
CURRENT SCOREPRISM - TOOLKIT FOR EVALUATING THE OUTCOMES AND IMPACTS OF SMALL/MEDIUM-SIZED CONSERVATION PROJECTS

4:3

INTRODUCTION &KEY CONCEPTS

DESIGNING & IMPLEMENTING
THE EVALUATION

PRISMMODULESMETHOD FACTSHEETS& ANNEXESPARTICIPATORY GOVERNANCE ASSESSMENT AS A
PLANNING TOOLLIVELIHOODS &GOVERNANCE
FACTSHEET

Note that a PGA can also be used as a planning tool.
Following an initial assessment, participants go on to
identify what the group must do in order to improve
performance against each indicator, and agreed actions
are recorded in a Governance Improvement Plan.

ExamplePARTICIPATORY
GOVERNANCEASSESSMENT

Acknowledgement
The method described here is a modified version that comprises
elements of the Natural Resource Governance Tool (structureof attributes, questions, scoring sheet) and the PGA method
developed by the WWF Hariyo Ban Program in Nepal19 (which
has a focus on assessing the governance of individual communitybased organisations, rather than on governance within alandscape).

19
WWF-Nepal (2013). Internal Governance Tool 3. Participatory Governance Assessment(PGA). WWF-Nepal, Hariyo Ban Program. http://pdf.usaid.gov/pdf_docs/PA00K352.pdf .

320

PRISM - TOOLKIT FOR EVALUATING THE OUTCOMES AND IMPACTS OF SMALL/MEDIUM-SIZED CONSERVATION PROJECTS

4:3

INTRODUCTION &KEY CONCEPTS

DESIGNING & IMPLEMENTING
THE EVALUATION

Factsheet: Participatory photo evaluation
Introduction
In some cases, photographs can be used to find out how a project
has affected livelihoods and well-being within a community.

LIVELIHOODS &GOVERNANCE
FACTSHEETPARTICIPATORY
PHOTO
EVALUATIONThis photographic method can be used as an impact assessment
exercise in its own right or can be used to identify key well-being
indicators for longer term monitoring and assessment.

Required steps
Step 1.

Provide members of the community with one or moredigital cameras, and leave the cameras with themfor a week or more. Ask them to take photographsthat illustrate how the project has affected them –positively or negatively.

STEPS 1 TO 3
Step 2.

321

PRISM
MODULES

Step 3.

METHOD FACTSHEETS& ANNEXES

Compare images taken by different people – men,
women, elders, youth, farmers, fishermen – to see
how different groups of people have been affected,
and what they consider to be key indicators of changebrought about by the project. Comparison of thepictures taken will allow differences and similarities
to be observed. The photographs also provide a
useful starting point for discussion – what changes/
impacts do the pictures symbolise, do others agree
or disagree with the opinion of whoever took the
picture, how widespread is the impact, what was thepathway to that particular impact etc? Also, do theimpacts recorded by the community correspond to
your expectations and assumptions according to your
Theory of Change?

Further reading
http://www.participatorymethods.org/method/participatoryvisual-methods-case-study .

Collect the photos and show them to participants,
asking them to discuss the pictures collectively.

PRISM - TOOLKIT FOR EVALUATING THE OUTCOMES AND IMPACTS OF SMALL/MEDIUM-SIZED CONSERVATION PROJECTS

4:3

INTRODUCTION &KEY CONCEPTS

DESIGNING & IMPLEMENTING
THE EVALUATION

Annex: Questions for assessment meeting

1.

PRISM
MODULES

4.

Legitimacy
1.1 What does the group do? What is its job?
1.2 Does it have the right to do this job? (De facto
community authority or de jure governmental
authority).

LIVELIHOODS &GOVERNANCE
ANNEX
QUESTIONS FORASSESSMENT
MEETING

4.3 If no, what could or should it do to make sure thatpeople have a chance to make suggestions, proposals
and complaints?

1.4 If no, who does have the right to do this job? Who should
be doing it?
Accountability
2.1 Who does the group answer to?

5.

2.2 If it makes a mistake, or does not do its job the way that
it is supposed to, who can tell the group to change?
3.

3.3 If no, what are the weaknesses and what should thegroup be doing to make sure that people are wellinformed about what it is doing?
322

Fairness
5.1 Does the group treat everyone in a fair way? Or does it
seem to favour some people over others?
5.2 If fair, what is it about how the group works that makes
it seem fair to you?

Transparency
3.1 Does the group do a good job of letting people knowwhat it does? (Does the group communicate what it isdoing?) Do most people feel like they know what thegroup is doing?
3.2 If yes, how does it keep people informed of its activities?
What does the group do to make sure that people knowwhat they are doing?

Participation
4.1 Are people living around here able to tell the groupwhen they have a suggestion about how it could do its
job better, or more fairly? Can people complain if they
feel that the group is not doing a good job and expectthat someone will pay attention?
4.2 If yes, what does it do to make sure people have achance to make suggestions and proposals or complain
about problems?

1.3 If yes, where did the right come from? (From thecommunity, from the State, etc.).

2.

METHOD FACTSHEETS& ANNEXES

5.3 If not fair, what is not fair about how the group works?
5.4 What could it do to be more fair in the way it does its
job?
6.

Knowledge/skills
6.1 Do people who work for the group know how to dotheir jobs well? Do they have the technical knowledge tomanage resource use or monitor natural resources, forexample? Do they know the policies? Examples?

PRISM - TOOLKIT FOR EVALUATING THE OUTCOMES AND IMPACTS OF SMALL/MEDIUM-SIZED CONSERVATION PROJECTS

4:3

INTRODUCTION &KEY CONCEPTS

7.

DESIGNING & IMPLEMENTING
THE EVALUATION

Resources
7.1 Does the group have the equipment it needs to do agood job? Examples?
7.2 Does the group have the personnel it needs to do its
job? Examples?
7.3 Does the group have the finances it needs to do its job?
Examples?

LIVELIHOODS &GOVERNANCE
ANNEX
QUESTIONS FORASSESSMENT
MEETING

8.

9.

Institutional framework
8.1 Are there laws or rules that are written down so thatthe people who work for the group and local people allknow what it is supposed to be doing and why?
Motivation
9.1 Do you think the group tries to do a good job? Whatmakes you say that?

10. Enacting decisions
10.1 When the group makes a decision to adopt a rule, doauthorities (for example chiefs, mayors, governors, civil
servants) respect the decision? Or do they just do what
they want to anyway? Why do you think that is?
10.2 Can the group protect, more or less, resources fromillegal use (for example from military poachers,
neighbouring jurisdictions or the population
themselves)? Explain.
11. Being held accountable11.1 If people whose resource use is managed think that thegroup is doing a bad job, is there anything the people
themselves can do to make it do better?
323

PRISM
MODULESMETHOD FACTSHEETS& ANNEXES

11.2 If yes, what can they do?
11.3 If no, what should people be able to do if the group is
doing a bad job?
12. Diversity
12.1 Are women represented equally in the group?
12.2 If yes, are their ideas listened to and adopted by thegroup?
12.3 If no, why is this?
12.4 Are minority peoples represented equally in the group?
12.5 If yes, are their ideas listened to and adopted by thegroup?
12.6 If no, why is this?

Resources
Wilkie D, Detoeuf D, Wieland M and Cowles P (2015). Guidelines
for learning and applying the Natural Resource GovernanceTool (NRGT) in Landscapes and Seascapes. Page 55. USAID,
Washington, D.C. and WCS, Bronx NY. USA https://programs.wcs.
org/DesktopModules/Bring2mind/DMX/Download.aspx?EntryId=28641&PortalId=97&DownloadMethod=attachment .
WWF-Nepal (2013). Internal Governance Tool 3. Participatory
Governance Assessment (PGA). WWF-Nepal, Hariyo Ban
Program. http://pdf.usaid.gov/pdf_docs/PA00K352.pdf .

PRISM - TOOLKIT FOR EVALUATING THE OUTCOMES AND IMPACTS OF SMALL/MEDIUM-SIZED CONSERVATION PROJECTS

4:3

INTRODUCTION &KEY CONCEPTS

DESIGNING & IMPLEMENTING
THE EVALUATION

Annex: Governance Assessment scoring sheet

Reproduced with permission from:

PRISM
MODULESMETHOD FACTSHEETS& ANNEXES

Tool (NRGT) in Landscapes and Seascapes. Page 55. USAID,
Washington, D.C. and WCS, Bronx NY. USA https://programs.wcs.
org/DesktopModules/Bring2mind/DMX/Download.aspx?EntryId=28641&PortalId=97&DownloadMethod=attachment .

Wilkie D, Detoeuf D, Wieland M and Cowles P (2015). Guidelines
for learning and applying the Natural Resource Governance

LIVELIHOODS &GOVERNANCE
ANNEX

ATTRIBUTES

Legitimacy

324

ASSESSMENTSCOREAUTHORITY

GOVERNANCEASSESSMENTSCORING SHEET

Developed by:
African Wildlife Foundation
World Wildlife Fund Wildlife
Conservation Society
Wildlife Conservation Society

CRITERIA

The group has no legal or traditional authority todefine rules or the ability to follow through with
implementation of plans.

-2

The process of legal or traditional recognition of thegroup to define the rules and implement procedures isunderway but not yet completed.

-1

The group has no legal or official recognition toparticipate in the process of creating rules andmonitoring procedures to ensure enforcement, but
its role is not accepted by all local stakeholders (ifcustomary) or there are disagreements over attribution(in the case of the government).

0

The group has legal or traditional recognition only todetermine what resource and land use is permitted,
to define who has access to certain resources, or can
implement certain land use practices. Its legitimacy inrelation to enforcement, however, is not recognized.

+1

The group is officially recognised by a legal act or
traditionally as having jurisdiction to determine what
resources or land use is permitted, to determine
resource access. It can also decide what penalties will be
imposed for violation of these rules.

+2

PRISM - TOOLKIT FOR EVALUATING THE OUTCOMES AND IMPACTS OF SMALL/MEDIUM-SIZED CONSERVATION PROJECTSCOMMENTS/
EXPLANATIONSRECOMMENDATIONS

4:3

INTRODUCTION &KEY CONCEPTS

ATTRIBUTESLIVELIHOODS &GOVERNANCE
ANNEX

Responsibility

GOVERNANCEASSESSMENTSCORING SHEET

Participation

325

DESIGNING & IMPLEMENTING
THE EVALUATIONCRITERIA

PRISM
MODULESASSESSMENTSCORE

The group has no set specifications or clearly
defined structure and no action is taken to guide themanagement of natural resources on behalf of thecommunities it is supposed to represent.

-2

The group has no specifications, no structure and takes
no active responsibility in decision-making (they are justthere in name and do not operate in practice,).

-1

The group has a clearly defined set of specifications
and a clearly defined structure, but its members do nottake any initiative in decision-making related to the
management.

0

The group has a set of specifications and a clearly
defined structure and is actively involved in the
management, and decision-making, but ignores theinterest of the communities they represent.

+1

The group has a set of specifications and a clearly
defined structure, and is actively involved inmanagement decisions while taking into account theinterested of the communities they represent.

+2

Indigenous and traditional peoples have no impact on
the process of developing policies that limit access anduse of resources, or in deciding penalties for those who
do fail to meet the standards sets.

-2

Indigenous and traditional people have little impact ondecisions regarding the management of the area, andplay no role in management.

-1

The participation of indigenous and traditional people indecision-making and management of the area is random
and on an ad hoc basis.

0

Indigenous and traditional people directly contribute tosome important decisions related to management but
their involvement should be improved.

+1

Indigenous and traditional people directly participate inall relevant decisions concerning the management andsuggestions, proposals and complaints are considered(continuous and constant interaction).

+2

PRISM - TOOLKIT FOR EVALUATING THE OUTCOMES AND IMPACTS OF SMALL/MEDIUM-SIZED CONSERVATION PROJECTSMETHOD FACTSHEETS& ANNEXES

COMMENTS/
EXPLANATIONSRECOMMENDATIONS

4:3

INTRODUCTION &KEY CONCEPTS

ATTRIBUTESLIVELIHOODS &GOVERNANCE
ANNEXTransparency

GOVERNANCEASSESSMENTSCORING SHEET

Equity326

DESIGNING & IMPLEMENTING
THE EVALUATIONCRITERIA

PRISM
MODULESASSESSMENT
SCORE

People have no information on the group’s activities anddecisions. Authorities are using the system for their own
interest.

-2

People receive information that is filtered, from asmall number of sources, and is of little use to different
member of the governance group.

-1

People receive scattered, irregular, information from thegovernance group.

0

The group shares decisions and informs the population
about their activities through regular forums
without giving them the opportunity to access all theinformation. There is less possibility for the authorities
to abuse the system for their own interests.

+1

The meetings of the governance group are open to thepress and the public, budgets can be found easily, andthe laws and decisions made are open for discussion.

+2

Rules and enforcement standards set by the group do
not provide fair management standards in defining rules,
their application, and the sharing of benefits arising fromthe management of natural resources-2

The group blatantly discriminates against stakeholders
in defining rules, their application, and the sharingof benefits arising from the management of natural
resources-1

Standards of fairness are provided in the rules put inplace, but are never enforced or applied to the “client’shead”.

0Standards of fairness are provided in the rules put inplace but are only partially implemented.

+1

The rules governing access and use of natural resources
are fair in terms of who benefits and who bears the costs
and are applied equally to all individuals and groups.
ie the costs and benefits are evenly distributed, equal
rights under the law and equal application of the law.

+2

PRISM - TOOLKIT FOR EVALUATING THE OUTCOMES AND IMPACTS OF SMALL/MEDIUM-SIZED CONSERVATION PROJECTSMETHOD FACTSHEETS& ANNEXES

COMMENTS/
EXPLANATIONSRECOMMENDATIONS

4:3

INTRODUCTION &KEY CONCEPTS

ATTRIBUTESDESIGNING & IMPLEMENTING
THE EVALUATIONCRITERIA

PRISM
MODULESASSESSMENTSCORECAPACITYLIVELIHOODS &GOVERNANCE
ANNEX
GOVERNANCEASSESSMENTSCORING SHEET

327

Knowledge
and Skills

The group has no competence and knowledge of a) factors
biological, economic, historical, socio-political and know what could
jeopardize long-term viability of the use of natural resources; b)
policies and practices that would be needed to remedy the situation
so that the resources of value are kept and used in a sustainable
manner; and c) means of a group to monitor the effectiveness of theimplementation of their conservation plans.

-2

The group has low skills and knowledge of a) factors- biological,
economic, historical, sociopolitical and know what could jeopardize
the long-term viability of the use of natural resources; b) policies
and practices that would be needed to remedy the situation sothat the resources of value are kept and used in a sustainable
manner; and c) a means of monitoring the effectiveness of theimplementation of their conservation plans.

-1

The group has good skills and knowledge of a) factors- biological,
economic, historical, sociopolitical and know what could jeopardize
the long-term viability of the use of the natural resources, but not b)
policies and practices that would be needed to remedy the situation
so that the resources of value are kept and used in a sustainable
manner.

0

The group has good skills and knowledge of a) factorsbiological,
economic, historical, sociopolitical and know what could jeopardize
the long-term viability of the use of natural resources and b)
policies and practices that would be needed to remedy the situation
so that the resources of value are kept and used in a sustainable
manner, but not c) means a group can monitor the effectiveness of
the implementation of their conservation plans.

+1

The group has good skills and knowledge of a) factors-biological,
economic, historical, sociopolitical and what could jeopardize
the long –term viability of the use of natural resources, b) polices
and practices that would be needed to remedy the situation sothat resources of value are kept so that the resources of value
are kept and used and used in a sustainable manner; and c) themeans to monitor the effectiveness of the implementation of theirconservation plans.

+2

PRISM - TOOLKIT FOR EVALUATING THE OUTCOMES AND IMPACTS OF SMALL/MEDIUM-SIZED CONSERVATION PROJECTSMETHOD FACTSHEETS& ANNEXES

COMMENTS/
EXPLANATIONSRECOMMENDATIONS

4:3

INTRODUCTION &KEY CONCEPTS

ATTRIBUTES

Resources

LIVELIHOODS &GOVERNANCE
ANNEX
GOVERNANCEASSESSMENTSCORING SHEETRegulatory
Framework

DESIGNING & IMPLEMENTING
THE EVALUATIONCRITERIA

PRISM
MODULESASSESSMENT
SCORE

There is no material or financial means, staff numbers are notsufficient-2

The budget, material and the staff available are inadequate to meetthe needs of the group.

-1

The budget and the material are inappropriate but the staff isadequate to respond to the basic needs of the group’s mission.

0

The budget, the material, and the staff available are acceptable tomeet the basic needs of the group’s mission, but it is not able tomeet all the basic needed.

+1

The budget, material and personnel available is sufficient to cover
all the needs of the group and to measure the plans that they put
into place and to evaluate and communicate the results and theimpacts of their efforts.

+2

There are no laws or traditional customs that define the rules ofmanagement-2

There is a profusion of laws with often-contradictory rules for
management-1

There are national laws and regulations or customary principles,
which are not adapted to the context managed by the governancegroup resources
There are specific rules and management standards, which are notused to drive the management by governance group.
The actions of the governance group are based on a set of rules andregulations that explicitly engage the informal or customary law
on who has access to what resources and to what these resources
are and how these resources can be used, though their actions areunlikely to be effective in the long term.

328

PRISM - TOOLKIT FOR EVALUATING THE OUTCOMES AND IMPACTS OF SMALL/MEDIUM-SIZED CONSERVATION PROJECTS

0

+1

+2

METHOD FACTSHEETS& ANNEXES

COMMENTS/
EXPLANATIONSRECOMMENDATIONS

4:3

INTRODUCTION &KEY CONCEPTS

ATTRIBUTESLIVELIHOODS &GOVERNANCE
ANNEX
GOVERNANCEASSESSMENTSCORING SHEET

Motivation

DESIGNING & IMPLEMENTING
THE EVALUATIONCRITERIA

PRISM
MODULESASSESSMENT
SCORE

Members of the group have no initiative and spontaneity. The groupexists in name only and even when support is given the expected
outcome is not obtained.

-2

Members of the group perform their duties sporadically because
they do not receive financial, material and technical assistance.
Once they have this support in the future, they will be able to act
more effectively.

-1

More than 20% of group members are willing to give their all in thework duties without external input (material, financial, technical)

0More than half of the group members have taken initiatives to give
body and soul and engage personally and unwavering commit to do
whatever is necessary to accomplish their mission with or without
external input (material, financial or other).

+1

With or without external input (material, financial, technical), thegroup and its members give body and soul and engage personallywith unwavering commitment to do whatever is necessary toaccomplish their mission.

+2

POWER
The group cannot make and enforce decisions.
-2
The group takes management decisions theoretically but neverimplements them.
Implementa- The group has the power to make decision and implement them ontion of group a portion of the species that exist on the resources it governs.
decisions
The group makes management decisions and implements andenforces them in harmony with the users and rights holders.
The group makes management decisions, has the authority andautonomy to enforce and apply all the rules in harmony with users
and right holders.

329

PRISM - TOOLKIT FOR EVALUATING THE OUTCOMES AND IMPACTS OF SMALL/MEDIUM-SIZED CONSERVATION PROJECTS

-1

0

+1

+2

METHOD FACTSHEETS& ANNEXES

COMMENTS/
EXPLANATIONSRECOMMENDATIONS

4:3

INTRODUCTION &KEY CONCEPTS

ATTRIBUTESLIVELIHOODS &GOVERNANCE
ANNEX
GOVERNANCEASSESSMENTSCORING SHEET330

DESIGNING & IMPLEMENTING
THE EVALUATIONCRITERIA

PRISM
MODULESASSESSMENT
SCORE

Users and rights holders of the resources under control of thegovernance group do not have access to management have noopportunity to access management information, make requests or
demand accountability.

-2

A platform for public access to information and formulation ofquestions to governance groups was planned but was never madeoperational.

-1

A public platform for information and formulation of questions to
the governance group exists, but users and resource rights holders
Accountability
do not have the knowledge, time or financial resources to exploit
and vice versa.

0

A public platform for information and formulation of questions to
the governance groups exists, users and rights holders have theknowledge, time and financial resources to operate, but are notinformed of the opportunity.

+1

There is a statutory or customary framework in place to facilitate
public access to information about the group’s work, to makerequests and to get answers in response.

+2

PRISM - TOOLKIT FOR EVALUATING THE OUTCOMES AND IMPACTS OF SMALL/MEDIUM-SIZED CONSERVATION PROJECTSMETHOD FACTSHEETS& ANNEXES

COMMENTS/
EXPLANATIONSRECOMMENDATIONS

4:4

INTRODUCTION &KEY CONCEPTS

DESIGNING & IMPLEMENTING
THE EVALUATION

PRISMMODULES

Policy factsheets and annexes

Media tracking
Media scorecards
Observation checklist for documenting meetings

POLICY
FACTSHEETSAND ANNEXES

Policymaker ratings
Bellwether methodology
Civil society tracking tool

331

PRISM - TOOLKIT FOR EVALUATING THE OUTCOMES AND IMPACTS OF SMALL/MEDIUM-SIZED CONSERVATION PROJECTSMETHOD FACTSHEETS& ANNEXES

4:4

INTRODUCTION &KEY CONCEPTS

DESIGNING & IMPLEMENTING
THE EVALUATION

Factsheet: Media tracking

POLICY
FACTSHEETMEDIA TRACKING

Summary
Used to evaluate media strategies that often are part of advocacy
efforts, this method examines whether media coverage of anissue changes over time. Basic media tracking reveals whether
advocacy issues or messages are mentioned more often but says
little else about their coverage. This method could also be used
where media strategies are not expressly employed – to evaluate/
gauge the level of media and public awareness and attention raisedfor the policy area/campaign. This need not be done as an isolated
event or evaluation, but could provide a system/methodology for
ongoing monitoring of media coverage.

PRISM
MODULESMETHOD FACTSHEETS& ANNEXES

Required steps

Step 1.

Identify the types of sources to track (print,
broadcast or electronic).

Step 2.

Choose the specific media outlets to track.

Step 3.

Select the relevant time periods to search.

Step 4.

Select the specific search terms (the advocacy
messages). All of these must be carefully chosen andtested. Search terms, for example, must be broad
enough to uncover relevant articles but specificenough to have value and be manageable.

Step 5.

Count the number of times the search terms orphrases appear.

STEPS 1 TO 5

Acknowledgement
Based on guidance in Annie E Casey Handbook of data collection
tools (pages 21-23).
http://dmeforpeace.org/sites/default/files/AEF_Data%20
Collection%20Tools.pdf

332

PRISM - TOOLKIT FOR EVALUATING THE OUTCOMES AND IMPACTS OF SMALL/MEDIUM-SIZED CONSERVATION PROJECTS

4:4

INTRODUCTION &KEY CONCEPTS

DESIGNING & IMPLEMENTING
THE EVALUATION

Factsheet: Media scorecards

POLICY
FACTSHEETMEDIASCORECARDSSTEPS 1 & 2

SummaryMedia scorecards are a method for comparing the media
coverage generated by different issues or organisations over time.
Combined with traditional media tracking that searches media
outlets for key words or phrases, media scorecards help to assessvery quickly the extent to which particular issues or organisations
are referenced in specific outlets, compared with the extent towhich other issues or organisations (allies or opponents) are also
referenced in those outlets. The “scorecard” in the title refers tothese comparisons.
Introduction
Many tools and services are available to access news content
rapidly via the internet. They range from free to expensive,and from quick to time-intensive. Some evaluators use a paid
subscription service offered by the online news database
LexisNexis (www.lexisnexis.com/en-us/gateway.page ), whichallows easy searching of hundreds of local, regional, national andinternational media sources. Another option is to use free searchtools such as Google News Search (news.google.com), which makes
it possible to search for specific terms from a specific online news
source. Searches can be set up for multiple sources and used tocreate ‘Google Alerts’, which automatically email updated results
on a pre-set basis. Although online media databases and searchengines return results very quickly, they are not instantaneous.
Depending on the source, there may be slight delays; it may takea day or two for news databases to upload new articles fromcertain sources. Really Simple Syndication (RSS) readers do notreceive instant updates, and even Google can take a few hours toindex new web pages. Delays, however, are usually not significant

333

PRISM
MODULESMETHOD FACTSHEETS& ANNEXES

in terms of the requirements and timeframes for overall projectmonitoring and evaluation.
Required steps
Step 1.
Identify the types of sources to track (web-pages,
broadcasts), the specific media outlets to track, therelevant time periods to search and the specificsearch terms to use.
Step 2.

Identify the issues or organisations to track as acomparison. The sources, outlets and timeframe
remain the same. Patterns and trends for the issue ororganisation of interest are then examined alongside
the comparison, to identify for example who or what
is generating more coverage, who is generating it firstand where the coverage is located.

Examples include:•
total number of stories;•

total area or time allotted to each story;•

total number of news stories above average length;•

total stories with pictures and graphics;•

total stories on the front page or in the local TV news
programme.

Acknowledgements
Based on guidance in UNICEF Monitoring and Evaluating
Advocacy Toolkit (page 26) http://www.unicef.org/evaluation/
files/Advocacy_Toolkit_Companion.pdf .

PRISM - TOOLKIT FOR EVALUATING THE OUTCOMES AND IMPACTS OF SMALL/MEDIUM-SIZED CONSERVATION PROJECTS

4:4

INTRODUCTION &KEY CONCEPTS

DESIGNING & IMPLEMENTING
THE EVALUATION

PRISMMODULESMETHOD FACTSHEETS& ANNEXES

Factsheet: Observation checklist for documenting meetings
Observation Checklist

POLICY
FACTSHEETOBSERVATIONCHECKLIST FOR
DOCUMENTINGMEETINGS

SummaryThis method can be used to gauge the degree to which community
members, key individuals or groups change their beliefs orpriorities about a particular issue. A simple observation checklist
can help to capture how often a particular issue is on a meeting
agenda, whether the issue was discussed, what the maincontent or emphasis was, the length or depth of the discussion
and perceptions of seriousness. See the example of a meetingobservation checklist below. It could also be used to record
whether or not a particular policy issue, organisations/network or
campaign was referenced or discussed. Checklists should then be
entered into an observation database, perhaps shared between
network partners, to track changes and influence over time.

For meetings
Date: __________________________

Length of meeting: ______________________

Setting: ________________________# of male attendees: ____

# of female attendees: ____

1.

What were the main issues discussed during this meeting (e.g. resource use, species
conservation, climate change, etc.)?
__________________________________________________________________________

2.
3.

Was conservation on the agenda?

YES

Was conservation discussed?

NO

YES

4.

What was the main content of the policy discussion?

__________________________________________________________________________

5.

Was agreement reached in this discussion?

YES

NO

What was the length of the discussion? ______________________________________

6.

Would you say that the issue(s) discussed were taken seriously by the attendees?
YESNO

Please explain: ____________________________________________________________

Acknowledgement
Based on guidance in Annie E Casey Handbook of data collection
tools (pages 21-23). http://dmeforpeace.org/sites/default/files/
AEF_Data%20Collection%20Tools.pdf

7.

Was any action planned related to conservation?

YES

NO

Please explain:_________________________________________________________________________

8.

Additional notes or comments:_________________________________________________________________________

334

NO

(If answered "yes" for question 3, please continue; if answered "no" for question 3, pleaseskip to question 8.)

PRISM - TOOLKIT FOR EVALUATING THE OUTCOMES AND IMPACTS OF SMALL/MEDIUM-SIZED CONSERVATION PROJECTS

4:4

INTRODUCTION &KEY CONCEPTS

POLICY
FACTSHEETPOLICYMAKER
RATINGSSTEPS 1, 2 & 3

DESIGNING & IMPLEMENTING
THE EVALUATION

PRISMMODULESMETHOD FACTSHEETS& ANNEXES

Factsheet: Policymaker ratings

Required steps

SummaryThis is a method for gauging political will or support for aparticular advocacy issue or proposal amongst a defined groupof policymakers (e.g. legislature, council). The approach wasdeveloped in response to the perceived inadequacy of indicators
commonly used to gauge policymaker support on issues (e.g.
number of new policies introduced on the issue; number of votes
for or against a specific policy). This method takes a different
approach to measuring such support, and capitalises on advocates’
insider knowledge about individual policymakers’ stances on policy
issues. It does not create extra work for advocates, but instead
usefully transfers what they already know through their regularintelligence-gathering and outreach. This method thus contains
an element of subjectivity, as it relies on individual advocate
knowledge, experience and judgement (although the “confidence
ratings” do go some way to ameliorating this); hence why it isadvised that a minimum of three project team members/advocatesshould make an input to the rating process.

Step 1.

Identify the issue.

Step 2.

Select specific policymakers/policymaking body for
analysis.
For example parliament, council, organisation, etc.

Step 3.

Familiarise yourself with the scoring system andcriteria.
Policymakers are rated according to three criteria.
Policymaker level of support — Individual policymakersupport for an issue based on his or her public
behaviours or actions in relation to the issue.
Policymaker level of influence — Policymaker influenceon the policy issue of interest (similar to the idea ofa power analysis). Ratings are based on criteria that
research shows relate to policymaker influence (seetable below).
Rater level of confidence — Confidence in the accuracyof ratings on the first two scales.

335

PRISM - TOOLKIT FOR EVALUATING THE OUTCOMES AND IMPACTS OF SMALL/MEDIUM-SIZED CONSERVATION PROJECTS

4:4

INTRODUCTION &KEY CONCEPTS

DESIGNING & IMPLEMENTING
THE EVALUATION

PRISMMODULESMETHOD FACTSHEETS& ANNEXESPOLICYMAKER RATINGS SCORING CRITERIASCALERATING

DEFINITION

Support1
Not at all supportive

No evidence that this person has spoken or taken any action in support of the policy
issue (includes opposition)

2
Somewhat supportive

Has indicated being favourably disposed to the policy issue (e.g. expresses support
for the issue or mentions it in one-to-one or small group conversations)3
Supportive

Occasionally takes action either publicly or behind the scenes beyond voting
in support of the policy issue (e.g. speaks out at public hearings, gets quoted in
the media, includes it in speeches, assigns bills to a power legislator, encourages
colleagues to support policies, plays a role in budget negotiations)

POLICY
FACTSHEETPOLICYMAKER
RATINGS

4
Extremely supportive

Influence
Criteria:
1. Majority party member
2. Relevant content expertise
3. Seniority/experience (record of
public service)
4. Reputation/respect (e.g. has
been able to exercise somepower/ leadership in thelegislature)
5. Key committee member
6. Formal leadership position(chairs a key committee or is a
Senate or Assembly leader)

Confidence336

1
Not very
influential
2
Somewhat influential
3
Influential

Has a well-known reputation for being a champion of the policy issue and regularly
takes leadership in advancing it (e.g. makes it a key part of their platform or agenda)
Meets none or only one criterion

Meets at least two criteria

Meets three or four criteria and/or is on a key committee

4
Extremely influential

Meets five or six criteria and/or holds a formal leadership position in the legislatureor and/or chairs a key committee

1
Not very confident

Ratings are a guess, based on third-hand, unverifiable or unreliable information
about the policymaker and their related interest or lack of interest (e.g. thepolicymaker or their staff saying they “love the issue” in a small meeting where they
feel pressure to speak positively)

2
Somewhat confident

Ratings are a fairly well informed guess (e.g. advocates have picked up consistent
information from more than one source, but sources may not be 100% verifiable orreliable, or the information collected is somewhat vague)

3
Confident

Ratings are based on advocates’ direct contact with the individual or informationfrom a trusted and reliable source

PRISM - TOOLKIT FOR EVALUATING THE OUTCOMES AND IMPACTS OF SMALL/MEDIUM-SIZED CONSERVATION PROJECTS

4:4

INTRODUCTION &KEY CONCEPTS

Step 4.

POLICY
FACTSHEETDESIGNING & IMPLEMENTING
THE EVALUATION

PRISMMODULESMETHOD FACTSHEETS& ANNEXES

Rate targeted policymakers on the three scales thatassess support, influence and confidence using thescorecard and criteria above. Brief notes as to thereason/evidence behind the ratings would be useful.

Step 5.

Discuss and provide some justification for thereasons behind the scores given in each completed
scorecard. It would be useful to note briefly the main
points of evidence used to inform this rating.

At least three project team members (the more thebetter, to increase robustness of the assessment) should
participate in the rating process. Advocates either
undertake the ratings as a group (arriving at a consensus
group rating), or do it independently and then average
their ratings.

Step 6.

Once ratings are complete, composite ratings arecalculated and aggregated across policymakers. Data,such as individual policymakers’ party affiliation,
district representation or community membership,
can be added to enable different ways of looking at
the analysis. This method is repeatable over time todetermine whether and how indicators shift.

POLICYMAKER
RATINGS
POLICYMAKER RATINGS SCORECARD.

STEPS 4, 5 & 6
POLICYMAKER RATINGS SCORECARD

Policymaker/policymaking body:Issue:
Scale

RatingDefinition

SupportInfluence
Confidence
Total
Notes/justification

Acknowledgement:
Based on guidance inInnovation Network Unique
Methods in Advocacy
Evaluation Brief. Examples
on pages 5-6. http://www.
innonet.org/resources/files/
Unique_Methods_Brief.pdf .

337

PRISM - TOOLKIT FOR EVALUATING THE OUTCOMES AND IMPACTS OF SMALL/MEDIUM-SIZED CONSERVATION PROJECTS

4:4

INTRODUCTION &KEY CONCEPTS

POLICY
FACTSHEETBELLWEATHERMETHODOLOGYSTEPS 1 & 2

338

DESIGNING & IMPLEMENTING
THE EVALUATION

PRISMMODULESMETHOD FACTSHEETS& ANNEXES

Factsheet: Bellweather methodology

Required steps

Introduction
This method was developed specifically for the evaluation of policy
advocacy efforts. It determines where a policy issue or proposal is
positioned on the policy agenda, how decision makers and otherinfluential actors are thinking and talking about it, and how likely
policymakers are to act on it.

Step 1.

Select the types of bellwethers to interview.
Categories might include: policymakers; media;
donors; UN agencies; other NGOs/CBOs;
researchers; and business and trade experts.
Categories should represent the types of individuals
whose opinions are important or influential on thepolicy issue of interest.

Step 2.

Selecting the bellwether sample requires thedevelopment of criteria for selecting individual
bellwethers.

The method involves structured interviews with “bellwethers”:
influential people in the public and private sectors whose positions
require that they be politically informed and that they track abroad range of policy issues. Bellwethers are knowledgeable andinnovative thought leaders, and political insiders whose opinions
about policy issues carry substantial weight and predictive value.
The bellwether methodology returns data that indicate how
effective advocates have been in communicating their messages,
and whether they have been successful in moving their issue onto
the policy agenda or in increasing its importance. Bellwether
data also informs advocates about specific gaps in bellwether
knowledge.

At least half of the sample should include bellwethers
who do not have a specific connection to the policy issue
being explored. This increases the probability that issue
awareness or knowledge detected during interviews
can be linked to advocacy efforts, rather than being
the result of personal experiences or other extraneous
variables. Other selection criteria might include gender,
cultural and geographical diversity. Once selection
criteria are developed, subject matter experts nominate
bellwethers who fit the criteria.

PRISM - TOOLKIT FOR EVALUATING THE OUTCOMES AND IMPACTS OF SMALL/MEDIUM-SIZED CONSERVATION PROJECTS

4:4

INTRODUCTION &KEY CONCEPTS

POLICY
FACTSHEETBELLWEATHERMETHODOLOGYSTEPS 3 & 4

339

DESIGNING & IMPLEMENTING
THE EVALUATION

Step 3.

Set up the interviews. Bellwethers must be unaware
before the interview begins that the interview will
focus on the specific policy issue. They are informed
about what the interview will generally cover but
they do not receive specific details. This helps toensure that bellwethers’ responses are authentic andunprompted.

Step 4.

Conduct interviews with questions that determine
what bellwethers know and think about the policyof interest. The interview might start by asking
bellwethers what issues they think are at the top of
the policy agenda.

PRISM
MODULESMETHOD FACTSHEETS& ANNEXES

Acknowledgement:
Based on guidance in UNICEF Monitoring and Evaluating
Advocacy Toolkit (page 26). http://www.unicef.org/evaluation/
files/Advocacy_Toolkit_Companion.pdf .

Their responses (which will be unprompted because
they do not know beforehand which specific policy issue
you are exploring) indicate whether the advocacy issue
of interest shows up in that agenda, and if so where,
and along with what other issues. Later questions canget more specific and ask about bellwethers’ familiarity
with the issue of interest and probe what they know,
allowing later content analysis to determine whether
advocates’ messages surface in bellwether discourse
about the issue. Bellwethers might also be asked to
predict whether they think the issue will advance in thenear future or in the longer term.

PRISM - TOOLKIT FOR EVALUATING THE OUTCOMES AND IMPACTS OF SMALL/MEDIUM-SIZED CONSERVATION PROJECTS

4:4

INTRODUCTION &KEY CONCEPTS

DESIGNING & IMPLEMENTING
THE EVALUATION

PRISMMODULESgovernments, the private sector, donors and otherimportant stakeholders, in pursuit of common
conservation and development objectives.

Annex: Civil society tracking tool
Goal
Local and national civil society groups dedicated to global
conservation priorities collectively possess sufficientorganisational and technical capacity to be effective advocates for,and agents of, conservation and sustainable development for atleast the next 10 years.

POLICYANNEX
CIVIL SOCIETY
TRACKING TOOL

340

Criteriai.

Human resources. Local and national civil society
groups collectively possess technical competenciesof critical importance to conservation, on topics that
include protected areas management; conservation
monitoring and analysis; sustainable financing; policy
analysis and influence; environmental education andmedia outreach; and threat mitigation and adaptation.

ii.

Management systems and strategic planning. Local
and national civil society groups collectively possess
sufficient institutional and operational capacity andstructures to raise funds for conservation and to ensure
the efficient management of conservation projects andstrategies.

iii.

Partnerships. Effective mechanisms (e.g. discussion
forums, round tables, mutual support networks,
alliances, etc.) exist for conservation-focused civilsociety groups to work in partnership with one
another and through networks with local communities,

METHOD FACTSHEETS& ANNEXES

iv.

Financial resources. Local civil society organisations
have access to long-term funding sources to maintainthe conservation results achieved via funding grants
and/or other initiatives, through access to new donor
funds, conservation enterprises, memberships,
endowments, and/or other funding mechanisms.

v.

Transboundary cooperation. In multi-country
situations, mechanisms exist for collaboration acrosspolitical boundaries at site, corridor and/or national
scales.

PRISM - TOOLKIT FOR EVALUATING THE OUTCOMES AND IMPACTS OF SMALL/MEDIUM-SIZED CONSERVATION PROJECTS

4:4

INTRODUCTION &KEY CONCEPTS

DESIGNING & IMPLEMENTING
THE EVALUATION

Indicators•

POLICYANNEX
CIVIL SOCIETY
TRACKING TOOL

•

•

•

•

341

Criterion (i) will be considered to have been partially
met if knowledge and capacity within local civil society
organisations is rated as satisfactory or above in atleast 50% of the technical competencies considered aspriorities.
Criterion (i) will be considered to have been fully met
if knowledge and capacity within local civil society
organisations is rated as satisfactory or above in atleast 90% of the technical competencies considered aspriorities.
Criterion (ii) will be considered to have been partially
met if at least 50% of the priority areas concerned haveat least one local, national, or international civil society
organisation dedicated to their conservation with atleast satisfactory institutional and operational capacity.
Criterion (ii) will be considered to have been fully met
if at least 90% of the priority areas concerned have atleast one local, national, or international civil society
organisation dedicated to their conservation with atleast satisfactory institutional and operational capacity.
Criterion (iii) will be considered to have been
partially met if fully institutionalised and sustainable
partnerships are dedicated to coordinatingconservation and development actions among key
stakeholder groups in at least 50% of the priority areas
concerned.

PRISM
MODULESMETHOD FACTSHEETS& ANNEXES•

Criterion (iii) will be considered to have been fully met
if fully institutionalised and sustainable partnerships
are dedicated to coordinating conservation anddevelopment actions among key stakeholder groups inat least 90% of the priority areas concerned.

•

Criterion (iv) will be considered to have been partially
met if 50% of the priority areas concerned have accessto stable and diversified long-term funding sources
for conservation through support to local civil society
organisations.

•

Criterion (iv) will be considered to have been fully met
if 90% of the priority areas concerned have access tostable and diversified long-term funding sources forconservation through support to local civil society
organisations.

•

Criterion (v) will be considered to have been partially
met if effective mechanisms for transboundary
collaboration exist in at least 50% of the countries
concerned.

•

Criterion (v) will be considered to have been fully met if
effective mechanisms for transboundary collaboration
exist in at least 90% of the countries concerned.

Acknowledgement:
Based on CEPF “tracking tool” for use in systematically monitoring
the impact of CEPF on civil society development. Available from:http://www.cepf.net/resources/publications/Pages/monitoring_
and_evaluation.aspx

PRISM - TOOLKIT FOR EVALUATING THE OUTCOMES AND IMPACTS OF SMALL/MEDIUM-SIZED CONSERVATION PROJECTS

4:4

INTRODUCTION &KEY CONCEPTS

DESIGNING & IMPLEMENTING
THE EVALUATION

PRISMMODULESMETHOD FACTSHEETS& ANNEXESCIVIL SOCIETY TRACKING TOOL WORKSHEET
CRITERIONi.

POLICYANNEX
CIVIL SOCIETY
TRACKING TOOL

Human resources. Local and national civil society
groups collectively possess technical competenciesof critical importance to conservation,, on topics
that include protected areas management;
conservation monitoring and analysis; sustainablefinancing; policy analysis and influence;
environmental education and media outreach; andthreat mitigation and adaptation..

ii. Management systems and strategic planning.
Local and national civil society groups
collectively possess sufficient institutional andoperational capacity and structures to raise
funds for conservation and to ensure the efficientmanagement of conservation projects andstrategies.
iii. Partnerships. Effective mechanisms exist forconservation-focused civil society groups to workin partnership with one another, and throughnetworks with local communities, governments,
the private sector, donors, and other important
stakeholders, in pursuit of common objectives.
iv. Financial resources. Local civil society
organisations have access to long-term funding
sources to maintain the conservation results
achieved via funding grants and/or otherinitiatives, through access to new donor funds,
conservation enterprises, memberships,
endowments, and/or other funding mechanisms.
v.

342

Transboundary cooperation. In multi-country
situations, mechanisms exist for collaboration
across political boundaries at site, corridor and/or
national scales.

BASELINE (YEAR)MID TERM(YEAR)

FINAL (YEAR)

Not metNot metNot metPartiallymet

Partiallymet

Partially
met

Fully met

Fully met

Fully met

Not metNot metNot metPartiallymet

Partiallymet

Partially
met

Fully met

Fully met

Fully met

Not metNot metNot metPartiallymet

Partiallymet

Partially
met

Fully met

Fully met

Fully met

Not metNot metNot metPartiallymet

Partiallymet

Partially
met

Fully met

Fully met

Fully met

Not metNot metNot metPartiallymet

Partiallymet

Partially
met

Fully met

Fully met

Fully met

PRISM - TOOLKIT FOR EVALUATING THE OUTCOMES AND IMPACTS OF SMALL/MEDIUM-SIZED CONSERVATION PROJECTSCOMMENTS

4:5

INTRODUCTION &KEY CONCEPTS

DESIGNING & IMPLEMENTING
THE EVALUATION

Method factsheets

Annexes

Field methods for species monitoring

Scorecard for evaluating action plan completion

Sampling approaches for monitoring different species toassess their status

Threat reduction scoring
Scorecard for evaluating changes in species’ status
Scorecard for evaluating changes in habitat status

Analytical issues when estimating abundance, distribution,
survival and reproduction
Field methods for habitat monitoring
Sampling approaches for monitoring habitat status

Using field assessment methods for evaluating changes in
species’ status
Using remote assessment methods evaluating changes inhabitat status
Using field assessment methods for evaluating changes inhabitat status

343

METHOD FACTSHEETS& ANNEXES

Scorecard for evaluating changes in knowledge gaps

Scorecard for evaluating action plan adequacy

SPECIES AND
HABITATMANAGEMENT
FACTSHEETS AND
ANNEXES

PRISM
MODULESPRISM - TOOLKIT FOR EVALUATING THE OUTCOMES AND IMPACTS OF SMALL/MEDIUM-SIZED CONSERVATION PROJECTS

4:5

INTRODUCTION &KEY CONCEPTS

DESIGNING & IMPLEMENTING
THE EVALUATION

PRISMMODULESabundance of populations at particular sites can be used
to identify the most important locations for particularspecies or habitats, and to help prioritise which actions
to implement at which locations.

Factsheet: Scorecard for evaluating knowledge gaps

SPECIES & HABITATMANAGEMENT
FACTSHEETSCORECARD
FOR EVALUATINGKNOWLEDGE GAPS

SummaryThis factsheet describes a simple method for assessing theoutcomes and impacts of project actions on knowledge gaps. This
involves first identifying the knowledge gaps relating to different
aspects of species/habitat status that the project actions aim toaddress, and then scoring the level of knowledge. The change
in scores during the course of a project can be compared to ahypothetical scenario in which the project did not take place, inorder to estimate the outcome and impact of project actions.

Required steps

Step 1.

STEP 1

344•

Trends (trends over time in population size/distribution/
habitat extent, etc.). Trends in the distribution orabundance of species, or in the extent of particular
habitat-types, provides important context to inform
conservation action, for example by prioritisinglocations where populations are suffering the greatest
rate of decline, or by identifying core areas of stability
for protection.

•

Threats (types of threats and their scope/area, severity/
intensity, timing/urgency, and hence impact/magnitude).
Identifying the cause of species population or range
decline is a key precursor to effective conservation
action.

•Ecology (habitat requirements, resource needs e.g. diet,nest-sites, etc., predators, social structure, reproductive
behaviour, movement patterns, phenology). A goodunderstanding of the ecology of a species can inform
conservation action, by identifying limiting factors andthe ways in which management may be used to counterthe impact of constraints. An assessment of habitat
requirements may be used to test whether changes inhabitat availability or quality may be limiting. Ecologicalunderstanding is particularly important where species
are being actively managed, or to inform the creation of
new habitats or reintroductions of species.

Identify knowledge gaps.
The main components of knowledge that project
actions may address can be summarised as follows:•

Distribution (including locations of populations,distribution limits, altitudinal preferences, occupancy
patterns, habitat extent). This is fundamental toidentifying the locations where conservation action may
benefit particular species or habitats, as well as helping
to prioritise where conservation action may best be
targeted.

•

Population size (including global/national/local
population size, relative abundance, population
density and territory/home range size). Variation in the

METHOD FACTSHEETS& ANNEXESPRISM - TOOLKIT FOR EVALUATING THE OUTCOMES AND IMPACTS OF SMALL/MEDIUM-SIZED CONSERVATION PROJECTS

4:5

INTRODUCTION &KEY CONCEPTS•

SPECIES & HABITATMANAGEMENT
FACTSHEETSCORECARD
FOR EVALUATINGKNOWLEDGE GAPSSTEP 2345•

DESIGNING & IMPLEMENTING
THE EVALUATION

Extinction risk (typically assessed using the IUCN Red
List categories and criteria (IUCN 2014), although othersystems are also used in different national contexts).
Quantifying extinction risk is an important componentof conservation prioritisation and a means of tracking
conservation success. It involves collating information
on distribution, population size, trends and ecology (lifehistory parameters) in particular.
Solutions (effectiveness of conservation action,
management, etc.). The most successful conservation
actions are based on interventions which are known to
work. This often requires research to test effectiveness,for example through experimental or monitoring
approaches.

PRISM
MODULES

Step 2.

METHOD FACTSHEETS& ANNEXES

Score the degree to which project actions improved
the knowledge base.
Once knowledge gaps have been identified, each should
be scored using Table 1 and following the scoring
criteria in Table 2. Scores should reflect:•

the knowledge base at the beginning of the project(scored retrospectively if not assessed at the beginning);•

the knowledge base at the end of the project; and•an estimate of what the knowledge base would
have been if the project had not taken place (thecounterfactual scenario).

Ideally the scoring should be carried out both by the project teamand by certain key informants who have relevant knowledge of the
project’s conservation target (e.g. experts on the species/site being
targeted). See the factsheet Key informant interviews for more
information on this approach. If scoring is only being carried out
by the project team you should attempt to do this as objectively as
possible, and justify the scores selected.

PRISM - TOOLKIT FOR EVALUATING THE OUTCOMES AND IMPACTS OF SMALL/MEDIUM-SIZED CONSERVATION PROJECTS

4:5

INTRODUCTION &KEY CONCEPTS

DESIGNING & IMPLEMENTING
THE EVALUATION

PRISMMODULESMETHOD FACTSHEETS& ANNEXESTABLE 1. KNOWLEDGE GAPS SCORECARD.
ASPECT OF KNOWLEDGE

BEFORE THE PROJECTAFTER THE PROJECTDistribution
Abundance

SPECIES & HABITATMANAGEMENT
FACTSHEETSCORECARD FOR
EVALUATINGKNOWLEDGE GAPS

Trend
Threats
Actions/interventions
Extinction risk
Justification:

Table 1. Knowledge gaps scorecard. The criteria for scoring eachknowledge aspect are given in Table 2.
The difference between (a) the change over time in the actual
project scores and (b) the hypothetical scores demonstrates theoutcome/impact of the project’s actions. The estimate without
the project is important to account for the fact that knowledge
may have improved due to external factors (e.g. work by otherindividuals/organisations that may have increased the knowledge
base for the species or ecosystem), rather than through the
project. Note that most projects do not address all of theknowledge gaps outlined in the table, and many only address one
or two (see examples below).

346

PRISM - TOOLKIT FOR EVALUATING THE OUTCOMES AND IMPACTS OF SMALL/MEDIUM-SIZED CONSERVATION PROJECTSESTIMATE WITHOUT THE PROJECT

4:5

INTRODUCTION &KEY CONCEPTS

DESIGNING & IMPLEMENTING
THE EVALUATION

PRISMMODULESMETHOD FACTSHEETS& ANNEXESTABLE 2. SCORES FOR THE LEVEL OF KNOWLEDGE ABOUT DIFFERENT ASPECTS OF INFORMATION ABOUT SPECIES/HABITATS.

SPECIES & HABITATMANAGEMENT
FACTSHEETASPECT OF KNOWLEDGESCORE

Distribution

1 – species/habitat distribution is unknown or based upon a small number of historical (>25 years old) records.
2 – some recent (<25 years old) location records or confirmed absences, or a large amount of historical data.
3 – large number of recent location records or confirmed absences, or an expert-based description of likely distribution is available.
4 – sample surveys from across a species or habitat’s potential range provide a good indication of the extent and spread of recent
location records or absences within the project area.
5 – published distribution model with good model fit covering entire project area, or extensive data across the project area produce
robust distribution information.

Abundance

1 – population size unknown.
2 – some abundance records exist but these are partial, unvalidated or old (>25 years).
3 – large number of recent (<25 years) abundance records exist from unstructured surveys, or unvalidated expert-based assessments of
population size within the project area.
4 – abundance quantified, but based on potentially biased methods (non-random locations, failing to account for detectability, etc.).
5 – Robust population estimate exists based on high quality methods (random sample locations, detectability accounted for, etc.).

Trend

1 – population/range trend/habitat extent unknown (i.e. data from one period only).
2 – some time-series of records or counts exists, but insufficient to infer more than anecdotal trend information, or data span 25 years
or less.
3 – large number of comparable records/counts exist from the project area, sufficient for at least some analysis of occurrence/
abundance between two time-periods to assess change through time over at least 25 years.
4 –monitoring scheme established and able to produce trends or change between at least two time-periods of 25 years or more, but
associated with some methodological biases (e.g. non-random locations, changes in methods, variation in effort).
5 –robust population and/or range change estimates over 25 or more years, derived from high quality methods.

Threats (for studies
of a single threat, orof a species/habitat
to examine a range of
threats)

1 – threats unknown.
2 – some ad hoc data or expert judgement about the importance of potential threat(s) is available.
3 –limited analysis of a single threat is available, but from a limited sample or a small proportion of the project area (i.e. may not be
representative) or large-scale expert-based assessment of key threats (e.g. IUCN Red List assessment).
4 – robust analysis of threats undertaken but key threat(s) still not identified.
5 –published analysis has identified key threats to species/habitat with high confidence.

Actions/interventions

1 – solutions untested.
2 – some anecdotal information about responses to interventions at one or a small number of locations.
3 – some quantitative information or expert-based assessment on the effectiveness of action.
4 – some formal analytical tests undertaken on the success of intervention from at least one location.
5 – analysis of the impact of intervention and solutions upon populations and distributions at multiple locations, which may besynthesised using meta-analyses.

Extinction risk

1 – extinction risk unknown.
2 – some information gathered to assess extinction risk.
3 – complete set of information available for extinction risk assessment.
4 – IUCN Red List assessment or equivalent categorical approach undertaken.
5 –quantitative assessment of extinction risk undertaken (e.g. Population Viability Analysis).

SCORECARD FOR
EVALUATINGKNOWLEDGE GAPS

347

PRISM - TOOLKIT FOR EVALUATING THE OUTCOMES AND IMPACTS OF SMALL/MEDIUM-SIZED CONSERVATION PROJECTS

4:5

INTRODUCTION &KEY CONCEPTS

Step 3.

DESIGNING & IMPLEMENTING
THE EVALUATION

PRISMMODULES

Discuss and provide justification for the scores given.

Discuss and record reasons for the scores given. Thisis just as important as the scoring itself, and provides
useful insights into how the project brought about (or insome cases did not bring about) the observed change.

SPECIES & HABITATMANAGEMENT
FACTSHEETSCORECARD FOR
EVALUATINGKNOWLEDGE GAPSSTEP 3348

PRISM - TOOLKIT FOR EVALUATING THE OUTCOMES AND IMPACTS OF SMALL/MEDIUM-SIZED CONSERVATION PROJECTSMETHOD FACTSHEETS& ANNEXES

4:5

INTRODUCTION &KEY CONCEPTS

DESIGNING & IMPLEMENTING
THE EVALUATION

EXAMPLE 1

SPECIES & HABITATMANAGEMENT
FACTSHEETSCORECARD FOR
EVALUATINGKNOWLEDGE GAPS

Project: A threatened species was known to occur within aprotected area from one or two historical records only. In
order to inform the future development of an action planfor the area, a project to re-survey the protected area forthe species was carried out. The specific aim of the project
was to confirm the continued persistence or absence of thespecies. By using a robust sampling approach, the project
confirmed the species persistence, and, using modelling ofwidely-available remote-sensing data, successfully predicted

PRISM
MODULESMETHOD FACTSHEETS& ANNEXES

the occurrence and abundance of the species acrossthe protected area, with the results being subsequently
published in a local journal.
Evaluation: In order to evaluate this, the project team asked
key informants with experience of the species and thetarget site to use the criteria in the PRISM knowledge gaps
factsheet to score the level of knowledge on the species
distribution and abundance before and after the project, andto estimate what the level of knowledge would have been
without the project (see table below).

ASPECT OF KNOWLEDGE

BEFORE THE PROJECTAFTER THE PROJECTESTIMATE WITHOUT THE PROJECTDistribution

1 – species/habitat distribution
is unknown or based upon asmall number of historical (>25years old) records.

5 – published distribution
model with good model fit
covering entire project area,
or extensive data across the
project area produce robust
distribution information.

1 – species/habitat distribution
is unknown or based upon asmall number of historical (>25years old) records.

Abundance

1 – population size unknown.

5 – robust population estimate
exists, based on high quality
methods.

1 – population size unknown.

Justification: Key informants scored the level of knowledge after the project as 5, highlighting the project’s publication of a model
predicting occurrence and abundance. They also scored the level of knowledge before the project and the estimate of what would
have happened without the project as 1, as no other projects or studies had looked at the target area or species in that time. Given
that the original project goal was to improve the score for distribution from 1 to 2 this represents a greater impact than originally
anticipated, and it would not have been achieved without the project.

349

PRISM - TOOLKIT FOR EVALUATING THE OUTCOMES AND IMPACTS OF SMALL/MEDIUM-SIZED CONSERVATION PROJECTS

4:5

INTRODUCTION &KEY CONCEPTS

DESIGNING & IMPLEMENTING
THE EVALUATION

PRISMMODULESMETHOD FACTSHEETS& ANNEXES

EXAMPLE 2

SPECIES & HABITATMANAGEMENT
FACTSHEETSCORECARD FOR
EVALUATINGKNOWLEDGE GAPS

Project: A conservation project converted some historicallyfarmed land to re-create wetland habitat adjacent to asmall protected area supporting a population of criticallyendangered frogs. It was not known whether the approachproduced habitat of sufficient quality for the frogs tocolonise, although the approach had been shown to work forother species in other contexts.

comparing it to the quality within the protected area, andchecked for the occurrence of the frogs within the recreated habitat, assuming that in the absence of habitat
restoration, the target species would not occur on farmland.

Evaluation: To evaluate the success of this action the project
team monitored habitat quality on the re-created habitat,

The project team undertook an evaluation both to assess
the project results and to assess whether they had beenable to fill the knowledge gap concerning whether or not
the approach worked for encouraging frogs to re-colonise a
restored area.

ASPECT OF KNOWLEDGE

BEFORE THE PROJECTAFTER THE PROJECTESTIMATE WITHOUT THE PROJECT

Actions/interventions

2 – some anecdotal
information about responses tointerventions at one or a small
number of locations.

4 – formal analytical tests,
including of counterfactuals,of the success of intervention
undertaken at least one
location.

2 – some anecdotal
information about responses tointerventions at one or a small
number of locations.

Justification: The restored habitat differed in water quality from habitat in the protected area, and the restored area was notcolonised by frogs. The conservation element of the project was therefore unsuccessful, but the research to fill the knowledge gap
was successful, achieving an actions/interventions score of 4. In the absence of the project, the level of information about the qualityof the re-created habitat and about the occurrence of the amphibians would have remained the same.

350

PRISM - TOOLKIT FOR EVALUATING THE OUTCOMES AND IMPACTS OF SMALL/MEDIUM-SIZED CONSERVATION PROJECTS

4:5

INTRODUCTION &KEY CONCEPTS

DESIGNING & IMPLEMENTING
THE EVALUATION

PRISMMODULESMETHOD FACTSHEETS& ANNEXES

Factsheet: Scorecard for degree of completion of an action plan

SPECIES & HABITATMANAGEMENT
FACTSHEET

SummaryThis factsheet describes a simple method for evaluating project
actions that aim to develop or contribute to the compilation of
species action plans or habitat conservation plans. It presents
a 1-10 scale for scoring the degree of completion of a plan. Thechange in scores during the course of a project is compared to ahypothetical scenario in which the project did not take place, inorder to estimate the outcome and impact of project actions.

Required steps

SCORECARD
FOR DEGREE OFCOMPLETION OF AN
ACTION PLAN
STEP 1351

Step 1.

Familiarise yourself with the scoring criteria.
Typically, compiling an action plan involves the ten
steps outlined in Table 1 (note that in some cases, actionplans may be compiled by soliciting stakeholder input
remotely, rather than in person at a workshop).

TABLE 1: SCORES FOR EACH STAGE IN CREATING A
SPECIES ACTION PLAN.
STAGE OF COMPLETIONSCORE

No existing action plan0

Stakeholders identified

1

Background information compiled

2

Workshop conducted with key stakeholders
reviewing background information, agreeing
status, threats and actions

3

Action plan drafted4

Action plan reviewed by workshop participants

5

Acton plan reviewed by wider stakeholder
network

6

Action plan finalised

7

Action plan published, launched, and distributed

8

Action plan officially recognised e.g. by IUCNSSC, government, multi-lateral environmental
agreement, other stakeholder groups

9

Action plan implemented (confirmed bymonitoring & evaluation)

10

PRISM - TOOLKIT FOR EVALUATING THE OUTCOMES AND IMPACTS OF SMALL/MEDIUM-SIZED CONSERVATION PROJECTS

4:5

INTRODUCTION &KEY CONCEPTS

Step 2.

SPECIES & HABITATMANAGEMENT
FACTSHEETSCORECARD
FOR DEGREE OFCOMPLETION OF AN
ACTION PLAN
STEP 2

DESIGNING & IMPLEMENTING
THE EVALUATION

Score the degree to which project actions
contributed towards the completion of theaction plan.

Following the scoring criteria in Table 1, fill in the scorecard in
Table 2 to categorise the degree to which the project contributed
to the completion of the species/site action plan. Scores shouldreflect:•

the degree of completion at the beginning of the project(scored retrospectively if not assessed at the beginning);•

the degree of completion at the end of the project; and•the estimated degree of completion if the project had
not taken place (the counterfactual scenario).

PRISM
MODULES

Ideally the scoring should be carried out both by the project teamand by certain key informants who have relevant knowledge of the
project’s conservation target (e.g. experts on the species/site being
targeted). See the factsheet Key informant interviews for more
information on this approach. If scoring is only being carried out
by the project team you should attempt to do this as objectively as
possible and justify the scores selected.

TABLE 2. ACTION PLAN COMPLETION SCORECARD.
ACTION PLAN

DEGREE OF COMPLETION
BEFORE THE PROJECTAFTER THE PROJECT

Insert name of action
plan here
Justification:

352

METHOD FACTSHEETS& ANNEXESPRISM - TOOLKIT FOR EVALUATING THE OUTCOMES AND IMPACTS OF SMALL/MEDIUM-SIZED CONSERVATION PROJECTSESTIMATED SITUATION IF THE
PROJECT HAD NOT HAPPENED

4:5

INTRODUCTION &KEY CONCEPTS

Step 3.

DESIGNING & IMPLEMENTING
THE EVALUATION

Discuss and provide justification for the scores given.

Discuss and record reasons for the scores given: this is just as
important as the scoring, and it provides useful insights into how
the project brought about (or in some cases did not bring about)the observed change.

SPECIES & HABITATMANAGEMENT
FACTSHEETSCORECARD
FOR DEGREE OFCOMPLETION OF AN
ACTION PLAN

PRISM
MODULES

Step 4.

Present the results.

The final difference between the actual scores and the
hypothetical scenario can be presented to stakeholders either asa narrative or as quantitative scores (see example 1) in a table ordiagram.

Note that many small/medium-sized projects, rather thancompleting a full action plan themselves, will instead generate
new information on a species population size and distribution, andfeed this into the action plan. In such cases, the scoring criteria
remain the same, with the difference between the scores given
for the situation after the project and those given for the estimateof what would have happened without the project reflecting thedegree to which the project contributed towards the action plan’scompletion (see example 2 below). In these cases it is particularly
important to provide justification and explain in as much detail as
possible how the project’s actions contributed to the production of
the wider plan.

STEPS 3 & 4353

METHOD FACTSHEETS& ANNEXESPRISM - TOOLKIT FOR EVALUATING THE OUTCOMES AND IMPACTS OF SMALL/MEDIUM-SIZED CONSERVATION PROJECTS

4:5

INTRODUCTION &KEY CONCEPTS

DESIGNING & IMPLEMENTING
THE EVALUATION

PRISMMODULESMETHOD FACTSHEETS& ANNEXES

EXAMPLE 1
Project: A project aimed to produce a species action plan fora rare plant.

SPECIES & HABITATMANAGEMENT
FACTSHEETSCORECARD
FOR DEGREE OFCOMPLETION OF AN
ACTION PLAN

Scoring and justification: This can either be stated as anarrative or scored as follows: score at start of project =0, score at end = 8, score in the absence of the project = 0;
outcome of project is 8-0 = 8.

EXAMPLE 1.
ACTION PLAN

Species action plan for rareplant

DEGREE OF COMPLETION
BEFORE THE PROJECTAFTER THE PROJECTESTIMATED SITUATION IF THE
PROJECT HAD NOT HAPPENED

0 – No existing action plan

8 – Action plan published,launched, and distributed

0 – No existing action plan

Justification: No action plan existed at the start of a project. The project implemented actions that resulted in publication of the planand its distribution, but no official recognition or implementation happened within the project timeframe. No other efforts by nonproject actors to produce an action plan occurred during the project timeframe.

354

PRISM - TOOLKIT FOR EVALUATING THE OUTCOMES AND IMPACTS OF SMALL/MEDIUM-SIZED CONSERVATION PROJECTS

4:5

INTRODUCTION &KEY CONCEPTS

DESIGNING & IMPLEMENTING
THE EVALUATION

PRISMMODULESMETHOD FACTSHEETS& ANNEXES

EXAMPLE 2

SPECIES & HABITATMANAGEMENT
FACTSHEETSCORECARD
FOR DEGREE OFCOMPLETION OF AN
ACTION PLAN

Project: An action planning process was already underway
to produce a species action plan for a critically endangered
shorebird along a migratory flyway. To support this process,a project carried out shorebird surveys at an importantstaging point on the flyway. This generated new informationon the species’ population size and distribution, which wasthen fed into the action plan.
Scoring and justification:

EXAMPLE 2.
ACTION PLAN

Species action plan for
shorebird

DEGREE OF COMPLETION
BEFORE THE PROJECTAFTER THE PROJECTESTIMATED SITUATION IF THE
PROJECT HAD NOT HAPPENED

0 – No existing action plan

6 – Action plan reviewed fromwider stakeholder network

5 – Action plan reviewed byworkshop participantsJustification: While the action plan had successfully completed its initial stages without the input of the project, the information
gathered by the project played an important role in allowing the action plan to progress to the next stages of development. Thisis reflected by the difference in scores between those given after the project and for the estimate of what would have happened
without the project.

355

PRISM - TOOLKIT FOR EVALUATING THE OUTCOMES AND IMPACTS OF SMALL/MEDIUM-SIZED CONSERVATION PROJECTS

4:5

INTRODUCTION &KEY CONCEPTS

SPECIES & HABITATMANAGEMENT
FACTSHEETDESIGNING & IMPLEMENTING
THE EVALUATIONSTEP 1356

METHOD FACTSHEETS& ANNEXES

Factsheet: Scorecard for adequacy of an action plan

Step 1.

SummaryThis factsheet describes a simple method for evaluating project
actions that aim to develop or contribute to the development ofspecies action plans or habitat conservation plans. It presents a
1-9 scale for scoring the adequacy of a plan. The change in scores
during the course of a project is compared to a hypotheticalscenario in which the project did not take place, in order toestimate the outcome and impact of project actions.

The adequacy of an action plan can be judged in three ways:
1. How comprehensively does it assess the status, threats
and actions needed?

Familiarise yourself with the scoring criteria.

2.

Does it have wide stakeholder buy-in and/or official
endorsement?

3.

Is it being used?

Each of these parameters can be scored as low (1), medium (2) orhigh (3) using the criteria in Table 1, with the sum of the scores
giving a maximum of 9 across all three parameters.

Required steps

SCORECARD FOR
ADEQUACY OF AN
ACTION PLAN

PRISM
MODULESTABLE 1. SCORES FOR THE ADEQUACY OF AN ACTION PLAN.
ADEQUACY OF ACTION
PLAN

SCORING CRITERIA

Comprehensivenessof assessment of thestatus, threats andactions needed

1 - Based only on a single source or missing many key sources of information.
2 - Missing some key sources of information.
3 - Fully comprehensive.

Breadth of stakeholder
buy-in and official
endorsement

1 - Based on limited stakeholder inputs. Some key stakeholder groups not consulted.
2 - At least one key stakeholder group not consulted or endorsement lacking by at least one key authority.
3 - Based on wide stakeholder inputs. Endorsed by relevant authorities (e.g. local/national government, international
bodies).

Degree of application
to coordinate actions
being implemented

1 - Action plan not being used. No actions being implemented, or those that are being implemented are done without
reference to the action plan.
2 - Action plan being partly used to define, prioritise and coordinate actions that are being implemented, but someactions or actors operating without reference to the action plan.
3 - Action plan being explicitly used to define, prioritise and coordinate actions that are being implemented.

PRISM - TOOLKIT FOR EVALUATING THE OUTCOMES AND IMPACTS OF SMALL/MEDIUM-SIZED CONSERVATION PROJECTS

4:5

INTRODUCTION &KEY CONCEPTS

Step 2.

SPECIES & HABITATMANAGEMENT
FACTSHEETSCORECARD FOR
ADEQUACY OF AN
ACTION PLAN
STEP 2

DESIGNING & IMPLEMENTING
THE EVALUATION

Using the scoring criteria in Table 1, score theadequacy of the action plan.

Ideally the scoring should be carried out both by the project teamand by certain key informants who have relevant knowledge of the
project’s conservation target (e.g. experts on the species/site being
targeted). See the factsheet Key informant interviews for more
information on this approach. If scoring is only being carried out
by the project team you should attempt to do this as objectively as
possible and justify the scores selected.

PRISM
MODULES

Fill in the scorecard in Table 2 following the scoring criteria in Table
1 to categorise the degree to which the project contributed to thecompletion of the species/site action plan. Score the situation:(1)

at the beginning of a project (this can be done
retrospectively if necessary);(2)

at the end of the project; and(3)

for a hypothetical scenario in which the project did nottake place (i.e. a counterfactual scenario).

TABLE 2. ACTION ADEQUACY SCORECARD.
ADEQUACY OF ACTION PLAN

BEFORE THE PROJECTAFTER THE PROJECT

Comprehensiveness ofassessment of the status,
threats and actions needed
Breadth of stakeholder buy-in
and official endorsement
Degree of application tocoordinate actions being
implemented
Total

Justification:

357

METHOD FACTSHEETS& ANNEXESPRISM - TOOLKIT FOR EVALUATING THE OUTCOMES AND IMPACTS OF SMALL/MEDIUM-SIZED CONSERVATION PROJECTSESTIMATED SITUATION IF THE
PROJECT HAD NOT HAPPENED

4:5

INTRODUCTION &KEY CONCEPTS

Step 3.

SPECIES & HABITATMANAGEMENT
FACTSHEETDESIGNING & IMPLEMENTING
THE EVALUATION

PRISMMODULES

Compare the difference between (a) the actualchange over time in scores and (b) the change inscores for the hypothetical scenario, to represent theoutcome and impact of project actions.

This third step is to account for the fact that scores may haveincreased during the period of the project owing to external
factors (e.g. work by other individuals/organisations may haveincreased the knowledge base for the species or ecosystem).
Justification notes should be provided for all scores given, in orderto support the assessment and to make it as objective as possible.

SCORECARD FOR
ADEQUACY OF AN
ACTION PLAN
STEP 3358

PRISM - TOOLKIT FOR EVALUATING THE OUTCOMES AND IMPACTS OF SMALL/MEDIUM-SIZED CONSERVATION PROJECTSMETHOD FACTSHEETS& ANNEXES

4:5

INTRODUCTION &KEY CONCEPTS

DESIGNING & IMPLEMENTING
THE EVALUATION

EXAMPLE

SPECIES & HABITATMANAGEMENT
FACTSHEETSCORECARD FOR
ADEQUACY OF AN
ACTION PLAN

Project: A project aimed to support the implementationof a site action plan for a small nature reserve. The action
plan had been developed as part of a previous project, butreports from conservationists working at the site indicatedthat it was not being implemented. The aim of the new
project was therefore to review the action plan and makesuggestions for its improvement and to promote uptake.
The project found the action plan to be sufficiently
comprehensive, having been reviewed and accepted byexperts on the target site. The action plan was also foundto have significant buy-in from the reserve authorities.
However the project identified that local communities were
either unaware of the plan or were unsure about

PRISM
MODULESMETHOD FACTSHEETS& ANNEXES

its implementation. The project worked with this key
stakeholder group to find ways of improving the plan and topromote its uptake. As a result, the action plan came to be
implemented at several of the target communities within thereserve, although logistical issues meant that not all target
communities could be reached.
Scoring and justification: Reserve managers andconservationists working at the site were asked to use thecriteria in the scorecard to rate the adequacy of the action
plan before and after the project and for a hypotheticalscenario where the project did not take place. Scoring
showed an increase in total score for action plan adequacy
from 5 to 8. If the project hadn’t happened, the score was
likely to have remained at 5.

ADEQUACY OF ACTION PLAN

BEFORE THE PROJECTAFTER THE PROJECTESTIMATED SITUATION IF THE
PROJECT HAD NOT HAPPENED

Comprehensiveness of assessmentof the status, threats and actions
needed

3 - Fully comprehensive.

3 - Fully comprehensive.

3 - Fully comprehensive.

Breadth of stakeholder buy-in andofficial endorsement

2 - At least one key stakeholder
group not consulted or endorsement
lacking from at least one key
authority.

3 - Based on wide stakeholder inputs. 2 - At least one key stakeholder
Endorsed by relevant authorities.
group not consulted or endorsement
lacking from at least one key
authority.

Degree of application to coordinate
actions being implemented

1 - Action plan not being used. No
actions being implemented, or thosethat are being implemented are done
without reference to the action plan.

2 - Action plan being partly used
to define, prioritise and coordinate
actions that are being implemented,
but some actions or actors operating
without reference to the action plan.

1 - Action plan not being used. No
actions being implemented, or thosethat are being implemented are done
without reference to the action plan.

Total

5

8

5

Justification: Increase in stakeholder buy-in and the fact that the action plan is now being implemented at some of the target communities are the main
reasons for the increase in score.

359

PRISM - TOOLKIT FOR EVALUATING THE OUTCOMES AND IMPACTS OF SMALL/MEDIUM-SIZED CONSERVATION PROJECTS

4:5

INTRODUCTION &KEY CONCEPTS

SPECIES & HABITATMANAGEMENT
FACTSHEETTHREAT
REDUCTIONSCORINGSTEPS 1 & 2

360

DESIGNING & IMPLEMENTING
THE EVALUATION

PRISMMODULESMETHOD FACTSHEETS& ANNEXES

Factsheet: Threat reduction scoring

Required steps

SummaryThis factsheet describes a simple method for assessing themagnitude of threats to species or habitats. It involves identifying
the relevant threats, scoring the timing, scope and severity of eachthreat, and summing these to calculate threat magnitude. Thechange in the magnitude of threats during the course of a projectis compared to a hypothetical scenario in which the project did nottake place, in order to estimate the outcome and impact of project
actions.

Step 1.

Classify which types of threats affect the species
or habitat, preferably using the IUCN/CMP Threatsclassification scheme.

Step 2.

Familiarise yourself with the scoring criteria.

The impact of a threat can be scored using the following criteria:a.

Scope - the estimated proportion of the population of
the species in the project area or the proportion of thehabitat in the project area that the threat affects.

b.

Severity - the rate of population decline of the species
or the rate of degradation of the habitat, within the areaof the project that is affected by the threat, over ten
years (or for species, three generations, if this is longer).

c.

Timing - whether the threat is happening currently, has
happened in the past or may happen in the future.

PRISM - TOOLKIT FOR EVALUATING THE OUTCOMES AND IMPACTS OF SMALL/MEDIUM-SIZED CONSERVATION PROJECTS

4:5

INTRODUCTION &KEY CONCEPTS

DESIGNING & IMPLEMENTING
THE EVALUATION

PRISMMODULESMETHOD FACTSHEETS& ANNEXESTABLE 1. SCORES FOR THE SCOPE, SEVERITY’ AND TIMING OF THREATS AFFECTING A SPECIES OR HABITAT.

SPECIES & HABITATMANAGEMENT
FACTSHEETTHREAT
REDUCTIONSCORING
STEP 3

THREAT PARAMETERSCORING CRITERIA

Scope of threat (% ofpopulation or area)

0 - Few individuals/small area (<10%)
1 - Some of population/area (10-49%)
2 - Most of population/area (50-90%)
3 - Whole population/area (>90%)

Severity of threat (%decline over 10 years or
3 generations)

0 - No or imperceptible deterioration (<1%)
1 - Slow deterioration (1–9.9%)
2 - Moderate deterioration (10–30%)
3 - Rapid deterioration (>30%)

Timing/risk of threat

0 - Past (and unlikely to return) and no longer limiting
1 - Likely in long term (beyond 4 years)
2 - Likely in short term (within 4 years)
3 - Happening now

Table 1. Scores for the scope, severity’ and timing of threats
affecting a species or habitat. (Figures given in brackets are for
guidance, rather than implying that quantitative data are essential
for applying this method).

Step 3.

Score the scope, severity and timing of each threat at
the beginning of the project.

Ideally the scoring should be carried out both by the project teamand by certain key informants who have relevant knowledge of the
project’s conservation target (e.g. experts on the species/site being
targeted). See the factsheet Key informant interviews for more
information on this approach. If scoring is only being carried out
by the project team you should attempt to do this as objectively as
possible and justify the scores selected.
Use the scorecard in Table 2 and the scoring criteria in Table 1.
Scores can be assigned retrospectively if they were not assessed at
the beginning of the project.

361

PRISM - TOOLKIT FOR EVALUATING THE OUTCOMES AND IMPACTS OF SMALL/MEDIUM-SIZED CONSERVATION PROJECTS

4:5

INTRODUCTION &KEY CONCEPTS

DESIGNING & IMPLEMENTING
THE EVALUATION

PRISMMODULESMETHOD FACTSHEETS& ANNEXESTABLE 2. THREAT ASSESSMENT SCORECARD.
THREAT

BEFORE THE PROJECTAFTER THE PROJECTESTIMATED SITUATION IF THE
PROJECT HAD NOT HAPPENED

Scope of threat (% ofpopulation or area)
Severity of threat (% decline
over 10 years or 3 generations)

SPECIES & HABITATMANAGEMENT
FACTSHEETTHREAT
REDUCTIONSCORINGSTEPS 4, 5 & 6

Timing/risk of threat

Total

Justification:

Sum the scores to produce an overall “impact score” for eachthreat at the beginning of the project (note that if any of the scoresfor scope, severity or timing for a given threat = 0, then the impact
score for that threat is set to 0, regardless of the scores for eitherof the other two parameters for that threat).

Step 4.

Score the scope, severity and timing of the threat at
the end of the project.

The change (if any) in the scores indicates whether the project had
a positive or negative impact (and its magnitude) or alternatively
that there was no impact that is quantifiable.
362

Step 5.

Estimate a score for what the scope, severity andtiming of each threat would have been if the project
had not taken place (the counterfactual scenario).

Step 6.

Compare and discuss the differences between
the scores to identify whether there has been achange in threat level and whether the change canbe attributed to project actions. Justification notes
should be provided for all scores given, in order tosupport the assessment and to make it as objective as
possible.

PRISM - TOOLKIT FOR EVALUATING THE OUTCOMES AND IMPACTS OF SMALL/MEDIUM-SIZED CONSERVATION PROJECTS

4:5

INTRODUCTION &KEY CONCEPTS

DESIGNING & IMPLEMENTING
THE EVALUATION

EXAMPLE

SPECIES & HABITATMANAGEMENT
FACTSHEETTHREAT
REDUCTIONSCORING

Project: A forest dwelling ungulate was known to bethreatened by hunting, primarily with snares (IUCN threatclass 5.1.1: “Hunting and collecting terrestrial animals(intentional use)”). The project targeted a national park
where the species was estimated to be declining rapidly.
The park contains several communities, and poor relations
between communities and park rangers mean that rangers
were unable to patrol effectively >90% of the park. The
project aimed to reduce the threat by working to build
relationships between rangers and communities to allow
rangers to conduct patrols. This was largely successful,allowing patrols to resume over 75% of the park area.

PRISM
MODULESMETHOD FACTSHEETS& ANNEXES

Subsequently the number of snares found by patrols
reduced from an average of six per patrol to two.
Evaluation: At the beginning of the project the team asked
key informants familiar with the species and the site to rate
the scope, severity and timing of the threat from hunting
at the beginning of the project, and this gave a threat scoreof 9. They were asked to repeat this scoring at the end of
the project and also for a hypothetical scenario where theproject had not taken place.
Informants rated the scope and severity of the threat
as being lower after the project, while timing remained
unchanged; giving a total threat score of 6. This contrasted
with the estimate for the “no-project” scenario where scope,
severity and timing remained the same.

THREATHUNTING AND COLLECTINGTERRESTRIAL ANIMALS(INTENTIONAL USE)

BEFORE THE PROJECTAFTER THE PROJECTESTIMATED SITUATION IF THE
PROJECT HAD NOT HAPPENED

Scope of threat (% ofpopulation or area)

3 - Whole population/area(>90%)

1 - Some of population/area(10-49%)3 - Whole population/area(>90%)

2 - Moderate deterioration(10–30%)3 - Rapid deterioration (>30%)

Severity of threat (% decline
3 - Rapid deterioration (>30%)
over 10 years or 3 generations)
Timing/risk of threat

3 - Happening now

3 - Happening now

3 - Happening now

Total

9

6

9

Justification: Hunting is probably still occurring in some areas so the timing of the threat remains the same. The fact that rangers areable to patrol a much larger area, and the fact that snares are being identified and removed, means that both the scope and severityof the threat from hunting could be scored lower after the project.

363

PRISM - TOOLKIT FOR EVALUATING THE OUTCOMES AND IMPACTS OF SMALL/MEDIUM-SIZED CONSERVATION PROJECTS

4:5

INTRODUCTION &KEY CONCEPTS

SPECIES & HABITATMANAGEMENT
FACTSHEETSCORECARD FOR
EVALUATINGCHANGES IN
SPECIES STATUSSTEPS 1, 2 & 3

364

DESIGNING & IMPLEMENTING
THE EVALUATION

PRISMMODULESMETHOD FACTSHEETS& ANNEXES

Factsheet: Scorecard for evaluating changes in species status

Required steps

SummaryThis factsheet describes a simple method that can be used tocategorise a change in species status (a) that is estimated orinferred to have occurred during the course of a project designed
to conserve the species, and (b) that would have occurred in ahypothetical scenario in which the project did not take place.
Comparing these provides a simple assessment of the project’s
outcomes and impacts and their magnitude. The scoring method
can be applied using information from observed data and/orfrom information obtained from interviews with key informants.
Justification notes should be provided for the categories chosen, inorder to demonstrate objectivity.

Step 1.

Identify the species whose status you want toevaluate. You should only evaluate one species at atime using this method.

Step 2.

Familiarise yourself with the scoring criteria in
Table 1.

Step 3.

Using the criteria outlined in Table 1 and thescorecard shown in Table 2, score the change in theabundance of the species since the project began
and for a hypothetical scenario in which the project
did not take place (and in which all other factors
remained the same). Species distribution extent or
occupancy can be used as proxies for abundance.

Ideally the scoring should be carried out both by the project teamand by key informants who have relevant knowledge of the species
population being assessed. See the factsheet Key informant
interviews for more information on this approach. If scoring is only
being carried out by the project team you should attempt to do thisas objectively as possible and justify the scores selected.

PRISM - TOOLKIT FOR EVALUATING THE OUTCOMES AND IMPACTS OF SMALL/MEDIUM-SIZED CONSERVATION PROJECTS

4:5

INTRODUCTION &KEY CONCEPTS

DESIGNING & IMPLEMENTING
THE EVALUATION

PRISMMODULESMETHOD FACTSHEETS& ANNEXESTABLE 1: SCORING CRITERIA FOR EVALUATING CHANGES IN SPECIES STATUS.

SPECIES & HABITATMANAGEMENT
FACTSHEETSCORECARD FOR
EVALUATINGCHANGES IN
SPECIES STATUS

365

How would the species’ abundance have changed in the absence of the project?

HOW DID THE SPECIES’ ABUNDANCE CHANGE DURING THE COURSE OF THE PROJECT?
LARGE DECREASE(>30%)

MODERATE
DECREASE (10-30%)

STABLE (<10%CHANGE)

MODERATE
INCREASE (10-30%)

LARGE INCREASE(>30%)

Large decrease (>30%)

Project action had
little or no impact
and failed to combat
a large decrease inabundance

Project action had
a minor positive
impact by changing
a large decrease inabundance into amoderate decrease inabundance

Project action had amoderately positive
impact by halting
a large decrease inabundance, resulting
in stable trends inabundance

Project action had asubstantial positive
impact by reversing
a large decrease inabundance, resulting
in a moderateincrease in abundance

Project action had
a very substantial
positive impact
by reversing alarge decrease inabundance, resulting
in a large increase inabundance

Moderate decrease(10-30%)

Project action had
a minor negative
impact by changing amoderate decreasein abundance intoa large decrease inabundance

Project action had
little or no impact
and failed to combat
a large decrease inabundance

Project action had amoderately positive
impact by halting amoderate decrease inabundance, resulting
in a stable trend

Project action had amoderately positive
impact by reversing amoderate decrease inabundance resulting
in a moderateincrease in abundance

Project action had asubstantial positive
impact by reversing amoderate decrease inabundance, resulting
in a large increase inabundance

Stable (<10% change)

Project action had amoderately negative
impact by causinga large decrease inabundance

Project action Had amoderately negative
impact by causing amoderate decline inabundance

Project action had
little or no impact andthe species continued
in a stable trend

Project action had amoderately positive
impact by causing amoderate increase inabundance

Project action had amoderately positive
impact by causinga large increase inabundance

Moderate increase (1030%)

Project action had asubstantial negative
impact by reversing amoderate increase inabundance, resulting
in a large decrease inabundance

Project action Had amoderately negative
impact by reversing amoderate increase inabundance, resulting
in a moderate decline
in abundance

Project action Had amoderately negative
impact by halting amoderate increase inabundance, resulting
in a stable trend

Project action had
little or no impact
and the species
continued to undergoa moderate increasein abundance

Project action had amoderately positive
impact by increasing
the observed rateof change from amoderate to a large
increase in abundance

Large increase (>30%)

Project action had
a very substantial
negative impact
by reversing alarge increase inabundance, resulting
in a large decrease inabundance

Project action had asubstantial negative
impact by reversing
a large increase inabundance, resulting
in a moderate decline
in abundance

Project action Had amoderately negative
impact by halting
a large increase inabundance, resulting
in a stable trend

Project action had amoderately negative
impact by reducing
the observed rate ofchange from a large tomoderate increase inabundance

Project action had
little or no impact andthe species continued
to undergo a large
increase in abundance

PRISM - TOOLKIT FOR EVALUATING THE OUTCOMES AND IMPACTS OF SMALL/MEDIUM-SIZED CONSERVATION PROJECTS

4:5

INTRODUCTION &KEY CONCEPTS

DESIGNING & IMPLEMENTING
THE EVALUATION

PRISMMODULESMETHOD FACTSHEETS& ANNEXESTABLE 2. SPECIES STATUS SCORECARD.
SPECIESHOW DID THE SPECIES’ABUNDANCE CHANGE DURING THE
COURSE OF THE PROJECT?

HOW WOULD THE SPECIES’ABUNDANCE HAVE CHANGED IN
THE ABSENCE OF THE PROJECT?

PROJECT IMPACT

Insert species name
Justification:

SPECIES & HABITATMANAGEMENT
FACTSHEETSCORECARD FOR
EVALUATINGCHANGES IN
SPECIES STATUS

Step 4.

Use the values in Table 1 to score for the impact ofproject actions on species status based on comparing
the observed change with the estimate for ahypothetical scenario in which the project did nottake place.
Dark green cells

STEPS 4 & 5

366

Cases where project actions had
a substantial or very substantial
positive impact.

Light green cells

Cases where project actions had
a minor or moderately positive
impact.

Amber cells

Cases where project actions had
a minor or moderately negative
impact.

Red cells

Cases where project actions had
a substantial or very substantial
negative impact.

Note that you would not normally carry out a project where it is
known in advance that the species targeted by the project will
increase in the absence of the project (lower two rows in Table 1),
but a species may increase for reasons unrelated to the project
once the project has started.

Step 5.

Compare the scores given by the different
experts engaged, and develop conclusions andrecommendations. When asking your chosen experts
to score you should also ask for and record theirjustifications for the answers given.

PRISM - TOOLKIT FOR EVALUATING THE OUTCOMES AND IMPACTS OF SMALL/MEDIUM-SIZED CONSERVATION PROJECTS

4:5

INTRODUCTION &KEY CONCEPTS

DESIGNING & IMPLEMENTING
THE EVALUATION

Factsheet: Scorecard for evaluating changes in habitat status

SPECIES & HABITATMANAGEMENT
FACTSHEETSCORECARD FOR
EVALUATINGCHANGES IN
HABITAT STATUS

SummaryThis method should be used to categorise a change in habitat
status (a) that is estimated or inferred to have occurred during
the course of a project designed to conserve the habitat, and (b)that would have occurred in a hypothetical scenario in which the
project did not take place. Comparing these provides a simple
assessment of the project’s outcomes and impacts and theirmagnitude. The scoring method can be applied using informationfrom observed data and/or expert opinion. Justification notes
should be provided for the categories chosen, in order todemonstrate objectivity.

STEPS 1, 2, 3 & 4
Step 2.

Step 3.

367

METHOD FACTSHEETS& ANNEXES

Ideally the scoring should be carried out both by the project teamand by certain key informants who have relevant knowledge of thehabitat being assessed. See the factsheet Key informant interviews
for more information on this approach. If scoring is only being
carried out by the project team you should attempt to do this asobjectively as possible and justify the scores selected.

Step 4.

Use the values in Table 1 to score the impact ofproject actions on habitat status, based on comparing
the observed change with the estimate for ahypothetical scenario in which the project did nottake place.

Dark green cells

Identify the habitat you want to evaluate. You should
only evaluate one type of habitat at a time using thismethod.

Cases where project actions had
a substantial or very substantial
positive impact.

Light green cells

Cases where project actions
had a minor or moderately
positive impact.

Familiarise yourself with the scoring criteria in
Table 1.

Amber cells

Cases where project actions had
a minor or moderately negative
impact.

Red cells

Cases where project actions had
a substantial or very substantial
negative impact.

Required steps
Step 1.

PRISM
MODULESUsing the criteria outlined in Table 1 and thescorecard shown in table 2, score the change inhabitat quality since the project began and for ahypothetical scenario in which the project did nottake place (and in which all other factors remained
the same).

PRISM - TOOLKIT FOR EVALUATING THE OUTCOMES AND IMPACTS OF SMALL/MEDIUM-SIZED CONSERVATION PROJECTS

4:5

INTRODUCTION &KEY CONCEPTS

DESIGNING & IMPLEMENTING
THE EVALUATION

PRISMMODULESMETHOD FACTSHEETS& ANNEXESTABLE 1: SCORING TABLE FOR EVALUATING CHANGES IN HABITAT STATUS

SPECIES & HABITATMANAGEMENT
FACTSHEETSCORECARD FOR
EVALUATINGCHANGES IN
HABITAT STATUS

368

How would the habitat status of the project site(s) have changed in theabsence of the project?

HOW DID THE HABITAT QUALITY OF THE PROJECT SITE(S) CHANGE DURING THE COURSE OF THE PROJECT?
LARGE
DETERIORATION(>30%)

MODERATE
DETERIORATION (1030%)

STABLE(<10% CHANGE)

MODERATE
IMPROVEMENT(10-30%)LARGE IMPROVEMENT(>30%)

Large
deterioration(>30%)

Project action had
little or no impact andfailed to combat a large
deterioration in habitat
quality

Project action had aminor positive impact
by changing a large
deterioration in habitat
quality into a moderate
deterioration in habitat
quality

Project action had amoderately positive
impact by halting alarge deteriorationin habitat quality,
resulting in a stable
trend in habitat quality

Project action had asubstantial positive
impact by reversing
a large deteriorationin habitat quality,
resulting in a moderate
improvement in habitat
quality

Project action had
a very substantial
positive impact byreversing a large
deterioration in habitat
quality, resulting in alarge improvement inhabitat quality

Moderate
deterioration(10-30%)

Project action had aminor negative impact
by changing a moderate
deterioration in habitat
quality into a large
deterioration in habitat
quality

Project action had
little or no impact andfailed to combat a large
deterioration in habitat
quality

Project action Had amoderately positive
impact by halting amoderate deteriorationin habitat quality,
resulting in a stable
trend in habitat quality

Project action had amoderately positive
impact by reversing amoderate deteriorationin habitat quality,
resulting in a moderate
improvement inhabitat quality

Project action had asubstantial positive
impact by reversing amoderate deteriorationin habitat quality,
resulting in a large
improvement in habitat
quality

Stable (<10%change)Project action had amoderately negative
impact by causing alarge deterioration inhabitat quality

Project action Had amoderately negative
impact by causing amoderate deteriorationin habitat quality

Project action had
little or no impact andstable trends in habitat
quality continued

Project action Had amoderately positive
impact by causing amoderate improvement
in habitat quality

Project action had amoderately positive
impact by causinga large increase inhabitat quality

Moderate
improvement(10-30%)

Project action had asubstantial negative
impact by reversing amoderate improvement
in habitat quality,
resulting in a large
deterioration in habitat
quality

Project action had amoderately negative
impact by reversing amoderate improvement
in habitat quality,
resulting in a moderate
deterioration in habitat
quality

Project action had amoderately negative
impact by halting amoderate improvement
in habitat quality,
resulting in a stable
trend in habitat quality.

Project action had
little or no impact andhabitat continued toundergo a moderate
improvement in quality

Project action Had amoderately positive
impact by increasing
the observed rateof change from amoderate to large
improvement in habitat
qualityLarge
improvement(>30%)

Project action had
a very substantial
negative impact byreversing a large
improvement in habitat
quality, resulting in alarge deterioration inhabitat quality

Project action had asubstantial negative
impact by reversing
a large improvement
in habitat quality,
resulting in a moderate
deterioration in habitat
quality

Project action had amoderately negative
impact by halting alarge improvement
in habitat quality,
resulting in a stable
trend in habitat quality

Project action had amoderately negative
impact by reducing
the observed rate ofchange from a large tomoderate improvement
in habitat quality

Project action had
little or no impact andhabitat continued
to undergo a large
improvement in quality

PRISM - TOOLKIT FOR EVALUATING THE OUTCOMES AND IMPACTS OF SMALL/MEDIUM-SIZED CONSERVATION PROJECTS

4:5

INTRODUCTION &KEY CONCEPTS

DESIGNING & IMPLEMENTING
THE EVALUATION

PRISMMODULESMETHOD FACTSHEETS& ANNEXESTABLE 2. HABITAT STATUS SCORECARD.
HABITATHOW DID THE HABITAT QUALITY
OF THE PROJECT SITE(S) CHANGE
DURING THE COURSE OF THE
PROJECT?

HOW WOULD THE HABITAT STATUS
OF THE PROJECT SITE(S) HAVECHANGED IN THE ABSENCE OF THE
PROJECT?

Insert details of habitat here (site/
habitat type etc.)

SPECIES & HABITATMANAGEMENT
FACTSHEETSCORECARD FOR
EVALUATINGCHANGES IN
HABITAT STATUS

Justification:

Note that you would not normally carry out a project where it is
known in advance that the habitat targeted by the project will
improve in the absence of the project (lower two rows in Table 1),
but a habitat may improve for reasons unrelated to the project
once the project had started.

STEP 5
Step 5.

369

Compare the scores given by the different
experts engaged, and develop conclusions andrecommendations. When asking your chosen experts
to score, you should also ask for and record theirjustifications for the answers given.

PRISM - TOOLKIT FOR EVALUATING THE OUTCOMES AND IMPACTS OF SMALL/MEDIUM-SIZED CONSERVATION PROJECTSPROJECT IMPACT

4:5

INTRODUCTION &KEY CONCEPTS

SPECIES & HABITATMANAGEMENT
FACTSHEETFIELD SURVEYSFOR EVALUATINGCHANGES IN
SPECIES STATUS

DESIGNING & IMPLEMENTING
THE EVALUATION

PRISMMODULES

Factsheet: Field surveys for evaluating changes in species status

Required steps

Summary
To evaluate changes in species status directly, robust andrepeatable field surveys are typically required. Deciding on an
appropriate field survey method will depend on the taxonomic
group being surveyed, for example surveying birds requires a very
different approach to that for monitoring aquatic insects. Notethat it often takes a long time for changes in species status tobecome measurable, so you must be confident that enough time
has passed to allow you to measure such changes.

Step 1.

Identify the species/taxonomic group(s) you want tomeasureThe species you want to measure will typically fall into one of thefollowing categories:•

The species being targeted by the project (e.g. surveying
the number of Snow Leopards in a particular area where
the conservation of this species is a priority).

•

Species important to the species being targeted by the
project (e.g. surveying species preyed upon by Snow
Leopards).

•

Species whose presence/abundance acts as anindicator of habitat quality (e.g. sampling freshwater
invertebrates to indicate water quality).

STEP 1370

METHOD FACTSHEETS& ANNEXESPRISM - TOOLKIT FOR EVALUATING THE OUTCOMES AND IMPACTS OF SMALL/MEDIUM-SIZED CONSERVATION PROJECTS

4:5

INTRODUCTION &KEY CONCEPTS

Step 2.

DESIGNING & IMPLEMENTING
THE EVALUATION

Define the species attributes you want to measure

Examples of species attributes that could be measured in a fieldsurvey include20 :Occurrence – Which species occur (e.g. presence/absence).

SPECIES & HABITATMANAGEMENT
FACTSHEETFIELD SURVEYSFOR EVALUATINGCHANGES IN
SPECIES STATUSSTEPS 2, 3 & 4Distribution – Where the species occur (e.g. area, physicalattributes).
Abundance – How many individuals/pairs of a species occur in agiven area (e.g. frequency, density).

PRISM
MODULES

Step 3.

Define the sample area.

If you are not able to measure change across the whole populationof interest, then you should select a sampling approach (see below)that will give you a representative measure.
If you are using a matching evaluation design which compareschanges in the species population being targeted by the projectwith a control population, then you need to ensure that the
populations are similar enough to allow you to draw usefulcomparisons, or that you have sufficient samples from across a
range of sites that are likely to be representative.

Population structure – Composition of the species population
being targeted (e.g. age, sex ratio, fragmentation or isolation).
Step 4.
Population dynamics – Recruitment, mortality, emigration,
immigration, seasonal factors.
Always think back to the main aims of the project and the question
you are trying to answer. For example if you want to know whether
a species is recolonising an area it had disappeared from, you may
wish to focus on its distribution. If the project involves eradicating
an invasive species that preys on the juveniles of a species ofconservation concern, you may wish to know whether successful
breeding of the target species is occurring (e.g. evidence ofhatchling success in albatross chicks).

Select data collection methods.

The methods available to you will depend on the species and thespecies attributes you are attempting to measure.
While it is not possible to cover all the available methods formonitoring and evaluating species status in this Toolkit, the Annex
Field methods for species monitoring describes some of the mostcommonly used methods and sampling approaches for different
taxonomic groups.
When selecting methods remember to consider potential sourcesof bias and how these could be reduced or minimised.

20
Further details of these characteristics can be found on pages 10-13 of the Cambridge
Handbook of Biodiversity Methods: https://sunsetridgemsbiology.wikispaces.com/file/view/
Biodiversity+Handbook.pdf .

371

METHOD FACTSHEETS& ANNEXESPRISM - TOOLKIT FOR EVALUATING THE OUTCOMES AND IMPACTS OF SMALL/MEDIUM-SIZED CONSERVATION PROJECTS

4:5

INTRODUCTION &KEY CONCEPTS

DESIGNING & IMPLEMENTING
THE EVALUATION

PRISMMODULESMETHOD FACTSHEETS& ANNEXESTABLE 1. POTENTIAL SOURCES OF BIAS DURING FIELD SURVEYS, AND POTENTIAL STRATEGIES FORMINIMISING/REDUCING THESE
POTENTIAL SOURCE OF BIAS

STRATEGIES FOR MINIMISING/REDUCING THE BIAS

Observers have different levelsof expertise

Use standard methodologies across years, surveys and sites.
Ensure all observers are trained to a certain standard.

SPECIES & HABITATMANAGEMENT
FACTSHEETFIELD SURVEYSFOR EVALUATINGCHANGES IN
SPECIES STATUS
STEP 5

Calibrate observers against each other before and during data collection
Record observer identity, allowing subsequent correction of bias analytically.
Variation in observer effort
between surveys

Use standard methodologies across years, surveys and sites.
Record observer effort during sampling periods, allowing analytical correlation of bias.

Certain species/habitats may
be more difficult to survey thanothersConsider approaches that account for variation in detectability between species.

Local conditions may affect datacollection (weather, politicalevents, etc.)

Record as much detail on local conditions as possible during surveys, allowing analyticalcorrection.

Lack of standardisation betweensites/observers

Agree and record definitions (e.g. sample size, type population unit) beforehand.

Stratify samples across habitats to increase coverage in difficult habitats.

Ensure all observers are trained to a certain standard.

Step 5.

Select the sampling approach.

As well as the method by which individuals are detected andcounted, it is also important to consider the sampling approach.
Although it may be possible to undertake a complete survey inwhich all individuals in the area of interest are counted, changes

372

in population size or range extent will typically be estimated fromsample surveys in which only a proportion of the population isrecorded. Table 2 outlines several different approaches to this.
The Annex Sampling approaches for evaluating changes in species
status provides further details on each of these and sources ofmore detailed methods for them.

PRISM - TOOLKIT FOR EVALUATING THE OUTCOMES AND IMPACTS OF SMALL/MEDIUM-SIZED CONSERVATION PROJECTS

4:5

INTRODUCTION &KEY CONCEPTS

DESIGNING & IMPLEMENTING
THE EVALUATION

PRISMMODULESMETHOD FACTSHEETS& ANNEXESTABLE 2. SUMMARY OF SAMPLING APPROACHES FOR ASSESSING THE STATUS OF SPECIES.

SPECIES & HABITATMANAGEMENT
FACTSHEETFIELD SURVEYSFOR EVALUATINGCHANGES IN
SPECIES STATUS

SAMPLING APPROACHNOTES AND CAVEATS

Complete counts

Rarely feasible unless individuals are easily detected and are concentrated within a relatively
small area. May use field counts or photos. Consider the most appropriate life stage.

Area-based sampling

Sample defined area and extrapolate to unsurveyed areas. Methods include point counts, linetransects and counts from fixed points. May need to correct for imperfect detectability. Samples
should be randomly distributed or appropriately stratified.

Timed sampling

Record numbers per unit time (e.g counts of migrating individuals, camera traps, passive
acoustic recording devices). Most appropriate for species that have low detectability, are widelydispersed, or require methods that are difficult to standardise. More appropriate for estimating
relative abundance than population size.

Live trapping

Mist nets (for birds/bats), mammal traps, fish nets, moth traps, invertebrate pitfall traps etc.
to estimate relative abundance in a similar manner to timed sampling. Rings/bands (birds),
microchips (vertebrates), colour marks (invertebrates) or unique appearance can be used to
identify individuals for demographic monitoring, and to estimate population size from the ratioof the number of individuals caught and marked to those recaptured in a subsequent sample.

Removal trapping21

A wide range of trapping approaches can be used if individuals must be killed for identification
purposes (e.g. pitfall, malaise or water traps for insects, netting for fish, snap traps for small
mammals), to estimate relative abundance in a similar manner to live trapping. Data can also be
used to estimate initial abundance from the decay in numbers of individuals trapped throughtime, although this depletes the target population and needs to standardise survey effort overtime.

Sampling eDNA

A developing technique for detecting occurrence or (with considerable caveats) abundance ofspecies, from DNA extracted from environmental or biological samples. May be particularly
useful for taxa that are difficult to survey using other approaches, such as amphibians or fish in
freshwater environments, cryptic mammals or micorrhizal fungi in soil. May be more relevant
for small and medium-sized conservation projects in future as technology develops.

These approaches may raise ethical issues. See Factsheet: Developing ethical protocols foran evaluation for more details
21

373

PRISM - TOOLKIT FOR EVALUATING THE OUTCOMES AND IMPACTS OF SMALL/MEDIUM-SIZED CONSERVATION PROJECTS

4:5

INTRODUCTION &KEY CONCEPTS

Step 6.

DESIGNING & IMPLEMENTING
THE EVALUATION

Plan for data management and analysis.

As a final step before you begin collecting data you should consider
how you will manage and analyse the data you collect.
To prepare for data analysis and presentation, think about thefollowing questions:

SPECIES & HABITATMANAGEMENT
FACTSHEETFIELD SURVEYSFOR EVALUATINGCHANGES IN
SPECIES STATUS•

Do you need datasheets and databases, and how should
they be designed?

•

Who will carry out the analysis, and when?

•

How will the data be analysed?

•

What other data are required prior to analysis?

•

What statistical tests are appropriate for analysing the
data?

•

What statistical packages are available for the analysisof data?

•

When should data analysis and presentation take place?

STEP 6374

PRISM
MODULESMETHOD FACTSHEETS& ANNEXES

Refer to the PRISM section Analyse data (in Step 3) for more
information.
Refer to the Species/Habitat Annex: Analytical issues whenestimating abundance, distribution, survival and reproduction
for specific guidance on analysis in relation to estimating species
abundance, distribution, survival and reproduction.

PRISM - TOOLKIT FOR EVALUATING THE OUTCOMES AND IMPACTS OF SMALL/MEDIUM-SIZED CONSERVATION PROJECTS

4:5

INTRODUCTION &KEY CONCEPTS

DESIGNING & IMPLEMENTING
THE EVALUATION

Factsheet: Remote assessment methods for evaluating changes
in habitat status

SPECIES & HABITATMANAGEMENT
FACTSHEET

Summary
Remote sensing data can be used to assess changes in land coverextent and fragmentation, and in some cases, condition (e.g. %canopy cover, vegetation structure). These measures can be used
as indicators of habitat status. (Remote sensing is also useful for
detecting some threats, so may be relevant to evaluation of threatrelated issues)

Required Steps

REMOTEASSESSMENTMETHODS FOREVALUATINGCHANGES IN
HABITAT STATUSSTEPS 1 & 2

375

Step 1.

Identify the area(s) you want to evaluate.

PRISM
MODULES

Step 2.

METHOD FACTSHEETS& ANNEXES

Identify the habitat characteristics you want tomeasure.

Examples of habitat characteristics that could be measured using
remote sensing include:
Extent – Area, physical attributes.
Composition – Extent, cover.
Structure – Vertical (e.g. tree layer topography).
Dynamics – e.g. regeneration, succession, cyclic change, patch
dynamics.
Threats/stresses – e.g. evidence of grazing, burning, erosion.

If you are not able to measure change across the whole area ofinterest then you need to adopt a sampling approach (see below)that will give you a representative measure of change across thearea of interest.
If you are using a matching evaluation design which compareschanges at the project site with a control site, then ensure that thesites are similar enough to allow you to draw useful comparisons
between them.

PRISM - TOOLKIT FOR EVALUATING THE OUTCOMES AND IMPACTS OF SMALL/MEDIUM-SIZED CONSERVATION PROJECTS

4:5

INTRODUCTION &KEY CONCEPTS

Step 3.

DESIGNING & IMPLEMENTING
THE EVALUATION

PRISMMODULESMETHOD FACTSHEETS& ANNEXES

Select data source.

There are three main sources of remote sensing assessment data:

STEP 3. SELECT DATA SOURCE.
DATA SOURCENOTES AND CAVEATSSatellite imagery

This is the most readily available form of remote sensing data. Many USGS LANDSAT images
are freely available to download and range in date back to the 1970s. ESA’s Copernicus
programme provides data every 3-4 days and started in 2015: it includes radar data whichmeans for example that forest under cloud can be monitored. Other potential sources include
images from France’s SPOT satellites, and commercial satellites IKONOS, Quickbird andOrbview; although these may require purchase. NASA has produced a detailed introduction
to the use of satellite remote sensing data, available at http://arset.gsfc.nasa.gov/ . The sectionof this at http://arset.gsfc.nasa.gov/land explains how to obtain images, process them in an
appropriate way before classification and undertake some basic classifications. Analysing
satellite images can be time-consuming and requires considerable GIS expertise, so it may be
worth considering academic collaboration with a university or research organisation to achieve
this efficiently.

Aerial photography

Small and medium-sized conservation projects are unlikely to have the funds to carry out anaerial survey. There may however be sources of existing data that can be used, for example fromrelevant government agencies or Google Earth.

Unmanned aerial surveys using“drones”

Unmanned aerial vehicles are increasingly used in monitoring for conservation. These tools are
becoming cheaper and may represent an efficient way to survey the impact of small/mediumsized conservation projects, especially if the cost can be shared across a number of projects. See
https://conservationdrones.org/ for further information.

SPECIES & HABITATMANAGEMENT
FACTSHEETREMOTEASSESSMENTMETHODS FOREVALUATINGCHANGES IN
HABITAT STATUS
STEP 3376

PRISM - TOOLKIT FOR EVALUATING THE OUTCOMES AND IMPACTS OF SMALL/MEDIUM-SIZED CONSERVATION PROJECTS

4:5

INTRODUCTION &KEY CONCEPTS

Step 4.

SPECIES & HABITATMANAGEMENT
FACTSHEETREMOTEASSESSMENTMETHODS FOREVALUATINGCHANGES IN
HABITAT STATUSSTEPS 4 & 5

DESIGNING & IMPLEMENTING
THE EVALUATION

Collect/download data.

Remote sensing assessment data can be used to measure change
in the extent of a particular land cover type which is representativeof a habitat (e.g. forest), or to compare rates of land cover loss/gain
before and after project interventions.
For satellite imagery and aerial photography this typically involves
downloading images from two or more points in time over theperiod for which you wish to evaluate change, for example apoint before the project, one during the project and one after the
project. For an unmanned aerial survey this will involve carryingout a drone survey of the required area.
In addition to measuring change before and after the project, it
is also important to consider what the land-cover extent or rateof change would have been in the absence of project actions(the counterfactual). To estimate this typically requires matching
the project location to suitable comparison sites selected to
be as similar as possible to the target site (similarity is based on
all available data and is tested by comparing average values ofcovariates). Once selected, the comparison sites will be comparedwith the project target sites to give a measurement of the project’s
outcomes and impacts.

PRISM
MODULES

Step 5.

Identify the parameters you want to measure, andanalyse data.

All three of the data sources listed in the previous step require
some post-collection processing to assess changes in habitat.
At a minimum they require visual assessment to determine theextent of each land-cover type and changes in them. A range ofincreasingly complicated methods can be used on these data toclassify them into land-cover types, to quantify land-cover change,
and to assess vegetation structure. Some knowledge of remotesensing is required for many of these analyses, but most arestraightforward.
There are open source software packages in which images can be
manipulated and analysed. These include QGIS http://www.qgis.
org/en/site/ (which has a wide range of free tutorials available
and an active online community to help new users). In addition, R(https://cran.r-project.org/) has spatial analysis functionality, andmany of the functions are described, with code, by Wegman et al.
(2016)22 . The European Commission’s Joint Research Centre (ECJRC) has developed a simple, semi automated land-cover and landcover change mapping tool which is open source and is targeted
towards protected areas and important sites for biodiversity(such as Important Bird and Biodiversity Areas and other Key
Biodiversity Areas (http://onlinelibrary.wiley.com/doi/10.1002/
rse2.14/full). A mostly automated system for classification offorest cover and change is available through http://claslite.
carnegiescience.edu/en/index.html .

Palumbo I, Rose, RA, Headley RMK, Nackoney J, Vodacek A and Wegmann M (2016).
Building capacity in remote sensing for conservation: present and future challenges. Remote
Sens Ecol Conserv. doi:10.1002/rse2.31 .
22

377

METHOD FACTSHEETS& ANNEXESPRISM - TOOLKIT FOR EVALUATING THE OUTCOMES AND IMPACTS OF SMALL/MEDIUM-SIZED CONSERVATION PROJECTS

4:5

SPECIES & HABITATMANAGEMENT
FACTSHEETREMOTEASSESSMENTMETHODS FOREVALUATINGCHANGES IN
HABITAT STATUS

INTRODUCTION &KEY CONCEPTS

DESIGNING & IMPLEMENTING
THE EVALUATION

There is an increasing number of post-processing products derived
from remote sensing which could also be useful to small- andmedium-sized projects. For example, the Global Forest Watch
website: http://www.globalforestwatch.org/ presents data ontree cover at 30 m resolution globally at annual intervals from2000 onwards. The dataset is updated annually. Users can carryout basic online analyses of tree cover and change in this over time
at a project site (either by uploading their own site boundary or,for projects implementing actions within a protected area, using
the relevant boundary which is available within the tool); or they
can download data to carry out more detailed analyses. Other
products are in development, including a water bodies dataset
which will be presented online, and an urban extent layer.
Fragmentation of habitat can be inferred from fragmentation
indices derived from categorical land-cover maps. Multiple
measures of fragmentation (and its inverse, connectivity), exist
and can be implemented in QGIS, R, or Arcmap.
For QGIS, http://conefor.org can be used as a plugin. This software
can undertake the majority of calculations needed.

PRISM
MODULES

Comparing results between sites
Simple t-tests on difference in means and cross-sectional
regression are two of the most common statistical methods forcomparing results between sites.
Jones and Lewis (2015) describe in more detail the steps to take inselecting a comparison site using remote sensing data, and how tobuild this into an evaluation design - see http://journals.plos.org/
plosone/article/asset?id=10.1371%2Fjournal.pone.0141380.PDF .

Further reading
Vihervarra et al. (2015) – combining remote and field assessment
data.: https://link.springer.com/article/10.1007/s10980-014-01375
Gottingen Forest Inventory Wiki (an excellent source of
information regarding forest plot design and monitoring).:
http://wiki.awf.forst.uni-goettingen.de/wiki/index.php/AWFWiki:Community_Portal

Aerial photographs can be treated in a similar way to satellite
images. In addition to processing in dedicated software, bothtypes of data can be interpreted visually. Polygons can be digitisedaround areas of land cover of interest by the user, rather thanhaving to classify images. This approach is valid if the focal land
cover type is distinct, or can be easily recognised by the human eye
and is comparable between images. The easiest way to do this isusing on-screen digitations, which can be carried out in QGIS.

378

METHOD FACTSHEETS& ANNEXESPRISM - TOOLKIT FOR EVALUATING THE OUTCOMES AND IMPACTS OF SMALL/MEDIUM-SIZED CONSERVATION PROJECTS

4:5

INTRODUCTION &KEY CONCEPTS

DESIGNING & IMPLEMENTING
THE EVALUATION

Factsheet: Field assessment methods for evaluating changes inhabitat status

SPECIES & HABITATMANAGEMENT
FACTSHEETFIELD ASSESSMENTMETHODS FOREVALUATINGCHANGES IN
HABITAT STATUSSTEPS 1 & 2

Further details of these characteristics canbe found on pages 10-13 of the Cambridge
Handbook of Biodiversity Methods - https://sunsetridgemsbiology.wikispaces.com/file/
view/Biodiversity+Handbook.pdf .
23

379

Summary
Field assessment methods can be used to confirm observations
made through remote sensing or to evaluate change in their own
right. They can generate a highly effective measure of project
impact, but note that there is often a trade-off between reliabilityof results and the amount of time and resources required.

Required Steps
Step 1.

Identify the habitat characteristics that you want tomeasure.

PRISM
MODULESMETHOD FACTSHEETS& ANNEXES

the target habitat should look like when in good condition, and its
status when the project began.
Examples of habitat characteristics:
Extent – Area, physical attributes.
Composition – e.g. communities, richness or diversity,
indicator species, cover, biomass.
Structure – Horizontal (e.g. plant community mosaics,
patches of short/tall vegetation), vertical (ground, shrub andtree layer topography).
Dynamics – e.g. regeneration, succession, reproduction,
cyclical change, patch dynamics.
Threats/stresses – e.g. evidence of grazing, burning, erosion.

There are often multiple habitat characteristics that could be
looked at, but it is typically most useful to focus on those that are
being targeted by the project.
First, think carefully about the question you are trying to answer.
For example if you want to know whether restoring habitat byplanting trees is working, you may wish to focus your evaluation
on whether the trees are growing as expected. If the project has
involved dealing with an invasive species, you may wish to knowwhether the habitat is showing a decrease in characteristics
relating to the impact of the species (e.g. evidence of browsing by
introduced goats).
You should also think carefully about what the defining
characteristics/attributes of the habitat are, try to determine what

Taking these factors into account, identify the habitat
characteristic(s) that you want to measure.

Step 2.

Define the sample area.

If you are not able to measure change across the whole area ofinterest then you should select a sampling approach (see below)that will give you a representative measure of change across thearea or habitat of interest.
If you are using a matching evaluation design which compareschanges at the project site with a control site, then ensure that thesites are similar enough to allow you to draw useful comparisons
between them.

PRISM - TOOLKIT FOR EVALUATING THE OUTCOMES AND IMPACTS OF SMALL/MEDIUM-SIZED CONSERVATION PROJECTS

4:5

INTRODUCTION &KEY CONCEPTS

Step 3.

DESIGNING & IMPLEMENTING
THE EVALUATION

Select data collection methods.

The methods you choose will depend on the habitat and thespecific characteristics you are attempting to measure.

SPECIES & HABITATMANAGEMENT
FACTSHEETFIELD ASSESSMENTMETHODS FOREVALUATINGCHANGES IN
HABITAT STATUSSTEPS 3, 4 & 5

While it is not possible to cover all the possible methods formonitoring and evaluating habitat status in this Toolkit, the Annex:
Field methods for habitat monitoring describes some of the mostcommonly used methods and sampling approaches for different
habitats and habitat characteristics.

Step 4.

Select a sampling approach.

In some cases it may be possible to measure change across anentire habitat (for example in an isolated stand of woodland).
However in most cases the project will require an appropriate
sampling strategy, where measurements are taken for a subset ofthe entire area potentially affected by project actions. The Annex
Sampling approaches for habitat monitoring outlines some of themost commonly used sampling strategies for habitat monitoring.

PRISM
MODULES

Step 5.

METHOD FACTSHEETS& ANNEXES

Plan for data management and analysis.

As a final step before you begin collecting data, you should
consider how you will manage and analyse the data you collect.
To prepare for data analysis and presentation, think about thefollowing questions:•
Do you need datasheets and databases?
•

Who will carry out the analysis, and when?

•

How will the data be analysed?

•

What statistical tests are appropriate for analysing the
data?

•

Is transformation of the data necessary before
statistical analysis can take place?

•

What statistical packages are available for the analysisof data?

•

When should data analysis and presentation take place?

Refer to the PRISM section Analyse data (in Step 3) for more
information

380

PRISM - TOOLKIT FOR EVALUATING THE OUTCOMES AND IMPACTS OF SMALL/MEDIUM-SIZED CONSERVATION PROJECTS

4:5

INTRODUCTION &KEY CONCEPTS

DESIGNING & IMPLEMENTING
THE EVALUATION

PRISMMODULES

EXAMPLE

SPECIES & HABITATMANAGEMENT
FACTSHEETFIELD ASSESSMENTMETHODS FOREVALUATINGCHANGES IN
HABITAT STATUS

381

Example: A matched sites, before-and-after trial in
1995-2001 in a Mediterranean Black Pine (Pinus nigra)
forest in the Pyrenees, Spain (Casasús et al. 2007)
found that grazing exclusion increased the biomass of
herbaceous plants and shrubs. Six years after treatment,
herbaceous plant and shrub biomasses (kg dry matter/ha) had increased in fenced areas (herbaceous plant: 501
to 1,730; shrub: 1,902 to 5,073) but not in grazed areas(herbaceous plant: 417 to 679; shrub: 1,120 to 1,207). At
the beginning of the study herbaceous plant and shrub
biomasses were similar in the grazed and fenced areas,
while six years later, both parameters were higher infenced areas.
Casasús I, Bernués A, Sanz A, Villalba D, Riedel J and
Revilla R (2007). Vegetation dynamics in Mediterranean
forest pastures as affected by beef cattle grazing.
Agriculture, Ecosystems & Environment 121: 365-370.

PRISM - TOOLKIT FOR EVALUATING THE OUTCOMES AND IMPACTS OF SMALL/MEDIUM-SIZED CONSERVATION PROJECTSMETHOD FACTSHEETS& ANNEXES

4:5

INTRODUCTION &KEY CONCEPTS

DESIGNING & IMPLEMENTING
THE EVALUATION

PRISMMODULES

Annex: Field methods for species monitoring

SPECIES & HABITATMANAGEMENT
ANNEX
FIELD METHODSFOR SPECIESMONITORING

382

This annex gives a very brief summary of field methods formonitoring different taxa that are relevant for small and mediumsized conservation projects. To indicate the level of expertise andequipment required, each method is categorised as: 1 (method
fairly simple to use i.e. does not require formal training or specialist
equipment), 2 (method requires some training and/or specialist
equipment), or 3 (method requires highly-specialised training andequipment).
More information on the methods in this annex can be foundin the Cambridge handbook of ecological survey methods
https://sunsetridgemsbiology.wikispaces.com/file/view/
Biodiversity+Handbook.pdf

PRISM - TOOLKIT FOR EVALUATING THE OUTCOMES AND IMPACTS OF SMALL/MEDIUM-SIZED CONSERVATION PROJECTSMETHOD FACTSHEETS& ANNEXES

4:5

SPECIES & HABITATMANAGEMENT
ANNEX
FIELD METHODSFOR SPECIESMONITORING

383

INTRODUCTION &KEY CONCEPTS

DESIGNING & IMPLEMENTING
THE EVALUATION

PRISMMODULESTAXACOMPLETE SURVEY

AREA-BASED SAMPLING

TIMED SAMPLING

Fungi

Mapping or counting offruiting bodies within study
area during peak fruiting
period. 1

Occurrence or countsof fruiting bodies within
quadrats or strip transects.
Distance-sampling alongtransects allows greater
coverage, useful for rarer
species. Fixed-point
photos may help assess
abundance. 1

Searches for fruiting bodies
within a defined period
may be useful for rarer
species. 1

Lichens

Mapping or counting
individuals possible where
species occupies small
area. Fixed-point photos
may be used to monitorchanges in % cover. 1

Counts or occurrence
within cells of quadrat.
Line-transects with
distance sampling allows
greater coverage andestimates density / true
abundance. Fixed-point
photos may be used tomonitor changes in %cover. 1

Searches for individuals
within a defined period
may be useful for rarer
species. 1

Bryophytes

Mapping or counting
individuals possible where
species occupies small
area. Fixed point photos
can be used to assesschanges in % cover or thenumber of colonies. 1

Visual estimates of coveror abundance possible
within defined areas,
quadrats or strip transects.
Fixed point photos can beused to assess changes in% cover or the number ofcolonies. 1

Aquatic plants

Mapping extent ofparticular species may bepossible, for example using
fixed-point photos. 1

Visual estimates of cover orabundance possible within
defined areas, quadrats
or strip transects. Thiscan be visual assessment,
fixed-point photos orfrom substrate samples ofknown area. 1

Timed water samples
can provide estimates ofphytoplankton abundance.
2

PRISM - TOOLKIT FOR EVALUATING THE OUTCOMES AND IMPACTS OF SMALL/MEDIUM-SIZED CONSERVATION PROJECTSLIVE TRAPPINGMETHOD FACTSHEETS& ANNEXESREMOVAL TRAPPINGEDNA
Important
new approachfor recording
occurrence
without
requiring
fruiting. 1

Colonisation of artificial
substrates left in thewater used to identify
occurrence. 2

Samples ofwater canbe used to
identify
occurrence ofspecies. 3

4:5

SPECIES & HABITATMANAGEMENT
ANNEX
FIELD METHODSFOR SPECIESMONITORINGINTRODUCTION &KEY CONCEPTS

DESIGNING & IMPLEMENTING
THE EVALUATION

PRISMMODULESTAXACOMPLETE SURVEY

AREA-BASED SAMPLING

TIMED SAMPLINGLIVE TRAPPING

Vascularplants

Mapping or counting
individuals only possible
for small populations.
Fixed point photos monitorchanges in cover of thegrowth and survival ofindividual large plants. 1

Selected colonies could
be monitored. Quadrats,
strip-transects widely used
to estimate abundance andcover. Line transects canbe used to count individual
plants. Fixed point photos
monitor changes in coverof the growth and survivalof individual large plants. 1
Quadrats with pin frames
provide more robust
information on cover andstructure. 2

Searches for individuals
within a defined period
may be useful for rarer
species. 1

Marking individual plants
can be used to monitor
recruitment and mortality.
1

Trees

Mapping or counting
entire populations may
only be feasible for small
populations. Fixed point
photos monitor changes
in cover of the growth andsurvival of individual large
plants. 1

Populations can bemonitored using quadrats,
strip and line transects
which are widely used
to estimate localised
abundance. Aerial or
fixed-point photography
can be used to countemergent canopy species(or those readily identified
when flowering/fruiting)and through time, tomonitor the survival andrecruitment of individual
trees within a study
area. Accompanying thiswith measurements ofindividual trees (such as thediameter at breast height),
can provide information
about population age
structure. 1

Searches for individuals
within a defined period
may be useful for rarer
species. 1

Trees can easily bemeasured and marked,
allowing age-structure,survival rates andrecruitment to bemonitored. 1

Remote techniquessuch as LiDAR and aerial
photography may allow
all individuals withinlarge areas to be counted,
although it may be difficult
to separate species in the
majority of circumstances.
23

Techniques using drones tosurvey canopy trees are indevelopment. 3

384

PRISM - TOOLKIT FOR EVALUATING THE OUTCOMES AND IMPACTS OF SMALL/MEDIUM-SIZED CONSERVATION PROJECTSMETHOD FACTSHEETS& ANNEXESREMOVAL TRAPPING

Repeated monitoring ofareas of ground of known
area can be useful forseed predation/dispersal
studies. 1

EDNA

4:5

INTRODUCTION &KEY CONCEPTSTAXACOMPLETE SURVEY

AREA-BASED SAMPLING

CoralsAerial photography orsatellite imagery can beused to map large-scale
reef geomorphology. 2

Mapping techniques canbe used to map cover ofdifferent species alongtransects (e.g. manta tow)
or in quadrats (althoughphysical quadrats can bechallenging given verticalcoral structure), whilst bothvideo and still photography
have been used to map
coral along transects orin quadrats respectively.
Alternatively, the line-pointintercept method allows
divers to record species
occurrence at intervals
along line-transects.
Various scoring systems
have been developed toscore condition alongtransects. 2

SPECIES & HABITATMANAGEMENT
ANNEX
FIELD METHODSFOR SPECIESMONITORING

385

DESIGNING & IMPLEMENTING
THE EVALUATION

PRISMMODULESTIMED SAMPLING

OdonataCounts of exuviae
provide index of adultabundance that can belinked to the quality ofa site for supporting aviable population. Countsof adults within strip
transects surveyed in goodconditions, particularly
along the waters’ edge, canbe used to monitor trends,
and may work better for
rare species, but may
not separate source/sink
populations. 1

Timed aquatic sampling(sweep netting or kicksampling) for larvae,
and timed counts for
adults surveyed in goodconditions, where strip
transects not possible. 1

Butterflies

Quadrats or strip
transects give egg orlarval density. 5m wide
strip transect surveyed
in good conditions usedas standard Butterfly
Monitoring Scheme
method in UK. 1

Timed counts for eggs
/ larvae can identify
presence / give index ofabundance. 1

PRISM - TOOLKIT FOR EVALUATING THE OUTCOMES AND IMPACTS OF SMALL/MEDIUM-SIZED CONSERVATION PROJECTSLIVE TRAPPING

Live-trapped adults, forexample by searching for
adults in suitable habitats
with a hand-net, can bemarked, and survival anddispersal monitored fromrecaptures. 2

METHOD FACTSHEETS& ANNEXESREMOVAL TRAPPING

EDNA

4:5

INTRODUCTION &KEY CONCEPTSTAXACOMPLETE SURVEY

DESIGNING & IMPLEMENTING
THE EVALUATION

AREA-BASED SAMPLING

PRISM
MODULESTIMED SAMPLING

Moths

Quadrats or strip
transects give egg orlarval density. 5m wide
strip transect surveyed
in good conditions usedas standard Butterfly
Monitoring Scheme
method in UK. 1

Other
terrestrial
invertebrates

Quadrats may be used
for searches for specialist
species where habitats
can be identified. Sometrapping devices are areabased, such as emergencetraps. Strip transects
may be used to monitor
pollinating insects on
flowers. Sweep nets canbe used to infer relative
abundance within aparticular area. 1

Provides index ofabundance for may species
groups with complex or
non-standard protocols. 1

Vegetation samples may
be taken and the numberof individuals per unitof vegetation assessed.
Counts of adults of someemergent insects within
strip transects surveyed
in good conditions,particularly along thewaters’ edge, can provide
estimates of abundance e.g.
for mayflies and stoneflies.
1

Timed aquatic sampling(sweep netting or kicksampling) can provide
estimates of relative
abundance. 1

Other aquatic
invertebrates

386

REMOVAL TRAPPINGEDNALight trapping provides
quantitative information
on species abundance,and occurrence for rare
species. Abundances
between species notnecessarily comparable
depending on attraction tolight. Pheremone traps canbe used for specific species.
Scented lures (sugaring andwine ropes) can also attract
species. Live-trapped
adults of larger species canbe marked, and survival
and dispersal monitoredfrom recaptures. 2

SPECIES & HABITATMANAGEMENT
ANNEX
FIELD METHODSFOR SPECIESMONITORING

LIVE TRAPPINGMETHOD FACTSHEETS& ANNEXES

Passive sound recording
may detect insects withauditory displays, such as
crickets and cicadas. 2

PRISM - TOOLKIT FOR EVALUATING THE OUTCOMES AND IMPACTS OF SMALL/MEDIUM-SIZED CONSERVATION PROJECTS

Artificial refugia can beused to attract individuals
to count them e.g. forslugs and snails. Where
appropriate, individuals
may be marked allowing
mark-recapture studies. 1

A wide range of lethal
trapping approaches exist
for different insect groups
e.g. pitfall traps, water
traps, malaise traps etc. andmay be used to estimate
abundance as the numberof individuals caught perperiod of sampling. 12

Artificial refugia can beused to attract individuals
to count them, and forsome species, markrecapture studies may bepossible. Light traps may
attract certain emergent
insect groups, e.g. caddis
flies. 2

Colonisation of artificial
substrates left in thewater used to identify
occurrence. 2

Samples ofwater canbe used to
identify
occurrence ofspecies. 3

4:5

INTRODUCTION &KEY CONCEPTS

TAXAMETHOD FACTSHEETS& ANNEXES

TIMED SAMPLINGLIVE TRAPPINGREMOVAL TRAPPING

Marineinvertebrates

Line transects or quadrats
can be used to countthe abundance of large
and detectable marine
invertebrates, using
distance sampling toestimate density. 2

Timed sampling canprovide estimates ofrelative abundanceor occurrence for lessdetectable species. 2

Artificial refugia can beused to attract individuals
to count them, and forsome species, markrecapture studies may bepossible. 2

Colonisation of artificial
substrates left in thewater used to identify
occurrence. 2

Fish in slowflowing or stillwatersUnderwater counts can bemade along transects orusing point counts, allowing
for distance sampling tocorrect for detectability. 2

Counts from banks inshallow waters may
provide a crude index
of abundance, underestimated for most species.
1

Wide range of fish-trapping
methods available formark-recapture studies orto index abundance. 12Gill, seine and trawl
netting, and angling, canbe used to provide catchreturns (fish caught perunit effort) as an index
of population size. This
may vary with season,conditions and migratory
status. Effort may bedifficult to record if fromcommercial fishing /angling data. 23

Techniques
to identify
species
occurrence
from water
samplesare beingdeveloped. 3

Wide range of fish-trapping
methods available formark-recapture studies,(dye or tagging) or toindex abundance. Traps on
weirs or fish ladders catch
migrating individuals. 2

Angling can be used toprovide catch returns (fishcaught per unit effort) as
an index of population size.
This may vary with season,conditions and migratory
status. Effort may bedifficult to record. 2

Techniques
to identify
species
occurrence
from water
samplesare beingdeveloped. 3

Hydroacoustic sampling(sonar) estimatesabundance of shoaling
fish. Electrofishing canbe used to estimate total
population size using
multiple catches between
stop nets across a river andanalysis using the removal
method, or to provide anindex of abundance basedon a single catch. Note itis highly dangerous andrequires training. 3

FIELD METHODSFOR SPECIESMONITORING

Fish inrunning
watersUnderwater counts can bemade along transects orusing point counts, allowing
for distance sampling tocorrect for detectability. 2
Hydroacoustic sampling(sonar) estimatesabundance of shoaling
fish. 3387

PRISM
MODULESAREA-BASED SAMPLINGSPECIES & HABITATMANAGEMENT
ANNEX

COMPLETE SURVEY

DESIGNING & IMPLEMENTING
THE EVALUATION

Timed counts underwater
will provide better
estimates of relative
abundance. 2

Counts from banks inshallow waters may
provide a crude index
of abundance, underestimated for most species.
1
Timed counts underwater
will provide better
estimates of relative
abundance. 2
Electronic counters canprovide effective estimatesfor migratory fish throughnarrow channels / passes.
3

PRISM - TOOLKIT FOR EVALUATING THE OUTCOMES AND IMPACTS OF SMALL/MEDIUM-SIZED CONSERVATION PROJECTS

EDNA

4:5

INTRODUCTION &KEY CONCEPTS

TAXAMarine fish

COMPLETE SURVEY

DESIGNING & IMPLEMENTING
THE EVALUATION

PRISMMODULESAREA-BASED SAMPLING

TIMED SAMPLINGLIVE TRAPPINGREMOVAL TRAPPING

Underwater counts can bemade along transects orusing point counts, allowing
for distance sampling tocorrect for detectability. 2

Counts of large species
from boats or shore(e.g. basking sharks)
may provide an index ofabundance. 1

Wide range of fish-trapping
methods available formark-recapture studies(dye or tagging), or to index
abundance. 2

Gill, seine and trawl
netting can be used toprovide catch returns (fishcaught per unit effort) as
an index of population
size. This may vary withseason, conditions andmigratory status. Effort
may be difficult to record iffrom commercial fishing /angling data. 23

Many amphibians areactive at night, so searchingfor individuals by torchlightin suitable habitat can
provide an index ofabundance, particularlyof breeding individuals(on land or underwater).
Given their sensitivity toenvironmental conditions
and date, surveys should
be conducted on idealnights, or under a rangeof conditions. Searchingof refuges during the day
may also work. Underwater
sweep netting may
provide an index of adultand larval abundance,
depending on time of year.
Occurrence or an index
of breeding abundance
may be obtained from egg
searches. 1

Fences and pitfall traps
may be used to trap andintercept individuals
traveling to or from abreeding site, and should
be checked at least daily.
Estimates of dispersing
juvenile abundance frombreeding ponds, or of egg
abundance, may be used
to estimate productivity /fecundity. Artificial refugia
can be used to attract
individuals to count them,either during hibernationor during the day. 1

Hydroacoustic sampling(sonar) estimatesabundance of shoaling
fish. 3
Amphibians

SPECIES & HABITATMANAGEMENT
ANNEX
FIELD METHODSFOR SPECIESMONITORING

Distance-sampling may
be used to correct counts
along transects to produce
densities, although
individuals hiding in refugia
may be undetectable. 1

Passive sound recording
may detect species withauditory displays, such as
many frogs and toads. 2

388

METHOD FACTSHEETS& ANNEXESPRISM - TOOLKIT FOR EVALUATING THE OUTCOMES AND IMPACTS OF SMALL/MEDIUM-SIZED CONSERVATION PROJECTSUnderwater traps (e.g.
funnel traps) can besubmerged or partiallysubmerged in marginal
habitat, and used to catch
individuals. They should
be checked regularly (atleast every 6 hrs) in warm
weather. Live-caught
amphibians may be suitable
for mark-recapture studies
to estimate abundance orsurvival, by photographing
unique markings (e.g. belly
marks), using skin stains(safe in some toads), or PIT
tagging, (although this may
increase the risk of diseasetransmission). 2

EDNA

Techniques
to identify
species
occurrence
from water
samplesare beingdeveloped. 3

4:5

INTRODUCTION &KEY CONCEPTS

TAXA

DESIGNING & IMPLEMENTING
THE EVALUATIONCOMPLETE SURVEY

Reptiles

PRISM
MODULESAREA-BASED SAMPLING

TIMED SAMPLINGLIVE TRAPPINGTransects of known length
can be walked to countindividuals, e.g. basking
individuals. For somespecies it may be possible
to correct for detection. 1

Timed transects canbe walked to countindividuals, e.g. basking
individuals. 1

Artificial refugia can beused to attract individuals
to count them. The precise
nature of the refugia willvary with the environment.
Refugia should be visitedfrequently during the main
period of activity (7 times
to determine occurrence
and 15-20 times toestimate abundance).
Fences and pitfall traps, orbaited funnel traps, may be
used to trap and intercept
individuals, and should bechecked at least daily. 1

SPECIES & HABITATMANAGEMENT
ANNEX

Mark-recapture studies
may be undertaken to
estimate abundance.
Live-caught individuals
may be suitable formark-recapture studies
by photographing uniquemarkings (e.g. head marks
on snakes), using nail polish/ paint on scales, or PIT
tags. 2

FIELD METHODSFOR SPECIESMONITORING

Marinereptiles389

Nests can be counted along
breeding beaches. 1

Early morning transects
along breedwing beaches
can be used to count turtle
tracks, and those which
end in nests give greater
accuracy. 1

PRISM - TOOLKIT FOR EVALUATING THE OUTCOMES AND IMPACTS OF SMALL/MEDIUM-SIZED CONSERVATION PROJECTSMETHOD FACTSHEETS& ANNEXESREMOVAL TRAPPING

EDNA

4:5

INTRODUCTION &KEY CONCEPTS

PRISM
MODULESTAXACOMPLETE SURVEY

AREA-BASED SAMPLING

TIMED SAMPLINGLIVE TRAPPING

Birds

Possible where individuals
are concentrated (e.g. at
breeding colony or nonbreeding aggregation) andcounted or photographed.
1

Territory mapping fromrepeat visits (usually 5-10)
within known areas used
to estimate the abundanceof breeding birds based
upon the interpretation ofmultiple maps. This doesn’t
work for non-territorial
species. Indices ofabundance can be derived
from a smaller number ofvisits. Line-transects with
distance sampling allow
coverage of a greater area
than area-based samples
and work well in open
habitats. Point-counts
work better in closed
habitats (e.g. woodland). 1

Estimates of abundance
using timed counts can be
useful for rapid assessmentof relative abundance. 1

A range of methods exist
for catching birds, ranging
from mist-netting totrapping individuals on
nests. Techniques require
training and in many cases,
licensing. Some of thesecan be standardised (e.g.
constant effort site (CES)
mist netting), allowing
robust estimates ofabundance to be derived.
Ringing (banding) schemes
are widely used for
demographic studies inbirds, allowing estimationof survival and recruitment.
Some countries also run
nest record schemes tomonitor breeding success.
3

SPECIES & HABITATMANAGEMENT
ANNEX
FIELD METHODSFOR SPECIESMONITORING

Passive sound-recording
may be used to detect
singing or calling birds.
Researchers are developing
algorithms for automatic
identification from soundrecording. Individualsof some species can berecognised from theirsong, enabling accurate
estimates of abundance. 2

High-definition aerial
photography can be used
for strip-transect surveysof larger species in open
habitats (sea, freshwater,
grassland). 2
Bats

Complete surveys of roots
possible when individuals
emerge at dusk. 1
Automatic devices may be
used. Counts may also be
undertaken at hibernationsites, although as usage
varies with temperatureand the availability ofother sites, rarely can suchcounts be regarded ascomplete. 2

390

DESIGNING & IMPLEMENTING
THE EVALUATION

Line-transects may be
used to record flying andforaging bats at night,
using a bat detector to aid
identification. Data will
provide an index of relative
abundance, but counts arehighly variable depending
upon date and conditions.
2

Passive sound recording
may be used to detectand estimate relative
abundance of many
species, depending on
the extent to which theirsonograms are known.
Algorithms are continually
being developed to allow
automated recognition. 2

PRISM - TOOLKIT FOR EVALUATING THE OUTCOMES AND IMPACTS OF SMALL/MEDIUM-SIZED CONSERVATION PROJECTS

Bat boxes may be installed
and used to estimate
relative abundance.
Repeat visits are required
as abundance may varywith conditions and date.
Mist-netting may also be
used with care to estimate
occurrence and relative
abundance. Techniques
require training and inmany cases, licensing.
Ringing can be used toestimate abundancefrom mark-recapture
techniques. 3

METHOD FACTSHEETS& ANNEXESREMOVAL TRAPPING

EDNA

4:5

SPECIES & HABITATMANAGEMENTANNEX

INTRODUCTION &KEY CONCEPTS

PRISM
MODULESMETHOD FACTSHEETS& ANNEXESTAXACOMPLETE SURVEY

AREA-BASED SAMPLING

TIMED SAMPLINGLIVE TRAPPINGREMOVAL TRAPPING

Large diurnal
mammals

Possible where individuals
are concentrated (e.g. at
breeding colony or nonbreeding aggregation) andcounted or photographed.
1Line-transects with
distance sampling allow
coverage of a greaterarea than area-based
samples and work wellin open habitats and for
low-density populations.
1 For large species, aerial
photography can be used
for strip-transect surveys.

Estimates of abundance
using timed counts
can be useful for rapid
assessment of relative
abundance for common
species. 1 Camera traps
can be deployed widely toestimate the frequency ofoccurrence and provide anindex of abundance. 2

Individuals may be
captured and marked usinga range of techniques
including recording uniquemarks, fur clipping (which
only lasts a few weeks),
using dyes and tagging. PIT
tags and microchips are
increasingly being used. 3

Mortality data fromhunting or other lethal
methods can be used
to estimate trends in
population size, or removal
techniques analysis can beapplied. 3

Breeding sites or othersigns (e.g. nests or roostsites) may be counted
within particular areas.
Other signs, such as
droppings, feeding signs,
tracks and hairs may all
be counted to infer an
index of abundance. Linetransects with distancesampling allow coverageof a greater area thanarea-based samples but
this is challenging in forest
conditions. 1

Estimates of abundance
using timed counts can be
useful for rapid assessmentof relative abundance forcommon species. 1

Individuals may be marked
using a range of techniques
including recording uniquemarks, fur clipping (which
only lasts a few weeks),
using dyes and tagging. PIT
tags and microchips are
increasingly being used. 3

Mortality data fromhunting or other lethal
methods can be used
to estimate trends in
population size, or removal
techniques analysis can beapplied. 3

Breeding sites (e.g.
burrows) may be counted
within particular areas.
Other signs, such as
droppings, feeding signs,
tracks and hairs may all becounted to infer an index ofabundance. Line-transects
with distance samplingallow coverage of a greaterarea than area-based
samples and work wellin open habitats and for
low-density populations.1
Spotlight counts at night
can be effective for
nocturnal or crepuscularspecies. 2

Estimates of abundance
using timed counts can be
useful for rapid assessmentof relative abundance forcommon species. 1

Live trapping using a
range of traps provide
quantitative information
on species abundance, andoccurrence for rare species.
Individuals may be marked
using a range of techniques
including recording uniquemarks, fur clipping (which
only lasts a few weeks),
using dyes and tagging. PIT
tags and microchips are
increasingly being used. 3

Mortality data fromhunting or other lethal
methods can be used
to estimate trends in
population size, or removal
techniques analysis can beapplied. 3

Arboreal
mammals

FIELD METHODSFOR SPECIESMONITORING
Cryptic or
nocturnal
mammals

391

DESIGNING & IMPLEMENTING
THE EVALUATION

Possible where individuals
are concentrated (e.g. at
breeding colony or nonbreeding aggregation) andcounted or photographed.
1

Passive sound recording
has the potential to be used
to detect and estimate
relative abundance of vocal
species. 2

Passive sound recording
has the potential to be used
to detect and estimate
relative abundance of vocal
species. Camera traps canbe deployed widely toestimate the frequency ofoccurrence and provide anindex of abundance. 2

PRISM - TOOLKIT FOR EVALUATING THE OUTCOMES AND IMPACTS OF SMALL/MEDIUM-SIZED CONSERVATION PROJECTS

EDNA

4:5

INTRODUCTION &KEY CONCEPTS

DESIGNING & IMPLEMENTING
THE EVALUATION

PRISMMODULESTAXACOMPLETE SURVEY

AREA-BASED SAMPLING

TIMED SAMPLINGLIVE TRAPPING

Marinemammals

Where individuals areconcentrated (e.g. at
breeding colony or nonbreeding aggregation),they may be counted or
photographed. 1

Aerial photography canbe used for strip-transect
surveys to provide
estimates of relative
abundance. 2

Counts of cetaceans fromboats or shore may provide
an index of abundance. 2

In species with individual
markings, photography
may be used to monitor
abundance anddemographic parameters.
2

Acoustic recording may
detect cetaceans over large
distances, and may be used
to identify individuals. 3

SPECIES & HABITATMANAGEMENT
ANNEX
FIELD METHODSFOR SPECIESMONITORING

392

PRISM - TOOLKIT FOR EVALUATING THE OUTCOMES AND IMPACTS OF SMALL/MEDIUM-SIZED CONSERVATION PROJECTSMETHOD FACTSHEETS& ANNEXESREMOVAL TRAPPING

EDNA

4:5

SPECIES & HABITATMANAGEMENT
ANNEX
SAMPLING
APPROACHESFOR EVALUATINGCHANGES IN
SPECIES STATUS

393

INTRODUCTION &KEY CONCEPTS

DESIGNING & IMPLEMENTING
THE EVALUATION

Complete counts - In some cases it will be possible to countaccurately all the individuals within a population. This is most likely
to be achievable where the individuals are easily detected andconcentrated within a relatively small area. Complete surveys canbe undertaken using counts in the field (although even counting by
an observer can be subject to a degree of error), or from images,
such as fixed-point photographs or aerial photographs, providing amore repeatable approach. When undertaking complete surveys
for a mobile species, it is important to do so at a time when theindividuals will be concentrated within the count location, such asat breeding colonies or roost sites; otherwise counts will have to
be corrected for the likely proportion of the population present.
For species with multiple life-stages, it is important to identify themost effective life-stage to count, which is often the adult stage.
Area-based sampling - (e.g. quadrats, fixed transects). Area-based
sampling approaches involve sampling or surveying defined areas,from which the results are extrapolated to unsurveyed areas.
Areas may be surveyed using a variety of techniques from samplebased approaches such as point counts and line-transects, mapping
methods such as common bird census techniques, and complete
counts using fixed-point photography. To identify true abundance
it may be necessary to correct for imperfect detectability, suchas using distance sampling or using data from repeat visits foroccupancy modelling. It is important that the locations of thesamples are randomly distributed across the area of interest, or
are appropriately stratified, in order to generate robust results.
If species abundance is highly variable between locations, it isimportant to ensure that there are sufficient samples to produce
robust population estimates. Correctly delimiting the sampling

PRISM
MODULESMETHOD FACTSHEETS& ANNEXES

area is also important to ensure that all potentially suitable areas
are covered. To assist with design, it may be helpful to conduct
some initial trials to estimate mean abundance and variance acrossa range of sites, before determining the sample size and power ofany final approach. Modelling the results appropriately will also be
important in order to estimate uncertainty correctly.
Timed sampling - The abundance of species with low detectability,that are widely dispersed, that require searching using methods
that are difficult to standardise, or for which abundance cannot
be easily related to a particular area, may be best estimated using
timed counts. Here the number of individuals is recorded perunit of time, rather than per unit area. This makes it difficult to
use this approach to estimate true population size, but it can beused to estimate relative abundances. Examples include counting
migratory individuals moving through particular bottlenecks,
or recording encounters using passive devices such as camera
traps and listening devices. Passive devices may be particularly
valuable for detecting the occurrence of rare and difficult to detect(particularly nocturnal) species. As there is likely to be temporalvariation in the numbers of individuals recorded, either within aday, or across a season, it is important to ensure that any samples
are appropriately timed within that range, and that this is done in
a comparable way between sites or visits. If an entire population
passes through a location during a particular period, then an
appropriate sample of timed counts may estimate true population
size.
Live trapping - (trapping, netting etc.). It is possible to live-trap
individuals of many species using a wide range of different
methods, including mist-netting birds, trapping small mammals,

PRISM - TOOLKIT FOR EVALUATING THE OUTCOMES AND IMPACTS OF SMALL/MEDIUM-SIZED CONSERVATION PROJECTS

4:5

SPECIES & HABITATMANAGEMENT
ANNEX
SAMPLING
APPROACHESFOR EVALUATINGCHANGES IN
SPECIES STATUS

INTRODUCTION &KEY CONCEPTS

DESIGNING & IMPLEMENTING
THE EVALUATION

netting fish, and light-trapping nocturnal insects. Trapping sessions
can be used to provide estimates of relative abundance in muchthe same way as the timed samples discussed above. As with all
trapping techniques, care has to be taken that the efficacy of thetrap does not vary between habitats or locations, and therefore
that such estimates of abundance are not subject to bias as aresult of other factors. Where it is possible to mark individuals,
for example by using rings (bands) on birds, PIT tags, microchips
in other vertebrates or colour marks on some invertebrates, or torecognise individuals from their appearance, then mark-recapture
techniques can be used to estimate the size of the population fromthe ratio of the number of individuals caught and marked which arethen recaptured in a subsequent sample. There can be challenges
in defining the area which the population of sampled individuals
is from, and the extent to which a particular population is “open”
or “closed”, or includes transient or mobile individuals, and theextent to which the “trappability” of individuals varies between
then, or even after capture. However, analytical solutions exist for
at least some of these issues, allowing this approach to provide
relatively robust estimates of abundance that should exclude manypotential biases. It can be the most effective way of estimating
the abundance of cryptic species, although it is resource-intensive
relative to other survey methods. Care needs to be taken thattrapping does not adversely affect the survival or behaviour of theindividuals that are caught.
Removal trapping - A greater range of trapping approaches isavailable if individuals do not have to be caught alive. These canbe used to provide estimates of relative abundance in much thesame was as live-trapping samples or timed samples as outlined

394

PRISM
MODULESMETHOD FACTSHEETS& ANNEXES

above, based purely on the number of individuals caught. Thus,
invertebrate sampling techniques such as pitfall traps or water
traps, which generally involve the death of the individuals caught,
or snap traps for small mammals, are widely used to estimate
the relative abundance of groups such as these. If such trapping
approaches are carefully planned, or specific methodologies are
adopted, then the data can be used to estimate abundance morerobustly, for example from the decay in the numbers of individuals
trapped through time.
eDNA - The field of documenting the occurrence or even theabundance of species within particular areas by means of theextraction of DNA from environmental or biological samples
is currently expanding rapidly. Broadly speaking, samples fromthe environment, such as water or soil, are collected and thenprocessed to extract the DNA, which is then multiplied andreplicated sufficiently for processing. The resulting DNA profiles
can then be compared with databases of known species in order to
identify the occurrence of those species within the environment.
This can prove particularly effective for monitoring taxa thatare difficult to survey, such as amphibians and fish in freshwater
environments, or micorrhizal fungi in the soil. Alternatively, theseapproaches can be used for rapid processing of biodiversity
samples, such as those obtained by destructive sampling, including
invertebrate samples: samples can be homogenised and processed
to identify the species present. This is an emerging technology
which may become more relevant for small and medium-sized
conservation projects as it develops further.

PRISM - TOOLKIT FOR EVALUATING THE OUTCOMES AND IMPACTS OF SMALL/MEDIUM-SIZED CONSERVATION PROJECTS

4:5

SPECIES & HABITATMANAGEMENTANNEX
ANALYTICAL ISSUESWHEN ESTIMATINGABUNDANCE,
DISTRIBUTION,SURVIVAL AND
REPRODUCTIONINTRODUCTION &KEY CONCEPTS

DESIGNING & IMPLEMENTING
THE EVALUATIONMETHOD FACTSHEETS& ANNEXES

Annex: Analytical issues when estimating abundance,
distribution, survival and reproduction

Ultimately, a rigorous survey across an entire project area (e.g. anatlas approach) will provide the best distribution data without theneed for statistical inference.

Estimating distribution - Information about species distribution
may be gained from a range of sources, from unstructured species
records to the results of standardised surveys. There are particular
problems associated with the interpretation of unstructured
records, as the distribution of these may be as much a function ofwhere people have looked for the species as its actual distribution.
Qualitatively, the location of records can be used to estimate a
species range, and where it has particular habitat or elevation
associations, to describe likely occurrence within that range
from maps of suitable habitat extent. Where possible, it will be
preferable to use statistical modelling approaches (e.g. species
distribution models, occupancy models) to develop quantitative
predictions from these records, which can be extrapolated acrossthe remainder of the project area, although in cases where survey
effort is non-random, care needs to be taken to ensure that this isdone robustly.

Estimating abundance - Abundance is more challenging
to estimate than occurrence. For many species, imperfectdetectability will reduce the number of individuals counted
relative to the true number present (see Borchers et al. 2001 fora consideration of these issues). A range of potential approaches
exists to account for this potential for under-estimation, ranging
from distance sampling to mark-recapture, each of which isaccompanied by its own assumptions and challenges. Providingthat comparable methods are used to compare variation inabundance in space and time, these issues should have a limited
impact on the ability to monitor project outcomes compared to asituation where the monitoring methods vary, either through timeor between sites.

If at all possible, it is advisable to make use of information about
the absence or inferred absence of a species to improve model
predictions. This can be done using the occurrence of otherspecies likely to be recorded at the same time as the focal species,
to identify null records or to indicate survey effort. A range ofstatistical techniques exists for modelling species distributions
from a series of occurrences and from underlying environmental
and spatial data. These techniques will produce the most robust
predictions when based upon a well-structured survey, where
effort is standardised or is well measured so it can be accounted
for.
395

PRISM
MODULES

Often extrapolation will have to be made from sample data tounsurveyed locations. Final estimates should be associated withan estimate of error (uncertainty) in order to identify whether
changes observed between repeat surveys are likely to bemeaningful. Any such extrapolation should take account ofpotential biases in the location of samples.
Estimating change - Changes in the occurrence or abundance ofa species may be inferred from unstructured data, or measured
more accurately from the collection of specific monitoring data.
The approaches described above for estimating distribution
and abundance may be used separately to assess occurrence
and population size in two or more time periods, and to quantify
change; or this may be more elegantly achieved in a single model

PRISM - TOOLKIT FOR EVALUATING THE OUTCOMES AND IMPACTS OF SMALL/MEDIUM-SIZED CONSERVATION PROJECTSDESIGNING & IMPLEMENTING
THE EVALUATION

to test for statistical differences between time-periods. In order
to attribute any observed changes to conservation intervention, it
will be necessary to examine differences in trend between site(s)subject to conservation intervention and control site(s).

ANALYTICAL ISSUESWHEN ESTIMATINGABUNDANCE,
DISTRIBUTION,SURVIVAL AND
REPRODUCTION

396

METHOD FACTSHEETS& ANNEXES

Effort required todetect a particulareffectLow

SPECIES & HABITATMANAGEMENT
ANNEX

Formal analysis of change should take account of the likely powerof such monitoring to detect significant change. This will be afunction of count accuracy, natural variability and the magnitude of
impact of the conservation intervention. In general, a larger sample
of data is required where count accuracy is low, natural variability
is high, or the magnitude of impact is low. In order to assess this, wewould recommend conducting a pilot study to identify the mosteffective monitoring regime for a given resource, in order to detecta particular magnitude of effect.

PRISM
MODULESHighINTRODUCTION &KEY CONCEPTS

Variability between sites

4:5

Figure 1. The sampling effort required to detect a particular effectincreases with both uncertainty / error at the site level (where
samples are taken from) and the degree of variability betweensites.

Low

PRISM - TOOLKIT FOR EVALUATING THE OUTCOMES AND IMPACTS OF SMALL/MEDIUM-SIZED CONSERVATION PROJECTSHigh

Uncertainty at site level

4:5

INTRODUCTION &KEY CONCEPTS

DESIGNING & IMPLEMENTING
THE EVALUATION

PRISMMODULES

Annex: Field methods for habitat monitoring

SPECIES & HABITATMANAGEMENT
ANNEX
FIELD METHODSFOR HABITATMONITORING

397

This annex gives a very brief summary of field methods formonitoring different habitat types that are relevant for small
and medium-sized conservation projects. To indicate the level ofexpertise and equipment required, each method is categorised as:
1 (method fairly simple to use i.e. does not require formal trainingor specialist equipment), 2 (method requires some training and/or specialist equipment), or 3 (method requires highly-specialised
training and equipment).
More information on the methods in this annex can be foundin the Cambridge handbook of ecological survey methods
https://sunsetridgemsbiology.wikispaces.com/file/view/
Biodiversity+Handbook.pdf

PRISM - TOOLKIT FOR EVALUATING THE OUTCOMES AND IMPACTS OF SMALL/MEDIUM-SIZED CONSERVATION PROJECTSMETHOD FACTSHEETS& ANNEXES

4:5

INTRODUCTION &KEY CONCEPTS

DESIGNING & IMPLEMENTING
THE EVALUATION

PRISMMODULESMETHOD FACTSHEETS& ANNEXES

HABITAT CHARACTERISTICSPECIES & HABITATMANAGEMENT
ANNEX

HABITAT TYPEEXTENTCOMPOSITIONSTRUCTURE

REGENERATIONEFFECTS OF EXTERNAL PRESSURE

Forest, woodlandand scrubSatellite imagery
to measure area ofindividual forest standand boundaries. 2

Plots, transects or
plotless sampling tomeasure changes in
species composition,richness and diversity.
1

Plots or plotless
sampling to measure
horizontal and verticalstructural diversity,thinning extent anddeadwood. 1

Aerial photography/
drones measureextent and location ofopen spaces. 3

Use plots to measure effects of harvesting
timber and non-timber forest products(NTFPs). Measure height, condition etc.
before & after harvesting and within/outside areas of harvesting. 3

Aerial photography/
drones to measureextent and location ofopen spaces. 3

Grassland

FIELD METHODSFOR HABITATMONITORING

Satellite imagery tomeasure extent ofarea. 2
Aerial photography ordrone imagery. 3

Measure grazing pressure by comparing
difference in tree mortality/condition
between grazed and ungrazed areas. 2

Use line transects or
quadrats to measurespecies richness,diversity and/or
presence/absence ofindicator species. 1

Quadrats/point
samples to measureheight, density andpercentage coverin different heightcategories. 1

Use vegetation height/condition tomonitor grazing intensity. 1

Quadrats or transects
to measure species/
species communities
characteristic of
the habitat, species
composition and/or
richness. 1

Quadrats/point
samples to measureheight, density andpercentage coverin different heightcategories. 1

Use vegetation height/condition tomonitor grazing intensity. 1

Quadrat surveys for
specific vegetation
types. 1
Mountain habitats

Satellite imagery tomeasure extent ofarea. 2
Aerial photography/
drones to measureextent and location ofarea. 3
Quadrats/transects
to measure exposed
rock, scree, bare soil
or snow cover. 1

398

Use distance sampling
to monitor specificindicator species ortrees in more open
habitats. 2

Measure seedling
regeneration andcomposition using
ground survey
methods. 1

Total counts, quadrats
or transects tomeasure presence/
absence of indicatorspecies. 1

PRISM - TOOLKIT FOR EVALUATING THE OUTCOMES AND IMPACTS OF SMALL/MEDIUM-SIZED CONSERVATION PROJECTS

Use occurrence/indicator species tomonitor changes in nutrient status. 2

4:5

INTRODUCTION &KEY CONCEPTS

DESIGNING & IMPLEMENTING
THE EVALUATION

PRISMMODULESMETHOD FACTSHEETS& ANNEXES

HABITAT CHARACTERISTIC
HABITAT TYPEEXTENTCOMPOSITIONSTRUCTURE

Still/slow-flowing
wetlands

Satellite imagery tomeasure extent ofarea. 2

Quadrats or transects
to measure species/
species communities
characteristic of
the habitat, species
richness, diversity
and/or presence/
absence of indicatorspecies. 1

Fixed point survey tomeasure vegetation
height and/or invasion
by non-wetland
species. 1

Use macroinvertebrate indicator species
to measure pollution. 2

Usemacroinvertebrate
indicator species
to measure waterchemistry. 2

General surveys,
quadrats or transects
to measure featuresof river morphology:
channel width, shape,pools etc. 1

Use macroinvertebrate indicator species
to measure pollution. 2

Aerial photography/
drones measureextent and location ofarea. 3SPECIES & HABITATMANAGEMENT
ANNEX

Rivers & streams

Aerial photography/
drones to measureextent and location ofarea. 3

FIELD METHODSFOR HABITATMONITORING
Coral reefs

399

Satellite imagery tomeasure extent of
individual area. 2

Quadrats ortransects to measurebankside andemergent vegetation
species richness,
and abundanceof vegetation andmacrophytes. 1
Belt transects orlong swim method
to survey indicatorspecies. 2

REGENERATION

Fixed point survey,
transect or quadrat
methods to measurevegetation height
and/or coverage ofbanks. 1
Point intercepttransects to measure
cover of sessile
benthic invertebrates.
2

Before/after or
Before-After ControlImpact (BACI) belttransects or long swim
method to survey
cover/indicatorspecies. 3

PRISM - TOOLKIT FOR EVALUATING THE OUTCOMES AND IMPACTS OF SMALL/MEDIUM-SIZED CONSERVATION PROJECTSEFFECTS OF EXTERNAL PRESSURE

4:5

INTRODUCTION &KEY CONCEPTS

DESIGNING & IMPLEMENTING
THE EVALUATION

Annex: Sampling approaches for habitat monitoring

SPECIES & HABITATMANAGEMENT
ANNEX
SAMPLING
APPROACHESFOR HABITATMONITORING

Plots
Plots are pre-defined square, rectangular or circular areas that areused to delineate an area of interest. These plots are then sampled
at one or more subsequent time points to provide data on changes
taking place. Care must be taken when using permanent plots toestimate wider-scale changes. Unrepresentative plots, or too few
plots, will give unreliable estimates of compositional and structuralchange, so it is important to define the locations of plots in a waythat minimises bias (preferably by locating them randomly). Thiscan be challenging in remote areas, but it is important that lessaccessible areas are sampled sufficiently, as habitat quality will
typically decline with ease of access.
Larger plots (>0.25 ha) are (1) better for validating satellite
imagery, (2) associated with lower variance between plots, (3)produce more representative estimates of vegetation parameters,(4) are better for measuring small-scale ecological processes, and(5) suffer less from “edge effects” (because they have a lower
perimeter : area ratio).

PRISM
MODULES

Plotless samplingThis is generally a much faster method for estimating attributes
than using permanent or temporary plots. However serious bias
can result if the distribution of species is not random (i.e. if it iseither clumped or uniformly distributed).
Types of plotless sampling include:“Bitterlich sampling” - Most commonly used for estimating tree
density, but any other information that can be recorded about
individual trees can be measured and averaged. It is rapid andunbiased, provided that the correct Basal Area Factor prism isselected. Too many trees counted systematically underestimates
the basal area; too few counted generates too much error.
For full details see http://wiki.awf.forst.uni-goettingen.de/wiki/
index.php/Bitterlich_sampling .
Other plotless methods include the Nearest Individual method, thePoint-centred Quarter method and the T-square sample method.
(Full details of these methods can be found in pages 233-234 of theCambridge Handbook of Biodiversity Methods).

Smaller plots (<0.25 ha) provide (1) better landscape-level
representation, but suffer from (2) higher random variation
between plots, (3) less representative vegetation parameters,and (4) greater edge-effects.
For further information, see Condit R (1996) http://www.stri.
si.edu/sites/publications/PDFs/Condit.TREE1995.pdf .

400

METHOD FACTSHEETS& ANNEXESPRISM - TOOLKIT FOR EVALUATING THE OUTCOMES AND IMPACTS OF SMALL/MEDIUM-SIZED CONSERVATION PROJECTS

4:5

INTRODUCTION &KEY CONCEPTS

SPECIES & HABITATMANAGEMENT
ANNEX
SAMPLING
APPROACHESFOR HABITATMONITORINGDESIGNING & IMPLEMENTING
THE EVALUATION

Transects
Transects are linear plots that are usually straight, but may follow
a natural course such as an existing footpath or river. They canbe treated as a long thin plot, e.g. 1 km long by 20 metres wide,
and only features within the transect boundary are counted andmeasured. Distance sampling is a form of transect which is usefulfor assessing features of high quality habitat that are too rare
to be effectively sampled using plot methods, e.g. frequency oflarge trees, or particular non-timber forest products. The method
involves measuring the distance of observed trees or otherfeatures from a straight transect line. Formulae, or the software
DISTANCE™, are used to calculate density.
Monitoring indicator species
This involves monitoring species that are characteristic ofparticular habitats. Their abundance (or in some cases, presence/
absence) can act as an indicator of habitat condition, for example
the perennial plants Mercurialis perennis and Anemone nemorosaare indicators of ancient woodland in a European context, while
the abundance of “disturbance-responsive species” (such as
grasses, herbs, ferns, bamboo, lianas and pioneer tree species) canbe useful for quantifying forest degradation. The Annex Sampling
approaches for evaluating changes in species status provides
specific guidance on monitoring species abundance.
Dominance indices
These can be used to provide a statistical measure for tracking
change over time and/or to make comparisons between sites.
Dominance indices are very quick to assess but they rely on having
staff who really understand the underlying principle and have goodexperience with the range of different abundances of disturbanceresponsive species.

401

PRISM
MODULESMETHOD FACTSHEETS& ANNEXES

Typically, habitat condition in forests is assessed by quantifying
the number of trees within particular size classes of “Diameter at
Breast Height” (DBH). The thresholds for these classes must be
calibrated to the forest type and condition in order to optimise
sensitivity and to be able to detect change over time: if they are
too high, few trees will be recorded in the class; if they are too low,
the effort required to count all the trees within the size class will be
excessively time-consuming.
References
The Cambridge Handbook of Biodiversity Methods - Survey,
Evaluation & Monitoring provides an overview of methods for all
taxa - https://sunsetridgemsbiology.wikispaces.com/file/view/
Biodiversity+Handbook.pdf .
Swinfield T and Harrison RD (2016). Optimising forest resource
assessments for ecosystem restoration: ftp://ftp.biotrop.org/
International_Conference_on_Tropical_Biology_2015/ORAL_PRESENTATION/SUB%20TEMA%201/S1ST1-O-02_Tom%20
Swinfield_lowres.pdf
The Gottingen Forest Inventory Wiki is an excellent source of
information regarding forest plot design and monitoring - http://
wiki.awf.forst.uni-goettingen.de/wiki/index.php/Main_Page .
Gill D and Daltry J (2014). Brief 3: How to make a monitoringplan for threatened tree species. Global Trees Campaign http://
globaltrees.org/wp-content/uploads/2015/03/GTC-Brief-3monitoring-plan-lo-res.pdf .

PRISM - TOOLKIT FOR EVALUATING THE OUTCOMES AND IMPACTS OF SMALL/MEDIUM-SIZED CONSERVATION PROJECTS

4:0

INTRODUCTION &KEY CONCEPTS

PRISM
MODULESMETHOD FACTSHEETS& ANNEXES

TERM

DEFINITION

ActionA specific activity or process undertaken over a specific period of time by a project in order to bring aboutchange.

Assumption

Expression of the logic underpinning the links between different stages of a project (e.g. why a particular
output could be expected to result in a subsequent outcome) or the external factors which could
positively or negatively influence project results.

Attribution

Ascribing a causal link between observed changes and project actions, taking into account the effects ofother actions and possible confounding factors.

Baseline

Information collected before or at the start of a project that provides a basis for planning and/or
assessing subsequent progress and outcomes/impacts.

Control group/site

A group or site that is not targeted by the actions of the project being evaluated.

Counterfactual

A hypothetical estimate of what would have happened (or not) had the project not been implemented.

Evaluation

The systemic assessment of an ongoing or completed project or project action. This involves looking at
specific elements of the project to see what difference the project has made and what lessons have beenlearnt.

Evaluation design

The methodology selected for collecting and analysing data in order to reach conclusions about project
results.

Impact

The long-term change(s) brought about by the project’s action(s).

Indicator

Quantitative or qualitative variable that provides a reliable means of measuring a particular outcome or
impact.

Logic model

Often a visual representation, this provides a “road map” showing the sequence of related events
connecting the need for a planned project with the project’ desired outcomes and results.

Logical framework(Logframe)

A management tool used to improve the design and evaluation of actions that is widely used bydevelopment agencies. It is a type of logic model that identifies strategic project elements (inputs,
outputs, outcomes, impact) and their causal relationships, indicators, and the assumptions or risks that
may influence success and failure.

Monitoring

The performance and analysis of routine measurements to detect changes in project status. Used to
inform the project team about the progress of an ongoing action or project, and to detect problems that
may be addressed through corrective actions.

Outcome

The change(s) brought about by the project’s actions.

OutputWhat is directly produced by the project’s actions.

GLOSSARY402

DESIGNING & IMPLEMENTING
THE EVALUATIONPRISM - TOOLKIT FOR EVALUATING THE OUTCOMES AND IMPACTS OF SMALL/MEDIUM-SIZED CONSERVATION PROJECTS

4:0

INTRODUCTION &KEY CONCEPTS

GLOSSARY403

DESIGNING & IMPLEMENTING
THE EVALUATION

PRISMMODULESMETHOD FACTSHEETS& ANNEXES

TERM

DEFINITION

Qualitative data

Observations or information expressed using non-numerical terms rather than numerical terms (e.g.
observations, answers to open questions, written, audio, video or other visual evidence).

Quantitative data

Information that can be expressed in numerical terms, counted, or compared on a scale.

Recommendations

Proposals based on findings and conclusions that are aimed at enhancing the effectiveness, quality or
efficiency of an action.

Theory of Change

A management tool that presents the logic of a project or programme in a diagrammatic form. It links
impacts and higher-level objectives to intermediate and lower-level objectives. The diagram (and related
description) may also indicate main actions, indicators, and strategies used to achieve the objectives. It
may also outline the main assumptions that must remain true in order for the project to work, and thepotential risks to success.

Threat reduction
outcome

Outcome relating to the reduction or mitigation of a threat. Typically it is the final change that needs tohappen in order to impact a conservation target.

Stakeholders

Entities (governments, agencies, companies, organisations, communities, individuals, etc.) that have a
direct or indirect interest in the project and any related evaluation.

Target(s)The specific individuals, groups, species, sites or organisations for whose benefit the project/project
action is being undertaken.

PRISM - TOOLKIT FOR EVALUATING THE OUTCOMES AND IMPACTS OF SMALL/MEDIUM-SIZED CONSERVATION PROJECTSINTRODUCTION &KEY CONCEPTS

DESIGNING & IMPLEMENTING
THE EVALUATION

PRISMMODULESMETHOD FACTSHEETS& ANNEXESPRISM - TOOLKIT FOR EVALUATING THE OUTCOMES & IMPACTS OF SMALL/MEDIUM-SIZED CONSERVATION PROJECTS404

PRISM - TOOLKIT FOR EVALUATING THE OUTCOMES AND IMPACTS OF SMALL/MEDIUM-SIZED CONSERVATION PROJECTS