Source document: https://docs-lawep.s3.us-east-2.amazonaws.com/1697630298139.pdf (Antigua and Barbuda)
Terminal Evaluation of the UNEP/GEF Project “Advancing the Nagoya Protocol in Countries of the Caribbean Region” (GEF ID 5774) Evaluation Office of UN Environment Programme April 2020 This report has been prepared by an independent consultant evaluator and is a product of the Evaluation Office of UNEP.

The findings and conclusions expressed herein do not necessarily reflect the views of Member States or the UNEP Senior Management.
For further information on this report, please contact: Evaluation Office of UNEP P. O.
Box 30552-00100 GPO Nairobi Kenya Tel: (254-20) 762 3389 Email: chief.eou@un.org Advancing the Nagoya Protocol in Countries of the Caribbean Region GEF ID: 5774 April 2020 All rights reserved © 2020 Evaluation Office of UNEP Terminal Evaluation: “Advancing the Nagoya Protocol in Countries of the Caribbean Region” Acknowledgements This Terminal Evaluation was prepared for the Evaluation Office of UNEP by David Simmons, as an independent evaluation consultant.
The report benefits from a peer review conducted within the Evaluation Office of UNEP.

The Evaluation Office of UNEP would like to thank the “Advancing the Nagoya Protocol in Countries of the Caribbean Region” project team, and in particular: Christopher Cox and Gloritzel Frangakis Cano at UNEP, as well as Domenique Finegan, María Pía Hernández and other colleagues at IUCN Mexico, for their contribution and collaboration throughout the evaluation process.
Sincere appreciation is also expressed to the various representatives from government ministries and departments in the participating countries, as well as representatives from GIZ, CARICOM and OECS, who took the time to discuss this project with the evaluation consultant.

Finally, the evaluation thanks all other stakeholders who shared their knowledge and comments, and in this way contributed to the preparation of this report.
Evaluation team David A. Simmons – Consultant Evaluation Office of UN Environment Pauline Marima – Evaluation Manager Mercy Mwangi – Evaluation Programme Assistant 3 | P a g e Terminal Evaluation: “Advancing the Nagoya Protocol in Countries of the Caribbean Region” About the Evaluation Joint Evaluation: No Report Language(s): English Evaluation Type: Terminal Project Evaluation Brief Description: This report is the terminal evaluation of the UNEP-GEF project “Advancing the Nagoya Protocol in Countries of the Caribbean Region”, implemented between July 2015 and May 2019.The project's overall development goal was to support countries of the Caribbean to facilitate access to their genetic resources and benefit sharing in a fair and equitable way, in line with the Convention on Biological Diversity and the Nagoya Protocol.
Its main objective was “seeking uptake of the Nagoya Protocol and implementation of key measures to make the protocol operational in Caribbean countries”.

The evaluation sought to assess project performance (in terms of relevance, effectiveness and efficiency), and determine outcomes and impacts (actual and potential) stemming from the project, including their sustainability.
The evaluation had two primary purposes: (i) to provide evidence of results to meet accountability requirements, and (ii) to promote learning, feedback, and knowledge sharing through results and lessons learned among UNEP, Executing Partner IUCN, and the GEF.
Key words: Antigua & Barbuda; Barbados; Grenada; Guyana; Jamaica; Saint Kitts & Nevis; Saint Lucia; Trinidad & Tobago; Terminal Evaluation; TE; GEF; IUCN; UNEP; GIZ; CARICOM; OECS; Access and Benefit Sharing; ABS; Genetic Resources, Indigenous and Local Communities; Convention on Biological Diversity; CBD 4 | P a g e Terminal Evaluation: “Advancing the Nagoya Protocol in Countries of the Caribbean Region” Project General Information Table ES1: Project Identification Table GEF Project ID: Implementing Agency: 5774 UNEP Sub-programmes: Ecosystems Management and Environmental Governance UNEP approval date: September 2015 Programme of Work Output(s): IMIS number: GFL-5060-2711-4E67 Executing Agency: Expected Accomplishment(s): International Union for the Conservation of Nature (IUCN) Ecosystem Management (expected accomplishment A), Environmental Governance (expected accomplishment B and C).
PoW biennia: 2016- 2017; 2018‒2019 Subprogramme 3: Healthy and productive ecosystems GEF approval date: July 2015 GEF Strategic Priority: BD – 5 Project type: Focal Area(s): Medium Size Project Biodiversity Expected start date: September 2015 Actual start date: February 2016 Planned completion date: September 2018 Actual completion date: May 2019 Planned project budget at approval: USD 5,635,257 GEF grant allocation: USD 1,826,000 Project Preparation Grant - GEF financing: USD 90,000 Actual total expenditures reported as of March 2019: GEF grant expenditures reported as of March 2019: Project Preparation Grant - co-financing: USD 4,690,999.61 USD 1,786,000.00 USD 41,356.41 Expected Medium-Size Project co-financing: USD 3,809,257 Secured Medium-Size Project co-financing: USD 2,904,999.61 (as of March 2019) First disbursement: February 2016 Date of financial closure: N/A No.
of revisions: No.
of Steering Committee meetings: 7 6 Date of last revision: 01/01/2019 Date of last/next Steering Committee meeting: Last: March 2019 Mid-term Review/ Evaluation (planned date): June 2017 Mid-term Review/ Evaluation (actual date): September 2017 Terminal Evaluation (planned date): Coverage - Country(ies): September 2018 Antigua & Barbuda, Barbados, Grenada, Guyana, Jamaica, Saint Kitts & Nevis, Saint Lucia, Trinidad & Tobago Terminal Evaluation (actual date): June 2019 Coverage - Region(s): Caribbean 5 | P a g e Terminal Evaluation: “Advancing the Nagoya Protocol in Countries of the Caribbean Region” Table of Contents Executive Summary .............................................................................................................. 2 1.0 Introduction .................................................................................................................... 9 2.0 Evaluation Methods ...................................................................................................... 10 2.1 Overall Approach ................................................................................................................ 10 2.2 Interviews ............................................................................................................................ 11 2.3 Field Visits ........................................................................................................................... 12 3.0 The Project
12 3.0 The Project ................................................................................................................... 13 3.1 Context ................................................................................................................................ 13 3.2 Objectives and components .............................................................................................. 14 3.3 Stakeholders ....................................................................................................................... 19 3.4 Project implementation structure and partners ............................................................... 20 3.5 Changes in design during implementation ....................................................................... 21 3.6 Project financing ................................................................................................................. 21 4.0 Theory of Change ......................................................................................................... 23 4.1 Reconstructed Theory of Change at Evaluation ............................................................... 23 4.2 Causal Logic: Outputs to Direct Outcomes ....................................................................... 23 4.3 Direct Outcomes (DO) to Intermediate State (IS) ............................................................. 24 4.4 Intermediate State to Impact .............................................................................................
24 5.0 Evaluation Findings....................................................................................................... 27 5.1 Strategic Relevance ............................................................................................................ 27 5.1.1 Alignment to the UNEP Medium Term Strategy (MTS) and Programme of Work (POW) ..................................................................................................................................... 27 5.1.2 Alignment to UNEP /GEF Strategic Priorities ............................................................ 27 5.1.3 Relevance to Regional, Sub-regional and National Environmental Priorities .......... 28 5.2 Quality of Project Design .................................................................................................... 28 5.2.1 Strengths ...................................................................................................................... 29 5.2.1 Weaknesses ................................................................................................................. 30 5.4 Nature of the External Context........................................................................................... 31 5.5 Effectiveness ....................................................................................................................... 32 5.5.1 Delivery of outputs ....................................................................................................... 32 5.5.2 Achievement of direct
 outcomes ............................................................................... 33 5.5.3 Likelihood of impact .................................................................................................... 34 5.6 Financial Management ....................................................................................................... 36 5.6.1 Completeness .............................................................................................................. 36 5.6.2 Communication ........................................................................................................... 38 5.7 Efficiency ............................................................................................................................. 40 i | P a g e Terminal Evaluation: “Advancing the Nagoya Protocol in Countries of the Caribbean Region” 5.8 Monitoring and Reporting .................................................................................................. 41 5.8.1 Monitoring design and budgeting............................................................................... 41 5.8.2 Monitoring of Project implementation ....................................................................... 42 5.8.3 Project reporting .......................................................................................................... 43 5.9 Sustainability
 ....................................................................................................................... 44 5.9.1 Socio-political Sustainability ....................................................................................... 44 5.9.2 Financial Sustainability................................................................................................ 44 5.9.3 Institutional Sustainability (including issues of partnerships) ................................. 45 5.10 Factors and Processes Affecting Performance ............................................................. 46 5.10.1 Preparation and Readiness ....................................................................................... 46 5.10.2 Quality of Project Management and Supervision .................................................... 47 5.10.3 Stakeholder Participation and Cooperation............................................................. 47 5.10.4 Responsiveness to Human Rights and Gender Equality......................................... 48 5.10.5 Country Ownership and Driven-ness ........................................................................ 49 5.10.6 Communication and Public Awareness ................................................................... 49 6.0 Conclusions, Lessons Learned and Recommendations ................................................ 52 6.1 Conclusions
52 6.1 Conclusions ......................................................................................................................... 52 6.2 Lessons Learned ................................................................................................................. 57 6.3 Recommendations.............................................................................................................. 59 Annexes Annex 1.
Response to stakeholder comments received but not (fully) accepted by the evaluator .................................................................................................................................... 61 Annex 2: List of Interviewees ................................................................................................... 63 Annex 3: List of documents consulted .................................................................................... 65 Annex 4: Evaluation TORs (without annexes) ......................................................................... 69 Annex 5: Brief Resumé of the consultant ................................................................................ 81 Annex 6: Quality Assessment of the Evaluation Report ......................................................... 82 List of Tables and Figures Table ES1: Project Identification Table ...................................................................................... 5 Table ES2: Evaluation Criteria and Rating Table ....................................................................... 5 Table ES3: Main Lessons Learned ............................................................................................. 6 Table ES4: Recommendations ................................................................................................... 8 ii | P a g e Terminal Evaluation: “Advancing the Nagoya Protocol in Countries of the Caribbean Region” Table 1: Proposed Country Visits ............................................................................................. 12 Table 2.
Summary: Components, Outcomes and Outputs ..................................................... 16 Figure 1.

Project Coordination Diagram................................................................................... 21 Table 3: Total Project Cost by Contributors ............................................................................ 22 Table 4.
Expenditure by Outcome/Output ............................................................................... 22 Figure 2: Theory of Change ....................................................................................................... 26 Table 5: Alignment of the Project to UN Environment MTS 2014-2017 ................................ 27 Table 6: Assumptions and Realities ......................................................................................... 35 Table 7: Co-financing Table ...................................................................................................... 37 Table 8: Total Project Cost ....................................................................................................... 37 Table 9: Financial Management Table ..................................................................................... 38 Table 10: Evaluation Criteria and Rating Table ....................................................................... 54 Table 11: Main Lessons Learned ............................................................................................. 57 Table 12: Recommendations .................................................................................................... 59 iii | P a g e Terminal Evaluation: “Advancing the Nagoya Protocol in Countries of the Caribbean Region” List of Acronyms and Abbreviations ABS Access and Benefit Sharing CARICOM Caribbean Community CBD GEF Convention on Biological Diversity Global Environmental Facility GEF-OFP GEF
 Operational Focal Points GIZ GR ILCs IUCN German Technical Cooperation (Deutsche Gesellschaft für Internationale Zusammenarbeit (GIZ) GmbH) Genetic Resources Indigenous and Local Communities International Union for the Conservation of Nature IUCN-ORMACC IUCN Regional Office for Mexico, Central America and the Caribbean MAT M&E MTE Mutually Agreed Terms Monitoring and Evaluation Mid-Term Evaluation NBSAP National Biodiversity and Action Plan NC NP National Coordinator Nagoya Protocol OECS Organization of Eastern Caribbean States PC PIC PIR PMT SMART TA ToC TOR Project Coordinator Prior Informed Consent Project Implementation Review Project Management Team Specific, Measurable, Achievable, Relevant, Timely Indicators Technical Assistance Theory of Change Terms of Reference 1 | P a g e Terminal Evaluation: “Advancing the Nagoya Protocol in Countries of the Caribbean Region” Executive Summary 1.
In July 2015 the UNEP received approval from the Global Environmental Facility for the Medium-Sized Project “Advancing the Nagoya Protocol in countries of the Caribbean Region” (GFL-5060-2711-4E67) for three years.

The project commenced in February 2016 and was completed in March 2019.
This is the final report of the Terminal Evaluation that took place between June and November 2019 and included field visits to three of the eight participating countries, Costa Rica, and Panama, the regional headquarters for the UNEP where the Task Manager is based.

2.
2.

The evaluation exercise was undertaken to assess project performance; provide evidence of results to meet accountability requirements; to promote operational improvement learning and knowledge sharing through results and lessons learned among UNEP and the project partners.

3.

This Medium Sized Project had a total budget of US$5,635,257.
That amount was divided between the GEF contribution of US$1,826,000 and US$3,809,257.00 in In-kind co- financing from governments of participating countries and other project partners.

The project had as the overall goal, “to support countries of the Caribbean to facilitate access to their genetic resources and benefit sharing in a fair and equitable way”.

4.
4.

This project was conceived out of a concern that Caribbean countries, though rich in biodiversity resources and traditional knowledge, and notwithstanding the passage of various environmental policies and legislation, still have challenges concerning institutional and legal arrangements at the national level to protect the environment and provide the basis for the implementation of basic provisions of the Nagoya Protocol.
Also, due to insufficient understanding of the implications of ratification of the Nagoya Protocol by politicians, limited knowledge of the resource base, and a dearth of experience operationalising the Access and Benefit Sharing mechanism associated with the Nagoya Protocol, progress was impeded, and they were unable to derive the benefits that would flow from ratification.
In response to these challenges, the project, which was undertaken in eight countries1 5. of the Caribbean, built around four main components.

These are as follows: • Component 1.

Identifying regional commonalities and assets and basic elements conducive to policy formulation.
The objective of this component is to build knowledge between countries of shared assets and technical information that may later be used by them to build cohesive policies at national level and collaboratively at regional level, such as a Scientific Study on Bioprospecting in the Caribbean Region and Stocktaking of main Applications of Traditional Knowledge in the region, which would be used to inform the formulation of National ABS Policies and a Regional ABS Policy.
The component also sought to identify, and where possible set up, sustainability mechanisms for supporting countries beyond the life of the project by creating networks and coordination mechanisms such as a Virtual ABS Policy Forum and project website.
• Component 2: Uptake and ratification of the Nagoya Protocol: The objective of this component was to provide support to participating countries in the development of the policy, legal, and regulatory frameworks governing ABS, to improve their understanding of the implications of the Nagoya Protocol ratification in terms of adjustments in the legal and institutional framework, assistance in the development 1 Antigua & Barbuda, Barbados, Grenada, Guyana, Jamaica, Saint Kitts & Nevis, Saint Lucia, Trinidad & Tobago 2 | P a g e Terminal Evaluation: “Advancing the Nagoya Protocol in Countries of the Caribbean Region” of draft ABS Bills and regulations, and in the development of regional strategic priorities for Nagoya Protocol implementation in the region.
• Component 3: Implementation of the Nagoya Protocol and establishing an enabling environment for the basic provisions of the NP.

The objective of this component was to assist countries in countries in developing the tools and guidelines required to implement the basic measures of the Nagoya Protocol.
These included assisting in building awareness among stakeholders that are key for Nagoya Protocol implementation to be effective, especially parliamentarians, officers of frontline ministries, indigenous communities and researchers.
Support will also be provided for the development of institutional agreements and administrative procedures for ABS Agreements such as Prior Informed Consent (PIC), Mutually Agreed Terms (MAT), and Benefit Sharing, and capacity building to create a Roster of Caribbean ABS Experts.
This component also sought to provide support in the drafting of methodologies that could be used by the countries for creating Traditional Knowledge and Genetic Resources inventories, and support strategies for the development of a regional database of research activities in the OECS and broader Caribbean region, linked to existing Clearing House Mechanisms (CHMs) or institutional web pages in the region.

• Component 4: Regional Coordination, technical support and capacity development.
This objective of this component was to create the capacity of participating countries to achieve a level of exchange and networking and allow the Executing Agency, the Implementing Agency, and the executing partners in countries and regionally to better coordinate actions to deliver assistance to the countries and come together in a coherent and united front on ABS related issues.

6.
6.

Though the design of the project successfully identified barriers to ratification and sought to have them addressed, through various interventions and pathways that would ultimately result in “Objectives”, it did not present an objective means of assessing how those impacts would be realised.
The intervention logic was based on a log frame analytical methodology, which adequately identified outputs and outcomes but was weak in terms of identifying the pathways through which Outputs would lead to Direct Outcomes, Intermediary State and Impacts.

In that regard, a Theory of Change was utilised in reconstructing the pathways through which intended impacts would be realised.

7.
7.

The evaluation exercise has found that the Project Management Team performed remarkably well in delivering on all its forty-one outputs, notwithstanding the relatively limited time-frame of thirty-six months allowed and the fact that the project was being implemented in eight countries and managed remotely by the Project Manager.
Much of that success was because the management structure involving the UNEP in its capacity as Implementing Entity, International Union for the Conservation of Nature (IUCN) Regional Office for Mexico, Central America and the Caribbean, acting as the Executing Agency, together with several partners and countries which made up the Regional Steering Committee, worked closely utilizing various modalities of communications (face to face and virtual meetings) to ensure that deliverables were satisfied, and consistent with the mandates of the project.
8.

However, a major shortcoming of the project was the fact that it set out to accomplish quite a lot (41 outputs) within a very short time frame (36 months).

Most significantly, the project listed some objectives, such as the adoption of ABS policies and legislative changes at the national level, which were outside the scope or ability of the project.

This was compounded by a set of assumptions that were not based on conditions or circumstances in the respective countries but, ambitious at best.
What this meant was that as time became a 3 | P a g e Terminal Evaluation: “Advancing the Nagoya Protocol in Countries of the Caribbean Region” factor, and some of those assumptions did not materialise in the anticipated time-frame, several of the direct outcomes did not materialise.

9.
9.

While outputs were achieved, knowledge of genetic resources confirmed and documented, awareness raised and tools provided to countries to point the way forward towards ratification and incorporation of Access and Benefit Sharing mechanisms, the number of countries that ratified the Nagoya Protocol increased from one to three.

However, not all Direct Outcomes and intended Impacts are clearly discernible, raising doubts regarding their sustainability, now that it has ended.

10.
10.

Though all outputs were seen as important in achieving the Direct Outcomes, some outputs can be deemed more important than others.

Those considered most important were the preparation of an inventory of genetic resources, the development of Access and Benefit Sharing Policy Documents, preparation of draft legislation, and the establishment of an online forum in realising the intended impacts.

11.

The project was successful in delivering on the Outputs and some of the Direct Outcomes.
Awareness of Access and Benefit Sharing implementation by identified stakeholders increased, and knowledge of the importance of genetic resources and traditional knowledge, particularly among Indigenous Persons and local communities, was heightened considerably; two additional countries, Antigua and Barbuda and Saint Kitts and Nevis, ratified the Nagoya Protocol; Access and Benefit Sharing mechanisms has been incorporated into legislation in Antigua and Barbuda, Prior Informed Consent (PIC) and templates for Mutually Agreed Terms (MAT) facilitating access to genetic resources and traditional knowledge are being used, and the use of online mechanisms for submitting licences have been adopted.
However, the absence of established national legislative and institutional frameworks and the slow pace at which the five countries that have not yet ratified the Nagoya Protocol are moving, suggests that sustainability and replication in the respective countries may not be realised.

12.
12.

While not all of the Direct Outcomes were realised, several countries, primarily OECS Member States, expressed a high level of interest in continuing with initiatives to raise awareness and ensure that the legislative architecture for the ratification of the Nagoya Protocol and implementation of the Access and Benefit Sharing mechanism is satisfied.
GIZ, IUCN and the Organization of Eastern Caribbean States (OECS), three of the partner organisations involved in the implementation, have indicated their willingness to provide further assistance if a request is forthcoming.

Meanwhile, GIZ is collaborating with the OECS to provide continued support to their sub-regional Member States.

13.
13.

The centrally coordinated management structure was hugely successful, particularly as it created opportunities for direct involvement of participating countries in providing project oversight, determining the direction of operations and establishing ownership of the project through their participation in the Regional Steering Committee.
This model of cooperation, utilising both direct, face to face contact, and virtual meetings, can serve as a model for the implementation of other regional initiatives.

14.

The training provided, both in terms of the establishment and operations of a regional ABS Clearing House Mechanism as well as the Trainer of Trainers initiative, was valuable in equipping a cadre of individuals with the skills needed for scaling up and passing on knowledge gained to other individuals in the respective countries.

15.
15.

The most significant complaint raised in respect of the failure to achieve greater success in respect of ratification of the Nagoya Protocol and the establishment of the 4 | P a g e Terminal Evaluation: “Advancing the Nagoya Protocol in Countries of the Caribbean Region” legislative and regulatory infrastructure was the time constraint.
While the outputs were delivered in thirty-six months, the engagement with High-Level stakeholders was inadequate due to the lack of time and insufficient financial resources.

16.

The adoption of policies and legislation, it was also noted, are activities that require additional time.
While it may also be argued that the efforts to engage with High-Level stakeholders could have occurred much earlier in the project implementation cycle, there was very little tangible evidence that, given the technical and financial constraints, without any additional assistance, any of the countries that have not ratified the Nagoya will now do so.

Any replication and scaling up is dependent on additional external support.

Table ES2: Evaluation Criteria and Rating Table Criterion A.
Strategic Relevance B.

Quality of Project Design C. Nature of External Context3 D. Effectiveness i.

Delivery of Outputs ii.

Achievement of Direct Outcomes iii.

Likelihood of Impact E. Financial Management F. Efficiency G. Monitoring and Reporting H. Sustainability i. Socio-political sustainability ii.

Financial sustainability iii.

Institutional sustainability I.

Factors Affecting Performance i.

Preparation and readiness ii.

Quality of project management and supervision iii.


Stakeholders participation and cooperation iv.

Responsiveness to human rights and gender equity v. Country ownership and driven-ness vi.
Communication and public awareness Overall Project Rating Rating2 HS S F MU HS U U MS HS S MU MU ML MU MS MU S MS S MU MS MS 2 The rating used for each section is as follows: HS - Highly Satisfactory, S – Satisfactory, MS - Moderately Satisfactory, MU - Moderately Unsatisfactory, U – Unsatisfactory, HU - Highly Unsatisfactory.
3 For Nature of External Context, the rating scale is changed to: HF - Highly Favourable, F - Favourable, MF - Moderately Favourable, MU - Moderately Unfavourable, U – Unfavourable, and HF - Highly Unfavourable.
(Note that this is a reversed scale) 5 | P a g e Terminal Evaluation: “Advancing the Nagoya Protocol in Countries of the Caribbean Region” Lessons Learned Table ES3: Main Lessons Learned Context: The most significant comment aired in respect of the failure of the project to realise all intended Direct Outcomes and Impacts was “insufficient time”.

There is validity to this assertion given the fact that several of those outcomes required changes in policy and regulations, both of which require time.
That time may be relative; however, when it requires the heightening of awareness of the NP and ABS mechanisms, the preparation and approval of policy documents, and the drafting and adoption of laws and regulations, that require the approval of Cabinet and the Parliament, these could take years to materialise.
Lesson #1: Projects intended to accomplish objectives that require structural changes and necessary supporting policy, legislative and regulatory must provide adequate time for those changes to be realised.

Application: UNEP must ensure that in the designing of projects, that sufficient acknowledgement is made of challenges inherent in adopting policy and necessary legislation and regulations.
In that regard, projects that require the adoption of new policies and legislation should identify and take into consideration best practices such that more practical implementation targets could be defined and executed.

Context: While there were not too many complaints about capacity constraints, it was, indeed a major concern as none of the countries was able to assign to the project, an officer tasked with the responsibility of ensuring its successful implementation.
Persons assigned to the project were already engaged with other activities.

This is not unusual, given the financial and staffing constraints faced by small countries.

Provision was made in the project to assign a Project Coordinator to work with each of the countries.

However, funding to engage such support was only provided for a total of 10 months out of the thirty-six-month implementation period of the project.
Lesson #2: Regionally executed projects should be provided with adequate technical and financial support to provide for the implementation of projects.

That support does not necessarily mean full-time engagement of a consultant (Project Coordinator), but the allocation of a Project Coordinator to adequately deal with issues such as engagement with stakeholders and raising public awareness-raising to obtain buy- in, particularly by High-Level stakeholders and decision-makers.
Application: Implementing Agencies should ensure that adequate Technical Support is provided for and clear sets of undertakings defined, to ensure that the efforts mainly target outputs which are likely to be the most difficult to implement.

6 | P a g e Terminal Evaluation: “Advancing the Nagoya Protocol in Countries of the Caribbean Region” Context: The project, both in the design and implementation phases, had a heavy bias towards the public sector.
While this is understandable, given the need to develop requisite policies, legislation and ABS implementation tools and templates, there was little if any provisions made for engagement with private sector entities whose commercial interests could have acted as a powerful driver of change.
Lesson #3a: Lesson #3b: Bioprospecting has great relevance and importance to both the overall economy and Indigenous Local Communities and can be a powerful driver of conservation and sustainable use if its commercial value can be readily identified and recognised.

However, for the bioprospecting regime to function effectively, the enabling conditions (institutional and legislative) must be established and made operational.
The private sector (commercial and research entities) as high-stake/ influence agents of change as well as key beneficiaries of a functional ABS system are integral to its implementation.

In this regard, initiatives to attract their involvement should be undertaken at an early stage or designed as a component of the project.

Application: National governments must create and establish the enabling conditions for bioprospecting to become operational.
This would include the necessary institutional and legislative frameworks and design incentives which could make bioprospecting attractive to private sector entities and research institutions.

Context: The implementation of projects in a multi-country (regional) format, as this one was, is attractive given the fact that they usually require the delivery of common outputs.
However, the high cost of travel between countries could limit the extent of participation by some participating countries, consume valuable resources that could be diverted to other uses, and further increase the carbon footprint associated with travel by project personnel.

Lesson #4: While the project only mandated the convening of three Steering Committee meetings, an additional three meetings were convened via the use of a virtual platform, resulting in cost-savings.
The project’s carbon footprint was also reduced as a result of the reduced travel and can become a feature of future projects that are similarly implemented on a multi-country format.

Application: Future projects can make similar use of virtual meetings.

Once this is recognised as an effective means of communications, and scheduled as part of project implementation, its effectiveness and participation rate could improve.

Context: The project was successful in completing all forty-one outputs.
However, by the time the last of these outputs were delivered, the project had come to an end, depriving countries of the key information which would have provided guided the development of potential revenue-earning bioprospecting initiatives.

Lesson #5a.
Lesson #5a.

Projects which have the potential to generate economic opportunities, particularly enhancing livelihoods for indigenous and local communities, should be delivered on 7 | P a g e Terminal Evaluation: “Advancing the Nagoya Protocol in Countries of the Caribbean Region” Lesson #5b time and where possible, set the stage for initiating projects in local communities which can be used as models for replication in other communities.
When projects are not results-oriented, there may be a risk of project teams becoming preoccupied with completion of activities and delivery of outputs, and checking them off, with no real consideration of the TOC and intended Impact Application Projects should always seek to identify and provide support to local communities in the establishment of community-based enterprises which can be used as models for scaling up and replicating in other communities.
Context: Awareness-raising, particularly among High-Level stakeholders, was seen as critical in achieving the uptake of the Nagoya Protocol.

However, adequate resources were not allocated for this initiative in terms of technical and financial support, and more importantly, engagement occurred too late in the project cycle to make it effective.
Lesson #6: Given the strategic importance of awareness-raising, adequate funding should be made available to ensure that the intended outcomes will be realised Application: In the design of projects, the critical path for the implementation of project activities should always be identified so that adequate resources could be applied to each of the activities to minimize delays in implementation.
Recommendations Table ES4: Recommendations Recommendation #1 UNEP Ecosystem Division and other international partner agencies must ensure that in the designing of projects, that sufficient acknowledgement is made of challenges inherent in adopting policy and necessary legislation and regulations.

In that regard, consideration should be given to the preparation of model guidelines, draft legislation and roadmaps to guide the eventual implementation.
Recommendation #2 Implementing Agencies should ensure that adequate Technical Support is provided for and clear sets of undertakings defined, to ensure that the efforts mainly target outputs that are most critical to producing the causal changes needed to achieve Impact (i.e.

a more results-oriented approach) Recommendation #3 For bioprospecting to become operational, National governments must create and establish the enabling conditions.
This will include the necessary institutional and legislative frameworks and the provision of incentives that could make bioprospecting attractive to private sector entities and research institutions.
Recommendation #4 UNEP Ecosystem Division should, in partnership with other international and regional institutions, provide additional assistance to countries, primarily aimed at raising awareness and providing support for the establishment and the delivery of community-based enterprises using some of recommendations contained in the Business Model for ABS prepared under the project.

8 | P a g e Terminal Evaluation: “Advancing the Nagoya Protocol in Countries of the Caribbean Region” 1.0 Introduction 17.
In July 2015 the Global Environmental Facility (GEF) granted approval for the undertaking of the Medium-Sized Project “Advancing the Nagoya Protocol in countries of the Caribbean Region”, hereafter referred to as the ABS-Caribbean Project for 36 months, with a total budget of US$5,635,257.

That amount was divided between the GEF contribution of US$1,826,000 and US$3,809,257.00 in In-kind co-financing from governments of participating countries and other project partners.
The project had as the overall goal “to support countries of the Caribbean to facilitate access to their genetic resources and benefit sharing in a fair and equitable way”.

18.

The Terminal Evaluation (TE) was undertaken after the project was completed to assess project performance (in terms of relevance, effectiveness and efficiency), and determine outcomes and impacts (actual and potential) stemming from the project, including their sustainability.
The evaluation has two primary purposes: ▪ ▪ to provide evidence of results to meet accountability requirements, and to promote operational improvement, learning and knowledge sharing through results and lessons learned among UNEP and the project partners (Organisation of Eastern Caribbean States (OECS), GIZ, International Union for Conservation of Nature- Regional Office for Mexico, Central America and the Caribbean (IUCN- ORMACC), UN CBD Secretariat, Caribbean Community (CARICOM).

19.
19.

In this regard, the evaluation sought to identify lessons of operational relevance for future project formulation and implementation, especially for follow-on projects pursuing the uptake of the Nagoya Protocol and implementation of key measures to make the protocol operational in the Caribbean.

20.
20.

Biodiversity conservation in the Caribbean is under threat, and high-valued genetic resources face insurmountable hurdles due to several factors including land degradation, climate change and invasive alien species.

While Caribbean countries are rich in traditional knowledge, several have enacted legislation for the protection of their environment and biodiversity, as well as laws governing forestry, land use and protected areas.
However, Caribbean countries lack the institutional and legal architecture to adequately manage their biodiversity resources and provide the basis for the implementation of basic provisions of the Nagoya Protocol.

Most significantly, due to limited resources and experience in the ABS area, there is still a lack of capacities (institutional, systemic and individual) within government, local communities and among all key stakeholders in this regard.
There is also a lack of awareness of issues concerning ABS as well as inadequate support to implement strategies and priority activities and limited resources for developing effective ABS measures and regimes.

21.
21.

The ABS-Caribbean project was implemented in eight countries4 of the region, all of whom are signatories of the Convention on Biological Diversity (CDB) and have made consistent efforts to meet their obligations under that agreement as expressed in national strategies, plans, regulations, and laws.
Most relevant to the proposed project objectives are the recent National Reports to the Convention on Biological Diversity, National Biodiversity Strategy and Action Plans (all of which make specific reference to some aspect of access and benefit sharing and traditional knowledge).
The Project is aligned with the 2014 – 2017 Medium-Term Strategy (MTS) Environmental Governance Sub-Programme, which has as its 4 Antigua and Barbuda, Barbados, Grenada, Guyana, Jamaica, St. Lucia, St. Kitts and Nevis, Trinidad and Tobago).
9 | P a g e Terminal Evaluation: “Advancing the Nagoya Protocol in Countries of the Caribbean Region” objective “to strengthen synergies and coherence in environmental governance, with a view to facilitating the transition towards environmental sustainability in the context of sustainable development”.

More specifically, it is aligned with Expected Accomplishment 1.

“Coherence and synergies” and Expected Accomplishment 2.
“Law”, both of which, are aimed at achieving Aichi Target 16 “Access and benefit-sharing”.

The Project also is aligned with GEF Strategic Objective BD-5 “Build Capacity on Access to Genetic Resources and Benefit Sharing (ABS)”; and with clear linkages to Aichi Targets (2) Biodiversity Value Integrated; and (16) Access and Benefit-Sharing; as well as with Articles 5, 6, 9 and 15 of the Nagoya Protocol.

22.
22.

This project was implemented by UNEP - Implementing Agency (IA) and executed - Executing Agency (EA) by the International Union for the Conservation of Nature - Regional Office for Mexico, Central America and the Caribbean (IUCN-ORMACC), with support from the UNEP Latin America and the Caribbean office.

In those various capacities, both the UNEP and IUCN had responsibility for overall coordination and project supervision.
A Regional Steering Committee (RSC) comprised of the IA and EA, along with representatives of all the participating countries and a select number of regional partners was selected to provide financial and technical oversight and ensure the project goals were achieved.
UNEP standard monitoring, reporting and evaluation processes and procedures were used to monitor and evaluate (M&E) progress, and this included regular reporting as well as a Mid-Term Management Review which took place (June – September 2017) at the mid-point stage (June 2017) of the project.

This Terminal Evaluation forms part of the overall M&E exercise being undertaken at the end of the project.

2.0 Evaluation Methods 2.1 Overall Approach 23.


The Terminal Evaluation involved an in-depth evaluation of the whole project utilising both desk and field research.

The desk research involved a literature review of the project documents and reports and interviews with key stakeholders with direct and indirect involvement in the project.

It was underpinned by a participatory approach whereby key stakeholders were kept informed and consulted throughout the evaluation process.
It also involved both quantitative and qualitative evaluation methods as appropriate to determine project achievements against the expected outputs, outcomes and impacts.

24.

The change process that the intervention was designed to achieve was assessed using the Theory of Change (ToC) analysis.

Given the fact that a ToC was not done during the project design phase, the Evaluator recreated a ToC using the project’s Result’s Framework and intervention logic.
In recreating the ToC, interviews with stakeholders in the participating countries were quite useful as it provided evidence of the change process initiated and the achievements which were realised.
The desk review part of the evaluation involved a review of relevant background 25. documentation, such as: ▪ project design documents (including minutes of the project design review meeting at approval) ▪ Annual Work Plans and Budgets or equivalent ▪ ▪ revisions to the project (Project Document Supplement) the logical framework ▪ project reports such as the six-monthly progress and financial reports 10 | P a g e Terminal Evaluation: “Advancing the Nagoya Protocol in Countries of the Caribbean Region” ▪ progress reports from collaborating partners ▪ relevant correspondence, including the Project Implementation Reviews (PIRs) ▪ supervision mission reports, etc ▪ GEF Tracking Tool ▪ Steering Committee Minutes ▪ quarterly expenditure reports ▪ co-financing records ▪ budget revisions ▪ technical reports, studies, publications, outreach material, etc ▪ Mid-Term Review/Evaluation of the project ▪ Terminal Report of the project including final project output ▪ audit report and final financial statements; and ▪ other reports deemed useful to the terminal evaluation of the project.
2.2 Interviews 26.

Interviews were undertaken with key stakeholders utilising various means of communication, including face-to-face interviews, telephone/Skype, and emails.

The interviews used as a guide, the semi-structured questionnaire developed by the evaluator, based on the template provided by the UN Evaluation Office (See Annex B).
The interviews also involved the asking of some strategic questions especially those pertaining to the ToC to determine the extent to which the drivers and assumptions, deemed most critical for the achievement of impact, were found to hold.

27.
27.

Among the key stakeholders interviewed were: ▪ UNEP Task Manager (TM) ▪ IUCN Project Management Team ▪ UNEP Fund Management Officer (FMO) ▪ Sub-Programme Coordinator ▪ Project Partners, including CARICOM, GIZ, CBD Secretariat, OECS ▪ National Focal Points, personnel in other Ministries (Attorney General’s Office, Foreign Affairs and Trade) ▪ National consultants ▪ Representatives of Indigenous Persons and Non-Governmental Organisations.

28.
28.

The face-to-face interviews required field visits to several countries given the regional physical spread, IA, EA and various partners (See Table 3).

The field visits, particularly with the Project Management Team was very important as it allowed for a focussed application of the various survey instruments and a complete interaction with the individuals directly involved in the implementation of the project.
It also provides an opportunity to observe first hand, what initiatives were undertaken which are direct outcomes of the project and obtain a better appreciation for the extent of the commitments expressed at the country or regional level for any scaling up or replication that is deemed necessary.

11 | P a g e Terminal Evaluation: “Advancing the Nagoya Protocol in Countries of the Caribbean Region” 29.
The evaluation engaged in assessing whether gender aspects were considered across the life of the project (design, implementation, monitoring reporting and evaluation) and by assessing the extent to which gender was considered and addressed.

That involved assessing the ProDoc to determine how gender issues were factored into the design of the project and how it was implemented in the field.

30.
30.

The evaluation assessed whether, and to which extent, challenges facing gender/marginalised groups were addressed by the Project.

It also assessed whether Project Monitoring and Reporting reflected gender-differentiated achievements/ challenges.

In that regard, the Project Implementation Reviews (PIR), MTR, Workshop Reports and Steering Committee minutes and other reporting material were consulted.

31.
31.

During the evaluation phase, most of the persons interviewed, including the representative of the IP, were women and their views in respect of Gender and Human Rights as it relates to the project were especially sought after.

2.3 Field Visits 32.

Based on the above, field visits were undertaken to Costa Rica, Panama, Antigua and Barbuda, Saint Lucia and Guyana.

Costa Rica was selected based on the fact that the regional headquarters for IUCN, the EA, is based there.
The IA and Task Manager are based in Panama, and given the fact that Panama is a mandatory stop on the route to Costa Rica, it made sense to stop there to meet with the IA Management team.

Visits to Costa Rica were scheduled for the last leg of the field.

This approach to evaluation allowed the Evaluator to cross-reference findings gained from reviewing the various project reports as well as comments from stakeholders in the various countries.

33.

The field visits commenced with a stop in Antigua.


Antigua and Barbuda was selected because of their advanced effort in ratifying the Nagoya Protocol and making the ABS regime an integral part of their national legislation.

From Antigua, the Evaluator visited Saint Lucia.

That selection was based on the fact that not only has Saint Lucia expressed great interest in the project but also the fact that the OECS’ headquarters is located there.
Both the OECS and GIZ have already indicated interests in utilising materials developed under the ABS Project and exploring opportunities for collaboration on a second phase to the project to allow for a continuation of the project in the OECS Member Countries.

It also offered an excellent opportunity to have discussions with ABS and GEF Focal Points and other stakeholders with interest in entrenching the provisions of the ABS regime.

34.
34.

Guyana was also chosen for a field visit given the fact that it has the largest indigenous population in the region.

Also, the CARICOM Secretariat, the regional governance Secretariat of 13 Member States of the Caribbean, is headquartered there.

CARICOM is of one of the earliest entities involved in biodiversity conservation and promoting adherence to the Nagoya Protocol and ABS regime.

See Table 3 below for the list of countries selected for field visits and the times spent there.
Table 1: Proposed Country Visits Country Antigua Saint Lucia Guyana Dates Visited September 9-10 September 11-12 September 15-19 No.

of Days 2 days 2 days 4 days 12 | P a g e Terminal Evaluation: “Advancing the Nagoya Protocol in Countries of the Caribbean Region” Country Costa Rica Panama Dates Visited September 22-24 September 25 – 27 No.

of Days 3 days 2 days 35.


The most significant limitation to the evaluation was the fact that the project did not have an explicit theory of change which therefore needed to be reconstructed by the Evaluator.

Some difficulties were also encountered in setting up interviews/meetings with key stakeholders.

36.

The Evaluator encountered difficulties in contacting some national ABS Focal Points.

Efforts to interview personnel at the Secretariat of the Convention on Biodiversity proved futile.
The IUCN Project Management Team (PMT) sent notices of the impending evaluation to all ABS Focal Points and Partner Organisations.

The Evaluator followed up with introductory letters (emails) requesting interviews (face to face or online).

While most of the countries responded and interview schedules established, no responses were received from Barbados and Jamaica.

37.
37.

The request for interviews with the partner organisation in question did not materialise because none of the persons contacted felt they were sufficiently knowledgeable about the project to speak on its behalf.

Their tenure with the project was brief, given the fact that they had just recently replaced someone more familiar with the project.

38.
38.

During country visits, the Evaluator interviewed stakeholders from several sectors (government, universities, contracted persons, ABS implementation trainees, NGO´s and representatives of Indigenous Persons).

While attempts were made to interview Indigenous Persons in Guyana regarding their participating in training activities and other aspects of the project, not much success was achieved due to logistical problems of either bringing them together or going out to their respective villages.
Either option would have involved time or transportation cost, both of which were in short supply.

In the end, the Evaluator met with one Indigenous Person, who participated in the workshops and is generally regarded as a voice of the indigenous community.

This person was able to offer considerable insight into the contribution that the project made to persons who are considered as having a wealth of information on genetic resources and traditional knowledge.

39.
39.

Stakeholders in countries not included for field visits were contacted.

However, not all responded to emails requesting interviews.

The result was that telephone interviews were conducted with the ABS Focal Points in Saint Kitts and Nevis and Grenada.

3.0 The Project 3.1 Context 40.

The habitat-rich tropical islands of the Caribbean islands support a wealth of biodiversity within their terrestrial and marine ecosystems, with a high proportion of species that are endemic, or unique.
Because of this uniqueness and high endemism, the Caribbean region is considered one of the world’s biodiversity hotspots, and their resources vulnerable to extinction.

These resources include about 11,000 plant species, of which 72 per cent are endemic and under threat primarily because 70% of the population live along the coast.
These Caribbean lives and livelihoods directly depend upon the use of their biological resources, both marine and terrestrial, given the economic and cultural importance these populations have attached to the use of these resources.

13 | P a g e Terminal Evaluation: “Advancing the Nagoya Protocol in Countries of the Caribbean Region” 41.
In addition to the above, the growing importance of Bioprospecting - the search for plant and animal species from which medicinal drugs and other commercially valuable compounds can be obtained- has heightened interest and awareness of the importance of these resources and the need for their protection and conservation.

The Caribbean region is also rich in Traditional Knowledge (TK) with a great variety of traditions that relate their folklore with biodiversity.
Amongst these practices, some of the most notorious are: traditional usage of fruits, plants and animals for medicinal purposes; traditional fishing methods, trapping, hunting and fishing techniques, traditional food culture and preservation techniques, handicraft and traditional environment preservation and conservation methods.

42.
42.

Although Caribbean Countries have enacted legislation for the protection of their environment and biodiversity, as well as laws governing forestry, land use and protected areas, there are still challenges concerning institutional and legal arrangements at the national level to protect the environment and provide the basis for accessing their genetic resources in the manner provided under the Nagoya Protocol.
Furthermore, due to limited resources and experience in the ABS area, there is still a lack of capacities (institutional, systemic and individual) within government, local communities and among all key stakeholders in this regard, and a lack of support to implement strategies and priority activities to ensure that issues concerning this discipline are adequately addressed at all levels.

All the eight participating countries5 are signatories to the Convention on Biological 43.
Diversity, and most have been exposed, albeit in a limited extent (through regional GIZ and Convention on Biological Diversity workshops), to varying levels of the ABS/Nagoya Protocol implementation requirements.

However, due mainly to their size, countries in the region have limited resources for developing effective ABS measures and regimes.

Of the eight participating countries, none of them had ratified the Nagoya Protocol, and only two (Antigua & Barbuda and Grenada) had signed it.
Notwithstanding this slow uptake of the Nagoya Protocol in the Caribbean region, they all have fully embraced the sustainable use of biological and genetic resources, through their unequivocal commitment to obligations contracted under the CBD.

44.
44.

The challenge for them is determining how to regulate access to genetic resources; how to implement the fair and equitable sharing of benefits resulting from the utilization of genetic resources and traditional knowledge associated with genetic resources; what enforcement measures will be required to ensure compliance by users, and what institutional and capacity building needs must be addressed to ensure all obligations under the NP are appropriately met.

45.
45.

The Caribbean ABS Project (hereafter referred to as ‘Project’) therefore sought to support countries to uptake, and where possible, ratify or accede to the Nagoya Protocol and take the first steps in implementation.

This was to be achieved by assisting countries in the development of regulatory frameworks for ABS, building capacity for its implementation, and sharing the experiences from these countries to catalyse similar processes in the Caribbean region.
This would require developing appropriate capacities and measures to ensure that countries have the requisite conditions to meet the obligations under the Nagoya Protocol.

3.2 Objectives and components 46.

The development goal of the Project is to support countries of the Caribbean to facilitate access to their genetic resources and benefit sharing in a fair and equitable way, in line with the Convention on Biological Diversity and the Nagoya Protocol.
Its main objective, 5 Antigua & Barbuda, Barbados, Grenada, Guyana, Jamaica, Saint Kitts & Nevis, Saint Lucia, Trinidad & Tobago 14 | P a g e Terminal Evaluation: “Advancing the Nagoya Protocol in Countries of the Caribbean Region” as stated in the Project Document, is “seeking uptake of the Nagoya Protocol and implementation of key measures to make the protocol operational in Caribbean countries”.

This objective is to be achieved through the implementation of four interdependent components.

47.
47.

This project is divided into four interdependent components as follows: ▪ Component 1.

Identifying regional commonalities and assets and basic elements conducive to policy formulation.

The objective of which is to build knowledge between countries of shared assets and technical information.

▪ Component 2: Uptake of the Nagoya Protocol.

The objective of this component is that participating countries take steps and decisions conducive to the ratification of the Nagoya Protocol.
▪ Component 3: Implementation of the Nagoya Protocol and establishing an enabling environment for the basic provisions of the NP, the objective of which is to assist countries in developing the tools and guidelines required to implement the basic measures of the Nagoya Protocol; and, ▪ Component 4.
Regional Coordination, technical support and capacity development; the objective of which is to bring together the participating countries at least twice during the life of the project to allow for the maximum level of exchange (e.g.

learned best practices etc.

), and networking.

Each component was to deliver several outputs and was expected to contribute to 48. several project outcomes, as presented in Table 2.
15 | P a g e Terminal Evaluation: “Advancing the Nagoya Protocol in Countries of the Caribbean Region” Table 2.

Summary: Components, Outcomes and Outputs Components Outcome Outputs 1.1.1 Scientific Study on Bioprospecting in the Caribbean Region produced and disseminated Outcome 1.1: Countries have a common understanding of shared assets/values, issues and needs on which to base ABS policy 1.1.2 Stocktaking of main Applications of Traditional Knowledge associated with biological resources.
1.1.3 Stocktaking of the expertise of non-regulatory organisations involved in promoting, protecting or documenting the Use of Biological Resources and associated Traditional Knowledge in the Caribbean.

COMPONENT 1: Identifying regional commonalities and assets, and basic elements conducive to policy formulation.

Outcome 1.2: Future directions of policy development for the region are defined.
Outcome 1.3: Countries understand their national assets/values and requirements in a regional context Component 2: Uptake of the Nagoya Protocol Outcome 2.1: National authorities take informed decisions on, and steps towards, the ratification of the protocol and future implementation.

1.1.4 Information Sheets about bioprospecting, common biological resources, Traditional Knowledge and related institutions in the Caribbean, distributed widely, both electronically and in print format.
1.2.1 ABS Policies produced as national drafts or updated versions of existing policies or strategies.

1.2.2 Draft Regional ABS Policy that describes a common vision and shared principles for ABS in the Caribbean.

1.2.3.

Summarised information items produced to disseminate policy-related progress at the national and regional levels, through websites, bulletins, annual reports and other means as relevant.
1.3.1 Project Website and Virtual Regional ABS Forum serving as openly accessible platforms for dissemination, exchanges, collaboration, and monitoring.

1.3.2 CBD COP side-event on Caribbean ABS with a high level of Caribbean participation and in collaboration with partner initiatives in the region.

1.3.3 Roster of ABS experts for the region.


1.3.4 Inter-institutional coordination included in ABS National Work Plans for at least five project countries 2.1.1 Assessment of existing national Legal Frameworks attending to legislative overlaps and mandates and the implications of ratification of the Nagoya Protocol completed and being used to prioritise interventions in project countries in support of the Nagoya Protocol.
2.1.2 Cabinet Papers produced to highlight legislative and regulatory needs and the benefits and opportunities of NP ratification.

2.1.3 Draft ABS Bill or Regulations formulated.


16 | P a g e Components Outcome Outputs Terminal Evaluation: “Advancing the Nagoya Protocol in Countries of the Caribbean Region” Component 3: Implementation of the Nagoya Protocol and establishing an enabling environment for the basic provisions of the NP Outcome 3.1: An enabling environment is created, which will lead to the implementation of the basic provisions of the Nagoya Protocol.
2.1.4 Nagoya Protocol ratification requests from the Executive Power to the Attorney General.

2.1.5 A Regional Strategy and Action Plan (2016-2021) that orients and converges regional efforts for ABS capacity building, sets common ABS capacity building goals for the Caribbean, collaboration and fundraising opportunities.
3.1.1 ABS Dialogues for Parliamentarians and In-Situ Exposure of Parliamentarians to Bioprospecting for awareness-raising regarding ABS and the Nagoya Protocol 3.1.2 Inter-institutional workshops for officials of ABS frontline ministries, as well as for consultations and awareness-raising with other relevant sectors.
3.1.3 Radio interviews and TV air-time discussions with researchers to highlight the risks, opportunities and challenges with ABS and bioprospecting 3.1.4 Posters and banners for targeted placement in all project countries.

3.1.5 Local radio spots produced and aired in indigenous language for ABS awareness-raising for Indigenous Peoples.

3.1.6 Operational Guidelines for Implementing ABS policies at the national level (institutional roles and responsibilities).
3.1.7 Standardized Training Manual for ABS Implementation developed and used among key line agencies engaged in ABS throughout the region.

3.1.8 At least twenty (20) trainers trained, with trainers identified on a Regional ABS Experts Roster, and available to provide expertise in the development of ABS capacity in the region.

3.1.9 Standardized Templates for ABS agreements for use through-out the Caribbean Region 3.1.10 Protocols for PIC developed with indigenous communities.
3.1.11 Standardized Methodology for the creation of national registers of marine and terrestrial biological resources.

3.1.12 Technical Assistance provided in the use of the ABS Clearing House as an exchange and monitoring mechanism (e.g.

for approved permits and agreements).


3.1.13 A searchable Regional Identification structured as web-based modules on Research into Caribbean Biological Resources and associated Traditional Knowledge, created on existing CHMs or institutional web pages (Linked to studies of Comp 1).
17 | P a g e Terminal Evaluation: “Advancing the Nagoya Protocol in Countries of the Caribbean Region” Components Outcome Outputs 3.1.14 Business Model for Countries of the Caribbean which highlights multiple economic scenarios possible through regulated bio-prospecting available as a tool for countries in their national ABS decision-making and negotiation processes.
4.1.1 Review to document and tally contributions and collaborations from national and regional institutions that contributed to ABS capacity building by promoting information and experience sharing and collaboration between institutions and countries.

Outcome 4.1: Countries share information and gain from the experiences of other countries.
4.1.2 Regional Project Inception Workshop completed with all project partners introduced to detailed project work plan, Logical project Framework, implementation timeline and procedures, monitoring and evaluation functions, and overall project governance.

Component 4.
Component 4.

Regional Coordination, technical support and capacity development Outcome 4.2: Effective project coordination and delivery, meeting agreed measurable outputs and indicators 4.1.3 Regional Project Closure Workshop for reviewing progress and planning of future activities, sharing lessons learned and best practices arising from the project.
4.1.4 Collaboration agreements reached with other key actors in the region resulting in joint planning and joint implementation of activities, avoidance of duplication, and optimisation in the use of resources available to the region.

4.2.1 National Work Plans (maximum 24 months) prepared and agreed for each project country on the basis of country “ABS Roadmaps”.

4.2.2 Technical assistance and feedback provided to all project countries for the implementation of their National Work Plans.
4.2.3 Project oversight and coordination structures established and functioning throughout the project lifetime.

4.2.4 Three (3) or more virtual or physical meetings carried out, involving Project Focal Points, national and regional organisations, and key partners, as appropriate, for project planning, coordination and oversight and to provide inputs to project implementation.
4.2.5 End-of-Project Survey to gauge, among project beneficiaries and partners, satisfaction levels regarding project results, management and technical assistance 4.2.6 Mid-Term Evaluation completed with project successes and lessons learned evaluated and used to inform the implementation of the rest of the project.

4.2.7 Terminal Evaluation completed with the achievement of project goals and objectives evaluated.
18 | P a g e Terminal Evaluation: “Advancing the Nagoya Protocol in Countries of the Caribbean Region” 3.3 Stakeholders 49.

Stakeholder participation was an integral part of the project, the objective being, to engage with a wide cross-section of individuals, in both the public and private sectors.
Among those targeted were parliamentarians, technical officers in ministries directly responsible for the implementation of NBSAP, local indigenous communities at the national levels and researchers, at both national and regional institutions, throughout the Caribbean.

50.

The identification and selection of stakeholders were subjected to a robust analytical exercise that commenced at the PIF preparation phase and later at the project preparation phase.
Stakeholders were identified using as a guide: ▪ ▪ those who could have the most relevant and direct impact on project activities and outcomes those who will be direct project beneficiaries.

▪ Regional institutions involved in promoting, protecting, documenting and coordinating the use of biological resources, and mainstreaming biodiversity conservation.

51.
51.

Among the High Level (HL) individuals targeted were parliamentarians and heads of departments in governments and national and regional organisations, as they were seen as being in a position of high power given their mandate and responsibilities.
More importantly, they were individuals who could help with facilitating the ratification of the NP by governments or even act as champions in assisting with and moving the policy formulation and institutional reform for ABS implementation forward, both at the national and regional levels.

52.

The consultations, undertaken over three months, involved one-on-one consultations with primary stakeholders and institutions in the project countries.
National CBD and GEF Focal Points were particularly targeted, as well as ABS Focal Points, where these have been appointed.

Research institutions were also approached as important stakeholders concerning bio- prospecting research and intellectual property.
Consultations with the CARICOM Secretariat and the GIZ Capacity Development Initiative were also conducted to understand the scope, challenges, and achievements to date of other initiatives in the region and to assess possible mechanisms for collaboration in future ABS interventions in the region.

53.

The project took special note of the interests of Indigenous People especially recognising their close relationship with genetic resources and as a source for and repository of TK.
The ProDoc sought to obtain an appreciation for their needs by consulting with local communities, and other institutions working closely with the ABS agenda, such as Iwokrama in the case of Indigenous Local Communities (ILCs) in Guyana.

54.

The issue of gender did not receive much attention in the design of the project as the only mention made was concerning “gender mainstreaming” in Component 4.
There it was stated that “gender considerations would be mainstreamed…, ensuring whenever possible equal opportunities for men and women in the implementation of all capacity building processes”.

While the Results Framework had, as one of its indicators the “number of gender-sensitive focus groups established particularly with indigenous people and local communities” (Results Framework 4.2.5), there was no mention made in any of the reports of what was achieved concerning gender.
19 | P a g e Terminal Evaluation: “Advancing the Nagoya Protocol in Countries of the Caribbean Region” 55.

Noticeably absent among the various stakeholder groups were entities from the private sector with a commercial interest.
This shortcoming has been noted, and in the project’s Final Report, mention was made of the need for increasing stakeholder awareness, especially within the less exposed sectors that the project was not able to reach, as well as at the executive level in the non-member countries of the Organisation of Eastern Caribbean States (OECS).

3.4 Project implementation structure and partners 56.
This project was implemented by UNEP - Implementing Agency (IA) and executed by the International Union for the Conservation of Nature - Regional Office for Mexico, Central America and the Caribbean (IUCN-ORMACC) - Executing Agency (EA), with support from the UNEP Latin America and the Caribbean Office.

In those various capacities, both the UNEP and IUCN had responsibility for overall coordination and project supervision.

57.
57.

A Regional Steering Committee comprised of the IA and EA, representation from all project countries, and selected regional partners to provide financial and technical oversight and ensure the project goals were achieved.

Also, recommendations produced by the Steering Committee were expected to improve implementation strategies, annual work plans and resources allocation budget and, when necessary, to adjust the project’s Result Framework (See Figure 1).
The UNEP standard monitoring, reporting and evaluation processes and procedures were used to monitor and evaluate (M&E) progress, and this included regular reporting as well as a Mid-Term Review which took place (June – September 2017) at the mid-point stage (June 2017) of the project.

This Terminal Evaluation forms part of the overall M&E exercise, undertaken at the end of the project.

20 | P a g e Terminal Evaluation: “Advancing the Nagoya Protocol in Countries of the Caribbean Region” Figure 1.
Project Coordination Diagram Implementing Agency UNEP Steering Committee Country representatives, UNEP, GIZ, CBD, IUCN, OECS + key partners Executing Agency IUCN National Focal Points Project Management Unit -Project Supervisor -Project Manager -Monitoring & Evaluation -Administrative Staff National consultants based in the project countries International consultants 3.5 Changes in design during implementation 58.
There were no events that affected the project’s scope or substantially altered the implementation of activities.

The Project had a planned start date of September 2015.

However, due to delays in the disbursement of funds, it commenced operations in February 2016.

Due to this initial delay, a no-cost extension request was made in the final year of the project.

That request was granted (Amendment No: 1/PCA/2015/010 GLF-5060-2711-4E67) allowing the Project to remain effective until 30 September 2019.
3.6 Project financing 59.

The total cost of the project was budgeted at US$5,635,257.

The original sum allocated in the budget for the GEF grant was US$1.826mn.

The actual expenditure of GEF Funds was US$1.786mn with the remaining sum of US$40,000 to be utilised to cover the cost of the Terminal Evaluation.
On average, expenditure on various project elements corresponded to the sums allocated within the original budget; however, there were a few exceptions, such as the following: ▪ Originally there was a budgeted amount of approximately US$200,000 for subcontracts to private consulting firms.

However, no funds were expended under this Sub- Component as a decision was taken to engage individual consultants instead.
Therefore, the funds were re-distributed mainly to the Training, and Equipment and Installation Components in subsequent budgets.

▪ Approximately US$187,850 more than originally estimated in the budget was spent on training and meetings and conferences ▪ Approximately US$81,819 more than the originally budgeted sum was spent on Consultants, both local and international.
21 | P a g e Terminal Evaluation: “Advancing the Nagoya Protocol in Countries of the Caribbean Region” ▪ Approximately US$82,167.11 was spent on miscellaneous items.

This is expected to increase by US$40,000 to $122,167.11 upon the submission of the Terminal Evaluation.

Adjustments were therefore made to project components in accordance with field 60. conditions, but the originally budgeted GEF Grand Total was not exceeded.

61.
61.

Concerning the Co-Financing Cash and In-Kind values, the budgeted contribution from Countries and institutions was originally set at US$3,809,257.

Information received from the Countries, and other participating institutions on Co-Financing contributions indicated that in total the co-financing was valued at US$2,904,999.61.

This value, however, did not include information from Jamaica or CBD.

Therefore, the In-Kind Contributions were approximately US$904,257 below the budgeted value.
Except for Barbados, Grenada and IUCN, the In-Kind contributions of the other countries and agencies was therefore significantly less than the originally budgeted values.

The total Project contributions are shown in Table 3 below.
Table 3: Total Project Cost by Contributors Parties GEF Trust Fund In-Kind Contribution from UNEP Cash Contribution from IUCN In-Kind Contribution from IUCN Third-Party Co Finance (Cash) Third-Party Co Finance (In-Kind) Total Cost of the Project Contribution ($ US) 1,826,000 90,000 289,993 211,081 12,000 3,206,183 5,635,257 Table 4: provides details of the actual expenditure from GEF compared with the original 62. budget for each Component/Outcome.

Table 4.
Expenditure by Outcome/Output Component/sub- component/output Estimated cost at design ($ US) Actual Cost/ expenditure ($ US) Expenditure ratio (actual/planned) Component 1: National Project Component2: Sub-Contracting Component 3: Training Component 4: Equipment and Installation Component 5: Miscellaneous GRAND TOTAL 1,085,043 199,163 362,400 38,600 140,794 1,826,000 1,109,869 0 550,250 43,714 82,167* 1.02 0.00 1.52 1.13 0.58/0.87* 1,786,000** 0.98/1.00** *The Component Miscellaneous will be increased by US$40,000 upon completion of this Terminal Evaluation.
Therefore, the Expenditure Ratio is expected to increase to 0.87.

**The Grand Total is expected to increase by US$40,000 upon submission of the Terminal Evaluation.

Therefore, the Expenditure Ratio for this total is anticipated to increase to 1.0 at that time.

22 | P a g e Terminal Evaluation: “Advancing the Nagoya Protocol in Countries of the Caribbean Region” 4.0 Theory of Change 4.1 Reconstructed Theory of Change at Evaluation 63.
Given the fact that no ToC was developed at the design phase of the project, one was reconstructed at the Mid-term Review.

Like at the MTR, a reconstruction was undertaken as part of this TE.

This reconstruction was facilitated through the use of the intervention logic in the Project Document (ProDoc), the Results Framework and the Project Identification Form (PIF) that was prepared at the project design stage.
While the Results Framework was useful in identifying Outputs to Outcomes, the Outcomes were not directly linked to the Outputs.

These also had to be regrouped in reconstructing the ToC (see section 4.2 below).

64.
64.

Following interviews conducted by the Evaluator, a further review of the reconstructed ToC was undertaken to help establish the project’s logical structure, identify causal links between Outputs, Direct Outcomes, and the Intermediary States, through to the realisation of Intended Impact, and determine the likelihood that the intended results will be realised.

Figure 1 shows the reconstructed ToC at evaluation.

4.2 Causal Logic: Outputs to Direct Outcomes 65.


The original ProDoc contained 4 Components under which 41 Outputs were identified.

From there, it went on to identify 6 Direct Outcomes (DO) resulting from those outputs.

In the reconstructed ToC, these 41 Outputs were regrouped into 16 Outputs, as a way of rationally clustering them based on their inherent commonalities and because of the logic of coordinating a large number of outputs and dispersed regional scope.

66.
66.

The Outcomes, the direct intended results stemming from the outputs of the project, were also regrouped to reflect those commonalities as well as to define the logical pathway clearly.

These Outputs show a clear linkage or the necessary activities that have to be undertaken to achieve the DO.
These Outputs included the undertaking of necessary studies to assess the level of the region’s stock of biological resources, bioprospecting activities, use and application of Traditional Knowledge (TK), and assess what capacities exist (human and technical) to facilitate and enable an environment for the sustainable use and sharing of genetic resources.

67.

For changes to happen along the causal pathways, certain external conditions and factors also needed to be in place.
Those external factors are “Assumptions” and “Drivers”.

Assumptions are external conditions necessary for project results to lead to next-level results; they are factors over which the project has no control.

Drivers are those factors that are necessary for the project results to lead to the next level result, over which the project has a certain level of control or influence and will be satisfied once present.
The ProDoc detailed several assumptions, which would provide the basis for the delivery of outputs to outcomes; however, no drivers were identified.

These drivers were, therefore, reconstructed in the preparation of the ToC.

68.
68.

The Assumptions are characterized by the strong support provided by all the stakeholders, including partner organizations, both at the national and regional levels to ensure that decision- makers, as well as users of GR and TK, are informed and aware of the need for new policies, legislation, regulations and other tools as they relate to the ABS agenda.
The realisation of these outcomes is also dependent on the assumption that partner organisations are willing to share 23 | P a g e Terminal Evaluation: “Advancing the Nagoya Protocol in Countries of the Caribbean Region” data and information on genetic resources and stakeholders are receptive to that information and willing to take action based on that information.

69.

For DOs to be realised, several internal factors must be seen to have some influence to allow the project to move beyond direct outcomes.
These internal factors or drivers (D1 and D2, influencing the causal link between Outputs and Direct Outcomes) include demonstrated interest in bioprospecting, technical understanding of the importance of common genetic resources which exist in the region, and continued commitment of the countries to the CBD and the NP, continue to be a driving force for scientists, politicians and their constituents to advocate for timely NP implementation.
A review of the reports submitted and discussions with the parties involved indicated that 70. all outputs were completed.

4.3 Direct Outcomes (DO) to Intermediate State (IS) 71.

The DOs are expected to lead to two Intermediate States (IS1 and IS2), which were the reformulated objectives and goals of the project as outlined in the ProDoc.
IS1 has been reformulated from the original goal to reflect the changed state - Caribbean countries facilitate access to their genetic resources and benefit sharing in a fair and equitable way, in line with the Convention on Biological Diversity - consistent with the CBD.
Likewise, IS2 has been reformulated from the original objective to reflect the changed state – the Nagoya Protocol and key measures are made operational in the Caribbean countries - resulting from operationalising the Nagoya Protocol and ABS.
Achieving these changed conditions will be influenced by the drivers DO1 – “technical capacity and finances improved to facilitate data collection and monitoring” and DO2 which sees that the “legal framework in most Caribbean countries is expanded to explicitly allow for PIC and participation of ILCs in biodiversity access agreements”.
That changed state is also dependent on the assumption that “National ABS policies, regulatory and legislative instruments and the Regional Strategy and Action Plan is adopted and used at the national level” (A2).

4.4 Intermediate State to Impact 72.

The ultimate impact is that local and global benefits deriving from ABS implementation are maximised through effective and transparent use of genetic resources and more assertive conservation of globally significant Caribbean biodiversity.
Achieving that impact is strongly linked to the assumption that the ABS regimes which have been institutionalised by countries in the region are operational and effective (A3) and that the regional institutions are actively involved in promoting, the sustainable use of biological resources and associated Traditional Knowledge (D3).
Because of these drivers and assumptions and the various outputs initiated by the project management team and other stakeholders, countries will inevitably realise benefits from the use of their GR and TK.

More importantly, awareness of the value of the resource will result in greater efforts at conservation with benefits manifested at the national, regional and global levels.
See Figure 2: below for a graphical indication of the causal pathways from output to outcomes through to intermediate states and impacts.

73.
While there is little doubt that the project was successful in delivering on the Outputs, and some of the Direct Outcomes, (e.g., level of awareness and knowledge of ABS implementation among stakeholders increased; two additional countries, Antigua and Barbuda and Saint Kitts and Nevis, in addition to Guyana ratified the Nagoya Protocol; ABS mechanisms incorporated into 24 | P a g e Terminal Evaluation: “Advancing the Nagoya Protocol in Countries of the Caribbean Region” legislation in some countries; Prior Informed Consent (PIC) and Mutually Agreed Terms (MAT) templates granting access to genetic resources being used; and, the use of online mechanisms for submitting licences adopted, were successfully realised, the absence of an established institutional framework for fully operationalising the ABS mechanism, and the slow pace at which the other five countries6 are moving with efforts to sign and ratify the NP suggests that sustainability, in the absence of continued technical and financial, may not be realised in the respective countries and consequently the Impact not realised.
6 Of the eight participating countries Barbados, Grenada, Jamaica, Saint Lucia, and Trinidad & Tobago have yet to ratify the Nagoya Protocol.
25 | P a g e Figure 2: Theory of Change Terminal Evaluation: “Advancing the Nagoya Protocol in Countries of the Caribbean Region” 26 | P a g e Terminal Evaluation: “Advancing the Nagoya Protocol in Countries of the Caribbean Region” 5.0 Evaluation Findings 5.1 Strategic Relevance 5.1.1 Alignment to the UNEP Medium Term Strategy (MTS) and Programme of Work (POW) The Project 74. is aligned with UN Environment Sub-Programme Environmental Governance objective: “The capacity of States to implement their environmental obligations and achieve their environmental priority goals, targets and objectives through strengthened laws and institutions is enhanced”.
More particularly, it is aligned with the Medium-Term Strategy (MTS) 2014-2017, Expected Accomplishment 1 and 2, as summarised in the following Table 5.
Table 5: Alignment of the Project to UN Environment MTS 2014-2017 Expected Accomplishment (EA) MTS 2014-2017, Sub-programme Environmental Governance Contribution of the Project Expected Accomplishment 1 (EA 1): Coherence and synergies: The United Nations system and the multilateral environmental agreements, respecting the mandate of each entity, demonstrate increasing coherence and synergy of actions on environmental issues.
Support for the adoption and ratification of the Nagoya Protocol Preparation and drafting of the ABS policy, and related legislation, which could be integrated into national laws Expected Accomplishment 2 (EA2) Law: The capacity of countries to develop and enforce laws and strengthen institutions to achieve internationally agreed environmental objectives and goals and comply with related obligations is enhanced.
Support for the drafting of the ABS Policy, Legislation and Regulations Overall support for the establishment and operations of competent national institutions including the Regional Clearing House 5.1.2 Alignment to UNEP /GEF Strategic Priorities 75.
The project is in full alignment with Articles 5, 6, 9 and 15 of the Nagoya Protocol which speak to the need for countries to derive the benefits arising from the utilisation of genetic resources in a fair and equitable way with the Party providing such resources.

It also laid the foundation for such sharing to be upon mutually agreed terms, and subject to the prior informed consent (PIC) of the Party providing such resources.

76.
76.

The project, though it made no specific mention of the Bali Strategic Plan and South- South Cooperation, one can draw from its configuration that it is linked to GEFs Strategic Objective BD-5.
This is reflected in its acknowledgement of the need to safeguarding the genetic diversity of global importance and will specifically contribute to the conservation and sustainable use of 2.6 per cent of the world’s 300,000 plant species and 3.5 per cent of the world’s 27,298 vertebrate species, all of which are endemic to the Caribbean.

See Paragraphs 26-29 of the ProDoc.


The project also sought to address many of the cross-cutting issues identified in the Bali Strategic Plan, such as strengthening national institutions, and the development of national law and regulations in compliance with the CBD.

27 | P a g e Terminal Evaluation: “Advancing the Nagoya Protocol in Countries of the Caribbean Region” 5.1.3 Relevance to Regional, Sub-regional and National Environmental Priorities 77.
The Project had great strategic relevance as it sought, not only to address challenges concerning the conservation of biodiversity but also to create a capacity for countries of the region to harness their genetic resources to enhance livelihood.

The Caribbean, as noted in the ProDoc, is rich in genetic resources, but these are under great threat due to land degradation, climate change, pollution from nutrients, unsustainable use and invasive alien species.
The region is also rich in traditional knowledge with a great variety of traditions that relate to the use of fruits, plants and animals for medicinal purposes.

However, many of the countries, lack the institutional, legal, and financial capacity to establish the mechanisms which would allow them to utilise their genetic resources in a manner that would not only meet their goals of conservation and sustainable development but also exploit them based on equity.

78.
78.

The project is consistent with national strategic priorities and plans as defined in their NBSAPs and in their 4th and 5th National Country Reports to the CBD in which they express the need to ratify the Nagoya Protocol and institute the ABS mechanism and address any further losses in their abilities to harness their genetic resources.
It also had great relevance from a regional perspective as it sought to build on initiatives commenced by the CARICOM Secretariat, in partnership with GIZ, to build capacity in the region for the mainstreaming of biodiversity.
Under the project “Capacity Building related to the Implementation of Multilateral Environmental Agreements (MEAs) in African, Caribbean and Pacific (ACP) Countries Project” several persons were provided training in the Drafting of Legislation for the Implementation of the Nagoya Protocol on Access and Benefit Sharing, and production of a Model ABS Legislation.
This is addressed in Section 3.6, Paragraphs 57-59 of the ProDoc, while regional linkages are described in Section 2.7 Criteria Summary Assessment Rating Strategic Relevance Overall Strategic Relevance rating Alignment to the UNEP Medium- Term Strategy (MTS) and Programme of Work (POW).

There is alignment to the UNEP MTS and POW Alignment to the UNEP/GEF/Donor Strategic Priorities.
There is alignment to the strategic priorities of UNEP/GEF/Donor Relevance to Regional, Sub-regional and National Environmental Priorities The project is highly relevant to Regional, sub-regional and National Environmental Priorities Complementarity with Existing Interventions The project was an ideal complement to other initiatives started by other regional organisations in collaboration with other international entities HS HS HS HS HS 5.2 Quality of Project Design 79.
The design of the project was based on a detailed problem analysis.

It highlights the significance of Caribbean biodiversity and genetic resources and the challenges facing the respective countries in terms of threats to biodiversity conservation, the absence of mechanisms to monetise and obtain the best value for their genetic resources and TK, particularly among local communities.

28 | P a g e Terminal Evaluation: “Advancing the Nagoya Protocol in Countries of the Caribbean Region” 80.
The design of the project took note of the fact that only one participating country (Guyana) had ratified the Nagoya Protocol, and only two (Antigua and Barbuda and Grenada) had signed it.

However, several had expressed an interest in supporting the Protocol’s objectives and ultimately in ratification.
Also, some countries had participated in various regional activities such as ABS workshops coordinated by the CBD and the ABS-initiative; and those participating of the project “Capacity building related to Multilateral Environmental Agreements in African, Caribbean and Pacific countries – ACP MEAs Phase II” led by UNEP.

81.

The ProDoc did not contain a ToC, but instead, had a Results Framework or Logical Framework.
While that framework is useful in identifying outputs and providing valuable indicators and detailing the results to be achieved, there is growing consensus that its inability to provide a clear, straightforward idea of the logical pathway from Outputs to Outcomes means it has limited usefulness in showing a clear pathway towards achieving outcomes.

82.

A major shortcoming of the project was the fact that it set out to accomplish quite a lot (41 outputs) within a very short time frame (36 months).
Most significantly, the project listed some objectives, such as the adoption of ABS policies and legislative changes at the national level, which were outside the scope or ability of the project.

This was compounded by some ambitious assumptions that were not based on conditions or circumstances in the respective countries.

What this meant was that as time became a factor, and some of those assumptions did not materialise in the anticipated time-frame, several of the direct outcomes did not materialise.
5.2.1 Strengths 83.

The project correctly identified and sought to address some of the systemic and structural challenges impeding progress in biodiversity conservation and the adoption of measures that would permit access to and sustainable use of genetic resources in a manner consistent with the provisions of the Nagoya Protocol and the ABS Mechanism.

84.
84.

The adoption of a regional approach to seeking ratification of the NP and ABS regime and the preparation of a regional strategy on ABS was sensible from a cost standpoint and rational, given the commonality of issues identified in the problem analysis.

The regional approach contributed to addressing the capacity constraints which limited the ability of most countries of the region to take full advantage of the value of genetic resources.

85.
85.

The centrally coordinated management structure allowed the IA and EA the opportunity to utilise limited resources to simultaneously benefit several countries by undertaking several studies which provided the background information on bioprospecting and developing the framework on which and ABS mechanism could be established in the participating countries.

86.

The choice of IUCN as an execution Agency was strategic and tremendously beneficial.
IUCN is a well-established, international organisation with a long track record of promoting, biodiversity conservation and supporting the ratification and establishment of ABS/ Traditional Knowledge mechanisms in other regions of the world.

IUCN had just concluded the undertaking of a similar project “LAC ABS: Strengthening the implementation of ABS regimes in Latin America and the Caribbean” (GFL-2328-2740-4C08), in eight Latin American countries.
More importantly, IUCN, given their vast network of resource personnel, was able to draw on the experience and to facilitate access to the well-resourced IUCNs Law Centre.

29 | P a g e Terminal Evaluation: “Advancing the Nagoya Protocol in Countries of the Caribbean Region” 87.

The project was built on initiatives commenced by CARICOM and GIZ and was aimed at strengthening the capacity of participating countries to ratify or accede to the Nagoya Protocol.
It also aimed to support countries in preparing for its implementation as well as widening awareness and deepening the understanding of the opportunities and benefits that could arise from participation in the global ABS framework.

88.

Only two countries, Antigua and Barbuda and St. Kitts and Nevis, ratified the NP during the tenure of this project.

Antigua and Barbuda submitted their instrument of ratification to the CBD in December 2016.

St. Kitts and Nevis submitted theirs on May 2018.


Guyana, who had ratified the NP in 2015, benefit from the development of the ABS mechanisms which were developed under the project.

Grenada and Saint Lucia have also indicated their intention to proceed with the ratification of the Nagoya Protocol and the respective Ministries have been provided with Draft Policy Documents for consideration by their respective Cabinets.

89.
89.

The project acknowledged the importance of indigenous people and gender issues in the conservation and management of genetic resources and further sought to ensure that gender consideration is mainstreamed in all capacity building activities.

90.

Several other positives included opportunities for networking, training of trainers on ABS to facilitate ongoing training at the national level, and convening workshops aimed at facilitating the use of the ABS Clearing House Mechanism and contract drafting.
91.

There was great support expressed for the Regional Operational Guidelines and Action Plan for implementing ABS policies as these would provide means for ensuring that prospective bio prospectors do not playoff one country against the other.

5.2.1 Weaknesses 92.

The project sought to do much in a very short period.
The requirement that several countries should have ratified the Nagoya Protocol and have legislation giving effect to the ABS/ Traditional Knowledge agenda within the life of the project was always going to be difficult given the fact that this was beyond the influence of the PMT.
While several other participating countries have expressed their desire to ratify the NP, that is a decision which has to be made by the Cabinet in the respective countries, and requires at best, amendments to existing legislation or new legislation.

93.

Insufficient allowance was made for the limited capacity constraints of the respective countries.
The National Focal Points and ABS Focal Points in the respective countries were already tasked with various assignments, were insufficiently resourced, and had difficulty providing the necessary support at the national levels.

While allowance was made to address that specific constraint through the provision of Technical Coordinators to the respective countries, that support was only for a total of ten months (part-time) engagement over 24 months.

94.
94.

The project, both in the design and implementation phases, had a heavy bias towards the public sector.

While this is understandable, given the need to develop requisite policies, legislation and ABS implementation tools and templates, there was little if any provisions made for engagement with private sector entities whose commercial interests could have acted as a powerful driver of change.
During actual implementation, efforts were made to engage with the private sector, particularly research entities such as the University of the West Indies (UWI), tour operators and non-governmental entities who work closely with indigenous persons.

However, to sustain and drive interest in bioprospecting, interest from commercial entities capable of investing in research is required.

30 | P a g e Terminal Evaluation: “Advancing the Nagoya Protocol in Countries of the Caribbean Region” 95.
While gender was mentioned as an issue which should be mainstreamed and equal opportunities ensured, there was no stated plan outlined for ensuring or monitoring its implementation.

There were no provisions for reporting on gender mainstreaming until it was included in the 2018/2019 PIR.

Notwithstanding, the PMT sought to ensure gender issues were addressed and reported on in the last PIR submitted.
Criteria Summary Assessment Quality of Project Design Barriers and constraints were correctly identified, and for the most part, the project sought to have them addressed.

There were some minor omissions in the design such as the limited support in respect coordination at the national levels, the omission of hurricanes as a major risk factor, the absence of an objective means of measuring the achievements of outcomes and impacts and just a slight reference to gender.
However, this was addressed during implementation.

Rating S 5.4 Nature of the External Context 96.

No mention was made in the ProDoc of any conflict or likelihood of conflict.

While this may have been an oversight, the Caribbean is generally a peaceful area and highly unlikely to be a staging area of conflict of any serious kind.

Elections and change of government are part of the political landscape of the Caribbean.
These electoral changes don’t usually negatively impact on projects, though the change of government may result in delays in implementation due to the realignment of ministries and some staff.

None of those risks materialised although elections were held in several countries and changes of government.

97.
97.

Despite the Caribbean region being situated in the hurricane belt and being the recipient of major storms and hurricanes on an annual basis, no mention was made, in the risk analysis, of the potential impacts of hurricanes and other natural disasters.

First, the Caribbean region is in the hurricane belt and prone to severe hurricanes.
Secondly, as was experienced in 2017, two Category 5 Hurricanes, Irma and Maria, wrought severe destruction (infrastructure, agriculture) on Antigua and Barbuda, and Dominica.

Not only do hurricanes and natural disasters destroy infrastructure and claim lives, but it could set an economy back because of the destruction of crops and disruption of air and sea communications.
In respect of this project, though none of the countries suffered the direct effects of the hurricanes which went through the Caribbean, it did cause some disruption to the communications with the result that air transportation in several of the participating islands was interrupted.

98.
98.

The Nature of the External Context is rated Moderately Favourable (F)7 Criteria Summary Assessment Nature of External Context External factors such as natural disasters could cause disruptions in travel and communications.

Two Rating Favourable 7 For Nature of External Context, the 6-point rating scale is changed to: Highly Favourable = 1, Favourable = 2, Moderately Favourable = 3, Moderately Unfavourable = 4, Unfavourable = 5 and Highly Unfavourable = 6.
(Note that this is a reversed scale) 31 | P a g e Terminal Evaluation: “Advancing the Nagoya Protocol in Countries of the Caribbean Region” Criteria Summary Assessment Rating Category 5 hurricanes struck the region during project implementation, but it had negligible impacts.

5.5 Effectiveness 99.


The Caribbean ABS project had great relevance and effectiveness given the state of biodiversity conservation and the need to establish protocols which would determine the conditions under which access to those resources would be permitted.
The objectives, as stated in the ProDoc was quite clear in its intended purpose which was to support countries of the Caribbean to facilitate access to their genetic resources and benefit sharing in a fair and equitable way, in line with the CBD and the Nagoya Protocol.

Its main objective is stated in the Project Document as “seeking uptake of the Nagoya Protocol and implementation of key measures to make the protocol operational in Caribbean countries”.
This objective was to be achieved through the implementation of four interdependent components, all of which had several outputs which, through implementation would result in a set of outcomes that would enable the project to achieve its intended impacts.

5.5.1 Delivery of outputs 100.

The original ProDoc contained 4 Components under which 41 outputs were identified.
In the reconstructed Theory of Change, these 41 outputs were regrouped into 16 Outputs (See Figure 2: ToC), as a way of rationally clustering them based on their inherent commonalities and also because of the logic of coordinating a large number of outputs and dispersed regional scope.

The Outcomes, the direct intended results stemming from the Outputs of the project, were also regrouped to reflect those commonalities as well as the logical pathway.
These Outputs show a clear linkage or the necessary preconditions that have to be undertaken to achieve the Direct Outcomes.
These Outputs included the undertaking of necessary studies to assess the level of the region’s stock of biological resources and bio-prospecting activities, and use and application of Traditional Knowledge, and an assessment of what capacities exist (human and technical) to facilitate and enable an environment for the sustainable use and sharing of genetic resources, and the preparation of ABS policies, legislation and regulations based on a commonly defined.

101.
101.

The Mid-term review noted that, notwithstanding a large number of Outputs, the project was well on its way towards completing all its Outputs.

The project reports confirmed that the outputs were completed.

The Evaluator also confirmed this following the review of documents made available for the review process.

The successful completion of some of these outputs was integral to the attainment of outcomes.

102.
102.

Each one of these outputs was aimed at contributing to the achievement of the planned outcomes under the four project components in which the project was divided.

Evaluative evidence gathered through documents reviewed, supported by interviews with project personnel, allowed the evaluator to conclude that these outputs were significant, and had to be realised if the outcomes were to be attained.

103.
103.

According to the countries, some of the main outputs of the project included the preparation of the regional bioprospecting study, the preparation of draft national and regional ABS policies and legislative guidelines documents and the establishment of the online platform and network for sharing of information.
This preparation of the Regional ABS 32 | P a g e Terminal Evaluation: “Advancing the Nagoya Protocol in Countries of the Caribbean Region” Guideline document was seen as important by the countries as it would provide several benefits as follows: ▪ Help countries in the Caribbean region avoid competing among themselves.
▪ Increase bargaining power for the region and help to avoid a race to bottom between countries seeking to get the benefits derived from the utilisation of genetic resources and traditional knowledge.

▪ Similar ABS frameworks will make it easier to navigate the access procedure, increasing researchers’ trust in the region and increasing the chances of receiving benefits derived from the use of genetic resources.
▪ Easier for countries to monitor compliance of the users if the countries are willing to maintain open channels of communication between each other.

▪ Help to devise new and innovative ways to deal with the issue of transboundary resources and traditional transboundary knowledge.

This issue can be of vital importance in a region as connected as the Caribbean.

104.

All forty-one of the Outputs were completed.


While it is fair to state that all outputs were completed, there is still no clear indication that all of the outputs have achieved or are likely to achieve the intended Direct Outcomes.

The Delivery of Outputs is rated Highly Satisfactory (HS).

5.5.2 Achievement of direct outcomes 105.

The achievement of Direct Outcomes was dependent on several drivers and assumptions occurring or holding.
The drivers are characterized by the strong support provided by all the stakeholders, including partner organizations who provided support, both at the national and regional levels to ensure that decision-makers, as well as users of genetic resources and TK, are informed and aware of the need for new policies, legislation, regulations and other implementing tools as they relate to the ABS agenda.
The realisation of these outcomes was also dependent on the assumption that partner organisations and ILC are willing to share data and information on genetic resources and stakeholders are receptive to that information and willing to take action based on that information.

106.

For DO to be realised, several internal and external factors must be seen to have some influence to allow the project to move beyond direct outcomes.
These external factors (Drivers) included the preparation of policy documents embracing the sustainable use of genetic resources and the assumption that stakeholders, including the legislators and implementing partners, take an interest in ABS and embrace the information conveyed and the messages delivered by the various media.

107.
107.

While it is fair to state that awareness of the knowledge and importance of genetic resources has been heightened and decision-makers are more aware of the importance of bioprospecting, only three countries have ratified the Protocol (Antigua and Barbuda, Guyana, and St. Kitts and Nevis).

Also, none of the other countries has revised their legislative framework to explicitly allow for PIC and participation of ILCs in biodiversity access agreements (DO2).
That absence of uptake is also evident in the fact that no significant strides have been made in terms of increasing technical capacities and finances to facilitate data collection (DO3) and there have been no strengthening of ABS legal and governance structures to facilitate bioprospecting in the region (DO5).

Also, the operationalisation of a harmonised regional mechanism to promote the exchange of relevant information among countries (DO6) is yet to materialise.
33 | P a g e Terminal Evaluation: “Advancing the Nagoya Protocol in Countries of the Caribbean Region” incorporating ABS/ Traditional Knowledge mechanisms 108.

Antigua and Barbuda have not only ratified the NP but have also revised their Environmental Act, local legislation.

Meanwhile, Guyana, Grenada, Saint Lucia and St. Kitts and Nevis have indicated plans are underway to make similar changes to their legislation.
Also, Guyana has instituted an online system for the submission of application and issuing of permits to access biological resources.

into 109.

Taking all of the above into consideration, there is little doubt that some countries made strides in helping the project achieve some of the Direct Outcomes are yet to materialise.

However, not all the Direct Outcomes were realised.
While it is stated that some countries have expressed a desire to continue working towards achieving these outcomes, at this stage of the evaluation, when the project has come to an end, these initiatives have not yet care yet to outcomes are yet commenced.

The rating for achieving direct outcomes is Moderately Unsatisfactory (MU).

5.5.3 Likelihood of impact 110.


The DOs were expected to lead to two Intermediate States (IS1 and IS2), which were the reformulated objectives and goals of the project as outlined in the ProDoc.

IS1 was reformulated from the original objective to reflect the changed state resulting from operationalising the NP and ABS.

Likewise, IS2 was reformulated from the original goal to reflect the changed state resulting from the implementation of the provisions of the NP and ABS mechanism consistent with the CBD.
Achieving these changed conditions will be influenced by the fact that there is increasing use of genetic resources and continued growth of BioTrade (D2) and the assumption that national ABS policies, regulatory and legislative instruments and the Regional Strategy and Action Plan is adopted and used at the national level (A2).

111.
111.

The ultimate impact is that local and global benefits deriving from ABS implementation are maximised through effective and transparent use of genetic resources and more assertive conservation of globally significant Caribbean biodiversity.
Achieving that impact is strongly linked to the assumption that the ABS regimes which have been institutionalised by countries in the region are operational and effective (A3) and that the regional institutions are actively involved in promoting the sustainable use of biological resources and associated Traditional Knowledge (D3).
Because of these drivers and assumptions and the various outputs initiated by the project management team and other stakeholders, countries will inevitably realise benefits from the use of their GR and TK.

More importantly, awareness of the value of the resource is likely to result in greater efforts at conservation with benefits manifested at the national, regional and global levels.

112.
112.

While it was assumed that the assumptions if held, would lead to Direct Outcomes (DO), and that in turn would lead to the realisation of Impacts following the attainment of the Intermediate States, very few of those assumptions held or materialised.

It is instructive to note that the overall rating of the likelihood of impact being realised at the Mid-Term Review was classified as ‘Highly Likely’.
However, the author of that report stated that since the drivers and assumptions were yet to be tested in moving towards the intermediate states and moving from intermediate states to long-term impact, there was a likelihood that some outcomes would not be fully achieved due to time constraints and differences in the rate of uptake among the project countries.

With these considerations, the Mid-Term Review noted, a more conservative rating of ‘Likely’ is easily justified.

113.
113.

Having now reviewed the project as part of the TE, it is discernible that while time constraint was indeed a factor, it was not the only issue of concern.

Also, of significance were 34 | P a g e Terminal Evaluation: “Advancing the Nagoya Protocol in Countries of the Caribbean Region” the facts on which some assumptions were based.

It does seem that there was an insufficient factual basis for some assumptions, while others were just ambitious and could not be easily justified.

114.
114.

Table 6 below presents a comparison between the assumptions and the realities that pertained, providing a clear indication as to why these assumptions constrained the realisations of the direct outcomes and intended impacts.

Table 6: Assumptions and Realities Assumptions Reality a.

Counterpart organisations and ILCs are willing to share information and recognise the usefulness of the data to be produced and knowledge to be generated.

b.
b.

National governments prioritise policy formulation as an essential first step and stakeholders and decision-makers are receptive to incorporating project results into policy formulation processes and value the importance of inter-institutional coordination for policy success.

c. The identification of common genetic resources can act as a driver for regional policy formulation and integration talks at the OECS and CARICOM levels.
d. Project stakeholders embrace online forum as a user-friendly, interactive and effective means of communication and information exchange The assumption that there was data to be shared was inaccurate.

Also, those organisations that had data (primarily universities) were reluctant to part with their data for various reasons.

The leap from policy formulation to legislation was quite huge and dependent on a host of financial and political decisions.
Without a clear cost recovery or sustainable financing roadmap, there was great reluctance on the part of some countries to proceed with ratification of the NP.

There was general agreement on the need for a common regional policy in respect of bioprospecting.

That approach is consistent with directives contained in the Revised Treaty of Chaguaramas and the Regional Biodiversity Policy being formulated by CARICOM.
While all agreed on the need for such an online tool, the cost of hosting and management of such a facility proved to be problematic.

e. Countries recognise the value of COP Side Event and network with each other to maximise presence and participation in the same Those meetings certainly helped in raising knowledge, awareness and value of bioprospecting.
f. Executive Power and Legislature exhibit political leadership in the NP ratification process such that Parliamentarians show interest in ABS and NP and participate in Parliamentarian Dialogues.

g. Entities such as IUCN, OECS, CARICOM, the GIZ ABS Capacity Development Initiative, and the CBD Secretariat contribute to the Regional Strategy to achieve common ABS capacity building goals in the region.
h. Scientists and researchers volunteer to participate in live on-air discussions on ABS and bioprospecting.

No one “Champion” was identified at the national and regional levels.

All of the entities identified played their part.

All have indicated their willingness to assist the countries as needed, both at the national and sub-regional levels.

The scientific community have been supportive but limited in what they can offer, given the propriety nature of some of their research activities.
35 | P a g e Terminal Evaluation: “Advancing the Nagoya Protocol in Countries of the Caribbean Region” Assumptions Reality i.

Local communities and indigenous peoples take an interest in ABS, trust the awareness- raising message and embrace its content Local communities and IP (Maroons in Jamaica and Guyana) all embraced their “new” knowledge, particularly as it promised wider acceptance and rewards for their traditional knowledge.

j.
j.

Countries fully embrace the outputs of the project and institutionalise required processes and strategies in all ABS related activities, including facilitation of personnel participation in training and capacity building opportunities They all recognised the need for Competent National Authorities but making this a reality is a slow process.

Given the paucity of resources in the respective countries, the approach seems to be aimed at working with existing institutions.
k. Institutions in the region take steps to institutionalise postings and updates of all ABS matters on ABS Clearing House and institutional web pages as part of a permanent internal activity geared towards information exchange.

The response to this initiative was poor.

115.

Following the realisation of Direct Outcomes, a clear pathway towards Intermediate State (IS) was also anticipated, based on several assumptions holding.
However, with very few of the assumptions holding and some of the DOs not yet realised, the only conclusion which can be drawn is that it is unlikely that the Intermediate States will be realised and likewise, the intended Impact.
Since one of the main constraints to achieving the DO, as put forward by project personnel, was time being insufficient, the question which has to be asked is would additional time have allowed for those assumptions to materialise and if so, how much more time would have been realistic?

This is not a question that can be answered by the Evaluator other than to indicate that without the assumptions holding and no guarantee that they will realise it is not likely there will be any scaling up and replication.
The Likelihood of Impact is rated Unsatisfactory (U), and overall Effectiveness is rated Moderately Unsatisfactory (MU).

Criteria Summary Assessment Rating Effectiveness Overall Effectiveness rating i.

Delivery of outputs All of the outputs were completed ii.

Achievement of direct outcomes Only two of the six DOs were realised, making it difficult to envisage that other intended impacts would be realised.

iii.
iii.

Likelihood of impact Three countries have ratified the NP, and several others have commenced incorporation of some of the ABS guidelines for PIC and MAT.

However, without the institutional support and a recognised legal framework in place, it is hardly likely the intended Impact will be realised.

MU HS U U 5.6 Financial Management 5.6.1 Completeness 116.

The total cost of the project was budgeted at US$5,635,257 (see Table 7).
The original sum allocated in the budget for the GEF grant was US$1.826mn.

The actual expenditure of 36 | P a g e Terminal Evaluation: “Advancing the Nagoya Protocol in Countries of the Caribbean Region” GEF Funds was US$1.786mn with the remaining sum of US$40,000 to be utilised to cover the cost of the Terminal Evaluation.

On average, expenditure on various project elements corresponded to the sums allocated within the original budget; however, there were a few exceptions.

117.
117.

Concerning the Co-Financing Cash and In-Kind values, the budgeted contribution from Countries and institutions was US$3,809,257.

Information received on Co-Financing contributions indicated that in total the co-financing was valued at US$2,904,999.61.

This value, however, did not include information from Jamaica or CBD.
Table 7: Co-financing Table UNEP own Financing (US$1,000) Actual Planned Government Other* Total (US$1,000) (US$1,000) (US$1,000) Planned Actual Planned Actual Planned Actual Total Disbursed (US$ 1,000) 1,826,000 1,786,000 12 289.99 2,127.99 1,786,000 1,786,000 2,508.94 1,599.96 998.32 1,305.04 3,507.26 2,905,000 Co-financing (Type/Source) − Grants − Loans − Credits − Equity investments − In-kind support − Other (*) - Totals 1,826,000 1,786,000 2,520.94 1,599.96 1,288.32 1,305.04 5,635.26 4,691,000 1,786,000 * This refers to contributions mobilised for the project from other multilateral agencies, bilateral development cooperation agencies, NGOs, the private sector and beneficiaries.
118.

Table 8 shows the variation in cost between Budgeted and Actual Total Project Cost broken down into GEF and each Co-Financier Cash and In-Kind Contributions.
Table 8: Total Project Cost Countries and Agencies Budgeted Contributions (US$) Actual Contributions (US$) Variation (US$) Cash In-Kind GEF Trust Fund Barbados Guyana Antigua and Barbuda St Kitts and Nevis Trinidad & Tobago Saint Lucia Grenada Jamaica UNEP CBD IUCN OECS GIZ Total Contribution 1,826,000 12,000 289,993 2,127,993 123,000 284,547 400,000 612,360 187,000 232,035 200,000 470,000 90,000 50,000 211,081 596,871 50,371 3,507,064 1,786,000 207,419 188,175 366,768 469,207 7,010 93.602 267,779 - 90,000 - 669,049 505,325 40,666 4,598,382 (40,000) 72,419 (96,372) (33,232) (143,153) (179,990) (138,433) 67,779 (470,000) - 50,000 167,975 (91,546) (9,705) (944,258) 119.
Though the EA received no report from the Secretariat of the CBDs in respect of their in-kind contribution, their “in-kind” contribution to the project activities, as detailed in their co- 37 | P a g e Terminal Evaluation: “Advancing the Nagoya Protocol in Countries of the Caribbean Region” financing letter to UNEP dated 11 April 2015 (Ref: SCBD/ABS/SBG/ET/jh184313), suggests that it would have included “provision of resource materials published by the Secretariat, staff time and co-organization of joint capacity-building activities”.
The CBD participated in Steering Committee (SC) Meetings, co-organizing meetings such as the side-events at COP 14 and 15, as well as participating in the provision of training to participating countries to demonstrate access to and use of the ABS Clearing House Mechanism.

120.

In-Kind Contributions amounted to US$904,257 below the budgeted value.
Except for Barbados, Grenada and IUCN, the In-Kind contributions of the other countries and agencies was therefore significantly less than the originally budgeted values.

The total Project contributions are shown in Table 7 above.

121.

The Actual Cost of the Project was, therefore, US$4,690,999 or US$944,258 less than the total Budgeted Cost.

This figure consists of Countries Contributions which were US$920,981 less than budgeted and the Contributions of Agencies which were $16,724 more than budgeted.
122.

Countries were provided with co-financing templates and instructions in completing their reports.

They were repeatedly urged to submit these reports on time.

Unfortunately, not all did, and several countries did not meet their full co-financing commitment.

123.

The general conclusion to be drawn from the completeness of financial information is that it was Moderately Satisfactory (MS).

5.6.2 Communication 124.


The PMT at IUCN-ORMACC, which included the Project Financial Assistant, and the UNEP Fund Management office (Panama) have expressed satisfaction with the responsibilities executed by each other.

Financial Reports were submitted on time and met the satisfaction of the Fund Management Office.

Some slight delays with disbursement caused a delay in the start-up of the project, and another disbursement delay midway through the project again slowed the pace of delivery of some outputs.
In the end, a request for a no- cost extension was made and approval granted.

125.

UNEP’s Task Manager, based at the Latin America and the Caribbean Office in Panama, provides continuous support and works closely with project personnel in project implementation aspects related to UNEP and the GEF implementation.

Good communications between the IA and EA were maintained throughout the Project implementation period, and both offices were adequately aware of the status of the project funding.
Table 9 summarises the financial management of the Project.

Table 9: Financial Management Table Financial management components: Rating8 Evidence/ Comments Completeness of project financial information: Provision of key documents to the evaluator (based on the responses to A-G below) A. Co-financing and Project Cost’s tables at design (by budget lines) B.
Revisions to the budget S HS Yes Yes Received copies of tables indicating Co-financing and Project Costs at design Received copies of all revisions 8 Ratings given on a 6-point satisfactory scale from ‘Highly satisfactory’ (HS) to Highly Unsatisfactory (HU).

38 | P a g e Terminal Evaluation: “Advancing the Nagoya Protocol in Countries of the Caribbean Region” Financial management components: Rating8 Evidence/ Comments C. D. E. F. G. H. All relevant project legal agreements (e.g.
SSFA, PCA, ICA) Proof of fund transfers Proof of co-financing (cash and in- kind) A summary report on the project’s expenditures during the life of the project (by budget lines, project components and/or annual level) Copies of any completed audits and management responses (where applicable) Any other financial information that was required for this project (list): Any gaps in terms of financial information that could be indicative of shortcomings in the project’s compliance9 with the UNEP or donor rules Project Manager, Task Manager and Fund Management Officer responsiveness to financial requests during the evaluation process Communication between finance and project management staff Project Manager and/or Task Manager’s level of awareness of the project’s financial status.
Fund Management Officer’s knowledge of project progress/status when disbursements are done.

Level of addressing and resolving financial management issues among Fund Management Officer and Project Manager/ Task Manager.

Contact/communication between by Fund Management Officer, Project Manager/Task Manager during the preparation of financial and progress reports.

Overall rating HS HS No All documents were shared with the Evaluator All documents were shared with the Evaluator Partial.
No information received from Barbados concerning their cash contribution.

Jamaica and the CBD provided no reports on co-financing.

Yes Summary Report received Audits for 2016, 2017 and 2018/2019 received Information on GEF spending and actual In-Kind Contributions received Yes.
Not all countries/ agencies provided data on their In-Kind or Cash Contributions Personnel, both at the EA and IA provided all the documents requested and expressed no concern about the financial management of the project Project personnel at both EA (IUCN- ORMACC) and the IA (UNEP) confirmed their awareness of the financial status having signed the necessary approvals Yes, these were confirmed by the Finance officer at the EA and by the Project Assistant There were no real issues of challenges On the completion of financial and progress reports, these would be submitted to the Project Assistant at UNEP before being shared with the Fund Management Officer and the Task Manager.
Yes Yes Yes S S S S S S 9 Compliance with financial systems is not assessed specifically in the evaluation.

Nevertheless, if the evaluation identifies gaps in the financial data, or raises other concerns of a compliance nature, a recommendation should be given to cover the topic in an upcoming audit, or similar financial oversight exercise.
39 | P a g e Terminal Evaluation: “Advancing the Nagoya Protocol in Countries of the Caribbean Region” Criteria Summary Assessment Financial Management Overall Financial Management rating 1.Completeness of project financial information All financial requirements between the EA and IA were satisfied.

Countries were provided with co- financing templates and instructions in completing their reports.
However, not all countries/agencies submitted their reports, and several did not meet their full co-financing commitment.

Rating MS MS 2.Communication between finance and project management staff Communications between the EA and IA were excellent.

There were two instances of disbursement delays by the IA, but good relations were maintained throughout the life of the project.

S 5.7 Efficiency 126.


From a design perspective, the project sought to achieve efficiency by building on the work commenced by other entities and existing institutions, capturing lessons and experiences of other projects including the GIZ-CDI, the SCBD, the CARICOM Secretariat, and the GEF-UNEP ABS LAC Project as well applying tools and practices learned from the implementation of a similar initiative in LAC as well as the GEF ‘Strengthening Access and Benefit Sharing (ABS) in the Bahamas.
In that regard, the baseline for this intervention offered substantial opportunities for cooperation, sharing of lessons, and the creation of strategic alliances.

The regional design was, therefore, an effective way of achieving efficiency in the implementation of activities of a similar nature in other participating countries.

127.
Given the fact that many of the participating countries share an adherence to the Common Law principles and have made commitments within various regional bodies under separate agreements10, the undertaking of a regional approach not only served to ensure that they achieved efficiency in the drafting of policy and legislative guidelines but also ensured that they were acting within the framework of their regional mandates by pursuing the harmonization of laws as it relates to the management of their natural resources.
128.

The project was initially intended to commence in 2015, but due to disbursement constraints, it became operational in February 2016.

Since that initial setback, there were no further delays that affected the implementation in any adverse manner.

That initial setback concerning disbursement and the delayed start meant that some additional time was needed towards the end of the project to complete all outstanding outputs.
That recommendation was made in the MTR, and a request for a three-month no-cost extension was granted.

129.

The holding of virtual meetings also achieves greater efficiency in the implementation of the project.
While this may have been planned at project inception, it not only served to reduce the project’s environmental footprint with less travelling of project staff, country personnel, and UNEP personnel but also became a valuable means of communications with some countries when they were unable to travel, either because of late approvals or when 10 The Revised Treaty of Chaguaramas establishing the Caribbean Community including the CARICOM Single Market and Economy (CSME) was signed in 2001.
The St. Georges Declaration of Principles for Environmental Sustainability in the OECS 40 | P a g e Terminal Evaluation: “Advancing the Nagoya Protocol in Countries of the Caribbean Region” flight among countries became disrupted in the aftermath of hurricanes which struck the islands in 2017.

130.

Efficiency is rated Highly Satisfactory (HS).

Criteria Efficiency Summary Assessment Efficiency was practised and achieved throughout the life of the project.
With a limited budget and tight time frame, the project delivered all outputs within thirty-six months.

Other measures of efficiency included the development of regional guidelines and templates, adopting a common approach the drafting of roadmaps and the use of virtual meetings to save cost on travel and reduce the carbon footprint of the project.

Rating HS 5.8 Monitoring and Reporting 131.

The project had a very robust monitoring and reporting framework.
That framework was based on UNEP standard monitoring, reporting and evaluation processes and procedures.

It was also consistent with the GEF Monitoring and Evaluation policy.

5.8.1 Monitoring design and budgeting 132.

The monitoring framework consisted of several tools, including a risk analysis, and the Results Framework, commonly referred to as the log frame, which included SMART indicators for each expected outcome.
These indicators, along with the key deliverables and benchmarks, provided a satisfactory means for tracking implementation and whether or not the expected results were being achieved.

The means of verification of these elements are summarised in the Project Result Framework and included time-frames (mid-term and end-of- project targets) as well as the criteria used to verify each achievement.

133.
133.

Since the approval of this project, new monitoring and reporting guidelines have been instituted, which require the use of a ToC in project formulation.

Since this project was not subjected to such a formulation, a ToC had to be recreated.

That reconstruction was based on the information contained in the Results Framework and intervention logic to capture the key elements of the project as it relates to outputs.
These outputs were easily extracted from the logical framework, allowing for the mapping of the causal pathways.

While no indicators were labelled as ‘Output Indicators’, the Mid-Term and End of Project Targets can be considered output indicators.

All indicators in the Results Framework are ‘Outcome’ indicators and embrace the project’s scope of work and ambitions.
Unfortunately, however, since the indicators, as contained in the log frame, were focused on the achievement of outputs, this is considered a shortcoming in the design of the project’s M&E framework since it limits the ability of project management to make adjustments and make decisions regarding the achievement of project’s expected outcomes.

134.

The project also made an allowance in the budget for the undertaking of monitoring and evaluation of project activities.
A Monitoring and Evaluation Plan (Appendix 7, ProDoc), and a “Summary of Reporting Requirements and Responsibilities” (Appendix 8), not only detailed the activities which had to be monitored, but also the time-frame in the life of the project when those activities should be undertaken, the cost of those undertakings and the parties responsible for reporting.
Two significant milestones in the M&E plan was the Mid- term Review which was executed between July – September 2017, at the half-way point of the 41 | P a g e Terminal Evaluation: “Advancing the Nagoya Protocol in Countries of the Caribbean Region” project and the Terminal Evaluation, which was completed between June and December 2019, following the completion of the project.
From all indications the funding allocated for these two items were adequate as the MTR was completed within the USD 25,000.00 allocated and the TE is likely to also come in within the USD 40,000.00 allocated for that item.

5.8.2 Monitoring of Project implementation 135.


Monitoring during implementation was primarily the responsibility of the Project Management Team lead by the Project Supervisor; however, several other stakeholders had various oversight responsibilities which all served to enhance and ensure the delivery of project outputs.
The Project Manager was responsible for the day-to-day management of the project and had a central role to play in reporting on the implementation of the different and the accomplishment of the various Outputs.
The Project Manager performed admirably in delivering all the outputs including the engagement of consultants and supervising the production of videos and other communications materials, convening meetings of the RSC, and training initiatives, responding to an individual request from countries to assist with the raising awareness at the national level, and preparation of semi-annual PIRs.
The Project Manager was adequately supported by other members of the Project Management Team in the preparation of quarterly financial reports and other logistical support which ensured the timely delivery of reporting materials and the facilitation of the Mid-Term Review and Terminal Evaluation exercise.

136.
136.

The Task Manager, operating out of the UNEP office in Panama, also provided valuable support in reviewing the quality of draft project outputs, providing feedback to the project partners, and establish peer review procedures to ensure adequate quality of project outputs in close collaboration with the PM.

All indications are that all aspects of the reporting system were completed as planned and on time.
These included the PIRs, half-yearly progress reports, financial reports, including the annual audits, and the MTR.

137.

The project Steering Committee Meetings, coordinated by the Executing Agency, provided a valuable opportunity to monitor the implementation of project activities.

Membership in the Regional Steering Committee (RSC) consisted of Country representatives, UNEP, IUCN as EA, and the project co-financiers (ABS CDI (GIZ), OECS, and CB).
In fulfilling its oversight role, the Regional Steering Committee convened six meetings, three of which were convened on their virtual platform.

138.

Though these meetings were not always well attended, they provided opportunities for members to obtain an update on progress in respect of activities being pursued by the PMT.
The PMT also helped to balance conflicting priorities and resources, ensuring that activities being undertaken by the various consultancies were meeting the needs of the respective states.
In that regard, their insistence on the need for common regional ABS policy and legislative guidelines was reflective of their desire to have guidelines which had some common elements but provided opportunities for the respective countries to make their inputs without adversely affecting the genetic resources of the region as a whole.

139.

The Mid-Term Review was carried out from July to September 2017.
At that stage, the Reviewer noted that the project was “on track and progressing well in almost all the activities and outputs” and that the participating countries were “all satisfied with the project’s performance”11.

This expression of satisfaction was confirmed during the terminal evaluation.

11 Pg.

11.

FINAL MID-TERM REVIEW REPORT Advancing the Nagoya Protocol in Countries of the Caribbean Region (GFL/5060-2711-4E67).
42 | P a g e Terminal Evaluation: “Advancing the Nagoya Protocol in Countries of the Caribbean Region” 140.

While there is no evidence that the project monitored the representation or participation of disaggregated groups, it, however, adopted an inclusive methodology that included gender mainstreaming to ensure the effective participation of women, indigenous peoples, local communities, particularly in the consultation and design of protocols for PIC and MAT.
Two national workshops where PIC and MAT templates were discussed, adopted, and shared, were held in January 2019 with the indigenous groups and local communities in Jamaica and Guyana.

141.

The Monitoring of Project implementation is rated Satisfactory (S) 5.8.3 Project reporting 142.
The project had a very rigorous reporting system as reflected in “Appendix 8: Summary of reporting requirements and responsibilities” which provided a detailed schedule of the time for delivery of the various reports inclusive of quarterly expenditure reports, co-financing reports, Audit Reports and Project Implementation Review.

These were complemented with minutes of the Regional Steering Committee, which provided project oversight as well as the approval of annual work plans and budgets.
While the PIR provided a detailed ongoing reporting of activities and milestones, it also provided opportunities for the Task Manager to undertake a careful analysis of achievements as well as provide updates on performance as far as risks were concerned.

143.

While the monitoring system was robust and generally satisfied the requirements laid out in the ProDoc, it did not provide opportunities to monitor, in an objective manner, the effectiveness of outputs and outcomes.
The Reporting System, followed by the Project Management Team, was mostly based on the implementation and completion of “Activities” (see PIR Format).

While there were milestones and targets, both for mid-term and end of the project (as stated in the revised work-plans of the Project), the fact that this approach does not allow for measuring progress towards Direct Outcomes and Impacts, meant that the monitoring system could not have adequately supported a results-based monitoring approach.

144.
144.

In the Terms of Reference for this evaluation exercise, it was mentioned that tracking tools would have been developed.

This, undoubtedly, would have gone a long way in tracking progress towards projects objectives throughout the project implementation period.

However, this was not done as a stand-alone exercise as it was stated in the ProDoc, under Component 5, Project Management (Pg.

92) that “GEF Tracking Tools are incorporated into Results Framework”.

145.
145.

The rating for project reporting is Highly Satisfactory (S).

Criteria Summary Assessment Rating Monitoring and Reporting Overall Monitoring and Reporting rating i.

Monitoring design and budgeting Monitoring and design are consistent with GEF/UNEP guidelines.

ii.

Monitoring of project implementation All reports, including PIRs and financial reports, were prepared.

The reporting format, however, did not allow for objective monitoring of progress in implementation.

S S S iii.
S S S iii.

Project reporting Reports were prepared and submitted on time.

HS 43 | P a g e Terminal Evaluation: “Advancing the Nagoya Protocol in Countries of the Caribbean Region” 5.9 Sustainability 146.

Sustainability, within the context of this evaluation exercise, is understood as the probability of direct outcomes being maintained and developed after the close of the project.
Several outputs, intended to ensure sustainability were delivered, including the development of ABS policy documents, ABS guidelines and an online platform for the exchange of information among participating countries.
As indicated in section 5.5.3 above, several of these outcomes were realised, however, given the fact that no sustainable legislative and institutional framework was created at the national level to sustain those initiatives, as well as the need for financial support, the likelihood of these outcomes being maintained is not high.

147.
147.

The countries expressed great satisfaction with the outcomes as they not only heightened their awareness of the importance of genetic resources and associated traditional knowledge, but provided them with tools (e.g.

ABS guidelines, Regional Strategy and Action Plan, and tools for facilitating access GR and Traditional Knowledge such as PIC and MAT Templates), and a model ABS Clearing House Mechanism.
All of the documents produced under this initiative, together with the training provided can be easily accessed and modified to suit the needs of the participating countries as well as other countries desirous of using those tools to enhance their capabilities in facilitating access to their GR/ Traditional Knowledge using existing national and regional institutions.

5.9.1 Socio-political Sustainability 148.


All of the participating countries expressed a high degree of support for the project and what it sought to achieve.

Three of the eight participating countries ratified the NP, however, only one, Antigua and Barbuda, has incorporated the ABS mechanism into national legislation.
Notwithstanding, at least four of the other countries, Grenada, Guyana, St. Kitts and Nevis, and Saint Lucia have stated their intention to proceed towards ratification or incorporation of ABS legislation and regulations into national laws.

149.

Notwithstanding the above, there have been concerns that without additional technical and financial support, progress towards ratification of the NP and the establishment of the required institutional and legislative framework will not materialise anytime soon.
The adoption and operationalisation of new policies and legislation carry costs which they could not readily determine how these would be offset.
While there was a high level of awareness generated and other user groups, particularly the ILC in Guyana and Jamaica, have embraced the interest in bioprospecting and rewards to be gained from sharing TK, without the active and sustained support of the political directorate it is difficult to envisage further progress being made in ratification of the NP, the incorporation of legislation and realisation of impacts.

150.

The rating for Socio-political Sustainability is Moderately Unlikely (MU).
5.9.2 Financial Sustainability 151.

The financial allocation, while adequate, did not allow for any extra-budgetary items or any activities beyond the life of the project.

There is strong justification for concluding that funding was inadequate, given the ambitious intentions of the project.

Though much was accomplished, there were several areas where additional funding could have made a difference in the direct outcomes desired.

152.
152.

The resources allocated for the National Project Coordinators were only to provide support for 10 out of the 36 months of the project.

An ideal amount would have been for 18 44 | P a g e Terminal Evaluation: “Advancing the Nagoya Protocol in Countries of the Caribbean Region” months.

The online platform for sharing information and the searchable database was never fully developed due to lack of resources for any country to host and manage that facility.
Likewise, much more resources were needed to make full use of electronic and social media to increase the reach of information sharing.

Towards the end of the project when it was agreed that the extent of information penetration was required, the lack of funds meant a consulting entity could not have been engaged to undertake an objective assessment of their communications and public awareness initiatives.

153.
153.

There is a great need for funding to further develop and expand on the initiatives started under this project.

Six of the countries have expressed their desire for a second phase.

Both the GIZ and OECS have expressed a desire to provide some continued support to the OECS sub-grouping to allow them to move ahead with the entrenchment of PIC and MAT and the incorporation of ABS in the legislative framework of their Member States.
The future of those initiatives is highly dependent on these institutions being provided with the pledged financial assistance.

154.

Given the stated interest of the institutions concerned a Rating of Moderately Likely (MU) seems justified.

5.9.3 Institutional Sustainability (including issues of partnerships) 155.

All of the participating countries have made improvements to their permitting system, a direct result of tools (templates) and training provided under this project.
However, more work needs to be done.

Countries have to commit to the initiatives that have been developed under the project to maximise the achievements and the momentum that has been garnered.

156.

The project has nurtured sustainability as efforts are underway to create a follow-up ABS project for the region, targeting other countries that were not part of this recently completed project.
Additionally, the policies and legislation/regulations that have been developed will now be used to move the countries forward towards formalising the process to access genetic resources in the region.

Furthermore, the contract templates (PIC and MAT) that were developed are currently used by the various departments in the preparation of contract documents.

157.

The methodology of the project is replicable in other regions.
As mentioned previously, including countries at the design stage of the project was a good approach as this helped to ensure they took responsibility for the outcome of the project and participated fully in ensuring it achieved its outcomes.

At least six countries within the region were interested in having a second phase of the project.

However, this is dependent on whether or not they can secure the funding to do so.

158.
158.

There is still an issue with commitment from the executive level on moving forward with the ABS agenda in the respective countries.

That is reflected in the fact that not much progress was made in creating the institutional framework for continued implementation.
While the countries do not have the capacity to have stand-alone entities, neither is it necessary, there is still the need for specific and visible structure that would ensure that the administration of an ABS mechanism will be smooth efficient.

These arrangements have yet to be established.

159.

The rating for Institutional sustainability is considered Moderately Unlikely (MU).
45 | P a g e Terminal Evaluation: “Advancing the Nagoya Protocol in Countries of the Caribbean Region” Criteria Sustainability Rating MU Summary Assessment Several outputs, intended to ensure sustainability were delivered, including the development of ABS policy documents, ABS guidelines and an online platform for exchange of information among participating countries.
However, others, such as ratification of NP, the incorporation of legislation and the establishment of the institutional framework are not likely to be realsed.

1.

Socio-political sustainability Three of the eight participating countries MU ratified the NP, however, only one, Antigua and Barbuda, has incorporated the ABS mechanism into national legislation.
Other countries have indicated their intention to proceed towards ratification or incorporation of ABS legislation and regulations into national laws.

There is a great need for funding to further develop and expand on the initiatives started under this project.

Six of the countries, with the support of OECS, GIZ and UN Environment are exploring those possibilities.

Not much progress made in creating the institutional framework for continued implementation.
Without additional support, it is unlikely the legislative framework will be further advance and further strengthening of the institutional framework ML MU MU 2.

Financial sustainability 3.

Institutional sustainability Overall Sustainability rating 5.10 Factors and Processes Affecting Performance 5.10.1 Preparation and Readiness 160.

The ProDoc detailed an extensive set of activities which suggests that careful thought and planning went into the preparation of the project.
A similar project was undertaken in LAC in which both the IA and EA were involved.

From a design perspective, it had great strategic relevance both in terms of its alignment with UNEP MTS and PoW and GEF Strategic Objectives.
It also had great relevance for the countries given the capacity, institutional and financial constraints which were negatively impacting on their ability to adequately conserve their biodiversity resources and establish the means for deriving benefits from the use of their genetic resources.

161.

The overall focus of the ProDoc and the components it proposed were appropriate and the project activities necessary to achieve the project objective.
However, a large number of outputs (41) to be achieved within the project time frame posed a challenge, and adaptive measures were required to have them completed on time.

That required pursuing some initiative simultaneously as well providing an additional three month for the national coordinators.

46 | P a g e Terminal Evaluation: “Advancing the Nagoya Protocol in Countries of the Caribbean Region” 162.
The shortcomings identified at the preparation stage had to do with some overambitious objectives which did not take into consideration the challenges in moving from the awareness phase to getting legislation in place to address problems identified.
Among the more obvious overambitious goals and incorrect assumptions were the following: • Data on GR in the Caribbean is sparse, and the few institutions involved in research, primarily the Universities or research entities guard their information very closely • All countries recognised the need for and expressed support for an online forum.
However, challenges in finding an institution to host the forum as well as provide ongoing management proved difficult • Development of a Roster of Experts never materialised as these were not filled in by the countries as was intended.

While this may appear to suggest a lack of experts and the absence of a large pool of resources, it has more to do with the effort required to generate such a database.

163.

The overall rating of preparation and readiness is Moderately Unsatisfactory (MU).
5.10.2 Quality of Project Management and Supervision 164.

The overall management and supervision of the project have been rated highly with all parties expressing satisfaction with the roles performed by the respective entities.

These expressions of approval were manifested in the fact that the PMT, and specifically, the Sub- Project Manager, who was responsible for the day to day management, was adaptive, and responded to the needs of participating countries.
All countries expressed satisfaction with the outputs delivered both at the national and regional levels, which resulted in draft policy documents, model templates for PIC and MAT and guidelines for the drafting of legislation.

165.

Under Component four, all management structures were put in place to ensure effective management of the project.
All work plans were developed, and the SC meetings were used to review accomplishments as well as undertake any necessary revisions to both the national work plans as well as the regional work agenda.

Six steering committee meetings were held, and the Mid-term Review completed at the half-way mark.

166.

Over the three-year life of the project, there were changes in the TM who was assigned to the project.
Without the input of the first TM, it was a little difficult to obtain a full appreciation of what were some of the challenges faced in the early phase of the project.

Notwithstanding, the supervision and guidance provided by UNEP from the perspective of the EA and other implementing partners were satisfactory.
While the TM did not attend all six of the SC meetings, communications with the PMT has maintained in-between those meetings, providing feedback on reports and ensuring that disbursement requests were addressed on time.

167.

The overall rating for Project Management and Supervision is Satisfactory (S).

5.10.3 Stakeholder Participation and Cooperation 168.
Stakeholders were identified at both the regional and national levels, their roles in relations to the project and potential contributions to the project.

Among the many stakeholders identified were parliamentarians, who, it was believed, could have the most relevant and direct impact on project activities and outcomes, direct project beneficiaries, persons in local communities, and other institutions working closely with the ABS agenda, such as Iwokrama in the case of ILCs in Guyana.
Special attention was paid to issues of 47 | P a g e Terminal Evaluation: “Advancing the Nagoya Protocol in Countries of the Caribbean Region” gender and Indigenous People, ensuring their consideration was mainstreamed in all capacity building processes.

169.
169.

While the PMT reached out to entities engaged in research activities (the University of the West Indies and ILC) through their public awareness programmes to tour operators and small scale business operators engaged in the sale of bio-pharmaceuticals and TK, there was no specific programme or sustained effort to involve the private sector.
That was due, in part, to the fact that the document “Business Model for Access and Benefit Sharing in the Caribbean Region” the blueprint for articulating and “maximizing the economic potential of biodiversity” resources was not completed until the latter part of the final year of the project.
Also, the regional database of research activities in the Caribbean region, which would have served as an information portal for sharing of information research and “the promotion of public-private sector interactions on the Caribbean” was never fully developed and remained underutilised.

170.

The PMT used various communication mechanisms to engage with stakeholders.

These included “direct meetings/interviews, consultations and teleconferences”.
Two side event meetings at UNCBD COP 14 and 15 in Cancun Mexico and Egypt were well attended by beneficiary countries, partner organisations and other interested stakeholders.

High-Level Meetings with Ministers proved a little more difficult.

However, during Caribbean ABS week (March 2019) the Minister of Agriculture, representatives from the Prime Minister’s office, Permanent Secretaries from around the region attended.
Also, several regional and national institutions were represented, along with representatives from indigenous people and local communities from Jamaica and Guyana, and local community groups and students participated in the meetings.

171.

The overall rating for Stakeholder Participation Moderately Satisfactory (MS).

5.10.4 Responsiveness to Human Rights and Gender Equality 172.
The ProDoc, while not explicitly outlining any issues or concerns relating to human rights, did specifically, however, express the need to mainstream issues of gender in all capacity building initiatives.

It also recognised the important contribution of IPs to biodiversity conservation and made allowance for their involvement in the implementation of the project.
In that regard, Component 3, Outcome 3.1 of the Logical project Framework, specifically targeted Indigenous communities and provided resources for the production of public awareness materials in the indigenous language, and in the development of templates for Prior Informed Consent (PIC) and Mutually Agreed Terms (MAT).

173.
173.

Concerning the implementation of the project, a gender methodology was adopted and implemented by the EA to ensure effective mainstreaming of gender by ensuring that IP was included workshops convened and in materials developed to heighten awareness of the ABS process.

Likewise, concerning participation by women, there was a high number of women involved in decision-making positions during the implementation of the project.
Women comprised almost 80 per cent of those in attendance at SC meetings, workshops and training initiatives undertaken by the project.

174.

Two workshops each were held in Jamaica and Guyana in January 2019, to ensure the effective participation of women, indigenous peoples, local communities, particularly in the consultation and design of protocols for PIC and MAT were convened under the project.
The workshops in Jamaica allowed for the participation of persons from the Rastafarians and Maroon Community.

The Guyana workshop was held in Lethem and Georgetown, and present 48 | P a g e Terminal Evaluation: “Advancing the Nagoya Protocol in Countries of the Caribbean Region” were representatives from all nine Amerindian districts: Wai Wai, Patamona, Arecuna, Macusi, Wapisiana, Carib, Warrau, Arawak and Akawaiothe.

175.

The overall rating on Gender and Human Rights is Satisfactory (S).
5.10.5 Country Ownership and Driven-ness 176.

The project had its genesis in the perceived need of countries to overcome challenges in respect of capacity building, inadequate legislative and an institutional framework which limited their abilities to derive maximum benefit from the use of their genetic resources while also conserving biodiversity.
Much of the outputs were, therefore, geared towards the realisation of these objectives and included the provision of information on which bioprospecting could be structured, raising of awareness among key stakeholders, and the preparation of policy and legislative instruments that would have ensured the realisation of intended outcomes and impacts.

177.
177.

The uptake of the Nagoya Protocol in Caribbean countries went beyond raising awareness to the need for a range of protection mechanism inclusive of capacity building legislative revision and institutional frameworks.

These mechanisms were dependent on governments of the region articulating new policies and making changes to legislation and institutional mechanisms.

Essentially, they had to demonstrate interest and commitment.
While the project sought to interest the decision-makers by creating and providing them with information that would guide them along, there was also a need for demonstration of by-in and country ownership.

That country ownership could have taken many forms – the articulation of new policies, the establishment of CNA, and legislative interventions 178.
At the national level, a considerable effort was made by the national focal points, and respective ministries in raising awareness and driving interest among various stakeholders, including among the indigenous communities, and ensuring that policy and legislative instruments were fashioned to their needs.

They have indicated their desire to continue with these initiatives with the support of the OECS and GIZ.

179.
179.

The embracing of the ABS policies as well as the adoption of PICs and MATs by the respective countries were small, but important steps towards country ownership.

Also, the ratification of the Nagoya Protocol by three countries and the integration of ABS policies in existing legislation is further evidence of their commitment.
However, the absence of a sustainable institutional programme or supporting national infrastructure and regional support mechanism, e.g., online communications platform, in several of the participating countries, raises doubt as to the level of country ownership and the sustainability of the project.

180.

The rating for Country Ownership and Driven-ness is Moderately Unsatisfactory (MU).

5.10.6 Communication and Public Awareness 181.


The project placed great emphasis and resources on communications and raising public awareness as the main objective of the project was geared towards supporting countries to uptake, and where possible, ratify or accede to the Nagoya Protocol.
That involved initiatives aimed at increasing understanding and importance of the NP among targeted stakeholders including Parliamentarians, researchers, ILCs, and representatives from ABS line ministries, agencies tasked with managing the permitting process and access to biological and genetic resources, users of the resource and the general public.

49 | P a g e Terminal Evaluation: “Advancing the Nagoya Protocol in Countries of the Caribbean Region” 182.
The communication channels were well established and observed during the implementation phase.

The role of the Steering Committee was very effective in acting as a medium for disseminating and communicate decisions and procedures among countries and key stakeholders.
The project also developed a virtual platform, including a permitting system and species database, for fostering effective communication amongst countries on ABS issues which will further enhance the ratification process through effective communication amongst parties on ABS.

183.
183.

Among the many initiatives undertaken by the project to achieve their objectives were the preparation of videos, for TV and Radio, interviews on radio and television, the airing of PSAs in all eight countries, the translation and production of videos in six different languages for the ILCs in Guyana, and the erection of signboards at strategic locations in three countries.

184.
184.

These initiatives were complemented with the convening of training workshops for two persons from each of the participating countries and over 30 inter-institutional workshops in the eight participating countries.

185.

Awareness-raising dialogues were held with environment ministers at two meetings of the OECS ministers of environment.

Cabinet notes were prepared for members of the cabinet for all eight countries.
Included in these notes was information on the Nagoya Protocol and the road to ratification for each country.

The convening of the Caribbean Access and Benefit Sharing Week, which took place in Trinidad and Tobago from January 29-31, was aimed at generating awareness of and support for the Nagoya Protocol, under the Convention of Biological Diversity (CBD).

186.
186.

While it is generally agreed that a considerable amount of activities was undertaken in raising awareness of and support for the NP, participants acknowledged that much more is still needed to increase awareness of the importance of genetic resources.
This was saliently captured by one participant who acknowledged that while there had been widespread public education that reinforced “...the value of genetic resources and the traditional knowledge that comes along ...there is so much more that has to be done in terms of raising awareness”.

187.

Other participants also noted that the esoteric way in which genetic resources and biodiversity conservation issues are being discussed does not facilitate easy uptake at a grassroots level.
There is, therefore, a need to make the language and the conversation much more user-friendly in and to make it more appealing, practical and hands on to persons at the local and community levels.

This point was reiterated by stakeholders in Antigua and Barbuda as well as a representative of the ILC in Guyana.

188.
188.

Given those concerns, it was also stated that while the use of videos and radio/TV announcements seems to have been the preferred choice of awareness-raising, there could have been greater use of social media.

189.

Notwithstanding those stated and observed successes, the project itself did not have an inbuilt mechanism for objectively assessing the effectiveness of these interventions.
This issue was raised at the level of the SC, but resources to initiate such activity came very late, and the lack of direct funding prohibited its implementation.

Also, the networks and communications channels established for sharing information was hardly used, the issues of hosting/management and funding being the primary constraining factors.

190.

The rating for Communications and Public Awareness is Moderately Satisfactory (MS).
50 | P a g e Terminal Evaluation: “Advancing the Nagoya Protocol in Countries of the Caribbean Region” Criterion Summary Assessment Factors and Processes Affecting Project Performance Preparation and readiness Overall Factors Affecting Performance rating An initial delay with disbursement suggests a lack of readiness.

However, it was not a significant delay.


Preparation concerns had more to do with the relatively large number of outputs, given the time frame for implementation and the overambitious nature of some assumptions.

Rating MS MU Quality of project management and supervision12 There were no real complaints with management and supervision.
S Stakeholders participation and cooperation Responsiveness to human rights and gender equity Country ownership and driven- ness Communication and public awareness Stakeholder participation has generally been positive.

However, there was little sustained engagement with the private sector.

Various media were used to engage stakeholders, ensuring that all targeted groups were engaged.

It was, however, expressed that engagement with HL decision-makers could have commenced earlier.
Gender mainstreaming ensured that women, IP and local community groups were acknowledged and involved in initiatives to raise awareness and provided with tools to facilitate ABS mechanisms were addressed.

Country representatives in the majority of participating expressed strong support for the outputs undertaken under the project.
However, the slow uptake of concrete measures to institutionalise and incorporate ABS in policies, legislation and across ministries, suggest some shortcomings in terms of ownership at the national level.

A range of communication media was used to raise to communicate with the public/ stakeholders, including radio and TV announcements, billboards and video clips.

The sustainability of the information shared is dependent on both more grass-root delivery and direct- engagement.
Time and resources did not permit for that level of engagement.

MS S MU MS 12 In some cases ‘project management and supervision’ will refer to the supervision and guidance provided by UN Environment to implementing partners and national governments while in others, specifically for GEF funded projects, it will refer to the project management performance of the executing agency and the technical backstopping provided by UN Environment, as the implementing agency.
51 | P a g e Terminal Evaluation: “Advancing the Nagoya Protocol in Countries of the Caribbean Region” 6.0 Conclusions, Lessons Learned and Recommendations 6.1 Conclusions 191.

The GEF funded, UNEP implemented, and IUCN executed project “Advancing the Nagoya Protocol in countries of the Caribbean Region”, sought to support countries of the Caribbean by assisting them in facilitating access to their genetic resources and benefit sharing fairly and equitably, in line with the CBD and the Nagoya Protocol.
More specifically, the project sought the uptake of the Nagoya Protocol in the respective countries by overcoming barriers linked to poor understanding of the Nagoya Protocol and ABS.

These barriers were identified as a lack of knowledge of and awareness of the real benefits associated with GR and TK, the absence of legislative instruments, weak institutional infrastructure to make the protocol operational in Caribbean countries.
It sought to do so through the strengthening of their policy, legal, and institutional frameworks and maximising their readiness for NP ratification.

192.

In seeking to achieve the objectives detailed above, forty-one outputs were delivered, resulting in several outcomes being realised.
Among them was the completion of an inventory of GR and TK, draft ABS regulations and guidelines which could be incorporated into their respective legislation to facilitate access to and sharing of benefits in respect of genetic resources, the establishment of an online platform for exchange of information on bioprospecting, training individuals in the use and operations of a bioprospecting CHM and a series of awareness-raising interventions targeted at various stakeholders including parliamentarians, and ILC.
193.

In the end, two countries, Antigua and Barbuda and St. Kitts and Nevis ratified the NP, bringing to three (and Guyana), the number of countries that have now ratified the NP.

Guyana had ratified the NP before the beginning of the project in 2015, but having not prepared regulations, were able to benefit from the development of ABS mechanisms which were developed under the project.

194.
194.

The achievements of the project can be linked to several factors, including the fact that the regional project was designed around a system of interventions targeting the systemic and structural challenges impeding progress.

This regional approach was instrumental in overcoming the capacity constraints which limited the ability of most countries of the region to take full advantage of the value of genetic resources.
It also allowed them to make the best use of limited resources and ensured that work is done at the regional level, such as the preparation of policy documents, legislative and regulatory guidelines and Action Plan, would benefit all the participating countries.

195.

Undoubtedly, the greatest achievement of the project can be linked to several factors, including the raising of awareness of the importance of GR and associated TK, particularly among the ILC.
Before the workshops targeting the IPs in Guyana and Trinidad and Tobago, and the Maroons and Rastafarians in Jamaica, very little was known about the value of GR/TK, and no mechanisms existed for them to derive benefits from the use of those resources.

Not only were they now informed, but they now had tools (PIC and MAT Templates) and the establishment of mechanisms requiring application and approval being given for all activities aimed at removing, extracting and utilising GR and TK.

196.
196.

Several other positives included opportunities for networking, training of trainers on ABS to facilitate ongoing training at the national level, and convening workshops aimed at facilitating the use of the ABS Clearing House Mechanism and contract drafting.
The most positive result was the overwhelming support expressed regarding the online forum due to its 52 | P a g e Terminal Evaluation: “Advancing the Nagoya Protocol in Countries of the Caribbean Region” use as an effective means for fostering the exchange of information among participating countries.

That exchange of information was seen as vital in prohibiting potential bioprospecting entities from playing off one country against the other.

197.
197.

Unfortunately, several outcomes were not realised or are yet to be realised, and consequently, the intended impacts were not being realised.

Several factors were identified as causal factors, most of which had their origin in the design of the project.

198.

The project, as noted above, had forty-one outputs, all of which were expected to contribute to achieving the direct outcomes and intended impacts.
These Outputs not only had to be delivered in thirty-six months, but the Outcomes which had to be realised, along with the Intended Impacts were dependent on some assumptions and drivers which did not hold or were not present.

199.

Several reasons were identified for these assumptions not holding.

The primary reason being that most of the outcomes and intended impacts could not have materialised in thirty-six months.
The requirement that several countries should have ratified the Nagoya Protocol and have legislation giving effect to the ABS/ Traditional Knowledge agenda within the life of the project was always going to be difficult given the fact that this was beyond the influence of the PMT.

200.

The articulation of new policy, legislation and regulations usually take time to come on stream in the participating islands.
Their adoption was dependent on increasing knowledge and awareness of the benefits of signing on to the NP, which would take time since the approval of the country’s Cabinet and Parliament is needed.

As a result, just two additional countries, Antigua and Barbuda, and St. Kitts and Nevis ratified the NP.

201.

Insufficient allowance was made for the limited capacity constraints of the respective countries.
The NFP and ABS Focal Points in the respective countries were already tasked with various assignments, were insufficiently resourced, and had difficulty providing the necessary support at the national levels.

While allowance was made to address that specific constraint through the provision of Technical Coordinators to the respective countries, that support was only for a total of ten months (part-time) engagement over 24 months.

202.
202.

While several other participating countries have expressed their desire to ratify the NP, that is a decision which has to be made by the Cabinet in the respective countries, and requires at best, amendments to existing legislation or new legislation.

203.

The project, both in the design and implementation phases, had a heavy bias towards the public sector.
While this is understandable, given the need to develop requisite policies, legislation and ABS implementation tools and templates, there was little if any provisions made for engagement with private sector entities whose commercial interests could have acted as a powerful driver of change.

204.

Great emphasis was placed on public awareness engagement.
However, while these are all of great quality and received considerable publicity, there is a general belief that the efforts to engage with high-level Stakeholders such as politicians could have commenced much earlier.

Also, a few persons interviewed noted that there was no use of social media and that sharing of that particular information and getting the message across will require a more direct engagement utilising a more grass-roots approach.
In the absence of any measure to assess the penetration and effectiveness, it is difficult to determine how well the information was received in the respective countries and what impact it will have on the overall effectiveness of the awareness-raising programme which was an integral part of the project.

53 | P a g e Terminal Evaluation: “Advancing the Nagoya Protocol in Countries of the Caribbean Region” 205.
Though very few countries have formally initiated action to incorporate regulations and guidelines given to them, several have indicated that they have already commenced use of the PIC and MAT templates in the guidelines used to permit bioprospecting in their respective countries.
They do recognise that formal amendments are required to ensure a legal basis for these measures, however until these can be achieved, they will continue using these instruments to ensure that access to their genetic resources and Traditional Knowledge are permitted on a fair and equitable basis.

Going forward, however, will require continued awareness-raising and it is anticipated that PIC and MAT Templates, along with online facilitation of bioprospecting request will be replicated.

206.
206.

Notwithstanding the failure to realise those intended outcomes and impacts, the project achieved a fair amount of success, some of which, if pursued by the countries could still result in the long-term benefits being achieved.

However, without financial support, and given the structural and capacity constraints which were not adequately addressed, it is hardly likely that the intended impacts will be fully realised.
Table 10: Evaluation Criteria and Rating Table Criterion Summary Assessment Rating13 A.

Strategic Relevance i. Alignment to the UNEP Medium Term Strategy (MTS) and Programme of Work (POW) ii.

Alignment to UNEP / Donor/GEF Strategic Priorities iii.

Relevance to Regional, Sub-regional and National Environmental Priorities iv.

Complementarity with Existing Interventions B.
Quality of Project Design There is alignment to the UNEP MTS and POW There is alignment to the strategic priorities of UNEP/GEF/Donor The project is highly relevant to Regional, sub-regional and National Environmental Priorities The project was an ideal complement to other initiatives started by other regional organisations in collaboration with other international entities Barriers and constraints were correctly identified, and for the most part, the project sought to have them addressed.
There were some minor omissions in the design such as the limited support in respect to coordination at the national levels, the omission of hurricanes as a major risk factor, the absence of an objective means of measuring the achievements of outcomes and impacts and just a slight reference to gender.

However, this was addressed during implementation.

C. Nature of External Context External factors such as natural disasters could cause disruptions in travel and communications.
Two HS HS HS HS HS S F 13 The rating used for each section is as follows: HS - Highly Satisfactory, S – Satisfactory, MS - Moderately Satisfactory, MU - Moderately Unsatisfactory, U – Unsatisfactory, HU - Highly Unsatisfactory.

For Nature of External Context, the rating scale is changed to HF - Highly Favourable, F - Favourable, MF - Moderately Favourable, MU - Moderately Unfavourable, U – Unfavourable, and HF - Highly Unfavourable.

(Note that this is a reversed scale).


54 | P a g e Terminal Evaluation: “Advancing the Nagoya Protocol in Countries of the Caribbean Region” Criterion Summary Assessment Rating13 D. Effectiveness i.

Delivery of Outputs ii.

Achievement of Direct Outcomes iii.

Likelihood of Impact E. Financial Management i. Completeness of project financial information ii.
Communication between finance and project management staff F. Efficiency Category 5 hurricanes struck the region during project implementation, but they had negligible impacts.

All of the outputs were completed, though there were some delays with the delivery of some critical outputs.

Only two of the six DOs were realised, making it difficult to envisage that other intended impacts would be realised.
Three countries have ratified the NP, and several others have commenced incorporation of some of the ABS guidelines for PIC and MAT.

However, without the institutional support and a recognised legal framework in place, it is hardly likely the intended Impact will be realised.

All financial requirements between the EA and IA were satisfied.

Countries were provided with co-financing templates and instructions in completing their reports.
Not all countries/agencies submitted their reports, and several did not meet their full co-financing commitment.

Communications between the EA and IA were excellent.

There were two instances of disbursement delays by the IA, but good relations were maintained throughout the life of the project.

Efficiency was practised and achieved throughout the life of the project.

With a limited budget and tight time frame, the project delivered all outputs within thirty-six months.
Other measures of efficiency included the development of regional guidelines and templates, adopting a common approach the drafting of roadmaps and the use of virtual meetings to save cost on travel and reduce the carbon footprint of the project G. Monitoring and Reporting Overall Monitoring and Reporting rating i.

Monitoring design and budgeting Monitoring and design are consistent with GEF/UNEP guidelines.

ii.
ii.

Monitoring of project implementation All reports, including PIRs and financial reports, were prepared.

The reporting format, however, did not allow for objective monitoring of progress in implementation.

iii.

Project reporting Reports were prepared and submitted on time.

H. Sustainability 1.

Socio-political sustainability 2.


Financial sustainability Several countries have expressed a desire for some form of continuation to allow more time to raise awareness and provide for the implementation of some MU HS U U MS MS S HS S HS S HS MU MU ML 55 | P a g e Terminal Evaluation: “Advancing the Nagoya Protocol in Countries of the Caribbean Region” Criterion Summary Assessment Rating13 3.
Institutional sustainability I Factors Affecting Performance Preparation and readiness Quality of project management and supervision14 Stakeholders participation and cooperation Responsiveness to human rights and gender equity Country ownership and driven-ness Communication and public awareness Overall Project Rating outputs.

However, without additional financial support, it is unlikely they will achieve much progress.
Not much progress made in creating the institutional framework for continued implementation.

Without additional support, it is unlikely the legislative framework will be further advance and further strengthening of the institutional framework.

Overall Factors Affecting Performance rating An initial delay with disbursement suggests a lack of readiness.

However, it was not a significant delay.


Preparation concerns had more to do with the relatively large number of outputs, given the time frame for implementation and the overambitious nature of some assumptions.

MU MS MU There were no real complaints with management and supervision.

S Stakeholder participation has generally been positive.

Various media were used to engage stakeholders, ensuring that all targeted groups were engaged.
It was, however, expressed that engagement with HL decision- makers could have commenced earlier.

Gender mainstreaming ensured that women, IP and local community groups were acknowledged and involved in initiatives to raise awareness and provided with tools to facilitate ABS mechanisms were addressed.

Country representatives in the majority of participating countries expressed strong support for the outputs undertaken under the project.
However, the slow uptake of concrete measures to institutionalise and incorporate ABS in policies, legislation and across ministries suggest some shortcomings in terms of ownership at the national level.

A range of communication media was used to raise to communicate with the public/ stakeholders, including radio and TV announcements, billboards and video clips.

The sustainability of the information shared is dependent on both more grass-root delivery and direct- engagement.
Time and resources did not permit for that level of engagement.

MS S MU MS MS 14 In some cases ‘project management and supervision’ will refer to the supervision and guidance provided by UN Environment to implementing partners and national governments while in others, specifically for GEF funded projects, it will refer to the project management performance of the executing agency and the technical backstopping provided by UN Environment, as the implementing agency.
56 | P a g e Terminal Evaluation: “Advancing the Nagoya Protocol in Countries of the Caribbean Region” 6.2 Lessons Learned 207.

The following table presents a summary of the main lessons learned from some of the project’s successes and challenges.

Table 11: Main Lessons Learned Context: The most significant comment aired in respect of the failure of the project to realise all intended Direct Outcomes and Impacts was “insufficient time”.
There is validity to this assertion given the fact that several of those outcomes required changes in policy and regulations, both of which require time.

That time may be relative; however, when it requires the heightening of awareness of the NP and ABS mechanisms, the preparation and approval of policy documents, and the drafting and adoption of laws and regulations, that require the approval of Cabinet and the Parliament, these could take years to materialise.
Lesson #1: Projects intended to accomplish objectives that require structural changes and necessary supporting policy, legislative and regulatory must provide adequate time for those changes to be realised.

Application: UNEP must ensure that in the designing of projects, that sufficient acknowledgement is made of challenges inherent in adopting policy and necessary legislation and regulations.
In that regard, projects that require the adoption of new policies and legislation should be phased such that more practical implementation targets could be defined and executed.

Context: While there were not too many complaints about capacity constraints, it was, indeed a major concern as none of the countries was able to assign to the project, one officer tasked with its dedicated responsibility ensuring its successful implementation.
Persons assigned to the project were already engaged with other activities.

This is not unusual, given the financial and staffing constraints faced by small countries.

Provision was made in the project to assign a Project Coordinator to work with each of the countries.

However, funding to engage such support was only provided for a total of 10 months out of the thirty-six-month implementation period of the project.
Lesson #2: Regionally executed projects should be provided with adequate technical and financial support to provide for the implementation of projects.

That support does not necessarily mean full-time engagement of a consultant (Project Coordinator), but the allocation of a Project Coordinator to adequately deal with issues such as engagement with stakeholders and raising public awareness-raising to obtain buy- in, particularly by High-Level stakeholders and decision-makers.
Application: Implementing Agencies should ensure that adequate Technical Support is provided for and clear sets of undertakings defined, to ensure that the efforts mainly target outputs which are likely to be the most difficult to implement.

57 | P a g e Terminal Evaluation: “Advancing the Nagoya Protocol in Countries of the Caribbean Region” Context: The project, both in the design and implementation phases, had a heavy bias towards the public sector.
While this is understandable, given the need to develop requisite policies, legislation and ABS implementation tools and templates, there was little if any provisions made for engagement with private sector entities whose commercial interests could have acted as a powerful driver of change.
Lesson #3a: Lesson #3b: Bioprospecting has great relevance and importance to both the overall economy and Indigenous Local Communities and can be a powerful driver of conservation and sustainable use if its commercial value can be readily identified and recognised.

However, for the bioprospecting regime to function effectively, the enabling conditions (institutional and legislative) must be established and operational.
The private sector (commercial entities and researchers) as high-stake/ influence agents of change as well as key beneficiaries of a functional ABS system, are integral to its implementation.

In this regard, initiatives to attract their involvement at an early stage of the project is critical to its successes.

Application: National governments must create and establish the enabling conditions for bioprospecting to become operational.
This would include the necessary institutional and legislative frameworks and design incentives which could make bioprospecting attractive to private sector entities and research institutions.

Context: The implementation of projects in a multi-country (regional) format, as this one was, is attractive given the fact that they usually require the delivery of common outputs.
However, the high cost of travel between countries could limit the extent of participation by some participating countries, consume valuable resources that could be diverted to other uses, and further increase the carbon footprint associated with travel by project personnel.

Lesson #4: While the project only mandated the convening of three Steering Committee meetings, an additional three meetings were convened via the use of a virtual platform, resulting in cost-savings.
The project’s carbon footprint was also reduced as a result of the reduced travel and can become a feature of future projects that are similarly implemented on a multi-country format.

Application: Future projects can make similar use of virtual meetings.

Once this is recognised as an effective means of communications, and scheduled as part of project implementation, its effectiveness and participation rate could improve.

Context: The project was successful in completing all forty-one outputs.
However, by the time the last of these outputs were delivered, the project had come to an end, depriving countries of the key information which would have provided guided the development of potential revenue-earning bioprospecting initiatives.

Lesson #5a.
Lesson #5a.

Projects which have the potential to generate economic opportunities, particularly enhancing livelihoods for indigenous and local communities, should be delivered on 58 | P a g e Terminal Evaluation: “Advancing the Nagoya Protocol in Countries of the Caribbean Region” Lesson #5b time and where possible, initiate projects in local communities which can be used as models for replication in other communities.
When projects are not results-oriented, there may be a risk of project teams becoming preoccupied with completion of activities and delivery of outputs, and checking them off, with no real consideration of the TOC and intended Impact Application Projects should always seek to identify and provide support to local communities in the establishment of community-based enterprises which can be used as models for scaling up and replicating in other communities.
Context: Awareness-raising, particularly among High-Level stakeholders, was seen as critical in achieving the uptake of the Nagoya Protocol.

However, adequate resources were not allocated for this initiative in terms of technical and financial support, and more importantly, engagement occurred too late in the project cycle to make it effective.
Lesson #6: Given the strategic importance of awareness-raising, adequate funding should be made available to ensure that the intended outcomes will be realised Application: In the design of projects, the critical path concerning the implementation of project activities should always be identified so that adequate resources could be applied to each of the activities along this critical path to minimize delays in implementation.

6.3 Recommendations 208.


The following table presents recommendations to the project, based on the findings of this evaluation.

Table 12: Recommendations Recommendation #1 UNEP Ecosystem Division and other international partner agencies must ensure that in the designing of projects, that sufficient acknowledgement is made of challenges inherent in adopting policy and necessary legislation and regulations.
In that regard, sufficient time should be built into the project to allow for a direct engagement at the national levels to heighten awareness and provide greater opportunities for the preparation and adoption of policies, legislation and regulations.
Recommendation #2 Implementing Agencies should ensure that adequate Technical Support is provided for and clear sets of undertakings defined, to ensure that the efforts mainly target outputs that are most critical to producing the causal changes needed to achieve Impact (i.e.

a more results-oriented approach) Recommendation #3 For bioprospecting to become operational, National governments must create and establish the enabling conditions.
This will include the necessary institutional and legislative frameworks and design incentives which could make bioprospecting attractive to private sector entities and research institutions.
59 | P a g e Terminal Evaluation: “Advancing the Nagoya Protocol in Countries of the Caribbean Region” Recommendation #4 UNEP Ecosystem Division should, in partnership with other international and regional institutions, provide additional assistance to countries, primarily aimed at raising awareness and providing support for the establishment and delivery of community-based enterprises using some of the recommendations contained in the Business Model for ABS prepared under the project.
60 | P a g e Terminal Evaluation: “Advancing the Nagoya Protocol in Countries of the Caribbean Region” Annex 1.

Response to stakeholder comments received but not (fully) accepted by the evaluator Statement and Stakeholder Comment Response of Evaluator #6.
The intervention logic was based on a log frame analytical methodology, which adequately identified outputs and outcomes but was weak in terms of identifying the pathways through which Outputs would lead to Direct Outcomes, Intermediary State and Impacts.

“I believe this is an oversimplification.

The pathway to achieving the outputs where identified, the reason why some of the outcomes where not fully achieved, in my opinion was a matter of timing and/or unrealistic planning”.
#8 “What this meant was that as time became a factor, and some of those assumptions did not materialise in the anticipated time-frame, several of the direct outcomes did not materialise.

“This is correct, but it is also correct to say that the executing agency together with the steering committee, did manage in most cases to substitute the unfeasible outcomes for new feasible ones, depending on the context”.

#9.
#9.

However, not all Direct Outcomes and intended Impacts are clearly discernible, raising doubts regarding their sustainability, now that it has ended.

“In deed there are some sustainability questions, but in most cases, the countries that did ratified the protocol, are continuously making improvement to apply the legislation (Antigua & Barbuda, Guyana and SKN).
In the case on Saint LUCIA (which did not ratify), for example, there is an initiative to continue the ABS regional project in order to work an ABS project at national level.

This is currently in discussion with UNEnv.” As noted in #63 and #64 that while the “Results Framework was useful in identifying Outputs to Outcomes, the Outcomes were not directly linked to the Outputs”.
Also “ the reconstructed ToC was undertaken to help establish the project’s logical structure, identify causal links between Outputs, Direct Outcomes, and the Intermediary States, through to the realisation of Intended Impact, and determine the likelihood that the intended results will be realised”.

That required linking assumptions and drivers with the respective Direct Outcomes, Intermediary State and Impacts.
While several assumptions were made, not all of them held, and timing and unrealistic planning were significant issues, but so too was the absence of drivers.

It is, therefore, not an “oversimplification” to say that the logical framework model was inadequate.

Reference is also made to Table 5: Assumptions and Realities where a detailed analysis is provided of the Assumptions and how far some of them were from reality and the extent to which there were gaps in the log frame model.
This has been acknowledged Personnel in Grenada, St. Kitts and Nevis and Saint Lucia all expressed their intention to pursue follow up activities to ensure ratification of the NP.

The OECS Commission as well as GIZ also indicated their willingness to support Member States that have expressed that desire to proceed with ratification.
However, it was also clear from interviews with personnel in the respective countries that some of the issues which hampered ratification previously, such as capacity constraints, and finances were still of some concern, notwithstanding the promised assistance from various agencies.

This is a sustainability issue which, at the time of evaluation, could not be ignored.
61 | P a g e Terminal Evaluation: “Advancing the Nagoya Protocol in Countries of the Caribbean Region” Statement and Stakeholder Comment Response of Evaluator #108 “Direct outcome refers as well said, to internal and external factors characterizing the level of performance or achievement that occurred because of the activity or services provided by the project.

Is this correct?
Is this correct?

In this case and considering what is stated in paragraphs 106,107 and 108 I do not agree the rating U. I am fully aware with the limitations faced to achieve all direct outcomes but considering the context in which the project stated (baseline) I strongly believe that the products and results obtained where successful.

I will kindly ask to reconsider this rating”.


The “Unsatisfactory” rating for Section 5.5.2 “Achievement of direct outcomes” is based on the determination that the Direct Outcomes are yet to materialise.

As stated in the report “while some countries have expressed a desire to continue working towards achieving these outcomes, at this stage of the evaluation, when the project has come to an end, these outcomes are yet to materialise”.
The failure of Direct Outcomes to materialise is not an assessment of the work of the project team.

This does not imply failure of the executing team.

One has to look at the project objectively and on the performance over the course of its implementation and why performance was the way it was.

Notwithstanding the above, a revision or upgrade from Unsatisfactory to Moderately Unsatisfactory is provided in acknowledgement that some Direct Outcomes were achieved.
62 | P a g e Terminal Evaluation: “Advancing the Nagoya Protocol in Countries of the Caribbean Region” Annex 2: List of Interviewees # Name 1 Diann Black Layne Country Antigua & Barbuda Chief Environment Officer Job Title Organisation Ministry of Lands, Marine Affairs, Housing and Environment Appointment Date September 9-10 2 Helena Jeffery Antigua & Barbuda Technical Coordinator Brown Ministry of Lands, Marine Affairs, Housing and Environment September 9-10 3 Nneka Nicholas Antigua & Barbuda Technical Officer / Legal Consultant Ministry of Lands, Marine Affairs, Housing and Environment September 9-10 Trisha Lovell Jonah Ormond Antigua and Barbuda Antigua & Barbuda Fisheries Officer Registrar Ministry of Lands, Marine Affairs, Housing and Environment Pesticides and Toxic Chemicals September 16 - 19 Vidyanand Mohabir Guyana Environmental Officer Environmental Protection Agency September 16 – 19 Toshao Loretta Fiedtkou David Persaud Julius Smith 10 Candace Amoroso 11 Marissa Moses 12 Eavin Parry 13 Jannel Gabriel 14 Donation Gustave Saint Lucia Guyana First People Trinidad and Tobago Trinidad and Tobago Trinidad and Tobago Trinidad and Tobago St. Kitts and Nevis St. Lucia Environmental Manager Environmental Biologist Biodiversity Specialist Ministry of Planning and Development Ministry of Planning and Development Ministry of Planning and Development Environmental Scientist Sustainable Development and Environment Officer Chief Forestry Officer Department of Environment Department of Sustainable
 Development Forestry Department August 21, 2019 August 21, 2019 August 21, 2019 September 11, 2019 15 Aria St Louis Grenada Head of Environment Division Ministry of Agriculture, Lands, Forestry, Fisheries and Environment 63 | P a g e Terminal Evaluation: “Advancing the Nagoya Protocol in Countries of the Caribbean Region” # Name 16 Hartmut Meyer Country Germany 17 Amrikah Singh Guyana 18 Norma Cherry Fevrier 19 Thérèse Yarde St. Lucia Barbados 20 Melesha Banhan 21 María Pía Hernández P. Antigua & Barbuda Costa Rica 22 Domenique Finegan Costa Rica 23 Michael Fung Costa Rica Job Title Advisor / Global Project ABS Capacity Development Initiative Programme Manager, Sustainable Development Program Officer Former Project Coordinator - Caribbean Hub Capacity Building ACP-MEAs Former Project Coordinator Coordinator, Biodiversity and Rights Technical Officer, Biodiversity and Rights Financial Officer 24 Gloritzel Frangakis, 25 Christopher Cox Panama Panama GEF Biodiversity Unit Task Manager Organisation GIZ CARICOM OECS CARICOM Appointment Date Friday August 16, 2019.
8:30 am September 17, 2017 September 12, 2019 N/A IUCN - Mexico, Central America and Caribbean IUCN - Mexico, Central America and Caribbean IUCN - Mexico, Central America and Caribbean UNEP UNEP September 23, 2019 September 23, 2019 September 23, 2019 September 25, 2019 September 25, 2019 64 | P a g e Terminal Evaluation: “Advancing the Nagoya Protocol in Countries of the Caribbean Region” Annex 3: List of documents consulted Project Design Documents 1.
Project Document (PRODOC) – Advancing the Nagoya Protocol in Countries of the Caribbean Region 2.

Proposed Biennial Programme of Work and Budget (POW) for 2014-2015, Governing Council of the United Nations Environment Programme September 2017 3.

Proposed Biennial Programme of Work and Budget (POW) for 2016-2017, United Nations Environment Assembly of the United Nations Environment Programme 4.

Request for CEO Approval - Advancing the Nagoya Protocol in Countries of the Caribbean Region 5.
United Nations Environment Programme, Evaluation Policy, Draft of March 2016 6.

Use of Theory of Change in Project Evaluations, Evaluation Office of UN Environment, Last Reviewed 16.12.16 7.

ABS Strategic Guidelines for Countries of the Caribbean 8.

Final Mid-Term Review Report Project Progress Reports 9.

(BTOR) - OECS Ministers of Environment Meeting presentation for the project Advancing the Nagoya Protocol in Countries of the Caribbean Region – [2nd May- 6th April 2016] 10.
Guidance on the Structure and Contents of the Main Evaluation Report, Evaluation Office of UN Environment, Last Reviewed 16.12.16 11.

IUCN Back-to-Office Report (BTOR) - ABS project Synergies Meeting–Bahamas: 24th – 27th August 2016 12.

IUCN Back-to-Office Report (BTOR) - Regional Meeting on Bioprospecting and Legislation and First National Meeting for Barbados for the Advancing the Nagoya Protocol in Countries of the Caribbean Region Project – [19th February- 24th February 2017] 13.
IUCN Back-to-Office Report (BTOR) - Clearing House Mechanism Meeting–Jamaica: 21st – 22nd July 2016 14.

IUCN Back-to-Office Report (BTOR) - In-Country Meetings with Project Countries for the Advancing the Nagoya Protocol in Countries of the Caribbean Region Project – [15th March- 18th March 2016] 15.

IUCN Back-to-Office Report (BTOR) - National Legislation Meetings–Jamaica: Aug 2nd 2016; Guyana- August 4th -5th; St Lucia- August 8th -9th, Grenada- August 11th and 12th; Antigua- August 15th and 16th 16.
IUCN Back-to-Office Report (BTOR) - CBD COP Side Event Presentation for the project Advancing the Nagoya Protocol in Countries in the Caribbean Region – [4th December to 8th December 2016] 17.

IUCN Back-to-Office Report (BTOR) - National Workshops on Traditional Knowledge and the Nagoya Protocol–Jamaica: 23rd July to Aug 1st 2016 18.
IUCN Back-to-Office Report (BTOR) - OECS Ministers of Environment Meeting presentation for the project Advancing the Nagoya Protocol in Countries of the Caribbean Region – [2nd May- 6th April 2016] 19.

IUCN Back-to-Office Report (BTOR) - Fourth Regional Meeting and First National Workshop for the Federation of Saint Kitts and Nevis for the Advancing the Nagoya Protocol in Countries of the Caribbean Region Project – [19th June- 22nd June 2017] 20.
IUCN Back-to-Office Report (BTOR) - Inception Meeting and First National Workshop for Trinidad and Tobago for the Advancing the Nagoya Protocol in Countries of the Caribbean Region Project – [23rd April- 29th April 2016] 21.

IUCN - Video marketing and dissemination plan, February 2017 22.

Medium Term Strategy 2014-2017, United Nations Environment Programme (UNEP), January 2015 23.
Minutes of the first Steering Committee Meeting April 26th 2016 65 | P a g e Terminal Evaluation: “Advancing the Nagoya Protocol in Countries of the Caribbean Region” 24.

Report of the Workshop on the ABS Clearing House of the Advancing the Nagoya Protocol in Countries of the Caribbean Region project July 21st – 22nd, 2016 25.
Inception Workshop Advancing the Nagoya Protocol in Countries of the Caribbean Region April 25, 2016 Hilton Hotel Port of Spain, Trinidad & Tobago report Four Seasons Hotel Kingston, Jamaica 26.

Report of the First National Workshop of Jamaica for the Advancing the Nagoya Protocol in Countries of the Caribbean Region project July 25-26 & 28-29, 2016 27.

Minutes of the First Steering Committee Meeting, April 26, 2016.

Hilton Trinidad & Conference Centre, Trinidad.

28.
28.

Minutes of the Second Steering Committee Meeting; Virtual Meeting via Zoom, March 27th, 2017 29.

Minutes of the Third Steering Committee Meeting; October 23rd, 2017 Jolly Beach Resort Bolans Village, Antigua & Barbuda 30.

Minutes of the Fourth Steering Committee Meeting; Virtual Meeting via Zoom April 18th, 2018 31.

Minutes of the Fifth Steering Committee Meeting; Starfish Jolly Beach Tuesday, October 30, 2018 32.
Minutes of the Sixth Steering Committee Meeting; Virtual Meeting via Zoom Wednesday, March 13, 2019 33.

PIR – July 1, 2015 to June 30, 2016 - Advancing the Nagoya Protocol in Countries of the Caribbean Region 34.

PIR – July 1, 2016 to June 30, 2017 - Advancing the Nagoya Protocol in Countries of the Caribbean Region 35.

PIR – July 1, 2017 to June 30, 2018 - Advancing the Nagoya Protocol in Countries of the Caribbean Region 36.
PIR – July 1, 2018 to June 30, 2019 - Advancing the Nagoya Protocol in Countries of the Caribbean Region 37.

Half Yearly Reports Progress Report, July 2016 - December 2016.

38.

Half Yearly Reports Progress Report, July 2017 To December 2017 39.

Half Yearly Reports Progress Report, July 2018 To December 2018 Project Deliverables 40.

Sylvia A. Mitchell Anthony Richards Kevel C. Lindsay) Advancing the Nagoya Protocol in Countries of the Caribbean Region Project (GFL/5060-2711-4E67) 41.
Assessment of Relevant Draft Legislation and Policy Framework for Implementation of the Nagoya Protocol in Saint Christopher and Nevis 42.

Antigua & Barbuda: ABS Legislative Assessment and Review Environmental Governance Consulting (2019) Development of a Business Model for Access and Benefit Sharing in the Caribbean Region, Prepared for IUCN, San Jose Costa Rica, 78 pages 43.
Accession to The Nagoya Protocol on Access to Genetic Resources and The Fair and Equitable Sharing of Benefits Arising from Their Utilization (Abs Protocol) to The Convention on Biological Diversity (CBD) By the Government of The Republic of Trinidad And Tobago 44.

Bioprospecting in the Caribbean Region – Caribbean ABS Institutional Mapping (DRAFT), Anthony Richards, Sylvia Mitchell, Kevel C. Lindsay, February 28, 2017 45.
Press Release - IUCN and UN Environment launch promotional videos for the Caribbean on the Nagoya Protocol 46.

Trinidad and Tobago: ABS Policy Skeleton 39.

Trinidad & Tobago – Initial Review of National Legislative Environment Relative to Access and Benefit- sharing and the Ratification of the Nagoya Protocol 66 | P a g e Terminal Evaluation: “Advancing the Nagoya Protocol in Countries of the Caribbean Region” 40.

Saint Lucia Assessment and Review of Draft ABS Legislation and Alternative Approaches 41.
Jamaica: ABS Assessment and Review (based on the work of Tomme Rosanne Young, consultant, in conjunction with the IUCN Environmental Law Centre and the IUCN Regional Office for Mexico Central America and the Caribbean) 42.

Barbados Assessment and Review of Natural Resource Laws, in the Context of ABS Implementation 43.

Melinda Janki, (2018) Review of the proposed templates on PIC and MAT and the explanation guide.

44.
44.

Melinda Janki Feb-19 Guide to the Template incorporating PIC & MAT for Access and Benefit haring Draft 11th February 2019 Agreement for Academic Research 45.

Morten Walløe Tvedt (2019) Guide to Developing ABS Contracts: Contract Example Tool, IUCN, Gland, Switzerland in collaboration with Regional Office for Mexico, Central America and the Caribbean 46.

Morten Walloe Tvedt in collaboration with Olivier Rukundo The ABS Contract Tool: Version 2.0.

GIZ, Germany 47.
GIZ, Germany 47.

Training of Trainer Manual: Advancing the Nagoya Protocol in the Countries of the Caribbean.

IUCN, Gland, Switzerland in collaboration with Regional Office for Mexico, Central America and the Caribbean 48.

Sylvia Mitchell, Kevel C. Lindsay, Anthony Richards.

(2018) Bio Prospecting in The Caribbean Region Caribbean Abs Institutional Mapping.

Iucn, Gland, Switzerland In Collaboration with Regional Office for Mexico, Central America And the Caribbean 49.
Sylvia Mitchell, Kevel C. Lindsay, Anthony Richards.

(2018) Bio Prospecting in The Caribbean Region Caribbean Abs Traditional Knowledge Stocktaking Traditional Knowledge in the Caribbean Region.

IUCN, Gland, Switzerland in collaboration with Regional Office for Mexico, Central America and the Caribbean 50.

Sylvia Mitchell, Kevel C. Lindsay, Anthony Richards.

(2018) Bio Prospecting in The Caribbean Region Biodiversity Summary of eight Caribbean countries.
IUCN, Gland, Switzerland in collaboration with Regional Office for Mexico, Central America and the Caribbean 51.

Sylvia Mitchell, Kevel C. Lindsay, Anthony Richards.

(2018) Standardized Methodology for Creation of Caribbean National Registers of Their Marine and Terrestrial Biological Resources 52.
Action Plan for The Implementation of Regional Strategic Guidelines for Countries in the Caribbean on Access to Genetic Resources and Associated Traditional Knowledge and the sharing of Benefits derived from their utilization draft text for discussion – February 2019 53.
(Eppd): January 2018 Government of the Republic of Trinidad and Tobago: Supplemental Note: Towards the Accession of the Nagoya Protocol on Access to Genetic Resources and the Fair and Equitable Sharing of Benefits Arising from their Utilization to the Convention on Biological Diversity Cabinet 54.

Information Sheets: #1 Is your Country Losing Out in the Global Search for New Medicines & Cosmetics?

55.
55.

Information Sheet #2: Tour Guides Access and Benefit Sharing Regulations Made Under the Environmental Protection Act Cap 20:05Revised Draft Guyana Regulations 56.

Saint Lucia: Draft Legislative Provisions: Access To Genetic Resources and Sharing of Benefits Arising From Their Utilization 57.

Annex 12 GEF ABS Project Final Report 1 FINAL REPORT Financial Reports 58.

Audit Reports PROJECT: “Advancing the Nagoya Protocol in Countries of The Caribbean Region”.

2016, 2017 2018-2019 59.


Budget Revisions #1 (end of 2015) - #7 (2018) 60.

Periodic (Quarterly) Expenditure Reports – Total 14.

67 | P a g e Terminal Evaluation: “Advancing the Nagoya Protocol in Countries of the Caribbean Region” 61.

Amendment No.1 to the PCA Between the United Nations Environment and IUCN 62.

Project Cooperation Agreement Between the United Nations Environment and IUCN 63.

CEO Endorsement Letters 64.

Co-Financing Letters 65.

Co-Financing Budget 66.

Co-Financing Reports 67.


Final Financial Report August 2019 68 | P a g e Terminal Evaluation: “Advancing the Nagoya Protocol in Countries of the Caribbean Region” Annex 4: Evaluation TORs (without annexes) SECTION 1: PROJECT BACKGROUND AND OVERVIEW GFL-5060-2711-4E67 International Union for the Conservation of Nature (IUCN) Ecosystem Management (expected accomplishment A), Environmental Governance (expected accomplishment B and C).
Medium Size Project (MSP) Biodiversity February 2016 Project General Information 1.

Table 1.
Project summary GEF Project ID: Implementing Agency: 5774 UN Environment IMIS number: Executing Agency: Sub-programmes: Expected Accomplishment(s): Ecosystems Management and Environmental Governance September 2015 UN Environment approval date: July 2015 GEF approval date: BD - 5 GEF Strategic Priority: Expected start date: September 2015 Planned completion date: September 2018 Planned project budget at approval: GEF grant allocation: USD 1,826,000 USD 5,635,257 Project Preparation Grant - GEF financing: Expected Medium-Size Project co-financing: First disbursement: No.
of revisions: No.
of Steering Committee meetings: Mid-term Review/ Evaluation (planned date): Terminal Evaluation (planned date): Coverage - Country(ies): USD 90,000 USD 3,809,257 February 2016 3 June 2017 September 2018 USD 1,061,141.76 USD 41,356.41 Programme of Work Output(s): Project type: Focal Area(s): Actual start date: Actual completion date: May 2019 Actual total expenditures reported as of [date]: GEF grant expenditures reported as of March 2019: Project Preparation Grant - co-financing: Secured Medium-Size Project co-financing: Date of financial closure: N/A Date of last revision: Date of last/next Steering Committee meeting: Mid-term Review/ Evaluation (actual date): Terminal Evaluation (actual date): Coverage - Region(s): Caribbean June 2019 Last: March 2019 Next: September 2017 USD 2,391,352.54 (as of March 2019) Antigua & Barbuda, Barbados, Grenada, Guyana, Jamaica, Saint Kitts & Nevis, Saint Lucia, Trinidad & Tobago 2.
Project rationale 1.

The Caribbean islands support a wealth of biodiversity within its terrestrial ecosystems, with a high proportion of species that are endemic, or unique, to the hotspot.

It includes about 11,000 plant species, of which 72 percent are endemic.

With 70% of the population living along the coast, Caribbean lives and livelihoods directly depend upon healthy marine and coastal resources.
Due to the rich marine ecosystems of the Caribbean region and the fact that the ocean's biodiversity is higher than that recorded on land, bioprospecting of new marine natural products (NMNP) is gaining importance.

Bioprospecting is already common in the insular Caribbean but generally goes unchecked.

This poses a challenge for the Caribbean countries on how to manage their natural and genetic resources ensuring fair access conditions and the sharing of benefits.

2.
2.

The Caribbean region is also rich in Traditional Knowledge (TK) with a great variety of traditions that relate their folklore with biodiversity.

Amongst these practices, some of the most notorious are: traditional usage of fruits, plants and animals for medicinal purposes; traditional fishing methods, trapping, hunting and fishing techniques, traditional food culture and preservation techniques, handicraft and traditional environment preservation and conservation methods.

3.
3.

Although Caribbean Countries have enacted legislation for the protection of their environment and biodiversity, as well as laws governing forestry, land use and protected areas, there are still challenges concerning institutional and legal arrangements at the national level to protect the environment and provide the basis for the implementation of basic provisions of the Nagoya Protocol.
Furthermore, due to limited resources and experience in the ABS area, there is still a lack of capacities (institutional, systemic and individual) within government, local communities and among all key stakeholders in this regard.
There is also lack of awareness of issues concerning ABS which often result in ad hoc actions 69 | P a g e Terminal Evaluation: “Advancing the Nagoya Protocol in Countries of the Caribbean Region” taken as well as lack of support to implement strategies and priority activities to ensure that issues concerning this discipline are adequately addressed at all levels.

4.
4.

All countries participating in this project are signatories to the Convention on Biological Diversity, and most have been exposed in a limited extent (through regional GIZ and Convention on Biological Diversity workshops) to varying levels of introduction to ABS/Nagoya Protocol implementation requirements.

Due mainly to their size, countries in the region have limited resources for developing effective ABS measures and regimes.
At the start if the project, none of the participating countries had ratified the Nagoya Protocol, and only two (Antigua & Barbuda and Grenada) had signed it.
While uptake of the Nagoya Protocol has been slow in the Caribbean region, there is an observable pattern of actions towards its implementation and a clear indication of political interests, as expressed by numerous policy documents which fully embrace the sustainable use of biological and genetic resources, and an unequivocal commitment to obligations contracted under the CBD.

5.
5.

These countries are faced with the challenge of determining how to regulate access to genetic resources, how to implement the fair and equitable sharing of benefits resulting from the utilization of genetic resources and traditional knowledge associated with genetic resources, what enforcement measures will be required to ensure compliance by users, and what institutional and capacity building needs must be addressed to ensure all obligations under the NP are appropriately met.

6.
6.

The Advancing the Nagoya Protocol in Countries of the Caribbean Region Project (hereafter referred to as ‘Project’) therefore sought to support countries to uptake, and where possible, ratify or accede to the Nagoya Protocol and take the first steps in implementation.
This was to be achieved by assisting countries in the development of regulatory frameworks for ABS, building capacity for its implementation, and sharing the experiences from these countries to catalyze similar processes in the Caribbean region.

This would require developing appropriate capacities and measures to ensure that countries have the requisite conditions to meet the obligations under the Nagoya Protocol.
Gender mainstreaming was also to be incorporated in the implementation of a gender-oriented methodology to promote inclusive spaces within the existing governance structure of indigenous peoples and local communities, in which women particularly have an active presence.

7.
7.

The project is expected to have a direct impact on safeguarding genetic diversity of global importance and will specifically contribute to the conservation and sustainable use of 2.6 percent of the world’s 300,000 plant species and 3.5 percent of the world’s 27,298 vertebrate species, all of which are endemic to the Caribbean.

3.

Project objectives and components 8.


The development goal of the Project is to support countries of the Caribbean to facilitate access to their genetic resources and benefit sharing in a fair and equitable way, in line with the Convention on Biological Diversity and the Nagoya Protocol.

Its main objective is stated in the Project Document as “seeking uptake of the Nagoya Protocol and implementation of key measures to make the protocol operational in Caribbean countries”.
This objective is to be achieved through the implementation of the following inter-dependent components.

9.

Component 1.

Identifying regional commonalities and assets, and basic elements conducive to policy formulation.
The objective of this component is to build knowledge between countries of shared assets and generate technical information that can later be used to build cohesive policies at the national level and collaboratively at the regional level, such as a Scientific Study on Bio-prospecting in the Caribbean Region and Stocktaking of the main Applications of Traditional Knowledge in the region, which would be used to inform the formulation of National ABS Policies and a Regional ABS Policy.
The component also seeks to identify, and where possible set up, sustainability mechanisms for supporting countries the project, by creating networks and coordination mechanisms.

The expected Outcomes are: • base ABS policy.

• Outcome 1.2: Future directions of policy development for the region are defined.

• Outcome 1.3: Countries understand their national assets/values and requirements in a regional context 10.

Component 2: Uptake of the Nagoya Protocol.


The objective of this component is that participating countries take steps and decisions conducive to ratification of the Nagoya Protocol.
This includes support for developing the policy, legal, and regulatory frameworks governing ABS, assistance to improve their understanding of the implications of the Nagoya Protocol ratification in terms of adjustments in the legal and institutional framework, assistance in the development of draft ABS Bills and regulations, and in the development of regional strategic priorities for Nagoya Protocol implementation in the region.

The expected Outcome is: • and future implementation.

11.
11.

Component 3: Implementation of the Nagoya Protocol and establishing an enabling environment for the basic provisions of the NP.

The objective of this component is to assist countries in developing the tools and guidelines required to implement the basic measures of the Nagoya Protocol.
These include providing assistance to build awareness among stakeholders that are key for NP implementation to be effective, especially parliamentarians, officers of frontline ministries, indigenous communities and researchers.

Support is also being provided for the development of institutional agreements and administrative procedures for ABS Agreements such as Prior Informed Consent, Mutually Agreed Terms, and Benefit Sharing, and capacity building to create a Roster of Caribbean ABS Experts.
This component also provides support in the drafting of methodologies that could be used by the countries for creating Traditional Knowledge and Genetic Resources inventories, and supports strategies that could bring sustainability to the project results, such as a regional database of Outcome 1.1: Countries have a common understanding of shared assets/values, issues and needs on which to Outcome 2.1: National authorities take informed decisions on, and steps towards, the ratification of the protocol future, well past life of the the in 70 | P a g e Terminal Evaluation: “Advancing the Nagoya Protocol in Countries of the Caribbean Region” Outcome 3.1: An enabling environment is created which will lead to the implementation of the basic provisions of research activities in the OECS and broader Caribbean region, linked to existing Clearing House Mechanisms (CHMs) or institutional web pages in the region.
The expected Outcome is: • the Nagoya Protocol.

12.

Component 4.

Regional Coordination, technical support and capacity development.

This component will bring together the participating countries at least twice during the life of the project to allow for the maximum level of exchange and networking.

These meetings will also assist in assessing common issues and how to cope with them collectively.
This component also covers monitoring, and evaluation activities, as well as strategic project planning and coordination.

The expected Outcomes are: • Outcome 4.1: Countries share information and gain from the experiences of other countries.

• Outcome 4.2: Effective project coordination and delivery, meeting agreed measurable outputs and indicators 13.

Component 5.

Project Management.

This includes basic services needed for project execution.
The main difference between component 4 and component 5 is that Component 4 embraces the overall project coordination through facilitation of regional meetings, promoting synergies among relevant initiatives and stakeholders as well as networking opportunities.

4.

Executing Arrangements 14.

UN Environment is the GEF Implementing Agency and the International Union for the Conservation of Nature (IUCN) is the Executing Agency.
The regional coordination and overall project supervision are the responsibility of the Implementing and Executing agencies (UN Environment and IUCN respectively).

The project Headquarters is located in IUCN Mesoamerica, Costa Rica.

15.

The project was managed centrally as one project rather than 8 separate sub-projects, though funds support was for delivery of national-level Outputs.

Project execution at a regional level was the responsibility of IUCN as the project`s Executing Agency.
The UN Environment-GEF Task Manager was responsible for providing support and issues.

The Task Manager for this working closely with IUCN personnel on all project management related project was stationed in Panama and remained in constant communication with the Project Manager and the project team during its execution period.

Supervision to ensure that the project meets UNEP and GEF policies and procedures was the responsibility to the Task Manager.
The Task Manager also reviewed the quality of draft project outputs, provided feedback to project partners, and ensured adequate quality of scientific and technical outputs and publications.

16.

A Regional Steering Committee was appointed with representation from all project countries, the GEF Implementing Agency, the Executing Agency, and selected regional partners.

The Steering Committee was responsible for ensuring that the project met its goals.
Recommendations produced by the Steering Committee were expected to improve implementation strategies, annual work plans and resources allocation budget and, when necessary, to adjust the project’s Result Framework.

17.

The national counterparts were represented by National Focal Points and National Teams who were responsible for ensuring that project outputs related to national interventions would be delivered.
Moreover, the National Focal Points liaised with the National Coordination Consultants, guiding them and providing the necessary means to execute activities at a national level.

The National Coordination Consultants provided coordination support for National Focal Points on a needs basis.

18.
18.

Due to its regional nature, and to ensure sustainability of the project results, the project also worked in coordination with regional entities such as the CARICOM and OECS Secretariats, which play and important role in implementing ABS in the region, through means such as a common policy and/or strategy, harmonized procedures and criteria, or information sharing and coordination mechanisms.

19.

Diagram 1.
19.

Diagram 1.

Project Coordination Diagram The decision-making flowchart and organizational scheme are presented in Diagram 1 below: 71 | P a g e Terminal Evaluation: “Advancing the Nagoya Protocol in Countries of the Caribbean Region” 5.

Project Cost and Financing 20.

This was a medium-sized project.


The overall project budget was USD 5,635,257 comprising USD 1,826,000 from the GEF Trust Fund and USD 3,809,257 in co-financing from the governments of the participating countries (Antigua & Barbuda, Barbados, Grenada, Guyana, Jamaica, Saint Kitts & Nevis, Saint Lucia, Trinidad & Tobago), with support from UN Environment, IUCN, Convention on Biological Diversity (CBD), Organization of Eastern Caribbean States (OECS) and the German Technical Cooperation (GIZ).
The funding distribution is shown in Table 3 below: USD % 1,826,000 3,809,257 32.40% 67.60% Table 3.
Estimated project cost Source of funds Cost to the GEF Trust Fund Co-financing Total UNEP 90,000 CBD 50,000 IUCN 501,074 OECS 596,871 GIZ 50,371 Countries 2,520,941 Barbados $12,000 (cash), $123,000 (in kind) Guyana $284,547 (in kind) Antigua and Barbuda $ 400,000 (in kind) St Kitts & Nevis $612,360 (in kind) Trinidad & Tobago $187,000 (in kind) Saint Lucia $232,035 (in kind) Grenada $200,000 (in kind) Jamaica $470,000 (in kind) Total Project Budget 5,635,257 100% 6.

Implementation Issues 21.


A Mid-Term Management Review was completed in September 2017 which indicated that the project was progressing favourably and was on course in delivering its objectives by mid-term.

The main recommendations revolved around the sustainability of Outcomes.

According to the MTR, there was a probability that the sustainability of project results would be hampered by weak institutional structures and mechanisms for ABS implementation at country level.
The findings suggest that stronger socio-political support and uptake could be achieved by promoting direct engagement among national focal points, the line ministries responsible for ABS, as well as other representatives of relevant institutions directly involved in ABS implementation within the project countries.

the MTR also proposed extension of the project duration to achieve the delivery of Outcomes as per the project’s Logical Framework.

22.
22.

Based on the project’s Final Report, there is still a need for increasing stakeholder awareness, especially within the less exposed sectors that the project was not able to reach., as well as at the executive level in the various Caribbean countries.

There is also a great need to develop revenue generation strategies that can fund the systems that were created under this project and maximize the achievements and maintain the momentum that has been created.
Without socio- political support, continued awareness and funding, the potential for moving forward of the ABS agenda in the region may not be effectively achieved.

23. the Nagoya Protocol, namely Antigua & Barbuda, St. Kitts and Nevis and Guyana.

By the time of the project’s operational/technical completion, three of the eight participating countries had ratified SECTION 2.

OBJECTIVE AND SCOPE OF THE EVALUATION 7.

Key Evaluation principles 24.


Evaluation findings and judgements should be based on sound evidence and analysis, clearly documented in the evaluation report.

Information will be triangulated (i.e.

verified from different sources) as far as possible, and when verification is not possible, the single source will be mentioned (whilst anonymity is still protected).

Analysis leading to evaluative judgements should always be clearly spelled out.

25.

The “Why?” Question.


As this is a terminal evaluation and a follow-up project is likely [or similar interventions are envisaged for the future], particular attention should be given to learning from the experience.

Therefore, the “Why?” question should be at the front of the consultants’ minds all through the evaluation exercise and is supported by the use of a theory of change approach.
This means that the consultants need to go beyond the assessment of “what” the project performance was, and make a serious effort to provide a deeper understanding of “why” the performance was as it was.

This should provide the basis for the lessons that can be drawn from the project.

26.

Baselines and counterfactuals.


In attempting to attribute any outcomes and impacts to the project intervention, the evaluators should consider the difference between what has happened with, and what would have happened without, the 72 | P a g e Terminal Evaluation: “Advancing the Nagoya Protocol in Countries of the Caribbean Region” Objective of the Evaluation project.
This implies that there should be consideration of the baseline conditions, trends and counterfactuals in relation to the intended project outcomes and impacts.

It also means that there should be plausible evidence to attribute such outcomes and impacts to the actions of the project.

Sometimes, adequate information on baseline conditions, trends or counterfactuals is lacking.
In such cases this should be clearly highlighted by the evaluators, along with any simplifying assumptions that were taken to enable the evaluator to make informed judgements about project performance.

27.

Communicating evaluation results.

A key aim of the evaluation is to encourage reflection and learning by UN Environment staff and key project stakeholders.
The consultant should consider how reflection and learning can be promoted, both through the evaluation process and in the communication of evaluation findings and key lessons.

Clear and concise writing is required on all evaluation deliverables.

Draft and final versions of the main evaluation report will be shared with key stakeholders by the Evaluation Manager.

There may, however, be several intended audiences, each with different interests and needs regarding the report.
The Evaluation Manager will plan with the consultant(s) which audiences to target and the easiest and clearest way to communicate the key evaluation findings and lessons to them.

This may include some or all of the following; a webinar, conference calls with relevant stakeholders, the preparation of an evaluation brief or interactive presentation.

8.
8.

In line with the UN Environment Evaluation Policy1 and the UN Environment Programme Manual2, the Terminal Evaluation (TE) is undertaken at completion of the project to assess project performance (in terms of relevance, effectiveness and efficiency), and determine outcomes and impacts (actual and potential) stemming from the project, including their sustainability.
The evaluation has two primary purposes: (i) to provide evidence of results to meet accountability requirements, and (ii) to promote operational improvement, learning and knowledge sharing through results and lessons learned among UN Environment and the main project partners i.e.

Organisation of Eastern Caribbean States (OECS), GIZ, International Union for Conservation of Nature- Regional Office for Mexico, Central America and the Caribbean (IUCN-ORMACC), UN CBD Secretariat, Caribbean Community (CARICOM).
Therefore, the evaluation will identify lessons of operational relevance for future project formulation and implementation, especially for follow-on projects pursuing the uptake of the Nagoya Protocol and implementation of key measures to make the protocol operational in the Caribbean.

9.

28. the evaluation will address the strategic questions listed below.
These are questions of interest to UN Environment and to which the project is believed to be able to make a substantive contribution: a. this intervention overcome the identified barriers, gaps and challenges?

b. most critical for the achievement of Impact been found to hold?
c. Pertaining to the results that can be attributed to this intervention, which opportunities exist or have already been set in motion, that are likely to stimulate replication of positive outcomes and best practice experiences on ABS implementation?

d. governments and other stakeholders to adopt ABS measures at the national or regional level?

e. implementation, and if so, what was it and how might it affect the intended Impact?
Based on the theory of change analysis undertaken at evaluation, to what extent have the assumptions deemed Has the evaluation identified any unintended results (positive or negative) deriving from the project’s Have the outputs generated by the project have adequate level of authority/credibility needed to catalyse action by In its efforts to promote the uptake of the Nagoya Protocol in the Caribbean region, to what degree of success has Key Strategic Questions to addition In the evaluation outlined in 10 below, Section criteria 10.
Evaluation Criteria 29.

All evaluation criteria will be rated on a six-point scale.

Sections A-I below, outline the scope of the criteria and a link to a table for recording the ratings is provided in Annex 1).

A weightings table will be provided in excel format (link provided in Annex 1) to support the determination of an overall project rating.
The set of evaluation criteria are grouped in nine categories: (A) Strategic Relevance; (B) Quality of Project Design; (C) Nature of External Context; (D) Effectiveness, which comprises assessments of the delivery of outputs, achievement of outcomes and likelihood of impact; (E) Financial Management; (F) Efficiency; (G) Monitoring and Reporting; (H) Sustainability; and (I) Factors Affecting Project Performance.

The evaluation consultants can propose other evaluation criteria as deemed appropriate.

A.
A.

Strategic Relevance 30.

The evaluation will assess, in line with the OECD/DAC definition of relevance, ‘the extent to which the activity is suited to the priorities and policies of the target group, recipient and donor’.

The evaluation will include an assessment of the project’s relevance in relation to UN Environment’s mandate and its alignment with UN Environment’s policies and strategies at the time of project approval.
Under strategic relevance an assessment of the complementarity of the project with other interventions addressing the needs of the same target groups will be made.

This criterion comprises four elements: i.Alignment to the UN Environment Medium Term Strategy3 (MTS) and Programme of Work (POW) 31.
The evaluation should assess the project’s alignment with the MTS and POW under which the project was approved and include, in its narrative, reflections on the scale and scope of any contributions made to the planned results reflected in the relevant MTS and POW.

ii.Alignment to UN Environment / Donor/GEF Strategic Priorities 32.

Donor, including GEF, strategic priorities will vary across interventions.
UN Environment strategic priorities include the Bali Strategic Plan for Technology Support and Capacity Building4 (BSP) and South-South Cooperation (S-SC).
The BSP relates to the capacity of governments to: comply with international agreements and obligations at the national level; promote, facilitate and finance environmentally sound technologies and to strengthen frameworks for developing coherent 73 | P a g e Terminal Evaluation: “Advancing the Nagoya Protocol in Countries of the Caribbean Region” international environmental policies.

S-SC is regarded as the exchange of resources, technology and knowledge between developing countries.
GEF priorities are specified in published programming priorities and focal area strategies.

iii.Relevance to Regional, Sub-regional and National Environmental Priorities 33.

The evaluation will assess the extent to which the intervention is suited, or responding to, the stated environmental concerns and needs of the country, sub-region or region where it is being implemented.
Examples may include: national or sub-national development plans, poverty reduction strategies or Nationally Appropriate Mitigation Action (NAMA) plans or regional agreements etc.

iv.Complementarity with Existing Interventions 34.


An assessment will be made of how well the project, either at design stage or during the project mobilization, took account of ongoing and planned initiatives (under the same sub-programme, other UN Environment sub-programmes, or being implemented by other agencies) that address similar needs of the same target groups.
The evaluation will consider if the project team, in collaboration with Regional Offices and Sub-Programme Coordinators, made efforts to ensure their own intervention was complementary to other interventions, optimized any synergies and avoided duplication of effort.

Examples may include UN Development Assistance Frameworks or One UN programming.
Linkages with other interventions should be described and instances where UN Environment’s comparative advantage has been particularly well applied should be highlighted.

Factors affecting this criterion may include: • Stakeholders’ participation and cooperation • Responsiveness to human rights and gender equity • Country ownership and driven-ness B.

Quality of Project Design 35.


The quality of project design is assessed using an agreed template during the evaluation inception phase, ratings are attributed to identified criteria and an overall Project Design Quality rating is established (www.unep.org/evaluation).

This overall Project Design Quality rating is entered in the final evaluation ratings table as item B.
In the Main Evaluation Report a summary of the project’s strengths and weaknesses at design stage is included, while the complete Project Design Quality template is annexed in the Inception Report.

Factors affecting this criterion may include (at the design stage): • Stakeholders participation and cooperation • Responsiveness to human rights and gender equity C. Nature of External Context 36.
At evaluation inception stage a rating is established for the project’s external operating context (considering the prevalence of conflict, natural disasters and political upheaval).
This rating is entered in the final evaluation ratings table as item C. Where a project has been rated as facing either an Unfavourable or Highly Unfavourable external operating context, and/or a negative external event has occurred during project implementation, the ratings for Effectiveness, Efficiency and/or Sustainability may be increased at the discretion of the Evaluation Consultant and Evaluation Manager together.

A justification for such an increase must be given.
D. Effectiveness i.Delivery of Outputs 37.

The evaluation will assess the project’s success in producing the programmed outputs (products, capital goods and services resulting from the intervention) and achieving milestones as per the project design document (ProDoc).

Any formal modifications/revisions made during project implementation will be considered part of the project design.
Where the project outputs are inappropriately or inaccurately stated in the ProDoc, reformulations may be necessary in the reconstruction of the TOC.

In such cases a table should be provided showing the original and the reformulation of the outputs for transparency.

The delivery of outputs will be assessed in terms of both quantity and quality, and the assessment will consider their ownership by, and usefulness to, intended beneficiaries and the timeliness of their delivery.
The evaluation will briefly explain the reasons behind the success or shortcomings of the project in delivering its programmed outputs and meeting expected quality standards.

Factors affecting this criterion may include: • Preparation and readiness • Quality of project management and supervision5 ii.Achievement of Direct Outcomes 38.
The achievement of direct outcomes (short and medium-term effects of the intervention’s outputs; a change of behaviour resulting from the use/application of outputs, which is not under the direct control of the intervention’s direct actors) is assessed as performance against the direct outcomes as defined in the reconstructed6 Theory of Change.

These are the first-level outcomes expected to be achieved as an immediate result of project outputs.
As in 1, above, a table can be used where substantive amendments to the formulation of direct outcomes is necessary.

The evaluation should report evidence of attribution between UN Environment’s intervention and the direct outcomes.
In cases of normative work or where several actors are collaborating to achieve common outcomes, evidence of the nature and magnitude of UN Environment’s ‘substantive contribution’ should be included and/or ‘credible association’ established between project efforts and the direct outcomes realised.
Factors affecting this criterion may include: • Quality of project management and supervision 74 | P a g e Terminal Evaluation: “Advancing the Nagoya Protocol in Countries of the Caribbean Region” • • • Stakeholders’ participation and cooperation Responsiveness to human rights and gender equity Communication and public awareness iii.Likelihood of Impact The evaluation will consider the extent to which the project has played a catalytic role or has promoted scaling up Quality of Project Management and Supervision (including adaptive management) Stakeholders participation and cooperation Responsiveness to human rights and gender equity Country ownership and driven-ness Communication and public awareness Financial Management Financial management will be assessed under 39.
Based on the articulation of longer-term effects in the reconstructed TOC (i.e.

from direct outcomes, via intermediate states, to impact), the evaluation will assess the likelihood of the intended, positive impacts becoming a reality.

Project objectives or goals should be incorporated in the TOC, possibly as intermediate states or long-term impacts.
The Evaluation Office’s approach to the use of TOC in project evaluations is outlined in a guidance note available on the Evaluation Office website, https://www.unenvironment.org/about-un-environment/evaluation and is supported by an excel- based flow chart, ‘Likelihood of Impact Assessment Decision Tree’.

Essentially the approach follows a ‘likelihood tree’ from direct outcomes to impacts, taking account of whether the assumptions and drivers identified in the reconstructed TOC held.
Any unintended positive effects should also be identified and their causal linkages to the intended impact described.

40.

The evaluation will also consider the likelihood that the intervention may lead, or contribute to, unintended negative effects.
Some of these potential negative effects may have been identified in the project design as risks or as part of the analysis of Environmental, Social and Economic Safeguards.7 41. and/or replication8 as part of its Theory of Change and as factors that are likely to contribute to longer term impact.

42.

Ultimately UN Environment and all its partners aim to bring about benefits to the environment and human well- being.
Few projects are likely to have impact statements that reflect such long-term or broad-based changes.

However, the evaluation will assess the likelihood of the project to make a substantive contribution to the high-level changes represented by UN Environment’s Expected Accomplishments, the Sustainable Development Goals9 and/or the high-level results prioritised by the funding partner.
Factors affecting this criterion may include: • • • • • E. 43. information and communication between financial and project management staff.

The evaluation will establish the actual spend across the life of the project of funds secured from all donors.

This expenditure will be reported, where possible, at output level and will be compared with the approved budget.
The evaluation will assess the level of communication between the Project/Task Manager and the Fund Management Officer as it relates to the effective delivery of the planned project and the needs of a responsive, adaptive management approach.

The evaluation will verify the application of proper financial management standards and adherence to UN Environment’s financial management policies.
Any financial management issues that have affected the timely delivery of the project or the quality of its performance will be highlighted.

Factors affecting this criterion may include: • Preparation and readiness • Quality of project management and supervision F. Efficiency 44.

In keeping with the OECD/DAC definition of efficiency the evaluation will assess the extent to which the project delivered maximum results from the given resources.
This will include an assessment of the cost-effectiveness and timeliness of project execution.

Focusing on the translation of inputs into outputs, cost-effectiveness is the extent to which an intervention has achieved, or is expected to achieve, its results at the lowest possible cost.

Timeliness refers to whether planned activities were delivered according to expected timeframes as well as whether events were sequenced efficiently.
The evaluation will also assess to what extent any project extension could have been avoided through stronger project management and identify any negative impacts caused by project delays or extensions.

The evaluation will describe any cost or time-saving measures put in place to maximise results within the secured budget and agreed project timeframe and consider whether the project was implemented in the most efficient way compared to alternative interventions or approaches.

45.
45.

The evaluation will give special attention to efforts by the project teams to make use of/build upon pre-existing institutions, agreements and partnerships, data sources, synergies and complementarities with other initiatives, programmes and projects etc.

to increase project efficiency.

The evaluation will also consider the extent to which the management of the project minimised UN Environment’s environmental footprint.

46.
46.

The factors underpinning the need for any project extensions will also be explored and discussed.

As management or project support costs cannot be increased in cases of ‘no cost extensions’, such extensions represent an increase in unstated costs to implementing parties.

Factors affecting this criterion may include: • • Preparation and readiness (e.g.
timeliness) Quality of project management and supervision themes: completeness of financial two 75 | P a g e Terminal Evaluation: “Advancing the Nagoya Protocol in Countries of the Caribbean Region” Stakeholders participation and cooperation Monitoring and Reporting The evaluation will assess monitoring and reporting across three sub-categories: monitoring design and budgeting, • G. 47. monitoring implementation and project reporting.

i.Monitoring Design and Budgeting 48.


Each project should be supported by a sound monitoring plan that is designed to track progress against SMART10 indicators towards the delivery of the project’s outputs and achievement of direct outcomes, including at a level disaggregated by gender, vulnerability or marginalisation.

The evaluation will assess the quality of the design of the monitoring plan as well as the funds allocated for its implementation.
The adequacy of resources for mid-term and terminal evaluation/review should be discussed if applicable.

ii.Monitoring of Project Implementation 49.

The evaluation will assess whether the monitoring system was operational and facilitated the timely tracking of results and progress towards projects objectives throughout the project implementation period.
This should include monitoring the representation and participation of disaggregated groups (including gendered, vulnerable and marginalised groups) in project activities.

It will also consider how information generated by the monitoring system during project implementation was used to adapt and improve project execution, achievement of outcomes and ensure sustainability.

The evaluation should confirm that funds allocated for monitoring were used to support this activity.

iii.Project Reporting 50.


UN Environment has a centralised Project Information Management System (PIMS) in which project managers upload six-monthly status reports against agreed project milestones.

This information will be provided to the Evaluation Consultant(s) by the Evaluation Manager.

Some projects have additional requirements to report regularly to funding partners, which will be supplied by the project team (e.g.

the Project Implementation Reviews and Tracking Tool for GEF- funded projects).
The evaluation will assess the extent to which both UN Environment and donor reporting commitments have been fulfilled.

Consideration will be given as to whether reporting has been carried out with respect to the effects of the initiative on disaggregated groups.

Factors affecting this criterion may include: • Quality of project management and supervision • Responsiveness to human rights and gender equity (e.g.

disaggregated indicators and data) H. Sustainability 51.
Sustainability is understood as the probability of direct outcomes being maintained and developed after the close of the intervention.

The evaluation will identify and assess the key conditions or factors that are likely to undermine or contribute to the persistence of achieved direct outcomes (i.e.

‘assumptions’ and ‘drivers’).


Some factors of sustainability may be embedded in the project design and implementation approaches while others may be contextual circumstances or conditions that evolve over the life of the intervention.

Where applicable an assessment of bio-physical factors that may affect the sustainability of direct outcomes may also be included.

i.Socio-political Sustainability 52.


The evaluation will assess the extent to which social or political factors support the continuation and further development of project direct outcomes.

It will consider the level of ownership, interest and commitment among government and other stakeholders to take the project achievements forwards.

In particular, the evaluation will consider whether individual capacity development efforts are likely to be sustained.

ii.Financial Sustainability 53.


Some direct outcomes, once achieved, do not require further financial inputs, e.g.

the adoption of a revised policy.

However, in order to derive a benefit from this outcome further management action may still be needed e.g.

to undertake actions to enforce the policy.

Other direct outcomes may be dependent on a continuous flow of action that needs to be resourced for them to be maintained, e.g.

continuation of a new resource management approach.
The evaluation will assess the extent to which project outcomes are dependent on future funding for the benefits they bring to be sustained.

Secured future funding is only relevant to financial sustainability where the direct outcomes of a project have been extended into a future project phase.

Even where future funding has been secured, the question still remains as to whether the project outcomes are financially sustainable.

iii.Institutional Sustainability 54.


The evaluation will assess the extent to which the sustainability of project outcomes (especially those relating to policies and laws) is dependent on issues relating to institutional frameworks and governance.

It will consider whether institutional achievements such as governance structures and processes, policies, sub-regional agreements, legal and accountability frameworks etc.
are robust enough to continue delivering the benefits associated with the project outcomes after project closure.

In particular, the evaluation will consider whether institutional capacity development efforts are likely to be sustained.

Factors affecting this criterion may include: • • may be undermined) • • I. Stakeholders participation and cooperation Responsiveness to human rights and gender equity (e.g.
where interventions are not inclusive, their sustainability Communication and public awareness Country ownership and driven-ness Factors and Processes Affecting Project Performance 76 | P a g e Terminal Evaluation: “Advancing the Nagoya Protocol in Countries of the Caribbean Region” (These factors are rated in the ratings table, but are discussed within the Main Evaluation Report as cross-cutting themes as appropriate under the other evaluation criteria, above) i.Preparation and Readiness 55.
This criterion focuses on the inception or mobilisation stage of the project (ie.

the time between project approval and first disbursement).

The evaluation will assess whether appropriate measures were taken to either address weaknesses in the project design or respond to changes that took place between project approval, the securing of funds and project mobilisation.
In particular, the evaluation will consider the nature and quality of engagement with stakeholder groups by the project team, the confirmation of partner capacity and development of partnership agreements as well as initial staffing and financing arrangements.

(Project preparation is included in the template for the assessment of Project Design Quality).

ii.Quality of Project Management and Supervision 56.
In some cases ‘project management and supervision’ will refer to the supervision and guidance provided by UN Environment to implementing partners and national governments while in others, specifically for GEF funded projects, it will refer to the project management performance of the executing agency and the technical backstopping and supervision provided by UN Environment.

57.
57.

The evaluation will assess the effectiveness of project management with regard to: providing leadership towards achieving the planned outcomes; managing team structures; maintaining productive partner relationships (including Steering Groups etc.

); communication and collaboration with UN Environment colleagues; risk management; use of problem-solving; project adaptation and overall project execution.

Evidence of adaptive management should be highlighted.
iii.Stakeholder Participation and Cooperation 58.

Here the term ‘stakeholder’ should be considered in a broad sense, encompassing all project partners, duty bearers with a role in delivering project outputs and target users of project outputs and any other collaborating agents external to UN Environment.
The assessment will consider the quality and effectiveness of all forms of communication and consultation with stakeholders throughout the project life and the support given to maximise collaboration and coherence between various stakeholders, including sharing plans, pooling resources and exchanging learning and expertise.

The inclusion and participation of all differentiated groups, including gender groups should be considered.

iv.Responsiveness to Human Rights and Gender Equity 59.
The evaluation will ascertain to what extent the project has applied the UN Common Understanding on the human rights-based approach (HRBA) and the UN Declaration on the Rights of Indigenous People.

Within this human rights context the evaluation will assess to what extent the intervention adheres to UN Environment’s Policy and Strategy for Gender Equality and the Environment.

60.
60.

In particular the evaluation will consider to what extent project design, implementation and monitoring have taken into consideration: (i) possible gender inequalities in access to, and the control over, natural resources; (ii) specific vulnerabilities of women and children to environmental degradation or disasters; and (iii) the role of women in mitigating or adapting to environmental changes and engaging in environmental protection and rehabilitation.

v.Country Ownership and Driven-ness 61.


The evaluation will assess the quality and degree of engagement of government / public sector agencies in the project.

While there is some overlap between Country Ownership and Institutional Sustainability, this criterion focuses primarily on the forward momentum of the intended projects results, i.e.. either a) moving forwards from outputs to direct outcomes or b) moving forward from direct outcomes towards intermediate states.
The evaluation will consider the involvement not only of those directly involved in project execution and those participating in technical or leadership groups, but also those official representatives whose cooperation is needed for change to be embedded in their respective institutions and offices.

This factor is concerned with the level of ownership generated by the project over outputs and outcomes and that is necessary for long term impact to be realised.
This ownership should adequately represent the needs of interest of all gendered and marginalised groups.

vi.Communication and Public Awareness 62.


The evaluation will assess the effectiveness of: a) communication of learning and experience sharing between project partners and interested groups arising from the project during its life and b) public awareness activities that were undertaken during the implementation of the project to influence attitudes or shape behaviour among wider communities and civil society at large.
The evaluation should consider whether existing communication channels and networks were used effectively, including meeting the differentiated needs of gendered or marginalised groups, and whether any feedback channels were established.

Where knowledge sharing platforms have been established under a project the evaluation will comment on the sustainability of the communication channel under either socio-political, institutional or financial sustainability, as appropriate.

SECTION 3.
SECTION 3.

EVALUATION APPROACH, METHODS AND DELIVERABLES 63.

The Terminal Evaluation will be an in-depth evaluation using a participatory approach whereby key stakeholders are kept informed and consulted throughout the evaluation process.

Both quantitative and qualitative evaluation methods will be used as appropriate to determine project achievements against the expected outputs, outcomes and impacts.
It is highly recommended that the consultant(s) maintains close communication with the project team and promotes information exchange throughout the evaluation implementation phase in order to increase their (and other stakeholder) ownership of the evaluation findings.

Where applicable, the consultant(s) should provide a geo-referenced map that demarcates the area covered by the project and, where possible, provide geo-reference photographs of key intervention sites (e.g.
sites of habitat rehabilitation and protection, pollution treatment infrastructure, etc.)

77 | P a g e Terminal Evaluation: “Advancing the Nagoya Protocol in Countries of the Caribbean Region” 64.

The findings of the evaluation will be based on the following: a.
A desk review of: • Relevant background documentation; • Project design documents (including minutes of the project design review meeting at approval); Annual Work Plans and Budgets or equivalent, revisions to the project (Project Document Supplement), the logical framework and its budget; • Project reports such as six-monthly progress and financial reports, progress reports from collaborating partners, meeting minutes, relevant correspondence and including the Project Implementation Reviews (PIRs); supervision mission reports, etc.
; • • • • • • b.

• • • • • • c. d. e. GEF Tracking Tool, Steering Committee Minutes; Quarterly expenditure reports, co-financing records, budget revisions, Technical reports on project Outputs, studies, publications, outreach material, etc.

; Mid-Term Review or Mid-Term Evaluation of the project; Terminal Report (or draft) of the project including final project output, audit report, and final financial statements; Other reports deemed useful to the terminal evaluation of the project.
Interviews (individual or in group) with: UN Environment Task Manager (TM); Project management team in IUCN; UN Environment Fund Management Officer (FMO); Sub-Programme Coordinator; Project partners, including CARICOM, GIZ, CBD Secretariat, OECS, national and regional focal points, etc.

; Other relevant resource persons, NGOs and private sector partners.
Survey (to be determined at the inception phase) Field visits to selected countries in the Caribbean (to be confirmed at the inception phase) Other data collection tools as may be deemed useful by the Evaluator 11.

Evaluation Deliverables and Review Procedures 65.


The evaluation team will prepare: • Inception Report: containing an assessment of project design quality, a draft reconstructed Theory of Change of the project, project stakeholder analysis, evaluation framework and a tentative evaluation schedule(see Annex 1 for links to all templates, tables and guidance notes).
• Preliminary Findings Note: typically in the form of a PowerPoint presentation, the sharing of preliminary findings is intended to support the participation of the project team, act as a means to ensure all information sources have been accessed and provide an opportunity to verify emerging findings.
• Draft and Final Evaluation Report: containing an executive summary that can act as a standalone document; detailed analysis of the evaluation findings organised by evaluation criteria and supported with evidence; lessons learned and recommendations and an annotated ratings table(see links in Annex 1).

• Evaluation Brief: a 2-page summary of key evaluation findings for wider dissemination through the EOU website.

66. Review of the draft evaluation report.


The evaluator will submit a draft report to the Evaluation Manager and revise the draft in response to their comments and suggestions.

Once a draft of adequate quality has been peer-reviewed and accepted, the Evaluation Manager will share the cleared draft report with the Project Manager, who will alert the Evaluation Manager in case the report contains any blatant factual errors.
The Evaluation Manager will then forward revised draft report (corrected by the evaluation team where necessary) to other project stakeholders, for their review and comments.

Stakeholders may provide feedback on any errors of fact and may highlight the significance of such errors in any conclusions as well as providing feedback on the proposed recommendations and lessons.

Any comments or responses to draft reports will be sent to the Evaluation Manager for consolidation.
The Evaluation Manager will provide all comments to the evaluation team for consideration in preparing the final report, along with guidance on areas of contradiction or issues requiring an institutional response.

67.

Based on a careful review of the evidence collated by the evaluation consultants and the internal consistency of the report, the Evaluation Manager will provide an assessment of the ratings in the final evaluation report.
Where there are differences of opinion between the evaluator and the Evaluation Manager on project ratings, both viewpoints will be clearly presented in the final report.

The Evaluation Office ratings will be considered the final ratings for the project.

68.

The Evaluation Manager will prepare a quality assessment of the first and final drafts of the main evaluation report, which acts as a tool for providing structured feedback to the evaluation consultants.
The quality of the report will be assessed and rated against the criteria specified in template listed in Annex 1 and this assessment will be appended to the Final Evaluation Report.

69.

At the end of the evaluation process, the Evaluation Office will prepare a Recommendations Implementation Plan in the format of a table, to be completed and updated at regular intervals by the Task Manager.

The Evaluation Office will track compliance against this plan on a six-monthly basis.

12.

The Evaluator 70.
For this evaluation, one consultant will work under the overall responsibility of the Evaluation Office represented by an Evaluation Manager (Pauline Marima), in consultation with the UN Environment Task Manager (Christopher Cox), Programme Assistant (Gloritzel Frangakis Cano), Fund Management Officer (Pooja Bhimjiani), the Coordinator of UN 78 | P a g e Terminal Evaluation: “Advancing the Nagoya Protocol in Countries of the Caribbean Region” be on found of UN Evaluation Office The consultant will be hired the over the period June 2019 to November 2019 during which time the evaluation Environment's the sub-programme on Environmental Governance (Cristina Zucca), Niklas Hagelberg - Coordinator, Ecosystem Management Subprogramme, and the Chief of the Ecosystem Services Unit (Pushpam Kumar).
The consultant will liaise with the Evaluation Manager on any procedural and methodological matters related to the evaluation.

It is, however, the consultant’s individual responsibility to arrange for their travel, visa, obtain documentary evidence, plan meetings with stakeholders, organize online surveys, and any other logistical matters related to the assignment.

The UN Environment Task Manager and project teams will, where possible, provide logistical support (formal introductions, meetings etc.)
allowing the consultant to conduct the evaluation as efficiently and independently as possible.

71. deliverables listed in Section 11 ‘Evaluation Deliverables’ above should be submitted.

72.

S/he should have: an advanced university degree, evaluation experience (preferably using a Theory of Change approach), at least 8 years’ experience in environmental management or a related field; expertise in the areas of ecosystems management and climate change adaptation is an advantage.
Knowledge of English language, along with excellent writing skills in English is required.

Working knowledge of Spanish language is desired.

Experience in managing partnerships, knowledge management and communication is desirable for all evaluation consultants.

73.

The consultant will be responsible, in close consultation with the Evaluation Office of UN Environment, for overall management of this evaluation and timely delivery of the outputs described in Section 11 Evaluation Deliverables, above.
The consultant will ensure that all evaluation criteria and questions are adequately covered.

Detailed guidelines for the Evaluation Consultant Environment website: the can (http://web.unep.org/evaluation/working-us/working-us ).

Specific Responsibilities: 74.

The Consultant will be responsible, in close consultation with the Evaluation Office of UN Environment, for overall management of the evaluation and timely delivery of its outputs, described in Section 11 ‘Evaluation Deliverables’, above.
The consultant will ensure that all evaluation criteria and questions are adequately covered.

S/he will be responsible for the evaluation design, data collection and analysis, and report-writing.
More specifically: Inception phase of the evaluation, including: -preliminary desk review and introductory interviews with project staff; -draft the reconstructed Theory of Change of the project; -prepare the evaluation framework; -develop the desk review, interview protocols, and data collection and analysis tools; -plan the evaluation schedule; -prepare the Inception Report, incorporating comments received from the Evaluation Office.
Data collection and analysis phase of the evaluation, including: -conduct further desk review and in-depth interviews with project implementing and executing agencies, project partners and project stakeholders; -conduct an evaluation mission to a selection of countries in the Caribbean region to visit the project locations, interview project partners and stakeholders, including a good representation of private sector stakeholders.
Ensure independence of the evaluation and confidentiality of evaluation interviews.

-regularly report back to the Evaluation Office on progress and communicate any possible problems or issues encountered and; keep the Project/Task Manager informed of the evaluation progress and engage the Project/Task Manager in discussions on emerging findings throughout the evaluation process.
Reporting phase, including: -draft the Main Evaluation Report, ensuring that the evaluation report is complete, coherent and consistent with the Evaluation Office guidelines both in substance and style; -liaise with the Evaluation Office on comments received and finalize the Main Evaluation Report, ensuring that comments are taken into account -prepare a Response to Comments annex for the main report, listing those comments not accepted by the Evaluation Consultant and indicating the reason for the rejection; and -prepare a 2-page summary of the key evaluation findings and lessons; Managing relations, including: -maintain a positive relationship with evaluation stakeholders, ensuring that the evaluation process is as participatory as possible but at the same time maintains its independence; -communicate in a timely manner with the Evaluation Office on any issues requiring its attention and intervention.
13.

75.

Table 5.
Tentative schedule for the evaluation Milestone Recruitment and Contracting Kick-off meeting (via Skype) Inception Report Data collection and analysis, desk-based interviews and surveys Field Mission (based on meeting arrangements and available budget) Draft report to UN Environment (Evaluation Manager and Peer Reviewer) Draft Report shared with UN Environment Task Manager and Project Team Tentative schedule* May 2019 June 2019 June 2019 June -July 2019 July 2019 August 2019 September 2019 Schedule of the evaluation The table below presents the tentative schedule for the evaluation.
79 | P a g e Terminal Evaluation: “Advancing the Nagoya Protocol in Countries of the Caribbean Region” October 2019 November 2019 Draft Report shared with wider group of stakeholders Final Report 14.

Contractual Arrangements 76.

Evaluation Consultants will be selected and recruited by the Evaluation Office of UN Environment under an individual Special Service Agreement (SSA) on a “fees only” basis (see below).
By signing the service contract with UN Environment/UNON, the consultant(s) certify that they have not been associated with the design and implementation of the project in any way which may jeopardize their independence and impartiality towards project achievements and project partner performance.

In addition, they will not have any future interests (within six months after completion of the contract) with the project’s executing or implementing units.
All consultants are required to sign the Code of Conduct Agreement Form.

77. deliverables.

The schedule of payment is as follows: Table 6: Schedule of Payment for the consultant: Deliverable Approved Inception Report (document 9 in Annex 1) Approved Draft Main Evaluation Report (document 16 in Annex 1) Approved Final Main Evaluation Report Fees will be paid on an instalment basis, paid on acceptance by the Evaluation Manager of expected key Percentage Payment 30% 40% 30% 78.
Fees only contracts: Air tickets will be purchased by UN Environment and 75% of the DSA for each authorised travel mission will be paid up front.

Local in-country travel will only be reimbursed where agreed in advance with the Evaluation Office and on the production of acceptable receipts.

Terminal expenses and residual DSA entitlements (25%) will be paid after mission completion.

79.
79.

The consultant may be provided with access to UN Environment’s Programme Information Management System (PIMS) and if such access is granted, the consultant agrees not to disclose information from that system to third parties beyond information required for, and included in, the evaluation report.
In case the consultant is not able to provide the deliverables in accordance with these guidelines, and in line with the expected quality standards by the UN Environment Evaluation Office, payment may be withheld at the discretion of the Director of the Evaluation Office until the consultants have improved the deliverables to meet UN Environment’s quality standards.

80.

If the consultant fails to submit a satisfactory final product to UN Environment in a timely manner, i.e.
before the end date of their contract, the Evaluation Office reserves the right to employ additional human resources to finalize the report, and to reduce the consultants’ fees by an amount equal to the additional costs borne by the Evaluation Office to bring the report up to standard.
80 | P a g e Terminal Evaluation: “Advancing the Nagoya Protocol in Countries of the Caribbean Region” Annex 5: Brief Resumé of the consultant David A. Simmons has more than 25 years of experience working in various areas related to Environmental Policy, Planning and Management and Sustainable Development.
Mr Simmons has considerable experience in the areas of institutional analysis and environmental policy planning and management having been contracted to undertake several related consulting assignments covering Climate Change Adaptation, National Biodiversity Strategies and Action Plans, and Coastal and Marine Policy, Planning and Management.
Mr Simmons has considerable experience of working on complex projects, e.g., the GEF-funded, and UN Environment executed “Integrating Water, Land and Ecosystems Management in Caribbean Small Island Developing States (IWECO)” being implemented in 9 Caribbean countries; The GEF funded, UNEP executed project to: “Increase Saint Lucia capacity to monitor MEA implementation and sustainable development”, based on a wide participatory process, and taking into account baseline and priorities for national information management; and, the GEF funded and World Bank executed “OECS Solid and Ship-generated Waste Management project”.
He was also the lead consultant for the preparation of the “Access and Benefit Sharing of Genetic Resources” policy document for Guyana (2009).

He has also undertaken Evaluation exercises on several national and regional projects including the Caribbean Challenge Initiative (2014), the Bahamas Network of Marine Protected Areas (2016) and the Mid-Term Evaluation of the “Energy for Sustainable Development in Caribbean Buildings” project (2017).
He has also facilitated national workshops which sought to mainstreaming Multilateral Environmental Agreements (MEAs) Into Policy Development: within the context of the BPOA, SAMOA Pathways, and the Sustainable Development Goals.

Mr Simmons has written and edited a number of publications and presented papers at several distinguished gatherings.
81 | P a g e Terminal Evaluation: “Advancing the Nagoya Protocol in Countries of the Caribbean Region” Annex 6: Quality Assessment of the Evaluation Report All UN Environment evaluations are subject to a quality assessment by the Evaluation Office.

This is an assessment of the quality of the evaluation product (i.e.

evaluation report) and is dependent on more than just the consultant’s efforts and skills.
Nevertheless, the quality assessment is used as a tool for providing structured feedback to the evaluation consultants, especially at draft report stage.

This guidance is provided to support consistency in assessment across different Evaluation Managers and to make the assessment process as transparent as possible.

Substantive Report Quality Criteria Quality of the Executive Summary: The Summary should be able to stand alone as an accurate summary of the main evaluation product.
It should include a concise overview of the evaluation object; clear summary of the evaluation objectives and scope; overall evaluation rating of the project and key features of performance (strengths and weaknesses) against exceptional criteria (plus reference to where the evaluation ratings table can be found within the report); summary of the main findings of the exercise, including a synthesis of main conclusions (which include a summary response to key strategic evaluation questions), lessons learned and recommendations.
I.

Introduction A brief introduction should be given identifying, where possible and relevant, the following: institutional context of the project (sub-programme, Division, regions/countries where implemented) and coverage of the evaluation; date of PRC approval and project document signature); results frameworks to which it contributes (e.g.
Expected Accomplishment in POW); project duration and start/end dates; number of project phases (where appropriate); implementing partners; total secured budget and whether the project has been evaluated in the past (e.g.

mid-term, part of a synthesis evaluation, evaluated by another agency etc.)

Consider the extent to which the introduction includes a concise statement of the purpose of the evaluation and the key intended audience for the findings?

II.
II.

Evaluation Methods This section should include a description of how the TOC at Evaluation14 was designed (who was involved etc.)

and applied to the context of the project?

A data collection section should include: a description of evaluation methods and information sources used, including the number and type of respondents; justification for methods used (e.g.
qualitative/quantitative; electronic/face-to-face); any selection criteria used to identify respondents, case studies or sites/countries visited; strategies used to increase stakeholder engagement and consultation; details of how data were verified (e.g.

triangulation, review by stakeholders etc.).

The methods used to analyse data (e.g.

scoring; coding; thematic analysis etc.)

should be described.
It should also address evaluation limitations such as: low or imbalanced response rates across different groups; extent to UN Environment Evaluation Office Comments Final Report Rating Adequate summary presenting the most pertinent findings of the evaluation in a clear and precise manner Precise, well written and captures all the main introductory points recommended in the TOR This section is complete, concise, and the approach and methods used for data collection and analysis have been described in great detail.
6 6 6 82 | P a g e Terminal Evaluation: “Advancing the Nagoya Protocol in Countries of the Caribbean Region” UN Environment Evaluation Office Comments Final Report Rating which findings can be either generalised to wider evaluation questions or constraints on aggregation/disaggregation; any potential or apparent biases; language barriers and ways they were overcome.
Ethics and human rights issues should be highlighted including: how anonymity and confidentiality were protected and strategies used to include the views of marginalised or potentially disadvantaged groups and/or divergent views.

III.

The Project This section should include: • Context: Overview of the main issue that the project is trying to address, its root causes and consequences on the environment and human well-being (i.e.

synopsis of the problem and situational analyses).
• Objectives and components: Summary of the project’s results hierarchy as stated in the ProDoc (or as officially revised) • Stakeholders: Description of groups of targeted stakeholders organised according to relevant common characteristics • Project implementation structure and partners: A description of the implementation structure with diagram and a list of key project partners • Changes in design during implementation: Any key events that affected the project’s scope or parameters should be described in brief in chronological order • Project financing: Completed tables of: (a) budget at design and expenditure by components (b) planned and actual sources of funding/co-financing IV.
Theory of Change A summary of the project’s results hierarchy should be presented for: a) the results as stated in the approved/revised Prodoc logframe/TOC and b) as formulated in the TOC at Evaluation.

The two results hierarchies should be presented as a two column table to show clearly that, although wording and placement may have changed, the results ‘goal posts’ have not been ’moved’.

The TOC at Evaluation should be presented clearly in both diagrammatic and narrative forms.
Clear articulation of each major causal pathway is expected, (starting from outputs to long term impact), including explanations of all drivers and assumptions as well as the expected roles of key actors.

V. Key Findings Strategic relevance: A.

This section should include an assessment of the project’s relevance in relation to UN Environment’s mandate and its alignment with UN Environment’s policies and strategies at the time of project approval.
An assessment of the complementarity of the project with other interventions addressing the needs of the same target groups should be This section is also complete and sufficiently covers all the required sub-topics in a detailed yet clear and concise manner.

The TOC diagram is a result of a consultative process.

The narrative is clear and provides a suitable explanation of causal pathways.
The diagrammatic representation needed to be simplified further and a significant amount of editing and summation was required at draft stage.

Drivers and Assumptions, as well as the change agents along these pathways are sufficiently described in the narrative.

Section is well done and covers the four main aspects of relevance prescribed in the TOR.
6 5 6 83 | P a g e Terminal Evaluation: “Advancing the Nagoya Protocol in Countries of the Caribbean Region” UN Environment Evaluation Office Comments Final Report Rating Alignment to the UN Environment Medium Term Alignment to UN Environment/GEF/Donor Strategic included.

Consider the extent to which all four elements have been addressed: 1.

Strategy (MTS) and Programme of Work (POW) 2.

Priorities 3.

Environmental Priorities 4.

B.


Quality of Project Design To what extent are the strength and weaknesses of the project design effectively summarized?

Relevance to Regional, Sub-regional and National Complementarity with Existing Interventions A summary of the project’s strengths and weaknesses at design stage are summarized, though not in sufficient enough detail to adequately explain the sub-optimal rating given for this criterion.
C. Nature of the External Context For projects where this is appropriate, key external features of the project’s implementing context that may have been reasonably expected to limit the project’s performance (e.g.

conflict, natural disaster, political upheaval) should be described.

The TE sufficiently describes the external operating context.
The implications on project performance has also been discussed in adequate detail D. Effectiveness (i) Outputs and Direct Outcomes: How well does the report present a well-reasoned, complete and evidence-based assessment of the achievement of a) outputs, and b) direct outcomes?

How convincing is the discussion of attribution and contribution, as well as the limitations to attributing effects to the intervention.
(ii) Likelihood of Impact: How well does the report present an integrated analysis, guided by the causal pathways represented by the TOC, of all evidence relating to likelihood of impact?

How well are change processes explained and the roles of key actors, as well as drivers and assumptions, explicitly discussed?

The delivery of outputs has been assessed in terms of both quantity and quality.
Minor inconsistencies were pointed out at earlier report versions but these have been rectified in the final draft; the omission of programmed activities and outputs is a significant flaw that should have be more clearly analysed.

Assessment of Direct Outcomes is well covered.

Reasons behind the success or shortcomings have been covered to varying degrees of detail.

The discussion follows logically from the assessment of Outputs and Direct Outcomes.
It is consistent with the TOC narrative and discusses the stakeholders and status of assumptions contributing to causal pathways from medium-term Outcomes to Impact.

5 6 5 5 E. Financial Management This section should contain an integrated analysis of all dimensions evaluated under financial management.

And include a completed ‘financial management’ table.

The section covers aspects of completeness, compliance and communication, as per guidance.
4.5 84 | P a g e Terminal Evaluation: “Advancing the Nagoya Protocol in Countries of the Caribbean Region” Consider how well the report addresses the following: • completeness of financial information, including the actual project costs (total and per activity) and actual co- financing used • management staff and • standards and procedures.
compliance with relevant UN financial management communication between financial and project F. Efficiency To what extent, and how well, does the report present a well- reasoned, complete and evidence-based assessment of efficiency under the primary categories of cost-effectiveness and timeliness including: Implications of delays and no cost extensions • • Time-saving measures put in place to maximise results within the secured budget and agreed project timeframe • Discussion of making use of/building on pre-existing institutions, agreements and partnerships, data sources, synergies and complementarities with other initiatives, programmes and projects etc.
• The extent to which the management of the project minimised UN Environment’s environmental footprint.

G. Monitoring and Reporting How well does the report assess: • Monitoring design and budgeting (including SMART indicators, resources for MTE/R etc.)

• Monitoring implementation (including use of monitoring data for adaptive management) • Project reporting (e.g.
PIMS and donor report) H. Sustainability How well does the evaluation identify and assess the key conditions or factors that are likely to undermine or contribute to the persistence of achieved direct outcomes including: • Socio-political Sustainability • Financial Sustainability • Institutional Sustainability (including issues of partnerships) I.

Factors Affecting Performance These factors are not discussed in stand-alone sections but are integrated in criteria A-H as appropriate.
To what extent, and how well, does the evaluation report cover the following cross-cutting themes: • Preparation and readiness • Quality of project management and supervision15 • Stakeholder participation and co-operation UN Environment Evaluation Office Comments Final Report Rating The quality of the assessment has been affected somewhat by data insufficiency (co-financing data was reported to have been lost, the original files were not made available to the evaluator).
(this section is rated poorly as a result of limited financial information from the project, this is not a reflection on the consultant, but will affect the quality of the evaluation report) Section has been covered as per guidelines although suggestions for improving the analysis were provided in earlier report versions.

Findings have been presented adequately and some examples and cross referencing provided to support the assessment.
5 A more analytical assessment of project monitoring and its implications on performance would have improved the assessment.

Suggestions for improving the analysis were provided in earlier versions.

5 Clear and concisely presented.

Provides a good idea of the status of each of the dimensions of sustainability form the analyses provided.

6 The required sub-criteria are all covered to varying levels of detail throughout the report.
Greater attention was needed for the following aspects: ‘Responsiveness to human rights and gender equity’ and 5 85 | P a g e Terminal Evaluation: “Advancing the Nagoya Protocol in Countries of the Caribbean Region” • Responsiveness to human rights and gender equity • Country ownership and driven-ness • Communication and public awareness VI.

Conclusions and Recommendations i.

Quality of the conclusions: The key strategic questions should be clearly and succinctly addressed within the conclusions section?
It is expected that the conclusions will highlight the main strengths and weaknesses of the project, and connect them in a compelling story line.

Conclusions, as well as lessons and recommendations, should be consistent with the evidence presented in the main body of the report.

ii) Quality and utility of the lessons: Both positive and negative lessons are expected and duplication with recommendations should be avoided.
Based on explicit evaluation findings lessons should be rooted in real project experiences or derived from problems encountered and mistakes made that should be avoided in the future.

Lessons must have the potential for wider application and use and should briefly describe the context from which they are derived and those contexts in which they may be useful.
iii) Quality and utility of the recommendations: To what extent are the recommendations proposals for specific actions to be taken by identified people/position- holders to resolve concrete problems affecting the project or the sustainability of its results.

They should be feasible to implement within the timeframe and resources available (including local capacities) and specific in terms of who would do what and when.
Recommendations should represent a measurable performance target in order that the Evaluation Office can monitor and assess compliance with the recommendations.

VII.

Report Structure and Presentation Quality i.


Structure and completeness of the report: To UN Environment Evaluation Office Comments ‘Communication and public awareness’ Final Report Rating The conclusions section is well developed and presents the most critical findings of the evaluation – both strengths and weaknesses are adequately discussed.

Responses to the key strategic questions are included and are anchored on findings in the report.
Summary of ratings table is complete 5 The lessons are relevant and based on findings presented in the report.

They have a potential for wider application and use.

6 The formulation of recommendations was improved from earlier versions to make a clearer distinction These recommendations are relevant though 2 out of 4 have the acting agents who are external agencies based in the country.

These recommendations will at best be communicated to the respective agencies.
5 what extent does the report follow the Evaluation Office guidelines?

Are all requested Annexes included and complete?

The report follows the prescribed structure, and meets all the requirements in the TOR 6 ii.

Quality of writing and formatting: Consider whether the report is well written (clear English language and grammar) with language that is adequate in quality and tone for an official document?

Do visual aids, such as maps and graphs convey key information?
Does the report follow Evaluation Office formatting guidelines?

OVERALL REPORT QUALITY RATING The report is well written in clear English language that is easy to comprehend.

Formatting is well done.

6 HS (5.5) 1.

A number rating 1-6 is used for each criterion: Highly Satisfactory = 6, Satisfactory = 5, Moderately Satisfactory = 4, Moderately Unsatisfactory = 3, Unsatisfactory = 2, Highly Unsatisfactory = 1.
The overall quality of the evaluation report is calculated by taking the mean score of all rated quality criteria.

86 | P a g e