Lessons learned from
evaluations
Series 2
MAY 2022i Performance Monitoring, and Evaluation
Planning,

Table of Contents
LIST OF TABLESIIACRONYMS AND ABBREVIATIONSIIIEXECUTIVE SUMMARYIVINTRODUCTION

1METHODOLOGY

1

LIMITATIONS

2

FINDINGS: USING LESSONS LEARNED2

FINDINGS: LESSONS LEARNED3KNOWLEDGE SHARING AND NETWORKINGNEW TECHNOLOGIESLOCAL OR NATIONAL OWNERSHIPLOCAL SUPPORTMONITORING TOOLS

3
5
7
10
11

KEY TAKE-AWAYS

12

THE WAY FORWARD

13REFERENCES

14

ANNEXES

16

ANNEX 1. LESSONS COMPILATION

16

List of Tables
Table 1: Number of lessons per category ......................................................................... 2ii

Acronyms and abbreviations
OECD/DAC

Organisation for Economic Co-operation and Development/Development
Assistance Committee

PPME

Planning, Performance Monitoring, and Evaluation Unit

REA

Rapid Evidence Assessment

ROI

Return on Investment

UN CC:Learn

One UN CC:Learn Partnership

UNITAR

United Nations Institute for Training and Researchiii

Executive summary
The Planning, Performance Monitoring, and Evaluation Unit (PPME) has developed this lessons
learned series to provide valuable insights for learning and improved decision-making related tofuture programming. This paper uses the rapid evidence assessment (REA) methodology toanalyse and synthetize lessons learned from past UNITAR independent and self-evaluations
undertaken since 2016.
This second edition focuses on lessons related to knowledge sharing and networking, local andnational ownership, monitoring tools, new technologies and local support. Fifty-one lessons wereanalyzed, resulting in 13 meta-lessons. A short survey was also deployed to UNITAR managers
and directors to better understand the use of lessons learned at UNITAR.

1.

Multisector and multidisciplinary groups help establishing a diverse network ofprofessionals and deepen the learning experience.

2.

Alumni networks and local coaches create a local support system for participants
after the training.

3.

E-learning is an effective tool for knowledge sharing and networking.

1.

Social media can be very advantageous to reach out to new beneficiaries andposition participants’ projects.

2.

Providing guidance to navigate digital learning platforms can help in reducing
dropouts in digital learning.

1.

Engagement of the community and local partners can improve project ownership.

2.

Sustainability of project results relies largely on local factors.

3.

Incorporation of cultural and local insights into the project/programme design canfoster local ownership.

4.

Ownership in country projects can be reached when the project goal is aligned withnational priorities, although timespan should be flexible to potential delays.

Local support

Lack of political will and support from supervisors can hinder behaviour change.

1.
2.

Local support can be constructed from inside the training programme.

Monitoring
tools

Local and national
ownershipNew
technologies

Knowledge sharing
and networking

The following meta-lessons were found:1.

Log frames are important but need to be complemented with regular monitoring foreffect.

2.

Monitoring attainment of learning objectives is a recurrent and significant process.

iv

Results from a survey deployed to directors and managers suggest that formulation andincorporation of lessons learned at UNITAR are a common practice that can occur through formal
documentation and be formally recorded, e.g., in narrative reports after conducting self-evaluations
and incorporated into the UNITAR lessons learned repository or more informally as internal
discussions of the project management team. This practice is, however, mostly done internally andthere is less opportunity to exchange lessons with other UNITAR programmes and divisions.
As next steps, a lessons learned forum will be organised to discuss findings from this assessment
and invite colleagues from Programme Units to share and exchange. Moreover, this paper series
will be continued, and future editions will focus on different categories of lessons learned. The
Lessons Learned database and this assessment shall inform new programming development.

v

Introduction
According to UNITAR’s Evaluation Policy (2021a), evaluation at the Institute serves accountability,
organizational learning, quality improvement and decision-making purposes. Lessons learned from
evaluations provide an evidence-based source of information on what has (and what has not)worked well from UNITAR programming, with results from such exercises contributing to thecreation and uptake of knowledge. This paper series aim to contribute to management anddissemination of knowledge generated from previous evaluations that can also contribute to quality
improvement of programming.
Using the REA methodology, the series analyzes and synthetizes lessons from UNITAR
independent and self-evaluations undertaken since 2016. Each issue of the series contains
lessons from different categories. This second issue focuses on lessons related to knowledge
sharing and networking, local and national ownership, monitoring tools, new technologies
and local support. Fifty-one lessons were analyzed, resulting in 13 meta-lessons. Furthermore,a short survey was deployed to UNITAR managers and directors to better understand the use of
lessons learned at UNITAR.
After this introduction, the purpose and methodology are presented, followed by the findings, takeaway messages from the process and a description of the next steps.

Methodology
This paper uses the definition of lessons learned proposed by the OECD/DAC Glossary (2010).
Lessons learned are defined as “Generalizations based on evaluation experiences with projects,
programs, or policies that abstract from the specific circumstances to broader situations.
Frequently, lessons highlight strengths or weaknesses in preparation, design, and implementationthat affect performance, outcome, and impact.” (OECD/DAC, 2010, p.26).
The guiding questions proposed for the analysis are the following:••

What can we learn from past programming to guide future design, planning,
implementation, monitoring and evaluation phases?
What factors can be identified as drivers of success or reinforcing challenges towards theachievement of project/programme goals?

The REA methodology provides a balanced assessment of what is known (and not known) from
evaluations about an intervention/programme/project by using a systematic methodology to searchand critically appraise evidence (Barends, Rousseau & Briner, 2017).
The lessons analyzed derive from independent and self-evaluations available in the UNITAR
repository of lessons learned as of March 2022. Lessons in the repository are classified under 19
categories of lessons according to their type. This series focuses on five of these categories:
knowledge sharing and networking, local or national ownership, local support, monitoring tools,
and new technologies. Data availability was also considered for inclusion in this series, but there
was not enough information for its appraisal, and was then omitted.
Excluding duplicated or miscategorized lessons, 51 lessons were analyzed for this paper.
Knowledge sharing and networking was the category with the largest number of lessons (21)followed by local or national ownership (17).
The distribution per category is presented in Table 1 below.

1Table 1: Number of lessons per category
Lesson category

NumberKnowledge Sharing and Networking
Local or National Ownership
Local Support
Monitoring Tools
New technologies

21
17
6
4
3

The analysis of the lessons learned followed a two-step procedure: first, lessons under eachcategory were reclassified in sub-categories; and second, lessons are extracted derived from theirrespective subcategory. Annex 1 presents the full list of lessons with their respective categories
and sub-categories.
Besides the analysis of lessons learned for the selected categories, a survey to directors andmanagers was deployed to collect experiences on lessons learned formulation and inclusion along
the programme/project cycle. Eight responses were received, with only one UNITAR division not
being represented in the results.

Limitations
In order to be ‘rapid’, an REA necessarily needs to compromise on the breadth, depth andcomprehensiveness of the search and, in this case, it does not allow for an in-depth study of thecontext of each project in which a lesson learned was formulated (Barends, Rousseau & Briner,2017). Another limitation is an unequal distribution of lessons by UNITAR division, with more thanhalf of the lessons (60 per cent, or 31 lessons) gathered from the Prosperity Division. The
remaining lessons belong to evaluation exercises from the divisions for Peace, Planet and Satellite
Analysis and Applied Research Divisions. Moreover, some of the categories selected contained
only a few lessons, which question their generalizability to the whole Institute.

Findings: Using lessons learned
Eight responses were obtained from directors and managers. The formulation of lessons learned
is done in all UNITAR programmes captured in the survey. Lessons are usually formulated forlarge projects (e.g., country programming or long-term projects), and to a lesser extent for small
projects. Lessons learned are documented both formally and informally. Formally, lessons
learned are included in narrative or project completion reports or in internal documents andrepositories. They can also be documented more informally as project notes obtained throughout
project implementation, e.g., feedback from participants, resource persons and staff, or in meetings
hold at the end of a project to discuss lessons learned either with UNITAR staff involved in the
project or with donors and partners. The small-scale and size of most projects was mentioned as
the main factor that hampered the identification and formulation of lessons learned.
Moving forward, almost all programmes represented in the survey (seven out of eight) haveincorporated lessons learned from past projects/programmes into new ones during 2020 and
2021. Four of them have used lessons from their own programme units and three have also applied
lessons from other programmes or divisions. These reflect diverse areas, such as training content
and delivery modality, strategic planning, budgeting or monitoring tools, and may inform futurephases of the same project or the design/development new projects. Box 1 below contains specificexamples on how lessons have been integrated.

2

Box 1. Examples of integration of lessons learned at UNITAR
•
•
•
•
•

•
•

The lessons learned informed us in improving our actions along the project cycle.
Lessons learned are often reflected in the formulation of concepts/proposals/initiationplans.
Extensive consultations with other teams that have engaged on similar activities,
particularly in similar geographies that we are undertaking programming in.
[…P]articular[ly] towards what training methods work best under what circumstances. Also,
more general in terms of when to submit proposals to which donors.
Adaptation of methodologies/timeframes/structure for transforming face to face to live andonline. Development of processes to facilitate increased participant exchange of knowledge
and experience.
The lessons learned identified were used to make 2022 onwards plan for project execution.
We also included some ideas on new projects based on the lessons learned identified.
Team discussion on implementation of new projects.

The Institute’s decentralized structure of project management has facilitated the application oflessons learned, where internal sessions dedicated to learning can be held to brainstorm on andstrategize from lessons that have been identified. On the other hand, the knowledge on theexistence of the centralized lessons learned repository and limited time disincentivize its use.

Findings: Lessons
evaluations
Knowledge sharing and networking

3

Learnedfrom1.

Multisector and multidisciplinary groups help establishing a diverse network ofprofessionals and deepen the learning experience
The first issue of the lessons learned series highlighted the importance of balancing theory
and practice (including group activities) and participants’ gender for successful training.
Similarly, an interactive methodology in a culturally diverse group of participants isimportant for participants to deepen their learning experience and strengthen theirprofessional networks.
UNITAR evaluations suggest that having participants from different backgrounds (e.g.,nationalities, organizational affiliations, areas of work) help creating a safe space forknowledge sharing, where participants can exchange their good practices and common
challenges faced in their different contexts. This has proved very valuable in regional
training for countries with similar contexts and challenges in which formal knowledge
sharing hubs and platforms are not established or are not well known by the relevant
stakeholders or there are other barriers for information exchange, e.g., geography. In thesecases, the UNITAR training are an opportunity to connect participants to fill these gaps byfacilitating South-South learning, exchange and cooperation.
“[…] One key benefit of the programme, reported by a number of participants, was the ability toengage face-to-face with counterparts from far-distant countries, facing similar issues withregards tsunami and DRR.”“The knowledge sharing among the participants from other countries provided invaluableknowledge and good practices that are not readily available elsewhere.”

Moreover, many professional and personal networks have been formed as a result of the
training programmes. In these cases, participants have recognized the value added ofbeing connected to professionals in other fields for their career development, especially for
mid-career and senior professionals. Likewise, this is mostly observed for national orregional training programmes. Examples of networks sustained after the training are thosederived from the “Entrepreneurship and Leadership Youth Training Fellowship
Programme” in Iraq and South Sudan, where the fellows created a space to exchange theirexperiences and connect to each other for working opportunities.1“The multi-sector approach enriched the entire programme and proved essential and effective
in building trust, friendship, and a network of like-minded Iraqi youth.”2.

Alumni networks and local coaches create a local support system for participants
after the training
Benefits and uses of alumni networks include social relations, career advancement andjob searches, group work, dissemination of information (formal and informal) and improved
organizational leadership (Fredricks, 2003). Alumni networks can also build a local support
system for participants, creating a reliable and comfortable space where they can shareprofessional information and experiences. UNITAR alumni is also a valuable resource for
the Institute. Former participants are also potential participants for future trainings. UNITAR
alumni can bring their own expertise and knowledge to improved future training editions
and they can be the first promoters of UNITAR events and activities, extending beneficiaryreach with limited resources.

1 Further reading with the Impact Stories for the programme in South Sudan (UNITAR, 2021b) and Iraq

(UNITAR, 2021c).
4

“The FAO/UNITAR partnership has created a network of development practitioners who arerelying on the joint initiative as their trusted source of knowledge and information relating toagriculture and trade.”

Incentives to enter and participate in an alumni network and continuous follow-up are key
to maintaining an alumni network.2 Another key ingredient for maintaining alumni networks
is the inclusion of local coaches, who are former participants of the training being delivered.
Local coaches are not only knowledgeable on the training delivered but also carry thepractical experiences shared by their colleagues during their programme cycle. Moreover,they can serve as a bridge to connect current with former participants, maintaining the
network alive. Although alumni network experiences at UNITAR have been more common
in programmes with multiple cycles, thematic networks can also be promoted.
“Building an alumni network creates a local support system for fellows even after the
programme ends. The presence of coaches helps connect current Fellows with Fellows fromprevious cycles.”3.

E-learning is an effective tool for knowledge sharing and networking
While multicultural groups are important for knowledge sharing, bringing together a
network of professionals is not always a feasible option, e.g., constrained monetary
resources do not allow for traveling, health restrictions, etc. In these case scenarios, elearning is an efficient option for connecting professionals. It can be relatively cheap
compared to face-to-face events and give the advantage that participants are alreadyfamiliar with online means.
“Using innovative tools such as eLearning have been demonstrated as an effective andefficient approach to share knowledge and strengthen networking of like-minded officials
from different countries and government ministries.”

New technologies

2 More information is available on UNITARnet on good practices to build an alumni network and online

community of practice.
51.

Social media can be very advantageous to reach new beneficiaries and position
participants’ projects
While dissemination of information is well known to be one of the benefits of social media,
this does not only need to occur from the UNITAR platforms, but participants can be
instrumental to disseminate UNITAR’s work to new beneficiaries, especially for
programmes targeting youth. Participant interaction in social media platforms when sharing
training experience can be a key communication tool to incentivize new beneficiaries’
enrolment.
Moreover, the use of social media to improve participants’ projects visibility need to beconsider since the training design. This is of particular interest in those training involvingsocial entrepreneurship or, more generally, project development.
“Throughout the 2019 programme cycle, UNITAR experienced first-hand how participants
utilized social media to spread awareness of the programme and of their own related activities.
Social networks are also highly relevant to entrepreneurship and social entrepreneurship,
where disseminated information about an organization or service, as well as interaction withthe community an organization is servicing, is critical to an enterprise’s effectiveness andsuccess”

2.

Providing guidance to navigate digital learning platforms can help reduce dropouts
from digital learning
Online training formats have represented an alternative to continue delivering training
under conditions of limited mobility and security. However, formative and substantive digital
divide3 is not always considered or addressed to assure that all stakeholders can accessand participate in the training under the same conditions. Whilst some participants might
be very familiar with online learning platforms (e.g., Moodle, mobile applications, etc.),others may find difficult to navigate them, to the point of not feeling adequate for the training
and resulting in high attrition rates. Having a preliminary session or providing detailed
guidance, as well as making ourselves available to assist on how to use the digitalplatforms/tools required for the training can be important to maintain participants interestin the training.
“[…] Future iterations of the Programme will need to include more hands-on introductory
training on how to use the platform. Access to online training is still limited in some countries.
Many participants may not have a stable internet connection or even access to the internet –especially as UNITAR works with people in least developed countries and fragile states. Since
this Programme focuses on “leaving no one behind,” we must further consider easier and lowerbandwidth tools, be they a mobile-learning platform or other technologies that fit suchenvironments.”

UNITAR leaning solutions have developed a guidance on how to make online events more
inclusive, including matters on accessibility and technological literacy levels.

3 The digital divide occurs at three interdependent levels: i) digital access divide meaning lacking accessto materials and digital tools (formative); ii) digital capability divide when users are limited in practicewith digital tools and equipment (substantive); and iii) digital outcome divide where users lack belief andself-efficacy that further inhibits their interaction with digital tools (substantive) (Mathrani, Sarvesh &Umer, 2021).
6

Local or national ownership1.

Engagement of the community and local partners can improve project ownership
One of the challenges of projects developed by actors external to the community (or the
local setting) is creating local ownership, which may be determinant for projectsustainability.
“[…] Engaging local staff and institutional partners is instrumental to support effective projectdelivery and ensure ownership and sustainability of results. This is particularly important for
projects implemented in geographic regions distant from the location of the main project
partners.”

Engaging local stakeholders can help build ownership by:4i)Ensuring that the project accurately reflects local needs and priorities. A project
tackling needs that are a priority for the community may increase their interest in
project participation and continuation of its implementation. Moreover, it reduces
overlooking any unintended impact of the project that may deter further
beneficiaries engagement.
“Community-based DRR and social inclusion concepts were well received andrelevant. Participants recognized and understood very well the concept of communitybased DRR, as most of their countries’ populations live in villages and settlements.
Community-based DRR is based on a system of cooperation with a diverse group ofsocial actors. It must include the most vulnerable, including women, persons withdisabilities, youth, the elders, minorities and others.”ii)

Securing local support. This factor is of great importance in contexts with marked
community decision making power in broader organizational structures wherelegitimacy of the project falls broadly on the communities. A question to be asked

4 Based on Community Ownership Support Service (2020), Caribbean Development Bank and Social

Development Commission (2019), and own analysis of lessons learned.
7

iii)

is who do you need to engage to build a conducive environment to local conditions?
Adapting the project to the established local organizational structures can easeownership engagement.
Identifying new (adequate) partners and collaborators. Local stakeholders have thegreatest knowledge of “who does what” in their local communities. Collaboratingwith the right people can assure that understanding of the project and the roles
assigned to each stakeholder are not obstacles for project ownership. Moreover,they are also the most knowledgeable about, and sometimes the ones owning, thechannels to increase beneficiaries outreach.
“Rural educational institutions and school professionals (teachers, social workers) havea key role as liaisons in connecting the [master trainers] and master organizations withyouth and their families in vulnerable communities”

Sustainability of project results relies largely on local factors

2.

As described above, community and local stakeholders’ engagement is a vehicle toimprove not only ownership but also project sustainability and greater outreach. Other local
factors that can also contribute to sustainability of project results are:i)
Stakeholders dialogue and local networks. Engaging different stakeholders since
project inception and promoting a dialogue among them is key to prepare thegrounds for future project phases where multiple actors need to work together todeliver and sustain project results. Creating a space for dialogue can also
strengthened local networks, which might create a value added for stakeholders toremain active in the project.
ii)
Formal agreements. Creating formal agreements, when possible,5 to embrace theproject spirit after project completion has proved useful to assure its sustainability.
“It is possible to say that GHS implementation is now fully “taken up” (institutionalised)and sustainable in the target countries. This result would not likely have been possible
without both phases of the GHS work: [… i)] ground work of bringing together the key
stakeholders in a continuing dialogue, preparing initial analyses of the existing situationand preparing draft regulatory instruments; and Phase 2, which solidified thispreparatory work via finalising and bringing legislation in to force and conducting broad
technical training and continued awareness raising.”

iii)

Bringing stakeholder assets into the project. Stakeholders, including beneficiaries,
came with new skills, knowledge and experiences that may be beneficial for the
project in the long run. These assets should be accounted for during project design
when building a successful sustainability strategy.
“National training institutions were selected as training beneficiaries because they are
well placed to integrate new content on climate policy and public finance within their
ongoing training programmes and because these programmes already target Ministryof finance officials.”

iv)

3.

Continuous support for long-term expected outcomes. Sustainability can turn achallenge when project activities and expected outcomes are “on-going” processes
instead of steady results to be expected after project implementation. In thesecases, project impacts and sustainability should be thought in a realistic manner,so that continuous support can only be provided from local sources.

Incorporation of cultural and local insights into the project/programme design canfoster local ownership
Incorporating cultural insights into the project or programmes is important to make surethat the new knowledge being transferred has validity and practical value when applying

5 It is worth noting that lessons highlighting this factor came mostly from projects involving training on

international agreements/procedures/regulations. However, non-institutionalized local commitments
can also produce analogous experiences.
8

knowledge and skills and in achieving expected results. For training-related projects,
inclusion of practical examples from local realities are also very valued by participants andfacilitate them taking ownership for transferring/applying the new knowledge. Similarly,
experts that were familiar or have some practical knowledge with the local situation was a
success factor for attaining the new technical competencies and planned actions.
“Many participants mentioned the importance of cross-country communication in the forums
and how local examples helped them to understand theoretical concepts.”“Over the years, research shows that conflict sensitivity, cultural and taboos sensitization andthe basics of the language and communication skills are critical for strengthening thepeacekeeping efforts and the safety of the civilians and the FPUs themselves.”

4.

Ownership in country projects can be reached when project goal is aligned withnational priorities, although timespan should be flexible to potential delays
For country projects, ownership is easier to build when the project goals are aligned withnational priorities and strategies which usually carries with it the existence of a national
apparatus with experience on the regard or willing to take over responsibilities that
contribute towards the project goal (and national strategy).
“Effective capacity development focuses on longer-term outcomes and impacts. Therefore, thecontent of the programme's curriculum need to be driven by national priorities and guided bystrategic national partners.”

However, the project timeframe should be flexible expecting some delays in processes ofconsultation or even approval. An example of flexibility to construct national ownership isthe UN CC:Learn project, as described in Box 2 below.
Box 2. National priorities and strategies in UN CC:Learn project (2014-2017 Phase)
The One UN CC:Learn Partnership (UN CC:Learn) is a collaborative initiative involving morethan 36 multilateral organizations which are involved in the development and/or delivery ofclimate change related learning. The project’s 2014-2017 implementation phase aimed tocreating sustainable individual and institutional capacities in developing and transitionalcountries to plan and implement effective climate change actions.
The mid-term evaluation of the 2014-2017 phase assessed the initiative as having a highdegree of relevance and ownership from national partners (partner countries). Strategy
development processes were always highly participatory and inclusive, drawing on theexperience and resources of institutions from across the economy (government, academia,
private sector, CSOs / NGOs etc.). This was related to the high relevance of the project tonational priorities and strategies concerning climate change.
The evaluation recognized that highly participatory processes, however, may cause somedelays in the project implementation timeframe, which should be considered since project
design.
[…P]rojects that support the development of national-level strategies(regardless of thematic focus) will commonly be subject to ‘delay’,particularly where a high degree of consultation, participation and politicalownership is sought. However, if ‘delays’ are the price of obtaining genuine
national ownership, then this will invariably be a price worth paying. As such,it is important that the length of time allocated to national strategy
development is generous and flexible, supported by project plans andactivities that are not overly dependent on formal strategies being in place.
(UNITAR, 2016, p. 27).

9

Local support

1.

Lack of political will and support from supervisors can hinder behaviour change
UNITAR applies the Kirkpatrick/Phillips approach to evaluate training. This approach
categorizes evaluation data based on five levels, including Level 1 (reaction and planned
action), Level 2 (learning and confidence), Level 3 (application and implementation), Level
4 (impact/organizational change) and Level 5 (Return on Investment-ROI), as shown in the
figure below. From Level 3,6 the aim is to gather evidence on participants’ application ofknowledge, skills and competencies to indicate changes in behaviour, i.e., whether
participants are doing things differently after the training programme that will lead toorganizational change or other results (Level 4) and, ultimately, demonstrate good value
for money with a return on investment (Level 5).
However, evidence from evaluations have indicated that expected changes might bedifficult to achieve when there is no immediate support from beneficiaries’ line managers
or decision makers. This lack of local support can come from top leadership or from otherstakeholders in the leadership/command chain. Building trust and relationships necessary
to obtain support and create interest is possible but takes time, for which we should not
expect to see immediate results in these particular cases.
“Lack of political will and attention from line managers and supervisors makes it difficult for
Fellows to win the support of their organizations and ministries.”

6 Guidelines on evaluating application of knowledge and skills, can be found in UNITARnet.

10

Along the same lines, lack of interest from decision makers renders stakeholder
engagement difficult and may lead to a lack of coherence across actors (e.g., duplicationof efforts with little impact from each of the actions implemented).
“[… A] lack of engagement between civil society and government was highlighted a number oftimes during the Programme as a key impediment to the incorporation of lessons learned andchanges in behaviour.”2.

Local support can be built from inside the training programme7
Local support cannot only be expected from stakeholders outside the training programme,
but it can also be built from the inside, taking into account the human capital of participants
and resource persons. Linked to the lessons derived in the “knowledge sharing andnetworking” section above, alumni networks are key instruments to develop local support
systems besides connecting participants with potential partners, namely former
beneficiaries and colleagues.
“Building an alumni network creates a local support system for participants even after the
programme ends”.

Monitoring tools

1.

Log frames are important but need to be complemented by regular monitoring foreffectLogical frameworks are useful instruments to support results-oriented project planning andmanagement and have positioned as the main monitoring tool in projects. Despite its
strengths, log frames can be too rigid to monitor external and contextual factors. In these7

Lessons under these meta-lessons were all referred to programmes with learning components.
Therefore, we prefer to not generalize for project with other development outcomes.
11cases, complementary tools, such as mapping and qualitative feedback, are useful tobetter monitor project results.
“Log frames are a useful day-to-day monitoring tool, but they are not always sufficient for
expressing the full-range of influences, contextual factors and assumptions that underpin a
project, particularly where work is primarily focused on attaining qualitative changes.”2.

Monitoring achievement of learning objectives is a recurrent and significant
process
Assessment of achievement of learning objectives is mainly done by obtaining feedback
from participants at the beginning and end of the training programme through selfassessments or objective assessments of learning (e.g., end-of-course quizzes or other
instruments). Nonetheless, achievement of learning objectives can also be monitored
throughout programme delivery. Monitoring accomplishment of learning objectives on aregular basis, e.g., at the end of each module (e.g., check for understanding and end of
module quizzes) facilitates the identification of learning barriers that may hinderedparticipants learning process, resulting in shortcomings on results.
Tools to monitor achievement of learning objectives include completion rates after each
module (high dropout rates or delays in completion can give signs of learning difficulties),
individual grade records or pre-post quizzes and tests (OECD, 2019).
“The training conducted online ensured that the participation of each participant can bemonitored, and the achievement of the learning objectives is measurable. Monitoring progressof each participant on a weekly basis allowed the available course slots to be fully optimized
and to provide support and encouragement to those lagging behind.”

Key take-aways
Lessons can help inform drivers of success and address challenges upfront for future
programming. This issue has highlighted five categories with 13 meta-lessons that can be used as
drivers for successful future programming. The rapid evidence assessment has led to the following
take-away messages:8
•

•

•
•

•

Formulation and incorporation of lessons learned are a common practice at UNITAR, but
the knowledge obtained through this process is not always shared with other programmes
and divisions. More effort needs to be undertaken to also learn from practices of otherdivisions and programme units.
Training sessions can be a good opportunity to engage in knowledge sharing andstrengthen participants’ social and professional networks when the groups are diverse andwhen local resource persons are present.
Use of digital tools and platforms bring many advantages but the appropriate guidance toparticipants who are less familiar with them needs to be provided.
Local ownership can be strengthened through the engagement of the local community andpartners, incorporation of cultural and contextual insights and ensuring that projects are
relevant to national (local) priorities and strategies. At the same time, improved ownership
can also contribute to the sustainability of project results.
Behaviour change can be difficult to occur when there is lack of political will and support
from line managers, hindering impact achievement.

8 Some key messages from the last exercise are still applicable and can be found on UNITARnet, butwere not included in this document to avoid redundancy.
12•Monitoring participants’ achievement of learning objectives and ensuring the presence ofa well-built log frame are good practices to help understand final project achievement. Log
frames should, however, be complemented with other information for explaining contextual
factors affecting project results.

The way forward
As next steps, the following actions are foreseen:••

A lessons learned forum will be organised to discuss findings from this assessment andinvite colleagues from Programme Units to share and exchange experiences.
This series will be continued, and future editions will focus on different categories oflessons learned.

13

References
Barends, E., Rousseau, D.M., & Briner, R.B. (Eds.) (2017). CEBMa Guideline for rapid evidence
assessments in management and organizations, Version 1.0. Center for Evidence-BasedManagement. https://cebma.org/wp-content/uploads/CEBMa-REA-Guideline.pdf
Caribbean Development Bank & Social Development Commission. (2019). Effective community
engagement across the project management lifecycle. Draft, unpublished version.
https://www.google.com/url?sa=t&rct=j&q=&esrc=s&source=web&cd=&cad=rja&uact=8&
ved=2ahUKEwj6h6_6usX3AhVSuKQKHYQKD3QQFnoECAQQAQ&url=https%3A%2F%
2Fwww.caribank.org%2FBNTFOM%2FC4.1&usg=AOvVaw0DDhlNG7VXbXGHNCnwxW_z
Community Ownership Support Service. (2020). Involving your community: A practical guide to
the benefits and techniques for involving and consulting local communities in community
asset based projects.
https://dtascommunityownership.org.uk/sites/default/files/Involving%20Your%20Commun
ity%202020.pdf
Fredrick, S.M. (2003). Creating and maintaining networks among leaders: an exploratory casestudy of two leadership training programs. Journal of Leadership and Organizational
Studies, 10(1), 45-54. https://doi.org/10.1177%2F107179190301000104
Mathrani, A., Sarvesh, T., & Umer, R. (2021). Digital divide framework: online learning indeveloping countries during the COVID-19 lockdown. Globalisation, Societies andEducation, ahead-of-print, 1-16. https://doi.org/10.1080/14767724.2021.1981253
OECD. (2019). Monitoring learning outcomes of adult learning programmes. A review ofEuropean best practices on monitoring instruments. OECD.
https://epale.ec.europa.eu/sites/default/files/oeso-rapport-2019-monitoring-learningoutcomes-of-adult-learning-programmes.pdf
OECD/DAC. (2010). Glossary of key terms in evaluation and results based management.
OECD/DAC. https://www.oecd.org/dac/evaluation/2754804.pdf
UNITAR. Repository of Lessons Learned.
https://unitaremail.sharepoint.com/unitarnet/Lists/Lessons%20Learned/AllItems.aspx?gro
upBy=Challenge%5Fx0020%5FCategory
UNITAR. (2016). Mid-term evaluation of the UN CC:Learn 2014-2017 implementation phase.
UNITAR. https://unitar.org/sites/default/files/uploads/pprs/mid-term-evaluation-reportuncclearnnov2016_final_20170113.pdf
UNITAR. (2020). Making online events more inclusive.
https://www.flipsnack.com/unitarcatalogue/making-online-events-more-inclusive.html
UNITAR. (2021a). Evaluation policy.
https://unitar.org/sites/default/files/media/file/AC.UNITAR.2021.07%20%20Evaluation%20Policy.pdf
UNITAR. (2021b). Training young professionals in South Sudan on leadership andentrepreneurship skills, 2015-2019. UNITAR. https://unitar.org/about/newsstories/stories/prosperity-training-young-professionals-south-sudan-leadership-andentrepreneurship-skills-2015-2019

14

UNITAR. (2021c). Enhancing leadership and entrepreneurship skills of young people in Iraq.
UNITAR. https://unitar.org/about/news-stories/stories/prosperity-enhancing-leadershipand-entrepreneurship-skills-young-people-iraq
UNITAR. (n.d.). Good practice document: establishing and maintaining an alumni platform or
network.
https://unitaremail.sharepoint.com/unitarnet/Documents/Good%20Practice%20Document%20-%20Alumni%20Networks.pdf
UNITAR. (n.d.). Guidelines on evaluating application of knowledge and skills.
https://unitaremail.sharepoint.com/unitarnet/Documents/Guidelines%20on%20Evaluating%20Application%20of%20Knowledge%20and%20Skills%20(Level%203).docx
UNITAR. (n.d). Online community of practice: good practice guidance.
https://unitaremail.sharepoint.com/unitarnet/Documents/Good%20Practice%20Document%20-%20Online%20Community%20of%20Practice.pdf

15

Annexes
Annex 1. Lessons compilation
Lessons
Fellows highlighted that they were working in multi-sector teams for the first time and noted that the
programme presented an opportunity to work together for a common goal. The multi-sector approachproved essential and effective in building trust, friendship and a network of like-minded South
Sudanese.

Category
Knowledge sharing andnetworking

Sub-category
Multisector/multidisciplinar
y teamBuilding an alumni network creates a local support system for Fellows even after the programme
ends. The presence of coaches helps connect current Fellows with Fellows from previous cycles.

Knowledge sharing andnetworking

Alumni network; Coaches
and mentors

Access to the network of like-minded field practitioners would have been difficult to replicate in aface-to-face environment.
Collaborative learning exposed the learners to different experiences of participants from different CIS
countries, thus enhancing and deepening the learning experience and placing the content modules
in a practical context.
The active involvement and effective mentoring of the international experts created an environmentthat encouraged the sharing of experiences.
The diverse, multi-sector group of Fellows encouraged peer learning and collaboration. Afghans andother regional experts also facilitated South-South Learning.

Knowledge sharing andnetworking
Knowledge sharing andnetworking

Multisector/multidisciplinar
y teamMultisector/multidisciplinar
y teamKnowledge sharing andnetworking
Knowledge sharing andnetworking

Coaches and mentors

The regional components (e.g. regional workshops, consumer campaigns, etc.) played a key
supporting role to keep other regional countries involved and acted as platforms for knowledge
sharing amongst the whole region and a variety of concerned stakeholders.

Knowledge sharing andnetworking

knowledge sharing
channels

Training fostered a dialogue between government and CSO representatives from the same country
and helped to forge working relationships at country and regional level which are likely to continue
beyond the programme.
Global projects offer a platform to engage in south-south cooperation activities, either between
project countries or by tapping the larger network of UNDP and UNITAR partners to seek the best
possible synergies.

Knowledge sharing andnetworking

Multisector/multidisciplinar
y teamKnowledge sharing andnetworking

knowledge sharing
channels

16

Multisector/multidisciplinar
y teamThe FAO/UNITAR partnership has created a network of development practitioners who are relying
on the joint initiative as their trusted source of knowledge and information relating to agriculture andtrade.
Access by the participants with the Mentors through the weekly discussion forums was an enrichingexperience for them. Interaction with the Mentors allowed the participants to get additional insightsthat have not been previously considered.

Knowledge sharing andnetworking

Alumni network

Knowledge sharing andnetworking

Coaches and mentors

Using innovative tools such as eLearning have been demonstrated as an effective and efficientapproach to share knowledge and strengthen networking of like-minded officials from different
countries and government ministries

Knowledge sharing andnetworking

knowledge sharing
channels

The lively involvement and effective mentoring by FAO experts resulted in an encouraging
environment in terms of sharing experiences and allowing individualized advice to participants.

Knowledge sharing andnetworking

Coaches and mentors

Collaborative learning enabled the participants to learn more about the topic through otherparticipants’ opinions, facts, and experiences from different countries within the region. As a result,
this led to a richer learning experience involving a healthy balance of theory and practice.

Knowledge sharing andnetworking

Multisector/multidisciplinar
y teamSeveral participants mentioned they had already established contacts and there are futurecollaborative projects in sight. Due to the engaging environment provided by the discussion forms,
some initiatives were brought to life such as writing articles and research papers.
The knowledge sharing among the participants from other countries provided invaluable knowledge
and good practices that are not readily available elsewhere.

Knowledge sharing andnetworking

Alumni network

Knowledge sharing andnetworking

Multisector/multidisciplinar
y team

The multi-sectoral background of the beneficiaries means that learners have access to a much wider
access to good practices and experiences by other practitioners from other regions. In addition, theparticipants were also exposed to a network of like-minded officials and professionals as well as
potential partners.

Knowledge sharing andnetworking

Multisector/multidisciplinar
y team17

THE DIVERSE, MULTI-SECTOR GROUP OF PARTICIPANTS ENCOURAGED PEER LEARNING
AND COLLABORATION
Participants came from various organisations and sectors and brought diverse knowledge andexperiences to the programme. Many highlighted that, through group work and assignments, they
were working in multi-sector teams for the first time and that they would continue to utilize a multisectors perspective after they finished the programme. They noted the programme presented an
opportunity to work together for a common goal: to create a better future for the people of Iraq.
The multi-sector approach enriched the entire programme and proved essential and effective inbuilding trust, friendship, and a network of like-minded Iraqi youth. Furthermore, the promotion of
public-private partnerships is essential to the long-term economic stability of Iraq, as the public sector
is responsible for creating an environment in which entrepreneurship and social entrepreneurshipcan flourish; such an environment can then have a tangible positive impact on the socio-economic
development of Iraqi society.
Although participants did not all begin the programme with the same level of knowledge about
entrepreneurship, which could have hindered teamwork and small group dynamics, in practice
participants cooperated as they undertook practical exercises; they helped each other contextualize
the learning to their various professional experiences.
The learning needs of both those who already had entrepreneurship skills and those with lessexperience needed to be incorporated in the programme’s design; UNITAR Staff and our Resource
Persons rose to the challenge and created a nuanced programme that addressed participants’ needs
and facilitated communication between participants from diverse sectors and professional
experiences.
Knowledge sharing plays an important role for small island developing states where some for thegood practices applied by one country can be relevant for many others.

Knowledge sharing andnetworking

Multisector/multidisciplinar
y teamKnowledge sharing andnetworking

knowledge sharing
channels

DEVELOPMENT OF ALUMNI NETWORK

Knowledge sharing andnetworking

Alumni network

2018 marked the Programme’s third cycle, bringing the total number of graduates of the Programme
to 95 to date. Given this growing network, UNITAR launched an official alumni network to allow
participants to share their ideas on the events and activities they are carrying out

18

Physical distances within the Pacific make collaboration more difficult
It must be remembered when discussing these entities that the Pacific Ocean covers some 165
million square kilometres, or 46% of the earths water surface, and approximately one-third of its total
surface area. Participants in the programme hailed from Micronesia, Melanesia, and Polynesia locations separated at times by enormous distances - with Palau and the Cook Islands for example,
separated by over 7800 kilometres.
One key benefit of the programme, reported by a number of participants, was the ability to engage
face-to-face with counterparts from far-distant countries, facing similar issues with regards tsunami
and DRR.
THE IRAQI CONTEXT IS CONDUCIVE TO ENTREPRENEURSHIP AND SOCIALENTREPRENEURSHIP
Compared to other conflict-affected countries, where patchy internet access and othercommunication issues often hinder organisations’ ability to function, Iraq has thriving social andbusiness networks, both online and offline. Throughout the 2019 programme cycle, UNITAR
experienced first-hand how participants utilized social media to spread awareness of the programmeand of their own related activities. Social networks are also highly relevant to entrepreneurship andsocial entrepreneurship, where disseminated information about an organisation or service, as wellas interaction with the community an organisation is servicing, is critical to an enterprise’s
effectiveness and success. Iraq’s young population suited to utilize the new technologies and social
networks present in the country to address their communities’ numerous and evolving needs throughsocial entrepreneurship.
VARIOUS LEVELS OF TOOLS AND TECHNOLOGIES TO ENHANCE PREPAREDNESSThe participants observed various high-tech facilities and technologies to prepare for potential
disasters, such as satellite warning system, emergency control rooms, sea level measuring devices,seacoast surveillance cameras, and tsunami evaluation towers. While such facilities andtechnologies are not readily available in many countries, alternative tools which can be relatively
quickly applied were also introduced, such as smart phone applications, home emergency kit,customized Go-Bags, playing cards and handkerchief for children to learn preparedness, DRR Songsand Diorama. Participants discussed how these tools can be customized so that it fits to theircountries’ context.

19

Knowledge sharing andnetworking

Multisector/multidisciplinar
y team

New technologies

Social media and social
networks

New technologies

Alternative technologies

Technical fragility and the digital divide need to be addressed
While online training is, in some aspects, more efficient than conventional in-person training and canhelp increase the total number of beneficiaries, there is concern that it exacerbates the digital divide.
Participants were relatively favourable to EdApp since it did not require a large bandwidth. However,some participants dropped out of the training because they were not familiar with EdApp and had
difficulty using it. Future iterations of the Programme will need to include more hands-on introductory
training on how to use the platform.
Access to online training is still limited in some countries. Many participants may not have a stableinternet connection or even access to the internet – especially as UNITAR works with people in least
developed countries and fragile states. Since this Programme focuses on “leaving no one behind,”
we must further consider easier and lower-bandwidth tools, be they a mobile-learning platform orother technologies that fit such environments.

New technologies

Digital divide

Effective capacity development focuses on longer-term outcomes and impacts. Therefore, thecontent of the programme's curriculum need to be driven by national priorities and guided by strategic
national partners.

Local or national
ownership

National Priorities andstrategies

Projects that support the development of national-level strategies will commonly be subject to delays,
particularly where a high degree of consultation, participation and political ownership is sought.

Local or national
ownership

National Priorities andstrategies

GHS implementation is an on-going activity and not one that is quickly "completed" within any given
timeframe. All project countries identified need continued training and awareness raising as the GHS
continues to be "rolled out."

Local or national
ownership

Sustainability

It is possible to say that GHS implementation is now fully “taken up” (institutionalised) and sustainable
in the target countries. This result would not likely have been possible without both phases of the
GHS work: Phase 1 that did the ground work of bringing together the key stakeholders in a continuing
dialogue, preparing initial analyses of the existing situation and preparing draft regulatory
instruments; and Phase 2, which solidified this preparatory work via finalising and bringing legislation
in to force and conducting broad technical training and continued awareness raising.

Local or national
ownership

Sustainability

The selection of the host country for the major regional events/conferences can provide important
incentives for those countries.

Local or national
ownership

Country incentives

National training institutions were selected as training beneficiaries because they are well placed tointegrate new content on climate policy and public finance within their ongoing training programmes,and because these programmes already target Ministry of finance officials.

Local or national
ownership

Sustainability

20

Cultural Context on the conflict of the peacekeeping operation is critical.
According to interviews, desk research and group discussions, police officers are often trained onpolice’s standard operating procedures, and other operational requirements but less on the culturalcontext and the drivers of the conflict where they will be deployed. Over the years, research showsthat conflict sensitivity, cultural and taboos sensitization and the basics of the language andcommunication skills are critical for strengthening the peacekeeping efforts and the safety of thecivilians and the FPUs themselves.

Local or national
ownership

Cultural and local context

Using a global project approach does not seem most appropriate to the development of national
MIAs, particularly given variations in capacity among targeted countries, and such projects should
consider a longer period of time for implementation.

Local or national
ownership

Implementation frameWith country projects it is important to involve local partners and engage with a wider community asmuch as possible to ensure local ownership and participation.

Local or national
ownership

Community and localorganizations engagement

Strengthening networks of local organizations and agents of change is essential to achieving
sustainable impact in individual and communities that leads to peaceful conflict resolution andreconciliation

Local or national
ownership

Sustainability

Rural educational institutions and school professionals (teachers, social workers) have a key role asliaisons in connecting the MT and master organizations with youth and their families in vulnerable
communities
Importance of local staff and partners in consortia. Engaging local staff and institutional partners is
instrumental to support effective project delivery and ensure ownership and sustainability of results.
This is particularly important for projects implemented in geographic regions distant from the locationof the main project partners.

Local or national
ownership

Community and localorganizations engagement

Local or national
ownership

Sustainability

The joint FAO and UNITAR initiative has created hope from the course participants that similar
initiatives will continue considering that the development challenges they face on the ground continue
to evolve.
Many participants mentioned the importance of cross-country communication in the forums and how
local examples helped them to understand theoretical concepts.

Local or national
ownership

Cultural and local context

Local or national
ownership

Cultural and local context

21

EXPERTS FAMILIAR WITH THE LOCAL SITUATION CONTRIBUTED TO CONTEXTUALIZEDLEARNING
UNITAR’s Iraq-based Staff, Coaches and Mentors were helpful in relating programme content to thecurrent local situation in the country through contextualisation of the discussion with local examples
and knowledge. It helped the learning atmosphere greatly by focusing on individual exercises, group
activities, role play. By applying what they have learned through their projects and in real life
scenarios have improved the knowledge retention and contributed to rich learning experience.
Local expertise is indispensable when working in conflict-affected areas, and the inclusion of said
expertise helped make the programme highly relevant to the participants’ needs. Partnership with
Zain Telecommunication, The Station and KAPITA, which run incubators and facilities inside Iraq
that provides aspiring young entrepreneurs with resources and space to work free of charge,
introduced participants to current entrepreneurship opportunities that exist close at hand.

Local or national
ownership

Cultural and local context

Community-based DRR and social inclusion concepts were well received and relevant
Participants recognized and understood very well the concept of community-based DRR, as most of
their countries’ populations live in villages and settlements. Community-based DRR is based on asystem of cooperation with a diverse group of social actors. It must include the most vulnerable,
including women, persons with disabilities, youth, the elders, minorities and others.
The 2020 Programme put ever more focus on social inclusion and addressed meeting not only theneeds of women in the communities but of the elderly and people with special needs. Participants
responded that these were new areas that they were eager to consider in their communities.

Local or national
ownership

Community and localorganizations engagement

Save yourself first
The importance of all stakeholders was stressed by a number of the study visits and engagementwith local partners. Everyone, from school children to the elderly, should be empowered andencouraged to make the decision to evacuate themselves, without waiting for official warnings. This
component figured heavily in participant discussions regarding evacuation planning andimplementation of lessons learned

Local or national
ownership

Community and localorganizations engagement

Lack of political will and attention from line managers and supervisors makes it difficult for Fellows
to win the support of their organizations and ministries.

Local support

Political will andhierarchies

Building the relationships and trust necessary to receive honest feedback to an Organizational Needs
Assessment takes time.

Local support

Social relations and trust

INTRODUCING LOCAL COACHES FOR THE 2017 AND 2018 CYCLESCoaches are standout graduates from previous cycle who are assigned with providing daily coaching
during workshops, as well as regular mentoring and overall guidance on the programme’s
assignments and requirements to participants. Coaches play an important role to help contextualize
the learning to local realities. Coaches receive additional “Coaching for Coaches” training as acontribution for their further personal and professional development.

Local support

Coaches22

BUILDING AN ALUMNI NETWORK CREATES A LOCAL SUPPORT SYSTEM FOR
PARTICIPANTS EVEN AFTER THE PROGRAMME ENDS
With each cycle of the programme, the network of alumni in South Sudan grows. Along with theknowledge and skills gained through UNITAR’s programme, former Participants have experience
working and developing projects in South Sudan, making every single alumni – not just Coaches or
local Staff – a valuable resource to both future Participants and their local counterparts.
Furthermore, the multi-sector nature of the programme means that Participants have access to amuch wider network of resources and potential partners than if they only had contacts in their own
sectors. The presence of Coaches also helps connect current Participants with Participants fromprevious cycles through Coaches’ participation in both groups.

Local support

Alumni newtork; Coaches

Participants discussed concerns and reservations that their organizations and institutions currently
pay less attention to timely Organizational Needs Assessments (ONA) that could potentially improve
the organizations’ goals as well as overall performance. This is particularly relevant for public-sector
organizations and ministries where lack of political will and attention from the line managers andsupervisors makes it difficult for Participants to win the support of their organizations and ministries.
Although there is no culture of conducting regular ONA in many organizations, part of the problem isa lack of supervisor support and a lack of communication between supervisors and an organization’s
top leadership. Linked to this are general problems such as of lack of funding, financial fiscal stability,
equality between public sector organisations, and the overall political environment. Participants noted
the idea of cultivating champions within organizations and ministries to win over supervisors’ support
and to bring the ONA to the attention of the top leadership. Participants should be encouraged tosuggest regular ONAs to their organization, to improve said organization’s service delivery andoverall performance.

Local support

Political will andhierarchies

A FURTHER NEED FOR COMMUNICATIONS TRAINING RELATING TO ENGAGEMENTBETWEEN CIVIL SOCIETY AND GOVERNMENT
While this is not unique to SIDS, a lack of engagement between civil society and government was
highlighted a number of times during the Programme as a key impediment to the incorporation oflessons learned and changes in behaviour.
Linked to this is the wasting of resources due to parallel and duplicative systems, processes, andtraining by government and civil societies, particularly within remote communities as encountered bymany SIDS.

Local support

Coherence and hierarquies

Logframes are a useful day-to-day monitoring tool, but they are not always sufficient for expressing
the full-range of influences, contextual factors and assumptions that underpin a project, particularly
where work is primarily focused on attaining qualitative changes.

Monitoring tools

Logframe

23

The training conducted online ensured that the participation of each participant can be monitored,
and the achievement of the learning objectives is measurable. Monitoring progress of eachparticipant on a weekly basis allowed the available course slots to be fully optimized and to provide
support and encouragement to those lagging behind.

Monitoring tools

Measurement of LOs

The training conducted online ensured that the achievement of the learning objectives is measurable

Monitoring tools

Measurement of LOs

TRUST-BUILDING IS KEY TO CONDUCTING A SUCCESSFUL A DATA COLLECTION THROUGHCUSTOMERS INTERVIEWS AND INSIGHTS
To compliment what participants learn during workshop II, one of their assignments is to conduct arobust and objective needs assessment through customers insights and observation in the field.
Participants highlighted challenges to conducting customers centred interviews due to many
challenges including the prevailing political and security environment in Iraq.
Building the relationships and trust necessary to receive honest feedback takes time and patience.
In future iterations of the programme, UNITAR Staff and Resource Persons will make sureparticipants understand the time required to thoroughly carry out data collection, as well as provide
advice for building relationships with a target community.

Monitoring tools

Measurement of LOs

24