Lessons learned from
evaluations
Series 1
DECEMBER 2021i
Planning,
Performance Monitoring and Evaluation

List of Tables

Table 1: Number of lessons per category .......................................................................... 2
Table 2: Lessons per Division and type of evaluation ...................................................... 3
Table 3: Adult learning principles. Adapted from Knowles (1973). .................................. 6

Table of Contents

List of Tables .......................................................................................................................... ii
Foreword ................................................................................................................................ iii
Executive summary .............................................................................................................. iv
Acronyms and abbreviations ............................................................................................... vi
Introduction ........................................................................................................................... 1
Purpose.................................................................................................................................. 1
Methodology.......................................................................................................................... 1
Limitations ......................................................................................................................... 2
Findings: Lessons Learned ................................................................................................. 3
Training methodologies .................................................................................................... 3
Tracking outcomes and impacts ................................................................................... 11
Gender inclusivity ........................................................................................................... 15
COVID-19 .......................................................................................................................... 17
Take-away messages ......................................................................................................... 19
The way forward .................................................................................................................. 19
References ........................................................................................................................... 21
Annexes ............................................................................................................................... 23
Annex 1. Lessons compilation ....................................................................................... 23ii

Foreword
Learning from evaluations is an important part of any evaluation exercise. This forwardlooking function of evaluation is often given less attention than the accountability function,
yet as a learning organization, UNITAR needs to seize opportunities to identify, share,
discuss and incorporate lessons from past experiences into future programming andpractices.
This lessons learned series provides the UNITAR community of professionals with structured
summaries on lessons learned from both self-evaluations and independent evaluation
exercises. This first series highlights lessons on training methodology, tracking outcomes
and impact, gender and COVID-19. It is hoped that this series will serve as a tool to promoteevaluation’s important learning function.
The able inputs from Katinka Koke, Roxana Gomez and Joyce Mukoma as well as fromcolleagues from programme units who have identified lessons from self-evaluation
undertakings are gratefully acknowledged.

Brook Boyer
Director, Division for Strategic Planning and Performance
Manager, Planning, Performance Monitoring and Evaluation Unitiii

Executive summary

The Planning, Performance Monitoring and Evaluation Unit (PPME) has developed thispaper series on lessons learned to provide valuable insights for learning and improved
decision-making related to future programming. This paper uses the rapid evidence
assessment (REA) methodology to analyse and synthetize lessons learned from past
UNITAR independent and self-evaluations undertaken since 2017.
This first edition focuses on lessons related to training methodologies, tracking outcomes
and impact, gender inclusivity and COVID-19. Seventy-eight lessons were analyzed,
resulting in fifteen lessons learned. The selection of lesson categories has been done
according to its relevance and to the number of lessons in the category.
The methodology chosen comprises nevertheless a few limitations. In order to be ‘rapid’, an
REA compromises on the breadth, depth and comprehensiveness of the search and in thiscase, it does not allow for an in-depth study of the context of each project in which a lesson
learned was formulated. Another limitation is an unequal distribution of lessons by UNITAR
division. Moreover, while evaluations derive from different sources as stated above, themajority has been formulated by UNITAR personnel in self-evaluation reports. While theLessons Learned database offers richness in lessons across categories, information on
actual use of those lessons to inform future project implementation is limited.
The following meta-lessons were found:Training methodologies

1.

Needs assessments help determine instructional design and ensure that thecontent is aligned with participants’ learning needs
Theory and practice need to be carefully balanced, and adult learning
principles can define the right mix.
Both self-paced e-learning and tailor-made other training events can lead tohigh participant engagement through flexibility in assigning learning time andadaptation to the audience’s needs and characteristics.
An appropriate selection of training experts and resource persons is key forensuring participants’ engagement.

Tracking outcomes andimpacts

Follow-up coaching and mentoring are key ingredients for ensuring
participants’ knowledge/skills application
Attitudes and behaviour change in capacity building projects are captured
better in the long-run.
Impact might occur through unplanned channels and unintended outcomescan be rewarding.
An adequate and valid Theory of Change (ToC) is essential for measuring
impact.
Theory-based approaches to evaluation are useful for measuring impact indemand-driven projectsivGender inclusivity
COVID-19

Achieving women empowerment requires a strategic approach with a needs
assessment that includes a thorough gender assessment.
Including different perspectives in group discussions can be enriching for allparticipants.
Depending on the cultural and national context, change management
strategies might also be required to target male participants’ attitudes
towards female participants or women in more general terms.
Participants’ engagement is crucial during the transformation of training
delivery formats
COVID-19 as an opportunity to strengthen online learning delivery methods.

Addressing the digital divide in the wake of the COVID-19 pandemic requires
adaptive management and availability of online tools
The paper concludes that while lessons are generalized, in some instances, they are limited
in applicability to all UNITAR programming and sometimes only applicable to more specificcontext. Moreover, the paper recognizes the usefulness of recording lessons learned fromexperience to inform drivers of success or challenges for future projects. Most of the lessons
examined as part of the assessment highlight strengths and only a few lessons highlight
weaknesses, though lessons learned can in general arise from both.
As next steps, a lessons learned forum will be organised to discuss findings from thisassessment and invite colleagues from Programme Units to share and exchange. Moreover,this paper series will be continued and future editions will focus on different categories oflessons learned. The Lessons Learned database and this assessment shall inform new
programming development. To track use of it, a survey to programme management will be
issued to measure use of lessons learned to design future phases of existing or new
programming.

v

Acronyms and abbreviations
CAS

Complex Adaptive Systems Approaches

FAO

Food and Agriculture Organization of the United Nations

GAANA

Gender-responsive Asset-based Approach to Needs Assessment

MSC

Most Significant Change

OECD/DAC Organisation for Economic Co-operation and Development/Development
Assistance Committee
PPME

Planning, Performance Monitoring and Evaluation Unit

QS

Quality Assurance Framework Quality Standards

REA

Rapid Evidence Assessment

TOC

Theory of Change

ToT

Training of Trainers

UNITAR

United Nations Institute for Training and Research

UNOSAT

United Nations Satellite CentreviIntroduction

Learning from past experiences to inform future programmes or organizational decisions is avital step of an evaluation process. Evaluations are useful not only for the topic or programme
under evaluation, but also other programmes/projects that may have similar characteristics(e.g., location, target audience, programme objective, etc.).
Aligned with UNITAR strategic framework’s learning approach to evaluation, the Planning,
Performance Monitoring and Evaluation Unit (PPME) has developed this paper series. Using
rapid evidence assessment (REA) methodology, this paper series analyzes and synthetizes
lessons from UNITAR independent and self-evaluations undertaken since 2017. Each of theseries contains lessons from different categories.
This first edition focuses on lessons related to training methodologies, tracking outcomes
and impact, gender inclusivity and COVID-19. Seventy-eight lessons were analyzed,
resulting in fifteen lessons learned. After this introduction, the purpose and methodology arepresented, followed by the findings, take-away messages from the process and a descriptionof the next steps.

PurposeThe Lessons Learned paper looks into past evaluations and captures lessons learned fromUNITAR’s work across its programmes from 2017-2020. This paper shall provide valuable
insights for improved decision-making and better development results. The first edition
focuses on lessons related to training methodologies, tracking outcomes and impact, genderinclusivity and COVID-19. The selection of lesson categories has been done according to its
relevance and to the number of lessons in the category. The lessons derive both fromindependent and from self-evaluations.

Methodology

This paper uses the definition of lessons learned proposed by the OECD/DAC Glossary(2010). Lessons or “lessons learned” are defined as: “Generalizations based on evaluation
experiences with projects, programs, or policies that abstract from the specific circumstances
to broader situations. Frequently, lessons highlight strengths or weaknesses in preparation,
design, and implementation that affect performance, outcome, and impact.” (OECD, 2010,p.26).
The guiding questions proposed for the analysis are the following:•••

What can we learn from past programming to guide future design, planning,
implementation, monitoring and evaluation phases?
What factors can be identified as drivers of success or reinforcing challenges towards
the achievement of project/programme goals?
What are the main lessons from the COVID-19 pandemic to inform future
programming?

The REA methodology provides a balanced assessment of what is known (and not known) in
the scientific literature (or in the lessons learned database) about an intervention, problem or1

practical issue by using a systematic methodology to search and critically appraise empiricalstudies. 1
The lessons analyzed derive both from independent and self-evaluations available in the
UNITAR repository of lessons learned from 2017-2020 2 as of October 2021. Lessons in therepository are classified under 19 categories of lessons according to their type. This edition
focuses on four of these categories: training/learning methodologies, tracking outcomes andimpact, gender inclusivity and COVID-19. As mentioned above, the selection of lessons
categories was made according to its relevance and the number of lessons classified in the
category. Training methodologies was the category with the larger number of lessons (48),while tracking outcomes and impact, gender inclusivity and COVID-19 have less than 15
lessons each, they were incorporated by their relevance for UNITAR programming. UNITAR
is committed to mainstream gender considerations into its programming, as stated in its
Strategic Framework and Gender Policy 3, as well as aligning its work from a results-based
perspective. On the other hand, the COVID-19 has required a shift in planning andimplementation of many of UNITAR’s projects which makes it relevant to learn from earlyexperiences of this new reality.
Seventy-eight lessons were analyzed for this paper. The classification per category ispresented in Table 1 below.
Table 1: Number of lessons per category

Limitations

In order to be ‘rapid’, an REA compromises on the breadth, depth and comprehensiveness ofthe search and in this case, it does not allow for an in-depth study of the context of each projectin which a lesson learned was formulated (Barends, Rousseau & Briner, 2017). Anotherlimitation is an unequal distribution of lessons by UNITAR division, with a total of 33 lessons
from the Prosperity Division but only 1 lesson from the Division for Multilateral Diplomacy, as
shown in Table 2. Moreover, while evaluations derive from different sources as stated above,the majority has been formulated by UNITAR personnel in self-evaluation reports (49 lessons).
While the Lessons Learned database offers richness in lessons across categories, information
1
Barends, E., Rousseau, D.M. & Briner, R.B. (Eds). (2017). CEBMa Guideline for Rapid Evidence
Assessments in Management and Organizations, Version 1.0. Center for Evidence BasedManagement, Amsterdam. Available from www.cebma.org/guidelines
2
Exception is made for the lessons under the COVID-19 category, which projects were implemented in
2020-2021.
3
Strategic framework:https://unitar.org/sites/default/files/media/publication/doc/unitar_strategicframework_web-new.pdf
Gender policy: https://unitar.org/sites/default/files/media/publication/doc/unitar-gendermainstreaming2016.pdf

2

on actual use of those lessons to inform future project implementation is limited. Finally, at thetime of this paper drafting, an independent evaluation of UNITAR programming response to
COVID-19 was in progress. Therefore, lessons of this assessment shall be read in conjunction
with the report.
Table 2: Lessons per Division and type pf evaluation

Findings: Lessons Learned
Training methodologies

The 48 lessons under this category can be categorized under five sub-categories which arealigned with the UNITAR Quality Assurance Framework (QAF) quality standards (QS) (2017):i) learning needs (related to QS 1); ii) content and structure (related to QS 5); iii) methodology
and target audience (related to QS 6 and QS 2); iv) training expertise (related to QS 8); andv) training follow-up (related to QS 10). Likewise, five main lessons are identified based on theabove.

3

1.

Needs assessments help determine instructional design and ensure that thecontent is aligned with participants’ learning needs

This lesson derives from the sub-category “Learning needs”. Needs assessments are key
training preparations in order to first answer the question whether training is the right answer
to the needs identified, how the training is addressing the identified needs, what instructional
design methods will be appropriate and what content shall be selected.
A needs assessment is an information-gathering process that assesses and analyses data
about an organizational need that could be met through training in order to determine thedifference between the current situation and the desired outcome (gap analysis) in regard to
the variables of interest. However, the response to a needs assessment is not necessarily atraining. Different processes are illustrated in a Good Practice document highlighting
experiences from two UNITAR Programme Units. 4“The needs assessment confirmed that the majority of participants were interested
in acquiring training skills and sharpening their communication and presentation
skills. However, it then became clear that some of the participants were more likely
to engage in other forms of outreach than training in their functions.”
The QAF 5 includes a set of ten quality standards. Quality Standard 1 focuses on learningneeds and includes the following sub-points:••••

The event responds to an identified learning need(s) for a specified target
audience.
Consideration is given to the relationship between learning needs andperformance needs of individuals/ organizations.
The event responds to an identified learning need(s) for a specified target
audience.
The learning need(s) are derived from and associated with evidence ofindividual and/or organizational capacity/performance needs.

For needs assessments to be effectively used, it is good practice to share the results from theassessment with trainers/resource persons and with participants. Box 1 presents an example
from the Hiroshima Office.
Organizational aspects such as advance notifications to participants, analysing participantlists, collecting expectations and assessing diversity in the audience can also lead to better
addressing training needs and expectations during the training event.
“Advance notification enables potential participants to prepare for the course and plan
accordingly.”“It’s helpful to have in advance the training participants list. This enables a better awarenessof the broad professional background and GIS knowledge level of participants to ensure thatthe training content takes into consideration the various learning needs.”“Some participants were approaching the training with very different expectations and fromvery different backgrounds. At first, it was challenging to create a collaborative work
environment amongst participants.”

Good Practice Document - Training Needs Assessment.pdf (sharepoint.com)
https://unitaremail.sharepoint.com//unitarnet/Documents/AC.UNITAR.2017.11%20%20Quality%20Assurance%20Framework,%20Revision,%202017.pdf
4
54

Box 1. “UNITAR Hiroshima Transforming the Future: Empowering Social Entrepreneurs andYouth Leaders for Iraq 2020” Self-Evaluation
The programme always undertakes a Needs Assessment with participants following theiracceptance to a programme. The assessment results help determine the instructional design of the
programme; they are also shared with Resource Persons for further tailoring the content to theparticipants learning needs.
All the participants undertook an initial needs assessment questionnaire regarding the content of
the programme during the orientation workshop in Baghdad. An anonymised summary of the needs
assessment results was shared with participants during workshop and was addressed during
presentations and practical exercises sessions. Participants indicated that exposure to these results
allowed them to identify issues they all faced in their work.
The needs assessment results indicated that participants wished for more focused training aroundkey elements of entrepreneurship, social entrepreneurship, communication techniques and cofounding team. The assessment data showed that more than half (62 per cent) of the participants
have “no knowledge” to “some knowledge” on key elements of entrepreneurship and social
entrepreneurship, which were addressed during workshop II. Furthermore, 92 per cent ofparticipants mentioned that effective communiques are relevant is to their current role, while 85 percent of participants mentioned that knowledge and skill around developing budgets will benefit them.
To address the gap in participants’ knowledge of entrepreneurship, introductory sessions wereincorporated in the programme to outline the concepts of entrepreneurship and social
entrepreneurship, as well sessions examining communication techniques and teambuilding andteamwork.

2.

Theory and practice need to be carefully balanced, and adult learning
principles can define the right mix.

The second sub-category focuses on “content and structure”. The selection and choice of
training topics, but equally having the right mix of theory and practice is of key importance.
Time allocations need to be made according to needs assessed (see lesson one).
“There is a continuous need to balance conceptual elements with technical, issue-specificactions to take.”“Theory without practice is less effective. The evaluation concluded that theoretical content –although critical to establish the foundational know-how – needs to be complemented bypractice. All informants insisted on this aspect. Practice helps build their muscle memory, theirprofessional ethics and behaviour and integrate all the notions through repetition and real-lifescenarios.”5

In this context, adult learning principles 6 are of use for defining the right mix. Table 3 below
provides an overview of adult learning principles.
Table 3: Adult learning principles. Adapted from Knowles (1973).

As we see in Table 3, adults enjoy being active during the learning process. Hence, the below
lesson concludes from a challenge that too-theory intensive events were not well received byadults:“The courses are considered too theory intensive. Some participants thought that given theprice paid for the course they were not getting enough in return in terms of content.”
Quality Standard 5 focuses on “Content and Structure” and consists of three sub-points.
••

6

Information on the content and structure of the event is presented in a clear and logicalsequence.
Learners are provided with a programme schedule or outline for face-to-face briefings,
workshops or seminars which details the sequencing of the contents and activities. For
courses, learners are provided with a syllabus.

See UNITAR Facilitation Toolbox for more information on training design.
6

•

3.

Methods, tools and interactive activities are structured in such a way as to facilitate theachievement of learning objectives.
Both self-paced e-learning and tailor-made other training events can lead tohigh participant engagement through flexibility in assigning learning time andadaptation to the audience’s needs and characteristics.

The third sub-category focuses on “methodology and target audience”. The lessons
contained in the category differentiate between e-learning, face-to-face and blended formats.
E-learning is considered as either complementary or an opportunity to reach larger groups
and ensure the learning is being continued during global health pandemics, such as COVID19 (see section below on COVID-19). When e-learning is in a self-paced modality, participants
can adjust training time in a flexible way to their respective working schedules.
“A self-paced and on-demand feature of the course ensured that the participants could
complete the course requirements without disruption of their professional activities.”
It is crucial to adjust to the target group by e.g., using experimental and dynamic approaches
when organizing training for youth and also adapting learning material according to theaudience’s characteristics. Other useful methods are case studies, study tours, mockcommittee meetings and feedback sessions which allow participants to practice their skills.
Teamwork and group assignments make adult participants feel more comfortable. Throughout
implementation, regular communication and interaction both with experts and peers was foundto be effective. Small repetitions of learning content can lead to fostering learning. Developing
projects as follow-up assignments were found to be great opportunities for participants to use
their skills and participants were highly engaged.
“More varied presentation techniques encourage learning. This includes increased use ofvisual aids, such as PowerPoint presentations and videos.”“Learning approaches structured around case-studies, mock Committee meetings andfeedback from a panel offered "real life" learning opportunities.”“Duplication is not a problem so long as the delivery methodology is effective.”“Working with youth in rural and urban settings requires adapted materials and methods that
help put knowledge in context and considers the differences across audiences in terms ofgender, age, culture and social norms.”

7

UNITAR’s QS 6 focuses on “Methodology” and refers to three sub-points:•••

The methods and tools to be used are defined, and relevant to the achievement oflearning objectives.
Learning is systematically assessed.
The mode of delivery selected is an effective medium for learners to achieve thelearning objectives.

UNITAR’s QS 2 focuses on “Target Audience” and refers to two sub-points:••

The target audience is defined with prerequisites (e.g., prior learning) or other selection
criteria clearly defined.
The event limits participation to the targeted audience (exceptions may be granted).

Backstopping services may be offered as additional add-on services that allow to adjust to
local needs. The below example illustrates how activities can be tailored to the target audienceto address individual institutional needs in the Pacific:“Integrating flexible mechanisms such as backstopping activities that can react to local
demands, particularly in rapidly changing environments such as the Pacific Islands, issupportive of successful project implementation.“

4.

An appropriate selection of training experts and resource persons is key forensuring participants’ engagement.

The fourth sub-category focuses on “training expertise”. The choice of resource persons is
key and, as impact story results have shown, 7 experts with great regional, national and local
expertise and knowledge are much appreciated by participants. Apart from trainers, also
mentors, coaches, master trainers and other resource persons, on international, national andon local levels, play an important role in framing the learning journey. It is also key that there
is a continuous and frequent exchange between the different personnel involved in the training
event.
UNITAR’s QS 8 focuses on “Training Expertise/Qualifications” and refers to three sub-points:•••

Individuals involved in the delivery / facilitation /moderation of training have therequired expertise given the level and nature of their involvement.
The bio profile/qualifications of the selected facilitators/moderators is communicatedto participants.
The event does not exceed a 30: 1 ratio of event participants to facilitators/trainers.

The QAF further specifies that the facilitator, tutor or moderator has the overall responsibility
to guide learners throughout the learning activity and assist them in the knowledge acquisition
process. They need to have the required expertise in the subject matter and adequate skills
as trainers to effectively facilitate the learning process. Online facilitators/tutors require in
additional specific online facilitation skills to adequately guide learners, in particular
coordinating and summarizing discussion threads, coordinating collaborative activities,
providing feedback, and assuring an adequate and well-balanced level of participation of thewhole cohort.

7

https://unitar.org/results-evidence-learning/stories
8

“Experts familiar with the local situation contributed to contextualized learning and were able
to relate programme content to the current local situation.”
Beyond training expertise, it is key how trainers/experts are being involved in discussion
boards, as the following lesson illustrates:“More involvement of the trainer on discussion boards would encourage the contribution of theparticipants. Posting two or three times a week, reacting to the participants’ contributions,would be good ways for the trainer to keep the discussions active.”

5.

Follow-up coaching and mentoring are key ingredients for ensuring
participants’ knowledge/skills application

The fifth sub-category focuses on “training follow-up”. Follow-up, coaching and mentoring
are recognized as part of efficient follow-up strategies to make sure learning can be appliedand used once back on the job. It is also considered part of exit strategies and shall lead toincreased sustainability. A learning reinforcement plan 8 can include communities of practices,simulation exercise, gamification, amongst others. It can start during the training (if there aredifferent modules) or after the training. In case of mobile reinforcement of learning for example,
there can be retrieval exercises (e.g., multiple-choice questions, open questions, flashcards)
every other day. Communities of practice, be they online or in person, can also lead toenhanced application and use of knowledge and skills as illustrates the below example:“Online communication and interaction with the participants on a regular basis (expert-toparticipant and peer-to-peer) provided invaluable access to practical information on the course
theme and a community of practice.”“Coaches greatly enhanced participants' confidence in using newly acquired policy appraisal
tools.”

8

See UNITAR good practice document on learning reinforcement plans
9Box 2: “UNITAR Hiroshima Developing Dreams: Entrepreneurship and Project Design in South
Sudan 2019 cycle” Self-Evaluation
Coaching and mentorship are key elements of this programme, augmenting the learning andproviding participants support as they complete their assignments. Volunteering their time, insight,
and experience, coaches and mentors are an important element of the programme and an invaluableasset to the timely delivery and quality of the programme. To promote South-South cooperation,international mentors from the UNITAR Hiroshima Afghanistan Programme were introduced to thenew participants in close coordination with the UNITAR Juba-based coaches, providing necessary
mentorship to the participants on a regular basis. Participants placed into five separate coaching andmentorship groups allowed mentors to provide focused, one-to-one and critical insights on
assignments and hold need-based online meetings. Most of the correspondence between
participants, mentors, and coaches took place through email exchanges. In evaluations, sharing
knowledge, experiences, and ideas attributed to an enriched learning experience and mentorship.
UNITAR will explore the possibility of organizing periodic face-to-face sessions between local mentors
and new participants in the UNITAR Juba Office to establish a good rapport between mentors andparticipants in future iterations of the programme.

Other follow-up activities can involve assignments to develop projects/plan to directly provide
participants with an opportunity to apply knowledge/skills as illustrated in Box 3.
Box 3: “Anti-Corruption Training Programme for North Africa” Self-Evaluation
One example of best practice was to ask the participants to develop an anti-corruption project/plan of
their choice using a template prepared by UNITAR. This was done as a follow-up assignment to thefirst workshop and was then presented at the second workshop. From the results it was clear that theexercise was done with interest and was used by most as an opportunity to develop existing ideas while
using newly acquired skills, and fostering new partnerships.

A specific type of training is “Training of Trainers (ToT)” which can lead to multiplier effects
when trained trainers use knowledge/skills to deliver training by trainers. An independent
cluster evaluation on Training of Trainers 9 has concluded some relevant lessons, specificallyon ToT:••

Institution-based. ToT initiatives that are institution-based are likely to produce more
enduring results, as trained trainers of learning centres (whether they are staff orconsultants) are more likely to have training opportunities in the medium to long term.
While an institutional basis for ToT is a necessary condition, it is not sufficient,however.
Partner ownership, buy-in and clarity in roles. Successful ToT requires a spirit of multistakeholder partnership, buy-in and clarity in roles by the ToT service provider, thebeneficiary institution/client, the donor (if applicable) and the individual trainers to be
trained. A lack of clarity in the ToT partnership concept or the roles of any of thesestakeholders can compromise the success of the engagement.

“A train-the-trainer approach can achieve good multiplier effects for outreach.”9

Cluster Evaluation of UNITAR’s Training of Trainers Programming | UNITAR
10

Tracking outcomes and impacts

Twelve lessons were classified under this category, leading to four sub-categories: i) definition,
follow-up and measurement of causal links; ii) data collection and management; iii)
measurement of behavioural change; and iv) project cycle and timeframe. Likewise, four main
lessons are identified based on the above.

1.

Attitudes and behaviour change in capacity building projects are captured
better in the long-run.

A large part of the Institute’s programming is focused on capacity building interventions withoutcomes related to either learning or behaviour change. Whereas there is evidence thatapplication and transfer of knowledge and individual behaviour change 10 can occur in shortertime periods (e.g., three months), institutional changes usually need longer time to appear,
and, therefore, measure and evaluate.
UNITAR has institutionalized the Kirkpatrick-Phillips approach 11 for evaluating training, aflexible approach that allows to capture both, learning and behaviour change, according to theprogramme objectives. Experiences from applying the framework has evidenced that
information related to outcomes and impacts at the institutional or other higher-level, e.g.,
impacts at broader geographical scale or with longer time span, is difficult to obtain in the
short-run. Depending on the nature of the training, information related to high-level behaviourchange should occur after the capacity building activities completion within an adequate
timeframe.
“Higher level outcomes and impacts relating to capacity development andbehavioural change invariably take a long time to deliver, and to gather evidence
against. When the intended outcome is systemic, national-level (or even globallevel) capacity development, it will almost certainly be unrealistic to expect such

10 Two examples of rapid change in behaviour at the individual level are captured in the Impact Stories: “Multilateral

Diplomacy - Enhancing Women’s Leadership and Mentoring Capacities” and “Hackathon - Strengthening Ruraland Urban Relationships while Achieving the SDGs”.
11 An approach that categorizes evaluation data based on five steps to evaluating training, including Level 1(reaction and planned action), Level 2 (learning and confidence), Level 3 (application and implementation), Level
4 (impact/organizational change) and Level 5 (ROI).

11

changes to be in place – and measurable – within a typical three to four year
project.”
Besides an adequate timeframe, measurement of behaviour change requires a monitoring
system that follows up on different stages of change. Previous independent evaluations havefound that programmes often lack adequate measurement for behaviour change. At theevaluation stage, qualitative indicators can help overcome gaps in quantitative indicators,
especially for retrospective guidelines to measure impact.

2.

Impact might occur through unplanned channels and unintended outcomescan be rewarding.

“UNITAR’s impact is evident through informal and unexpected channels such as the sharingof knowledge by participants through personal networks.”
Past evaluations indicate that outcome achievement from UNITAR training can go beyond the“individual trained” and their organization, and occur through informal and unplanned channels
such as knowledge sharing within a participant’s (informal) network. These knowledge transfer
experiences include knowledge sharing with the community; inclusion of content from the
UNITAR training in lectures, seminars and webinars; and some experiences of ToT format
and personal projects organized and managed by participants outside their professionalsettings. Transfer of knowledge can also occur informally with colleagues at work (e.g., during
lunch breaks) 12, usually when participants do not have the opportunity to share their new
knowledge using formal channels. Tracing alternative channels of knowledge sharing (andresults from it) derived from the training can help better understand and measure UNITAR’s
scope of impact.
Unintended outcomes can also derive from the training dynamics themselves. Capacity
development projects with an interactive component (e.g., teamwork and discussion panels)
can incentivize collateral knowledge sharing and well-established professional networks
among participants. An example from the latter includes the FAO-UNITAR joint e-Learning
series in Eastern Europe and Central Asia. The opportunity of interaction between participants
provided by the course has created a network of professionals from the region in the areas oftrade, international trade agreements and food security, interested in continuous learning andknowledge sharing. Participants have remained in contact after the course completion outside
the UNITAR-FAO official channels, i.e., participants’ personal networks.
“Planning for collateral outcomes in training events: When people are brought
together for a training event, there will always be ‘collateral’ benefit that comesthrough participants making contact and sharing their ideas and experiences withothers. The theory around “boundary objects” can help better understand and planfor these less visible but important knowledge-sharing processes to happen andbe maintained”
Planning for collateral outcomes in projects with learning outcomes is, nevertheless, possible,
allowing the possibility to create mechanisms for follow-up. Building on existing theoreticalknowledge, such as complex adaptive systems approaches (CAS) 13 of capacity development
projects, some methods to measure unintended impacts can be applied during the planning
12 Data from a survey applied for the independent evaluation of UNITAR’s Strategic Framework Fund indicates that87 per cent of survey respondents have shared knowledge from UNITAR training informally with their colleagues.
13 “CAS focuses on capturing changes in the behavior of and relationships between the direct participants of the
CD process, rather than specified outcomes in terms of traditional technical performance indicators. […]. CAS
approaches are also open to recognizing intended and unintended changes resulting from CD” (Vallejo & When,2016). https://www.sciencedirect.com/science/article/pii/S0305750X15002739

12and evaluation process. Some recurrent methods to do so are the Most Significant Change(MSC) approach, outcome mapping and outcome harvesting (Vallejo & When, 2016). Other
theories from the social sciences identified in previous evaluations is the “boundary objects
theory”, as described in Box 4.
Box 4. IOMC toolbox for decision making in chemicals management – Phase III project.
The Toolbox project was created to set up a ‘one-stop-shop’ where national staff responsible for thesound management of chemicals could quickly and easily find the resources they needed. The
Toolbox was designed also as a problem identification and problem-solving tool to enable countries
to identify the most appropriate and efficient actions to address specific national problems related
to chemicals management.
A participants survey indicated that 40 per cent considered that meeting and learning from eachother was the most useful aspect of the workshop. Experiences cited in the evaluation report aslisted: “The interaction with representatives from other countries allowing (us) to know how they are
currently managing their chemicals”, “Connection with other colleagues in the same field ofexperience (toxicology) from other countries”, and “Know what others are doing in my country about
the treatment of chemicals”.

Moreover, unintended impacts can be highly beneficial for both beneficiaries and project
development (or future phases of a project). An example is presented in the “CommonSensing
project” managed by UNOSAT, where the COVID-19 realities created the need to move thecapacity development activities to online or blended modalities, improving the digital literacyof beneficiaries. The knowledge of new online tools did not only create advantages forbeneficiaries (i.e., technical knowledge) but it also broadened possibilities for future training.

3.

An adequate and valid Theory of Change (ToC) is essential for measuring
impact.

The evaluations reviewed confirm the importance of ToC for measuring impacts and resultswith greater precision. Defining a ToC in the early stages of the project is as important asvalidation and adjustment during project implementation.
Benefits of defining a sound and realistic ToC, including risks and assumptions around it, onachieving and measuring impact can be summarized in three points:•••

A well-designed ToC enhances project efficiency as resources are better focalized.
It is easier to measure progress and tackle challenges when there are clear causal
mechanisms towards a specific goal. Review of risks and assumptions of the proposed
ToC is necessary to identify these challenges and understand progress.
A clear ToC can create greater engagement from partners and implementing partners
as it provides clarity of the project and how it contributes to an overall goal.
“A well-articulated Theory of Change can ensure Partner engagement and projectsuccess. UN CC:Learn’s partners are committed to the success of the project andattribute their own success to its support. There are opportunities to strengthen
this engagement through clarifying purpose and methodology through developing
a well-articulated agile Theory of Change.”

Furthermore, the factors identified for a functional ToC related to impact measurement include
the following:•

Coherence and alignment between the ToC and log frame is key for a clear pathway
to impact.
13

•

•

Results and intended impacts should be aligned to the project resource
constrains/availabilities since project inception. Resources to consider include theallocated budget, staff designated to project implementation and their capabilities anddata availability (e.g., secondary data available to track impact or budget allocated fordata collection).
Define potential risks and assumptions when drafting the initial ToC.
“Importance of defining realistic, measurable results. While project log frames are
dynamic instruments and may be subject to review and modification, it is importantthat project metrics have means of verification and can be measured within the
project’s resource constraints”

4.

Theory-based approaches to evaluation are useful for measuring impact indemand-driven projects

Although the definition and initial measurement of progress indicators should take place since
the project planning stages, the nature of a project might limit this from happening. Tracking
progress and measuring impact in demand-driven projects is difficult, considering that project
outputs are not necessarily planned since its inception. In this event, theory-based approaches
to evaluation 14 represent a suitable approach for identifying results and impacts. Box 5 below
presents an example from an independent evaluation.
“Effective management of documentation, information and data is critical in highly activitydriven undertakings.”
Box 5. Reconstructing and testing a ToC for measuring impact: Evaluation of UNOSAT Rapid
mapping service
The UNOSAT Rapid Mapping Service provides satellite imagery analysis during humanitarian
emergencies. The Service has been created to meet the demand of United Nations agencies,
Member States and other humanitarian agencies for rapid mapping and satellite derived analysis in
the wake of disasters and complex emergencies.
In the absence of such results-based planning, results were previously measured by counting thenumbers of activations and using a user feedback survey with varying response rates.
An evaluation of the Services identified impact on end-beneficiaries by reconstructing a ToC using
theoretical contribution where possible.

Even though demand-driven projects do not represent a large part of the UNITAR project
portfolio, previous experiences can facilitate future programming.

https://www.canada.ca/en/treasury-board-secretariat/services/audit-evaluation/evaluation-governmentcanada/theory-based-approaches-evaluation-concepts-practices.html#toc5
14

14

Gender inclusivity

The seven lessons under this category can be categorized under three sub-categories: i)strategy and assessment, ii) enriching discussions; and iii) participant attitude. Likewise, threemain lessons are identified based on the above.

1.

Achieving women empowerment requires a strategic approach with a needs
assessment that includes a thorough gender assessment.

The lessons under this sub-category conclude that it is important to have a strategic approachin order to empower women through training, that goes much beyond a simple participant
selection.
“Achieving women empowerment in Fellowship cohorts requires a more strategic approachbeyond requesting national partners to select more female candidates.”

Empowerment of women does also not just happen in the training room and one needs to becreative to also target women outside of office and training room spaces. One way to start a
strategic gender approach is by integrating gender assessments as part of the needs
assessments. The Gender-responsive Asset-based Approach to Needs Assessment(GAANA) is designed to help to conduct needs assessment by focusing on assets/capacities
of different gender groups by examining their capacities in terms of access to resources aswell as their roles and power relations. GAANA helps to identify the existing accessmechanisms, underlying power relations, organizational, agency, community, fiscal, andindividual people skills. This approach is useful as it goes beyond analyzing existing
assets/capabilities available in a community or among a group of people and examines accessmechanisms as well as existing structures and power relations that may make it hard for othermembers of the community or groups to utilize the existing assets (CFAES, 2020).
“A thorough gender assessment is important to undertake as part of the needs assessment
and analysis to ensure that gender mainstreaming in project design is relevant and adequate
and can be realistically delivered throughout project implementation.”
As part of good practices in gender-responsive monitoring and self-evaluation that were
presented by UNOSAT, 15 it was highlighted that conducting a gender assessment during the
15 Good practice document on “Gender-responsive Monitoring and Self-Evaluation“, publication forthcoming.

15

baseline helps having baseline data on specific gender-related indicators and can inform the
project implementation. Moreover, combining self-assessments and objective assessments oflearning and disaggregating results by sex helps understanding whether different scores are
related to confidence-related matters or mirroring acquisition of knowledge.
Box 6: “Women's Leadership in Tsunami-based Disaster Risk Reduction Training Programme
for World Tsunami Awareness Day 2016” Self-Evaluation
Patriarchy and cultural norms make women’s participation in DRR planning difficult:
While it was indicated by participants that the number of men in DRR related positions outweighs thatof females, there was also an acknowledgement that women tend to “overcommit” to families andchurches. A key line of discussion that developed during the programme was for this commitment to infact be channelled toward increasing DRR related education and best-practice like sharing of DRR
related preparedness information at church events; at PTA events; and through family connections.

2.

Including different perspectives in group discussions can be enriching for allparticipants.

Opinions from different genders and sexes may differ. It is hence recommended to strive forgender-balance in all training activities, when possible. However, in order to create an enabling
environment for women, one needs to go beyond gender parity amongst participants. In somecases, additional efforts may be needed to actively include women into discussions against
the principle of “one-size fits all”. Encouragement and ways to strengthen women’s confidence
could be two ways to further motivate women to actively participate to discussions.
“The majority of female Fellows were vocal in sharing their experiences, which contributed toconstructive discussion and informative debate within the group.”“Gender balance among Participants enriched the quality of workshops through women’s
active engagement and input in practical exercises, group discussions, and teamwork.”
UNITAR’s strategic framework 2018-2021 includes the objective to achieve gender-parity
amongst its participants by 2021. In 2020, for the first time, UNITAR positively reversed its
female-male gender ratio. UNITAR’s Executive Director for example is part of the GenderChampions network and one of the key pledges is the Panel Parity Pledge, through whichChampions commit to no longer take part in any single-sex panels. This hence also applies totrainers and organizing personnel.

3.

Depending on the cultural and national context, change management strategies
might also be required to target male participants’ attitudes towards female
participants or women in more general terms.

This sub-category’s lesson applies to the male-dominated peacekeeping training context but
also in entrepreneurship programmes such as those delivered in Afghanistan and Iraq,
amongst other locations. Particularly for projects targeting women empowerment, but also for
projects that do not have women empowerment as a primary objective, change management
strategies may be required in addition to a gender analysis, gender strategy and a genderresponsive needs assessment.
An ITC-ILO handbook on “Gender and Organisational change” provides ideas for assessing
the readiness for organizational change for gender equality. 16
Arcand (2007) explains that for change to happen four conditions must be met:
16

ITC-ILO handbook on Gender and Organisational change
16

•
•
•
•

Creating awareness among staff and management;
Increasing the willingness of actors;
Improving individual and collective skills;
Allowing mobilisation of actors.

Each of Arcand’s four conditions needs to be met for change to take place. Change agents,
whether they be at the top, mid-level or in operations, and who are firm promoters of genderequality, may have an overly positive view of the change process and should therefore beaware of myths relating to change in general and the challenges facing gender equality change
agents.
Box 7: “UNITAR Hiroshima Transforming the Future: Empowering Social Entrepreneurs andYouth Leaders for Iraq 2020” Self-Evaluation
Gender balance among participants enriched the quality of workshops by women and men’s active
engagement and input in practical exercises, group discussions, and teamwork. It is observed by Staff
that male participants respected their female counterparts’ strength and accepted women’s current andpotential contributions to the areas of entrepreneurship, leadership, and development in Iraq. To
increase the number of skilled female leaders who can have a positive impact on Iraqi society.

COVID-19

The eleven lessons under this category (7 classified as COVID-19 and 4 classified in othercategories but related to COVID-19) lead to three subcategories: i) efficiency; ii) achievementof learning objectives; and iii) format delivery and methodologies, and likewise three lessons
learned were derived from the analysis.

1.

Participants’ engagement is crucial during the transformation of training
delivery formats

In the wake of the COVID-19 pandemic, many UNITAR training events needed to shift from aface-to-face to an online delivery format. The success of transformation of face-to-face toonline modalities did not only depend on UNITAR staff and trainers adaptability but also on
participants engagement in seeking alternatives to complete the training and their commitment
to follow these solutions. Active communication and consultation with participants about new17

alternatives for online learning resulted in greater participant’s responsibility and ownership
under the new modality and smoothed the adaptability process.
“[…]Robust engagement and communication with participant[s]allowed for the smooth
implementation and identifying alternative solutions. Participants showed high level ofcommitment through completing assignments and video recording their Pitches which were
submitted on time as the final assignment.”

2.

COVID-19 as an opportunity to strengthen online learning delivery methods.

While the COVID-19 pandemic brought many challenges during programme implementation,it also enabled UNITAR to leverage the adoption and use of online training methodologies.
The evaluations revised for this analysis suggest that online training formats were, in somecases, more time and money efficient than traditional formats (i.e., face-to-face). The identified
advantages include:••••

COVID-19 helped position online learning as an advantageous tool to reach broader
audiences while being cost-effective from a project perspective.
The reduction of travel required for face-to-face training and print materials reduced
the project’s carbon footprint.
COVID-19 provides the opportunity for programmes to update (or create) internal
guidelines and procedures for effective online training delivery.
Online training self-paced features allowed participants to combine their professional
activities with learning.

“Online e-learning resources can have a wide reach and broad and diverse engagement,
enabling project goals to be achieved at low cost. E-learning approaches can continue
or be scaled up during times of global health crises, where face to face activities are
suspended.”“The self-paced and on-demand feature of the course (asynchronous learning) ensured
that the participants could complete the course requirements without disruption of theirprofessional activities which is extremely important in the current COVID-19 crisis.”3.

Addressing the digital divide in the wake of the COVID-19 pandemic requires
adaptive management and availability of online tools

One of the main concerns during the transformation of training delivery format due to COVID19 was ensuring the principle of leaving no one behind and how to address the digital divide.
Two influential factors were identified during this review: adaptive management and existenceof previous tools able to reach broader audiences.
••

Adaptive management is crucial for testing and adopting alternative training tools thatfavour the achievement of learning objectives and inclusion of participants access andknowledge of digital tools.
Previous experiences of training delivery using less traditional formats of training
delivery (e.g., mobile-based or radio broadcasting formats) can speed up adoption of
resources that help closing the digital divide in online training.

“Unintended outcomes can be highly relevant, appreciated and rewarding. Adaptive
management is crucial to address important niche areas for capacity support, such as
the case for responding to demand-driven and tailored backstopping support and18

addressing digital divide by supporting online and blended learning solutions in the wakeof the COVID-19 pandemic.”“The app-based mobile-ready course delivery enhanced the learning experiencethrough greater engagement, easy access of the courses, while ensuring that eachparticipant has opportunity to interact Experts to support their learning journey.”Take-away messages
The rapid evidence assessment has led to the following take-away messages:•
Lessons are generalized but, in some instances, they are limited
in applicability to all UNITAR programming and sometimes only
applicable to more specific context. We recognize a weakness oflessons learned where e.g., lessons may not apply between learningand non-learning events or be limited to a specific national context.
• Lessons learned are very useful to record learning fromexperience. While personnel at UNITAR is rotating, lessons frompast project implementation can help informing future projects, bethey in a similar or different context. They hence present an excellent
opportunity for organizational learning.
• Most of the lessons examined as part of the assessment
highlight strengths and only a few lessons highlight weaknesses,
though lessons learned can in general arise from both. In thefindings, we tried to include both positive and not so positive lessons,but a full balance was not possible due to an imbalance in the
database. This might be a common finding from self-evaluations
given that it is more difficult to write about challenges and failures
than about positive experiences. Independent evaluations include
more lessons focusing on challenges. Nevertheless, we recognize
the importance of encouraging colleagues to also develop lessons
based on challenges as part of self-evaluations.
• Lessons can help inform drivers of success or challenges for future
programming. This paper has highlighted four categories with fifteen
sub-categories that can be used as drivers for successful future
programming.

The way forward

As next steps, the following actions are foreseen:• A lessons learned forum will be organised to discuss findings
from this assessment and invite colleagues from Programme Units
to share and exchange.
• This paper series will be continued and future editions will focus
on different categories of lessons learned.
• The Lessons Learned database and this assessment shallinform new programming development. To track use of it, a survey to programmemanagement will be issued to measure progress on the following two indicators:

19

o
o

Managers from projects confirming that lessons learned informed the design offuture phases of those projects;
Percentage of projects in which lessons learned inform the design of futurephases of those projects.

20References

Arcand, M. (2007). La gestion du changement, Vingt et une questions pour mieuxappréhender et gérer les transformations organisationnelles. Editions Nouvelles.
Barends, E., Rousseau, D.M., & Briner, R.B. (Eds). (2017). CEBMa Guideline for rapid
evidence assessments in management and organizations, Version 1.0. Center for
Evidence Based Management, Amsterdam. www.cebma.org/guidelines
CFAES. (2020). Points for consideration. Gender-responsive asset-based approach toneeds assessments (GAANA). Ohio State University. https://lod.cfaes.ohiostate.edu/sites/lod/files/imce/Documents/EvalMENA2020/GANNA_Points%20for%20
Consideration_no%20worksheets.pdf
Government of Canada. Theory-based approaches to evaluation: concepts and practices.
https://www.canada.ca/en/treasury-board-secretariat/services/auditevaluation/evaluation-government-canada/theory-based-approaches-evaluationconcepts-practices.html#toc5
ITC-ILO. (2017). Handbook on gender and organizational change. ITC-ILO.
https://www.itcilo.org/sites/default/files/inlinefiles/Handbook%20on%20Gender%20rev%202019.pdf
OECD/DAC. (2010). Glossary of key terms in evaluation and results based management.
OECD/DAC. https://www.oecd.org/dac/evaluation/2754804.pdf
UNITAR. Good practice document for training needs assessments.
https://unitaremail.sharepoint.com/unitarnet/Documents/Good%20Practice%20Docu
ment%20-%20Training%20Needs%20Assessment.pdf
UNITAR. Good practice document learning reinforcement plan.
https://unitaremail.sharepoint.com/unitarnet/Documents/Good%20Practice%20Docu
ment%20-%20Learning%20Reinforcement%20Plan.pdf
UNITAR. Repository of Lessons Learned.
https://unitaremail.sharepoint.com/unitarnet/Lists/Lessons%20Learned/AllItems.aspx?groupBy=Challenge%5Fx0020%5FCategory
UNITAR. (2016). UNITAR gender mainstreaming, gender equality and the empowerment of
women policy. https://unitar.org/sites/default/files/media/publication/doc/unitargender-mainstreaming2016.pdf
UNITAR. (2017). Quality assurance framework. UNITAR.
https://unitaremail.sharepoint.com/unitarnet/Documents/AC.UNITAR.2017.11%20%20Quality%20Assurance%20Framework,%20Revision,%202017.pdf
UNITAR. (2018). Strategic Framework 2018-2021.
https://unitar.org/sites/default/files/media/publication/doc/unitar_strategicframework_
web-new.pdf
UNITAR. (2019). UNITAR facilitation toolbox.
https://www.flipsnack.com/unitarcatalogue/facilitation-toolbox-en-fr.html
UNITAR. (2020). Cluster evaluation of UNITAR’s training of trainers programming. UNITAR.
https://unitar.org/results-evidence-learning/evaluation/cluster-evaluation-unitarstraining-trainers-programming21

UNITAR. (2021a). Hackathon - Strengthening Rural and Urban Relationships While
Achieving The SDGs. https://unitar.org/about/news-stories/stories/hackathonstrengthening-rural-and-urban-relationships-while-achieving-sdgs
UNITAR. (2021b). Multilateral Diplomacy - Enhancing Women’s Leadership and Mentoring
Capacities. https://unitar.org/about/news-stories/stories/multilateral-diplomacyenhancing-womens-leadership-and-mentoring-capacities
Vallejo, B., & Wehn, U. (2016). Capacity development evaluation: The challenges of theresult agenda and measuring return on investment in the Global South. World
Development, 79, 1-13. https://doi.org/10.1016/j.worlddev.2015.10.044

22Annexes

Annex 1. Lessons compilation

Lessons
Coaching and mentoring were key elements of the programme,
augmenting the learning and providing Fellows support as they
completed their assignments.
A self-paced and on-demand feature of the course ensured that theparticipants could complete the course requirements without
disruption of their professional activities.
There is a continuous need to balance conceptual elements withtechnical, issue-specific actions to take.
Coaches greatly enhanced particpants' confidence in using newly
acquired policy appraisal tools.
Strengthening local capacities in the long run relies on a more
frequent exchange that goes beyond a limited amount of individual
trainings sessions and shared curricula.
International mentors and resource persons contributed to global,
regional, and national-level discourse. Their inclusion added multiple
perspectives to Fellows' learning and brought various insights andexpertise.
Proposal writing and presentation skills are highly relevant to Fellows'needs.
Experts familiar with the local situation contributed to contextualized
learning and were able to relate programme content to the current
local situation.
Learning approaches structured around case-studies, mockCommittee meetings and feedback from a panel offered "real life"
learning opportunities.
Daily monitoring of progress of each of the participants ensure thatthey are aware of the support and guidance available to themthroughout the course. This contributed to the high completion rate.
A train-the-trainer approach can achieve good multiplier effects foroutreach.
Advance notification enables potential participants to prepare for thecourse and plan accordingly.
More varied presentation techniques encourages learning. This
includes increased use of visual aids, such as PowerPoint
presentations and videos (Reformulated by PPME).
It is helpful to have in advance the training participants list. This
enables a better awareness of the broad professional backgroundand GIS knowledge level of participants to ensure that the training
content takes into consideration the various learning needs.
Most of the participants encountered some difficulty capturing thebasic concepts of remote sensing in two days. Most of themrecommended a bit more time to grasp remote sensing theories andapplication.
Most of the participants are professionals working in governments
and they are less comfortable with exams and more comfortable withgroup assignments or teamwork. The exams made them nervous
and to a slight extent, shifted their focus from the course work to the
preparation of the exam.
The courses are considered too theory intensive. Some participants
thought that given the price paid for the course they were not getting
enough in return in terms of content.
More involvement of the trainer on discussion boards would
encourage the contribution of the participants. Posting two or threetimes a week, reacting to the participants’ contributions, would begood ways for the trainer to keep the discussions active.
23

Category
Training/Learning
methodologies

Sub-category
Training follow-up

Training/Learning
methodologies

Methodology and target audience

Training/Learning
methodologies
Training/Learning
methodologies
Training/Learning
methodologies

Content and structure

Training/Learning
methodologies

Training expertise

Training/Learning
methodologies
Training/Learning
methodologies

Content and structure

Training/Learning
methodologies

Methodology and target audience

Training/Learning
methodologies

Methodology and target audience

Training/Learning
methodologies
Training/Learning
methodologies
Training/Learning
methodologies

Training follow-up

Training/Learning
methodologies

Learning needs

Training/Learning
methodologies

Content and structure

Training/Learning
methodologies

Methodology and target audience

Training/Learning
methodologies

Content and structure

Training/Learning
methodologies

Training expertise

Training follow-up
Training follow-up

Training expertise

Learning needs
Methodology and target audienceSome participants were approaching the training with very different
expectations and from very different backgrounds. At first, it waschallenging to create a collaborative work environment amongst
participants.
One example of best practice was to ask the participants to develop
an anti-corruption project/plan of their choice using a template
prepared by UNITAR. This was done as a follow-up assignment to
the first workshop and was then presented at the second workshop.
From the results it was clear that the exercise was done with interest
and was used by most as an opportunity to develop existing ideas
while using newly acquired skills, and fostering new partnerships.
The needs assessment confirmed that the majority of participants
were interested in acquiring training skills and sharpening theircommunication and presentation skills. However, it then became
clear that some of the participants were more likely to engage in otherforms of outreach than training in their functions.
Building on the change management presentations, discussion
focused on the need for more structured processes for stakeholder
identification and any related role allocation. The added legitimacy
and perceived results from such a focus were outlined as were
methodologies participants could use to realize this, particularly withregard to transition modelling.
The methodology for the programme, blended so as to appeal todiffering adult learning styles, included participant presentations,expert presentations, study tours, and small group-, paired- andindividual exercises. In particular, the DiSC Personality Profile test, abehaviour assessment tool which was then linked to the role ofparticipants as leaders in their communities, was commented on by
participants as being both unique, and inspirational.
Theorywithout
practiceislesseffective.
The evaluation concluded that theoretical content – although criticalto establish the foundational know-how – needs to be complemented
by practice. All informants insisted on this aspect. Practice helps build
their muscle memory, their professional ethics and behaviour andintegrate all the notions through repetition and real-life scenarios.
Duplication is not a problem so long as the delivery methodology iseffective.
Informants mentioned that some of the training materials wererepetitive and redundant considering other trainings they received intheir country of origin. However, according to interviews and group
discussions, repetition is part of capacity building and learning as
long as the methodology is effective. In this project, the methodology
was said to be adapted to the context and effective.
Digitalpresenceisefficient
UN CC:Learn’s and UNITAR’s strong digital presence andexperience of developing digital content has been core to theproject’s success and should be built on in future phases and in otherprojects.
Online e-learning resources can have a wide reach and broad anddiverse engagement, enabling project goals to be achieved at low
cost. E-learning approaches can continue or be scaled up during
times of global health crises, where face to face activities are
suspended.
E-learning is an efficient mean to reach larger groups.
Institution-based. ToT initiatives that are institution-based are likely
to produce more enduring results, as trained trainers of learning
centres (whether they are staff or consultants) are more likely to havetraining opportunities in the medium to long term. While aninstitutional basis for ToT is a necessary condition, it is not sufficient,however.
24

Training/Learning
methodologies

Learning needs

Training/Learning
methodologies

Training follow-up

Training/Learning
methodologies

Learning needs

Training/Learning
methodologies

Content and structure

Training/Learning
methodologies

Methodology and target audience

Training/Learning
methodologies

Content and structure

Training/Learning
methodologies

Methodology and target audience

Training/Learning
methodologies

Methodology and target audience;
COVID-19

Training/Learning
methodologies

Learning needs

Integrating flexible mechanisms such as backstopping activities thatcan react to local demands, particularly in rapidly changing
environments such as the Pacific Islands, is supportive of successfulproject implementation.
Master trainers make a difference as key drivers of knowledge andchange in peace and reconciliation if given the adequate tools andincentives
The potential role of youth as agents in peace and reconciliation isenhanced if there is a combination of dynamic capacity building
approaches, experiential learning in context and linkages to networks
that catalyze leadership.
Working with youth in rural and urban settings requires adaptedmaterials and methods that help put knowledge in context andconsiders the differences across audiences in terms of gender, age,
culture and social norms
Virtual and distance learning mechanisms are effective if seen ascomplementary to the in-person methods, and if enough guidance isprovided to the trainees and organizations to apply best practices infacilitation to make this as closely experiential, relational andsensitive as the face-to-face methods
Online training methodologies have been fully embraced as aneffective
training
methodology given the current COVID-19 pandemic.
While online training delivery is a viable approach, the participants
dohave
preferences.
The FAO/UNITAR approach of delivering mentor-facilitated training
with interaction with fellow learners and receiving a Certificate ofParticipation is an appealing format for the participants
Online communication and interaction with the participants on aregular basis (expert-to-participant and peer-to-peer) provided
invaluable access to practical information on the course theme and acommunity of practice. This also greatly expanded the visibility of
FAO and UNITAR in the region.
The flexibility (i.e. no scheduling constraints) of the course ensured
that the participants could participate at any time or review the course
materials whenever needed.
Using innovative tools such as eLearning have been demonstrated
as an effective and efficient approach to share knowledge andstrengthen networking of like-minded officials from different countries
and government ministries.
Face-to-face training delivery is still the preferred training
methodology, but blended training (face-to-face and online training)has been increasingly accepted based on the survey results.
The availability of the course across different countries and theflexibility accorded to the participants ensured that the online training
can be offered to a greater number of participants which they canfollow at their own pace. 13 of the 17 countries that participated in
the course were Least Developed countries in Sub-Saharan Africa
where electricity and internet connection is unreliable.
The self-paced and on-demand feature of the courses(asynchronous learning) ensured that the participants could follow
the programme without disruption of their professional activities. The
learning tools and daily monitoring of progress allowed the learners
to achieve the weekly milestones in their learning journey and toaccess support as and when needed.
In spite of the global lockdown and restricted movement brought by
the COVID-19 pandemic, the training methodologies and use of
information and communication technologies allowed thebeneficiaries uninterrupted access to quality training and to connect
and learn from peers and experts.
25

Training/Learning
methodologies

Methodology and target audience

Training/Learning
methodologies

Training expertise

Training/Learning
methodologies

Methodology and target audience

Training/Learning
methodologies

Methodology and target audience

Training/Learning
methodologies

Methodology and target audience

Training/Learning
methodologies

Methodology and target audience;
COVID-19

Training/Learning
methodologies

Training expertise

Training/Learning
methodologies

Methodology and target audience;
training follow-up

Training/Learning
methodologies

Learning needs

Training/Learning
methodologies

Methodology and target audience

Training/Learning
methodologies

Methodology and target audience

Training/Learning
methodologies

Learning needs

Training/Learning
methodologies

Methodology and target audience;
training follow-up

Training/Learning
methodologies

Methodology and target audience;
COVID-19

VISITING HIROSHIMA INSPIRED PARTICIPANTS AND GAVE
THEM
HOPE
FOR
IRAQ’SRECOVERYA major component of the programme’s third international workshop,
held in Hiroshima, is exposing Participants to the story of Hiroshima’s
atomic bombing and subsequent reconstruction and recovery. Many
Participants remarked on the emotional impact of spending timelearning about the horrors of Hiroshima’s destruction in the Peace
Memorial Museum, then stepping back out into the verdant, beautiful
Peace Park and seeing the transformation Hiroshima accomplished.
Alongside this simple visual impact, UNITAR Staff and Resource
Persons also gave a lecture on Japan’s post-war reconstruction, as
well as a guided tour of the Park. Although the situations of post-war
Japan and current Iraq have significant differences, seeing
Hiroshima’s transformation with their own eyes inspires Participants
to continue working to give their country a bright future.
THE SHARING OF NEEDS ASSESSMENT RESULTS POSITIVELYIMPACTEDTHE
PROGRAMME

Training/Learning
methodologies

Methodology and target audience

Training/Learning
methodologies

Learning needs

Training/Learning
methodologies

Content and structure

The programme always undertakes a Needs Assessment withparticipants following their acceptance to a programme. The
assessment results help determine the instructional design of the
programme; they are also shared with Resource Persons for further
tailoring the content to the participants learning needs.
All the participants undertook an initial needs assessment
questionnaire regarding the content of the programme during theorientation
workshopin
Baghdad.
An anonymized summary of the needs assessment results wasshared with participants during workshop and was addressed during
presentations and practical exercises sessions. participants
indicated that exposure to these results allowed them to identify
issues
they
all
faced
in
their
work.
The needs assessment results indicated that participants wished formore focused training around key elements of entrepreneurship,social entrepreneurship, communication techniques and co-founding
team. The assessment data showed that more than half (62%) of theparticipants have “no knowledge” to “some knowledge” on key
elements of entrepreneurship and social entrepreneurship, whichwere addressed during workshop-II. Furthermore, 92% participants
mentioned that effective communiques are relevant is to their current
role, while 85% participants mentioned that knowledge and skill
around developing budgets will benefit them. To address the gap inparticipants’ knowledge of entrepreneurship, introductory sessions
were incorporated in the programme to outline the concepts ofentrepreneurship and social entrepreneurship, as well sessions
examining communication techniques and teambuilding andteamwork.
BUSINESS PROPOSAL WRITING AND PRESENTATION SKILLSARE HIGHLY RELEVANT TO PARTICIPANTS’ NEEDS
Both UNITAR Staff and the Iraqi participants understood the skill
level and preparation necessary for writing and pitching effective
business proposals to donors and investors. Elements of effective,
comprehensive proposals, such as the results of an objective Needs
Assessments, a sound budget, and detailed risk mitigation andmonitoring and evaluation sections, were emphasized throughout the
programme. Participants also honed their presentation skills throughmultiple opportunities to present their assessment results and project26

proposals, along with presentations related to group exercises
undertaken during workshops. Constructive comments onpresentation style were always part of the feedback from UNITAR
Staff and Resource Persons.

COACHING AND MENTORING WAS A KEY ELEMENT OF
FACILITATINGPARTICIPANTS
LEARNING

Training/Learning
methodologies

Training follow-up

Coaching and mentorship is a key element of this programme,
augmenting the learning and providing Participants support as they
complete their assignments. Volunteering their time, insight, andexperience, Coaches and Mentors are an important element of the
programme and an invaluable asset to the timely delivery and qualityof the programme. To promote South-South cooperation,international Mentors from the UNITAR Hiroshima Afghanistan
Programme were introduced to the new Participants in close
coordination with the UNITAR Juba-based Coaches, providing
necessary mentorship to the Participants on a regular basis.
Participants placed into five separate coaching and mentorshipgroups allowed Mentors to provide focused, one-to-one and criticalinsights on assignments and hold need-based online meetings. Mostof the correspondence between Participants, Mentors, and Coaches
took place through email exchanges. In evaluations, sharing
knowledge, experiences, and ideas attributed to an enriched learning
experience and mentorship. UNITAR will explore the possibility oforganizing periodic face-to-face sessions between local Mentors and
new Participants in the UNITAR Juba Office to establish a good
rapport between Mentors and Participants in future iterations of the
programme.
ONLINE TRAINING CREATES NEW OPPORTUNITIES IN TIMESOFUNCERTAINTYTO
CONNECTAND
LEARN

Training/Learning
methodologies

Methodology and target audience;
COVID-19

The use of online training tools and methodologies such as, online
webinar was out of necessity to deliver our commitment to theprogramme and our participants. However it turned very usefulduring large-scale shutdown and quarantine. It allows for learningfrom the comforts of one’s home without any travel. The programme
will make efforts to utilise such methodologies and new technologies
to connect participants in the future and to maximize the number ofbeneficiaries without impacting the quality of the training delivered byadhering to well established guidelines for effective online training
delivery.

27

SHARING

OF

NEEDS

ASSESSMENTRESULTS

The programme undertakes a learning needs assessment with
Participants following their final selection. These assessments
influence the instructional design processes of the Division and areshared with Resource Persons and UNITAR Staff for constructive
feedback on designing needs-based learning modules. Beginningwith this Programme, an anonymized summary of the needs
assessment results was shared with Participants through theorientation workshop and addressed during presentations andpractical exercises sessions. Participants indicated that exposure tothese results allowed them to identify common issues amongst
themselves. The results also functioned as discussion points when
examining
lessons
learnedandchangemanagement.

Training/Learning
methodologies

Learning needs

Training/Learning
methodologies

Learning needs

Genderinclusivity

Strategy and assessment

The results of the learning needs assessment indicated the need formore focused training on the project management cycle. The data
showed that while 68% of Participants have a role related to project
management, including project proposal writing, developing actionplans, and monitoring and evaluation, 77% Participants reported thatthey had no formal training in the areas of project management,
which is negatively affecting their work performance and the overall
servicesdeliveredto thepeopleofSouth
Sudan.
To fill the gap, introductory sessions were incorporated to outline theconcepts of the project management cycle and sessions examining
Organizational Needs Assessment, in order to provide all
Participants with a baseline knowledge. Additional sessions weredelivered on proposal writing and action plan development. With
regard to the communication and leadership aspects ofthe programme, the assessment showed that while the Participants
were, in general, quite comfortable within their field and/or
communities, there was a need for further training around effective
communication, risk mitigation, conflict identification, and conflict
resolution.
The multi-stakeholder nature of the workshops helped to discuss
some of the important intergovernmental coordination topics that canhelp improve data availability and dissemination at low cost
Achieving women empowerment in Fellowship cohorts requires amore strategic approach beyond requesting national partners to
select more female candidates.
Gender balance among participants enriched the quality ofworkshops and female fellows were often vocal in sharing theirexperiences.
The majority of female Fellows were vocal in sharing theirexperiences, which contributed to constructive discussion andinformative debate within the group.
Patriarchy and cultural norms make women’s participation in DRR
planning difficult: While it was indicated by participants that the
number of men in DRR related positions outweighs that of females,
there was also an acknowledgement that women tend to“overcommit” to families and churches. A key line of discussion thatdeveloped during the programme was for this commitment to in-fact
be channelled toward increasing DRR related education and bestpractice like sharing of DRR related preparedness information atchurch events; at PTA events; and through family connections.

28

Genderinclusivity
Genderinclusivity

Enriching discussions

Genderinclusivity

Enriching discussions

Genderinclusivity

Strategy and assessment

Importance of gender analysis to ensure gender mainstreaming. A
thorough gender assessment is important to undertake as part of theneeds assessment and analysis to ensure that gendermainstreaming in project design is relevant and adequate and can berealistically delivered throughout project implementation.
GENDER EQUALITY HAD A POSITIVE INFLUENCE ON THE
GROUP
DYNAMIC

Genderinclusivity

Strategy and assessment

Genderinclusivity

Enriching discussions; Participant's
attitude

Genderinclusivity

Enriching discussions; Participant's
attitude

COVID-19

Effficiency

COVID-19

Achievement of learning objectives

One of the key characteristics of the programme was the gender ratio
among participants, which is closely related to the Sustainable
Development Goal 5. UNITAR took concrete steps to improve thegender balance in the 2019 cycle of the programme compared to the2018 cycle. Nine (9) out of sixteen (16) of the 2019 programmeparticipants were women, coming from various public sector, private
sector, academic institutions, and civil society organizations. The
majority of female participants were highly educated, competent, andvocal in sharing their experiences and opinions, which contributed toconstructive discussion and informative debate within the group.
Occasionally participants were more reserved due to a lack ofconfidence in their English ability, but this issue affected participantsof both genders. In the future, UNITAR will make sure to nurture
participants’ confidence along with their knowledge and skills.
Gender balance among participants enriched the quality ofworkshops by women and men’s active engagement and input inpractical exercises, group discussions, and teamwork. It is observedby Staff that male participants respected their female counterparts’
strength and accepted women’s current and potential contributions
to the areas of entrepreneurship, leadership, and development in
Iraq. To increase the number of skilled female leaders who can havea positive impact on Iraqi society.
GENDER EQUALITY HAD A POSITIVE INFLUENCE ON THE
GROUP
DYNAMICOne of the notable characteristics of the programme was the genderratio among Participants. Almost half of the 2019 ProgrammeParticipants were women, who came from various public sector,
private sector, academic institutions, and civil society organizations.
Gender balance among Participants enriched the quality ofworkshops through women’s active engagement and input inpractical exercises, group discussions, and teamwork. Genderbalance also closely related to the Sustainable Development Goal 5,
Gender Equality. It is observed by the Coaches and Staff that male
Participants respected their female counterparts’ strength andaccepted women’s current and potential contributions to the areas ofentrepreneurship, leadership, and development in South Sudan.
Female Participants were often vocal in sharing their experiences
and opinions, which contributed to constructive discussion andinformative debate within the group.
The COVID-19 pandemic did not negatively impact any stage of the
project implementation. It has rather strengthened the utility of online
training as a means to deliver capacity building activities to a wider
audience in a flexible, measurable, and cost-effective manner
The self-paced and on-demand feature of the course (asynchronous
learning) ensured that the participants could complete the course
requirements without disruption of their professional activities whichis extremely important in the current COVID-19 crisis.

29

THE OUTBREAK OF INFECTIOUS DISESASES CAN POTENTILLYDELAY THE IMPLEMENTATION AND HARM THE HEALTH OF
PARTICIPANTS
The outbreak of infectious diseases such as Malaria and Typhoid,which are very common disease in many parts of South Sudan andare affecting the health of so many people including the programmeParticipants. This is not limited to the in-country workshops andevents but also international workshops, which requires travel andstay in another country. Some countries have very stringent
infectious disease control procedures, that If a Participant is infected
and remain asymthematic for a certain of period time, could potential
be barred during entry or exit to and from a country, should theParticipant develops fever or exhibits other symptoms. Similarly, theoutbreak of Ebola caused serious concerns and fear among the
programme Participants. In addition, the COVID-19 outbreak which
has started at the end of 2019 and continued through March 2020,
has caused serious health and socio-economic impacts around theworld. While there is no known cases of COVID-19 in South Sudanso far, however the subsequent quranteens put in place andrestriction on international travel and affected our face-to-faceactivities in South Sudan. Considering such a situation, the future
programme will develop and test alternative implementation
methodologies including webinars, and e-learning courses.
Conducting a course via e-Learning resulted in a significant reductionof the cost per participant and the carbon footprint of a widelydispersed audience compared with traditional face-to-face training.
Besides, face-to-face training delivery is not a practical option in thecurrent circumstances.
Access to knowledge and skills development delivered online
resulted is a significant reduction of the cost per participant and thecarbon footprint of a widely dispersed learners compared withtraditional face-to-face training. As a result, the programme was able
to accommodate greater number of beneficiaries. The app-based
mobile-ready course delivery enhanced the learning experiencethrough greater engagement, easy access of the courses, while
ensuring that each participant has opportunity to interact Experts to
support their learning journey.
THE FLUID AND CHANGING HEALTH, POLITICAL AND
SECURITY SITUATION IN IRAQ AND THE WORLD
During the implementation phase of the programme, a number ofchallenges arose such as, total shutdown and curfew in Iraq whichcoincided with the timeline of Workshop-II (Oct 2019), and the global
COVID-19 virus pandemic outbreak (Mar 2020) during workshop-IV.
These challenges while enormous and beyond the control of the
programme, however the programme managed and overcome thenegative impacts of such events by resourceful solutions,collaboration and adaptability of the Participants. For instance, theworkshop-IV as the final event of the 2019 programme wasconducted online to eliminate the health risks if participants were togather. Robust engagement and communication with participant
allowed for the smooth implementation and identifying alternative
solutions. Participants showed high level of commitment throughcompleting assignments and video recording their Pitches whichwere submitted on time as the final assignment.
Although the programme supports individual, and by extension,
institutional transformation, no tracking of its output to outcome to
impact has been undertaken. Tracking is essential for learningrelated programming.
30

COVID-19

Format delivery and methodologies

COVID-19

Efficiency

COVID-19

Efficiency; Format delivery andmethodologies

COVID-19

Achievement of learning objectives

Tracking
outcomes
impactandDefinition and follow-up of causal
linksEffective management of documentation, information and data is
critical to track progress in activity-driven projects. The lack ofsystematic documentation for all activations and the unclear
identification of the actual users of the Service's deliverables maketracking the Service's effects challenging.
Higher-level outcomes and impacts related to capacity development
and behavioral change take a long time to deliver and to gather
evidence for.
Measuring attitude and behaviour change is important in capacity
building
projects.
UNITAR showed innovation by wanting to measure the behaviouralchange in the trained FPUs and PCCs. It should be reinforced andfind the means to improve the data collection process so it can thenbe replicated to other projects.
A well-articulated Theory of Change can ensure Partner engagement
andproject
successUN CC:Learn’s partners are committed to the success of the project
and attribute their own success to its support. There are opportunities
to strengthen this engagement through clarifying purpose andmethodology through developing a well-articulated agile Theory of
Change.
Projects with ambitious global goals can achieve significant reachwhen designed to build local partnerships and to build capacity toempower local action, using methodologies developed to mobilize
local engagement and access or leverage local resources.
Implementing a sizeable project without a clear evidence based ToC
creates lack of clarity, lack of focus and inefficiencies and appears to
be contributing to less than optimal GP collaboration.
Achievement of a project goal seeking global change throughcapacity building will never be possible in a 4 year project cycle.
Projects of such ambition should be established against at least a 10
year timeframe, with systematic processes for evidence gathering,evaluation, and amendments to ensure effective progress towards
the
goal.
A good ToC will include regular review of risks and assumptions, planfor piloting and innovation and build evidence of what works and what
doesn’t, in order to strengthen the project with every subsequent
stage.
Needs assessments and medium to long-term engagement.
Successful ToT initiatives take time and require a careful andthorough assessment of needs and medium to long-term
engagement in which capacities of learning centres can be
developed through training, mentoring and coaching strategies, aswell as through regular monitoring and follow-up before
disengagement of the ToT service provider. The timing ofdisengagement needs to be carefully considered and discussed by
the service provider and the beneficiary: disengagement too early
could compromise impact and sustainability; disengagement too latecould lead to reliance on the service provider and long-term
inefficiencies in the ToT strategy for the beneficiary.
Planning for collateral outcomes in training events: When people arebrought together for a training event, there will always be ‘collateral’
benefit that comes through participants making contact and sharing
their ideas and experiences with others. The theory around“boundary objects” can help better understand and plan for these lessvisible but important knowledge-sharing processes to happen and be
maintained
Developing and using a theory of change: Including a theory ofchange in a project document and revisiting it during the mid-term
and final evaluations is generally seen as good practice. The lesson
31

Tracking
outcomes
impactTracking
outcomes
impactTracking
outcomes
impactandData collection and management

andMeasurementchangeofbehaviouralandMeasurementchangeofbehavioural

Tracking
outcomes
impactandDefinition and follow-up of causal
links; project cycle and timeframe

Tracking
outcomes
impactandTracking
outcomes
impactandDefinition and follow-up of causal
links

Tracking
outcomes
impactandDefinition and follow-up of causal
links

Project cycle and timeframe

is that such a use of theory of change should be considered in otherprojects.
Coherence and alignment between the log frame and ToC is key fora clear pathway to impact.
UNITAR’s impact is evident through informal and unexpectedchannels such as the sharing of knowledge by participants throughpersonal networks
Importance of defining realistic, measurable results. While project log
frames are dynamic instruments and may be subject to review andmodification, it is important that project metrics have means ofverification and can be measured within the project’s resource
constraints.
Unintended outcomes can be highly relevant, appreciated andrewarding. Adaptive management is crucial to address importantniche areas for capacity support, such as the case for responding to
demand-driven and tailored backstopping support and addressing
digital divide by supporting online and blended learning solutions in
the wake of the COVID-19 pandemic.

32

Tracking
outcomes
impactTracking
outcomes
impactTracking
outcomes
impactTracking
outcomes
impactandDefinition and follow-up of causal
links

and

Measurementchange

and

and

of

behavioural

Definition and follow-up of causal
links;
data
collection
andmanagement
Definition and follow-up of causal
links; COVID-19