GUIDANCE NOTEGLOBAL EVALUATION REPORTASSESSMENT AND ANALYSIS SYSTEM(GERAAS)

GUIDANCE NOTEGLOBAL EVALUATION REPORTASSESSMENT AND ANALYSISSYSTEM (GERAAS)

Independent Evaluation Service (IES)
Independent Evaluation and Audit Services (IEAS)
UN Women

New York, August 2019

TABLE OF CONTENTS
I. INTRODUCTION

1

II.WHAT IS THE UN WOMEN INDEPENDENT EVALUATION REPORT QUALITY ASSESSMENT?

1

III. USERS OF THE GUIDANCE

2

IV. APPLICABILITY

2

V. APPROACHES AND METHODS

2

VI. STEPS FOR REVIEWING AND RATING INDIVIDUAL EVALUATION REPORTS

2Stage 1: Report classification and filtering

3

Stage 2: Report review and rating

3

Stage 3: Rating the parameters and overall score of the report

4Stage 4: Sharing final reviews and executive feedback on individual reports

5VII. META-EVALUATION REPORT

5

VIII. ROLE OF IES

6

ANNEX 1: UN WOMEN GLOBAL EVALUATION QUALITY ASSESSMENT AND RATING MATRIX

7

I.

INTRODUCTION

The evaluation function is a priority for UN Women as it ensures institutional accountability,
learning and informed decision-making. This is achieved, inter-alia, through high-quality
evaluations that inform evidence-based decision-making to help advance achievement of genderequality and women’s empowerment. The UN Women Evaluation Policy gives greater pertinence
to the credibility and quality of evaluation processes and products.
While the UN Women Independent Evaluation Service (IES) provides leadership and quality
assurance of the evaluation function, the planning, commissioning and management of mostevaluations are decentralized. Approximately 90 per cent of annual evaluation reports aredecentralized, commissioned by Business Units at the decentralized level and headquarters
divisions.
To ensure good quality and credible evaluations, particularly at the decentralized level, in 2013
IES established the Global Evaluation Reports Assessment and Analysis System (GERAAS). Within
the framework of the Global Evaluation Strategy 2018–2021, the GERAAS guidance and the
Evaluation Quality Assessment (EQA) matrix have been revised to further enhance the quality andcredibility of evaluations cognizant of UN Women’s institutional maturity. The GERAAS guidance
and the EQA are also aligned with the revised UNEG norms and standards (2016).
Through the GERAAS, all completed evaluations within UN Women are independently assessed,
and the rating and review feedback are posted in UN Women’s publicly accessible database, theGlobal Accountability and Tracking of Evaluation Use (GATE).

II. WHAT IS THE UN WOMEN INDEPENDENT EVALUATION REPORT QUALITYASSESSMENT?
The GERAAS is an organization-wide system established to assess the quality of UN Women’s
evaluation reports. The GERAAS is a central tenet of the IES strategy to strengthen the quality,transparency, credibility and utility of UN Women evaluations. The GERAAS uses the UNEG
evaluation report standards as a basis for review and assessment, while ensuring specificstandards relevant to UN Women.
The GERAAS is complemented by a range of evaluation quality assurance mechanisms, which
provide quality assurance of the evaluation process and products. In the long term, the GERAAS
will constitute a mechanism for near time independent assessment and feedback of the quality ofevaluation reports provided to offices to allow quality improvement of reports. Details of quality
assurance standards aligned with different stages of the evaluation phases are provided in the
UN Women Evaluation Handbook on “How to manage gender-responsive evaluations”.

1III. USERS OF THE GUIDANCEThe GERAAS guidance and EQA matrix seek to provide more clarity and guidance on the quality
standards required for evaluation reports. The standards should be used by evaluation managers, evaluators and independent assessors to ensure that the evaluations produced by
UN Women are high quality and credible. By providing constructive feedback to commissioning
offices, IES aims to provide an incentive for evaluation managers and evaluators to improve thequality of future evaluation reports. In this sense, the EQA matrix also serves as a self-assessment tool and a means of communication between all actors involved in evaluation (evaluation
managers, consultants/experts, evaluation reference group, Regional Evaluation Specialists etc.).

IV. APPLICABILITY
The GERAAS is an organization-wide system and the pre-defined standards should be applied andused for all types of evaluations (corporate, decentralized and joint evaluations) commissioned by
UN Women headquarters Business Units, Decentralized Offices and IES.

V. APPROACHES AND METHODSGERAAS uses the UNEG and United Nations System-Wide Action Plan for Gender Equality
Evaluation Performance Indicator (UN-SWAP EPI) evaluation report standards as a basis forassessment and rating, while ensuring specific standards relevant to UN Women. The assessment
acknowledges the wide variety of contexts in which evaluation reports are produced and theresources available. In doing so, the GERAAS seeks to focus on developing constructive insights
and capacity building in offices to improve the quality and usefulness of future evaluations.
Through its annual meta-evaluation report, the GERAAS also aims to contribute towards
capitalizing on the knowledge produced from evaluation reports and on capturing trends withdifferent parameters across regions.
The GERAAS assesses final evaluation reports and accompanying annexes posted in GATE. To
ensure credibility and objectivity, the quality assessment of final evaluation reports is undertaken
by an external and independent firm or an individual selected through an open bidding process.
The rating needs to give greater weight to the quality of the parameters and subrubrics rather
than the extent to which the parameters or subrubrics are present in the body of the report.

VI. STEPS FOR REVIEWING AND RATING INDIVIDUAL EVALUATION REPORTS
The assessment process includes reviewing reports against pre-defined standards, completing the
EQA matrix (including UN-SWAP Scorecard) and the executive feedback for each report. The EQA
matrix is composed of eight parameters:
1. Object and context of the evaluation
2. Evaluation purpose, objectives and scope
3. Evaluation methodology
4. Findings2

5. Conclusions and lessons learned
6. Recommendations
7. Gender and human rights considerations (UN-SWAP EPI)
8. Report presentation.
In general, the independent assessment and rating process consist of four main stages: 1
0F

Stage 1: Report classification and filtering

This comprises basic information such as title, region/country, type, costs, geographic andthematic coverage, stage/timing and management of the evaluation.
Stage 2: Report review and rating

Every assessment begins with a thorough reading of the evaluation report. In particular, theassessor is expected to:• Review the report and accompanying annexes in detail using the GERAAS EQA matrix.
• Perform a quality review of each report based on the EQA matrix template. The assessment
matrix provides eight pre-defined parameters including the UN-SWAP Evaluation
Performance Indicator Scorecard.
• Conduct the review based on what is written in the evaluation report to ensure consistency
in the rating of each parameter and the overall report.
General information on the EQA matrix• Each parameter/section has been associated with a weighting (or a multiplying factor)
which is proportionate to and illustrates its relative importance to the overall quality of
the report.
• While all parameters are important, the assessment gives more weight to findings,
conclusions, methodology and recommendations. Taken together, these four criteria
constitute nearly 70 per cent of the total quality score.
• Each parameter is further disaggregated into 29 rubrics. Under each quality parameter,
the relative importance of the subrubrics varies, and the score given to each subrubric isweighted.
• To support transparency in the rating, the EQA tool displays the criteria weight given toeach rubric within the parameter, the weighted increments and the raw point score against
all 29 rubrics.
• Usually, it is important for a report written in Spanish or French, for example, to bereviewed by a native speaker or someone with a high degree of proficiency.

1 This section and the steps outlined are provided mainly to guide the external assessor to independently review and rate thequality of final evaluation reports.

3

Table 1: The eight parameters/sections and associated weighting
Parameter/section

Parameter Weight (%)

1: Object and context

5

2: Purpose, objectives and scope

5

3: Methodology

15

4: Findings

20

5: Conclusions and lessons learned

20

6: Recommendations

15

7: Gender equality and human rights(UN-SWAP)
8: Report presentation

10
10

Stage 3: Rating the parameters and overall report score• Each parameter/section has several rubrics that are weighted against their relative
importance within the parameter. In the matrix, each rubric is scored as Fully (3),Mostly (2), Partially (1) or Not at all (0).
• Based on the rating of the subrubrics under each parameter, the overall rating for eachparameter is automatically assigned by the aggregation of the weighted score as Very
good, Good, Fair or Unsatisfactory.
• Provide executive feedback under each parameter. Comments should focus on the section
overall rather than on a particular rubric. The justification for the overall rating of eachsection/parameter should be included in the written feedback space provided under eachparameter (more guidance on feedback to commissioning offices is provided under Stage 4
below).
•The UN System-Wide Action Plan for Gender Equality (UN-SWAP Evaluation Performance
Indicator) criteria are rated according to the methods set by UNEG, with results integrated
into the GERAAS rating. The UN-SWAP Evaluation Scorecard is a reporting tool organizedaround three scoring criteria that capture the overall elements related to integrating
gender equality dimensions in evaluation reports. For this specific subparameter (Section
7 of the matrix), each criteria is rated as Fully integrated (3), Satisfactorily integrated (2),Partially integrated (1) or Not at all integrated (0). Based on the rating of the criteria, theoverall rating for the parameter is automatically assigned by the aggregation of the score,
as Meeting requirements (7 and above), Approaching requirements (4 and above), andMissing requirements (less than 4). UNEG endorsed the technical note and scorecard on
the UN-SWAP EPI which should be referenced for further details.

•

Once all rubrics are assessed, the total weighted score and overall rating are automaticallygenerated in Part III The Overall Rating of the matrix. The overall rating of an evaluation isbased on an aggregation of the weighted values of the parameters and the subrubrics
against a four-scale rating, which is Very good (85% and above), Good (65% and above),
Fair (50% and above) or Unsatisfactory (less than 50%).

4•The overall rating and assessment of the strengths and weaknesses of the evaluation
report gives an indication of the relative reliability of the results and determines the extent
to which the report can be used with confidence to feed into future programming and toserve other purposes.
Table 2: The quality rating scale2

GERAAS
Quality Rating
Scale

UN SWAP
Evaluation
Performance
RatingImplication

GERAASscore

Description of UN-SWAP scores

Very goodMeets

The report can be used withconfidence and is considered a good
example.

3 – Fully

3 – Fully integrated. Applies when allof the elements under a criterion aremet, used and fully integrated in theevaluation.

requirement

Good

Approaches
requirements

The report adheres to UN Women
evaluation standards and can beused with confidence.

2–
Mostly2 – Satisfactorily integrated. Applies
when a satisfactory level has beenreached and many of the elements
are met but improvements could still
be made.

Fair

Missesrequirement

The report meets certain standards,
but some elements are missing or
inadequately addressed. The report
should be used with caution andsubstantive improvements in someareas are needed.

1–
Partially

1 – Partially integrated. Applies whensome minimal elements are met but
further progress and remedial action
are needed to meet the standard
required.

The report has serious limitations
and therefore cannot be used withany level of confidence.

0 – Not
at all0 – Not at all integrated. Applies
when none of the elements under acriterion are met.

Unsatisfactory

Stage 4: Sharing final reviews and executive feedback on individual reports

Inherent in the GERAAS is provision of specific executive feedback to commissioning offices about
the quality of evaluation reports they managed. This is mainly designed to strengthen internal
evaluation capacity by providing practical recommendations to improve future evaluations and to
inform assessment of external consultants’ performance who might be hired for futureevaluations. Where possible, feedback should be provided in the original language of the report.

VII. META-EVALUATION REPORTThe GERAAS includes an annual analysis of the overall quality of a series of evaluation reports.
This meta-evaluation report focuses on presenting insights from analyses that can most usefully
support future action to improve the quality of evaluation reports. The report needs to illustrateand summarize the strengths and possible weaknesses of evaluations in the relevant period todevelop recommendations for future evaluation work. The meta-evaluation report also identifies
good practices.

2 GERAAS uses a four-scale rating system based on the extent to which reports meet the assessment criteria.

5

VIII.

ROLE OF IES

IES oversees, coordinates and supports the GERAAS review process from the selection of
consulting firm/individual consultant to finalization and provision of executive feedback to theoffices concerned. While the external reviewer primarily takes sole responsibility for ensuring thecompleteness, quality and adherence to established standards, IES closely monitors the consistent
application of approaches, rating and finalization of the exercise in a timely manner.
IES presents the findings of the review at the Annual Session of the Executive Board and to senior
managers and the Global Evaluation Advisory Committee. The report is also shared with theheadquarters divisions, Regional Offices and Country Offices concerned to improve the quality
and utility of evaluations by highlighting the strengths, good practices and areas that require
improvement. The report is posted on the GATE along with the rating and executive feedback
system, which allows access to the general public. This contributes to the transparency andcredibility of UN Women when reporting on its performance.

6

ANNEX 1. UN WOMEN GLOBAL EVALUATION QUALITY ASSESSMENT AND RATING MATRIX (sample)

Independent Evaluation and Audit Services (IEAS)
UN WOMEN Global Evaluation Quality Assessment and Rating
RatingScale

Ratingexplanation

Parameter
Weight (%)

Very Good

Good

Fair

Unsatisfactory

The report canbe used with highlevel ofconfidence and isconsidered agood example.

The report canbe used withcertain degreeof confidence.

Partiallymeetsrequirementswith somemissing
elements. The
report can beused withcaution.

Misses out theminimum
qualitystandards.

1: Object andcontext
2: Purpose andscope

Reviewer Guidance :- Overall reports are rated against a 4-point scale (Very Good, Good, Fair
and Unsatisfactory), which is an aggregated rating of eight parameters.
- Each overarching parameter is rated against a 4-point scale (Fully,
Mostly, Partially and Not at all).
- Parameters such as evaluation methodology, findings, conclusions andrecommendations are given more weight.
- Executive feedback - provide summary of the extent to which thereport meets or fails to meet the criteria provided under eachparameter. Please also include suggestion on how to improve futureevaluation practice. The overall review, rating , and the executive
feedback will be provided to the evaluation commissioning office.

5

5: Conclusions and lessons
learned20

5

6: Recommendations

15

3: Methodology

15

7: Gender Equality and Human
Rights (UN-SWAP)

10

4: Findings

20

8: Presentation

10

Are weightings equal to 100%?
OK

PART I: REPORT DETAILS
Report title

Corporate Evaluation on Strategic Partnerships for Gender Equality andthe Empowerment of Women
Sequence number
Evaluators[Female]RegionEurope andCountry(ies)Central Asia

Geographical Coverage[Male]

Year
Type of intervention evaluatedGlobal
2018
CPE

7

Portfolio Budget (USD)
Strategic Plan Thematic Area(select all that apply)

Women’sleadershipandparticipation

Evaluation
Budget (USD)
Women’sleadership inpeace,
security andhumanitarian
response

Reviewer
Women’sleadership inpeace, security
andhumanitarian
response

Review Date

February 22nd, 2018

Women’saccess to
economic
empowerment
andopportunities

PART II: THE EIGHT KEY PARAMETERSSECTION 1: OBJECT AND CONTEXT OF THE EVALUATION (weight5%)

RATING

Good

Does the report present a clear and full description of the 'object'of the evaluation?

50%

Executive Feedback on Section 1

1.1 The report clearly specify the object of the evaluation, and provides
clear and complete description of the intervention's logic or theory ofchange, intended beneficiaries by type and by geographic location(s)as well as resources from all sources including humans and budgets,and modalities.
1.2 The context includes factors that have a direct bearing on theobject of the evaluation: social, political, economic, demographic, andinstitutional. This also includes explanation of the contextual genderequality and human rights issues, roles, attitudes and relations.
1.3 The key stakeholders involved in the implementation, including
the implementing agency(s) and partners, other stakeholders and theirroles are described.
1.4 The report identifies the implementation status of the object,including its phase of implementation and any significant changes (e.g.
plans, strategies, logical frameworks) that have occurred over time andexplains the implications of those changes for the evaluation.
SECTION 2: PURPOSE, OBJECTIVES AND SCOPE (weight 5%)

Criteria
WeightWeighted
increments

Raw
point
score

Fully

25%

0.42

1.25

Mostly

25%

0.42

0.83

Partly

25%

0.42

0.42

Not at all

25%

0.42

0.00

RATING

Very Good

8

Are the evaluation's purpose, objectives and scope sufficiently
clear to guide the evaluation?

83%

2.1 Purpose, objectives and use of evaluation: The evaluation
report provides clear explanation of the purpose and the objectives ofthe evaluation including the intended use and users of the evaluationand how the information will be used.
2.2 Evaluation Scope:
The evaluation report provides clear
description of the scope of the evaluation, including justification ofwhat the evaluation covers and did not cover (thematically,geographically etc) as well as the reasons for this scope (eg.,specifications by the ToRs, lack of access to particular geographic areasfor political or safety reasons at the time of the evaluation, lack ofdata/evidence on particular elements of the intervention).
SECTION 3 : METHODOLOGY (weight 15%)

Fully

Is the methodology used for the evaluation clearly described andappropriate, and the rationale for the methodological choicejustified?
3.1 Methodology: The report specifies and provides complete
description of a relevant design and sets of methods including thechosen evaluation criteria, questions, and performance standards. The
methods employed are appropriate for analyzing gender and rights
issues identified in the evaluation scope.
3.2 Data collection, analysis and sampling: The report clearly
describes the methods for the data sources, rationale for theirselection, data collection and analysis methods. The report includes
discussion of how the mix of data sources was used to obtain a diversityof perspectives, ensure data accuracy and overcome data limitations.
3.3 Stakeholders Consultation: The evaluation report gives acomplete description of stakeholder’s consultation process in theevaluation, including the rationale for selecting the particular level andactivities for consultation.
3.4 Limitations: The report presents clear and complete description oflimitations and constraints faced by the evaluation, including gaps in
the evidence that was generated and mitigation of bias.
3.5 Ethics: The evaluation report includes a discussion of the extent towhich the evaluation design included ethical safeguards andmechanisms and measures that were implemented to ensure that theevaluation process conformed with relevant ethical standards
including but not limited to informed consent of participants,
confidentiality and avoidance of harm considerations.
SECTION 4: FINDINGS (weight 20%)

Executive Feedback on Section 2

Criteria
WeightWeighted
increments

Raw
point
score

50%

0.83

2.50

50%

0.83

1.67

Criteria
WeightWeighted
increments

Raw
point
score

35%

1.75

5.25

40%

2

4.00

10%

0.50

1.50

5%

0.25

0.25

10%

0.5

1.50

MostlyRATING

Very Good

83%

Executive Feedback on Section 3

Fully

MostlyFully

PartlyFully

RatingGood

9

Are the findings clearly presented, relevant and based onevidence?

70%

4.1The evaluation report findings provide sufficient levels of highquality evidence to systematically address all of the evaluationquestions and criteria.
4.2 Findings are clearly supported by and respond to the evidence
presented, reflecting systematic and appropriate analysis andinterpretation of the data; they are free from subjective judgements
made.
4.3 The causal factors (contextual, organizational, managerial, etc.)
leading to achievement or non-achievement of results are clearly
identified.
4.4 Findings are presented with clarity, logic and coherence (e.g., avoid
ambiguities).
SECTION 5: CONCLUSIONS AND LESSONS LEARNED (weight 20%)

PartlyAre the conclusions clearly presented based on findings andsubstantiated by evidence?

40%

5.1 Conclusions are well substantiated by the evidence presented andare logically connected to evaluation findings.

Fully

5.2 The conclusions reflect reasonable evaluative judgments that add
insight and analysis beyond the findings
5.3 Conclusions present strengths and weaknesses of the object(policy, programmes, project's or other intervention) being evaluated,based on the evidence presented and taking due account of the viewsof a diverse cross-section of stakeholders.
5.4 Lessons Learned: When presented, the lessons learned section
stems logically from the findings, presents an analysis of how they canbe applied to different contexts and/or different sectors, and takes into
account evidential limitations such as generalizing from single point
observations.

Not at all

SECTION 6: RECOMMENDATIONS (weight 15%)

Rating

Are the recommendations relevant, useful, and actionable andclearly presented in a priority order?

100%

6.1 Recommendations are logically derived from the findings and/or
conclusions.

Fully

Executive Feedback on Section 4

Criteria
WeightWeighted
increments

Raw
point
score

30%

2.00

2.00

30%

2.00

4.00

20%

1.33

4.00

20%

1.33

4.00

Criteria
WeightWeighted
increments

Raw
point
score

40%

2.67

8.00

40%

2.67

0.00

15%

1.00

0.00

5%

0.33

0.00

Criteria
WeightWeighted
increments

Raw
point
score

30%

1.50

4.50

MostlyFully
Fully
RatingFair
Executive Feedback on Section 5

Not at all

Not at all

Very Good
Executive Feedback on Section 6

10

6.2 The report describes the process followed in developing therecommendations including consultation with stakeholders.

Fully

6.3 Recommendations are clear, realistic (e.g., reflect an understandingof the subject's potential constraints to follow-up) and actionable.

Fully

6.4 Clear prioritization and/or classification of recommendations to
support use.
SECTION 7: GENDER AND HUMAN RIGHTS (weight 15%)

Fully

Does the evaluation meet UN SWAP evaluation performanceindicators? Note: this section will be rated according to UN SWAP
standards.

67%

7.1 GEWE is integrated in the evaluation scope of analysis andevaluation criteria and questions are designed in a way that ensures
GEWE related data will be collected.
7.2 A gender-responsive methodology, methods and tools, and data
analysis techniques are selected.
7.3 The evaluation findings, conclusions and recommendation reflect a
gender analysis.

Partially
integrated (1)SECTION 8: THE REPORT PRESENTATION (weight 10%)Score

Is the report well structured, written in accessible language andwell presented?

40%

8.1 Report is logically structured, well written and presented withclarity and coherence (e.g. the structure and presentation is easy toidentify and navigate (for instance, with numbered sections, clear titles
and sub-titles; context, purpose and methodology would normally
precede findings, which would normally be followed by conclusions,lessons learned and recommendations) and written in an accessible
language with minimal grammatical, spelling or punctuation errors.
8.2 The title page and opening pages provide key basic informationon the name of evaluand, timeframe of the evaluation, date of report,
location of evaluated object, names and/or organization(s) of theevaluator(s), name of organization commissioning the evaluation, tableof contents -including, as relevant, tables, graphs, figures, annexes-; listof acronyms/abbreviations, page numbers.
8.3 The Executive Summary is a stand-alone section that includes an
overview of the intervention, evaluation purpose, objectives andintended audience, evaluation methodology, key findings, conclusions
and recommendations. The Executive summary should be reasonably
concise.

Mostly

1.00

3.00

30%

1.50

4.50

20%

1.00

3.00

Criteria
WeightWeighted
increments

Raw
point
score

UNSWAP
score

33%

1.11

1.11

1

33%

1.11

2.22

2

33%

1.11

3.33

3

Criteria
WeightWeighted
increments

Raw
point
score

40%

1.33

2.67

10%

0.33

0.33

40%

1.33

0.00

Approaching Requirements
Executive Feedback on Section 7

Satisfactorily
integrated (2)Fully
integrated (3)Rating20%

Fair
Executive Feedback on Section 8

PartlyNot at all11

8.4 Annexes should include, when not present in the body of thereport:Terms of Reference, Evaluation matrix, list of interviewees, list of site
visits, data collection instruments (such as survey or interview
questionnaires),listof
documentary
evidence.
Other appropriate annexes could include: additional details on
methodology, copy of the results chain, information about theevaluator(s).
Additional Information

Fully

10%

0.33

1.00

Identify aspects of good practice of the evaluation

PART III: THE OVERALL RATINGKey Guiding Question

Is this a credible report that addresses the evaluation purposeand objectives based on evidence, and that can therefore be usedwith confidence?

Total
weighted
score %Overall Rating

66.83

Good

Other reviewer's
comments

12