![](media/image1.png){width="4.4375in" height="3.8229166666666665in"}

Policy and Program Coordination Bureau

Contract Number: AEP-C-00-99-00034-00

Last Revised: April 2003

#  

# Foreword

This Toolkit was originally developed in the summer and fall of 2000 to
accompany the one-week USAID \"Performance Management Workshop\". To
date, this workshop has been held over 30 times in Washington and all of
the USAID regions and over 900 Agency staff have been trained. This
version of the Toolkit was produced in April 2003 to reflect the changes
in ADS Chapters 200-203 that went into effect on January 31, 2003.

We would like to acknowledge the technical and administrative support
provided by the staff of the USAID Bureau for Policy and Program
Coordination, especially Dan Blumhagen and John Haecker of the Center
for Development Information and Experience.

We would also like to thank the staff of the USAID/Africa Bureau,
particularly Kathie Keel and Ruth Buckley from the Office of Development
Planning, Shirley Erves from the Regional Economic Development Support
Office and Dana Ott from the Office of Sustainable Development. Their
contributions, during various iterations of the Toolkit and its
implementation during the pilot workshop, have been significant. In
addition, the participants of the pilot workshop provided valuable
insights and comments that contributed to the overall effectiveness of
this Toolkit.

Also, we would like to thank Rita Owen and her colleagues from the
Office of Human Resources/Learning Services for their cooperation and
support. They have contributed significantly to the dissemination of
these materials to hundreds of USAID staff.

And lastly, our sincere thanks to all of the Operating Units in the
field and in Washington who shared their knowledge and experience with
us during the development of their Performance Management Plans.

To all those who have contributed to this effort, we hope that this
Toolkit will further the Agency\'s Managing for Results efforts and lead
to more sustainable and effective programs that improve the lives of the
people that USAID staff work so hard to serve.

**Integrated Managing for Results Team**

**IBM Business Consulting Services**

**January 10, 2003**

+-----------------------------------+-----------------------------------+
| *For more information about the   |                                   |
| Integrated Managing for\          |                                   |
| Results Contract,                 |                                   |
| AEP-C-00-99-00034-00, contact:    |                                   |
| (1/10/03)*                        |                                   |
+-----------------------------------+-----------------------------------+
| **Dan Blumhagen**, M.D., Ph.D.,   | **Perry Pockros, Shiro            |
| M.P.H.                            | Gnanaselvam**                     |
|                                   |                                   |
| Cognizant Technical Officer       | Program Managers                  |
|                                   |                                   |
| USAID                             | IBM Business Consulting Services  |
|                                   |                                   |
| 1300 Pennsylvania Ave. NW         | 1616 North Fort Myer Drive        |
|                                   |                                   |
| Washington, DC 20523              | Arlington, VA 22209               |
|                                   |                                   |
| Office: 202-712-5816              | Office: 703-741-1000              |
|                                   |                                   |
| <dblumhagen@usaid.gov>            | <perry.pockros@us.ibm.com>        |
|                                   |                                   |
|                                   | [shiranthi.gn                     |
|                                   | anaselvam@us.ibm.com]{.underline} |
+-----------------------------------+-----------------------------------+

***FACING PAGE:** This Toolkit is organized around a three-part process
for developing and implementing PMPs. Each task is fully detailed with
supporting materials, tools, and resources.*

# PMP Development and Implementation Process

![](media/image1.wmf)

**Table of Contents**

**Foreword [iii](#foreword)**

PMP Development and Implementation Process
[iv](#pmp-development-and-implementation-process)

Toolkit Quick Reference Guide [viii](#toolkit-quick-reference-guide)

List of Acronyms [x](#list-of-acronyms)

Overview [11](#overview)

Introduction to the Performance Management Plan (PMP)
[11](#introduction-to-the-performance-management-plan-pmp)

How to Use This Toolkit [13](#how-to-use-this-toolkit)

Part 1: Plan for PMP Development [15](#part-1-plan-for-pmp-development)

Task 1 -- Assemble a PMP development team
[15](#task-1-assemble-a-pmp-development-team)

Task 2 -- Collect and review background materials
[16](#task-2-collect-and-review-background-materials)

Task 3 -- Develop a workplan [17](#task-3-develop-a-workplan)

Task 4 -- Conduct team briefing meeting
[18](#task-4-conduct-team-briefing-meeting)

Part 2: Develop the PMP [19](#part-2-develop-the-pmp)

Task 1 - Review Results Framework
[21](#task-1---review-results-framework)

Key Definitions, ADS Guidance and Helpful Resources
[21](#key-definitions-ads-guidance-and-helpful-resources)

1.1 Assess quality of results statements
[23](#assess-quality-of-results-statements)

1.2 Validate logic [24](#validate-logic)

1.3 Verify results are within USAID's manageable interest
[27](#verify-results-are-within-usaids-manageable-interest)

1.4 Ensure critical assumptions are identified
[27](#ensure-critical-assumptions-are-identified)

Task 2 -- Develop Performance Indicators
[29](#task-2-develop-performance-indicators)

Key Definitions, ADS Guidance and Helpful Resources
[29](#key-definitions-ads-guidance-and-helpful-resources-1)

2.1 Develop list of potential indicators
[30](#develop-list-of-potential-indicators)

2.2 Assess potential indicators [31](#assess-potential-indicators)

2.3 Select best indicators [38](#select-best-indicators)

2.4 Document indicators in the PMP [38](#document-indicators-in-the-pmp)

Task 3 -- Identify Data Source and Collection Method
[40](#task-3-identify-data-source-and-collection-method)

ADS Guidance and Helpful Resources
[40](#ads-guidance-and-helpful-resources)

3.1 Identify potential data sources
[41](#identify-potential-data-sources)

3.2 Generate data collection options
[42](#generate-data-collection-options)

3.3 Select data collection option [43](#select-data-collection-option)

3.4 Develop data collection tools [45](#develop-data-collection-tools)

3.4 Develop data collection tools [46](#develop-data-collection-tools)

Task 4 -- Collect Baseline Data and Verify Quality
[47](#task-4-collect-baseline-data-and-verify-quality)

Key Definitions, ADS Guidance and Helpful Resources
[47](#key-definitions-ads-guidance-and-helpful-resources-2)

4.1 Collect baseline data [49](#collect-baseline-data)

4.2 Conduct a data quality assessment
[50](#conduct-a-data-quality-assessment)

4.3 Build commitment to and capacity for quality
[54](#build-commitment-to-and-capacity-for-quality)

Task 5 -- Establish Performance Targets
[57](#task-5-establish-performance-targets)

Key Definitions, ADS Guidance and Helpful Resources
[57](#key-definitions-ads-guidance-and-helpful-resources-3)

5.1 Establish indicator targets [58](#establish-indicator-targets)

5.2 Input baselines and targets into performance data tables
[59](#input-baselines-and-targets-into-performance-data-tables)

Task 6 -- Plan for Other Assessing and Learning Activities
[61](#task-6-plan-for-other-assessing-and-learning-activities)

Key Definitions, ADS Guidance and Helpful Resources
[61](#key-definitions-ads-guidance-and-helpful-resources-4)

6.1 Plan for data analysis and use [67](#plan-for-data-analysis-and-use)

6.2 Plan for performance reviews [68](#plan-for-performance-reviews)

6.3 Plan for evaluations and special studies
[70](#plan-for-evaluations-and-special-studies)

6.4 Plan for performance reporting [73](#plan-for-performance-reporting)

Conclusion [76](#conclusion)

Part 3: Use, Review and Revise the PMP
[78](#part-3-use-review-and-revise-the-pmp)

Task 1 -- Use the PMP [78](#task-1-use-the-pmp)

Task 2 -- Review the PMP [79](#task-2-review-the-pmp)

Task 3 -- Revise the PMP [80](#task-3-revise-the-pmp)

Appendix A: Worksheets [81](#appendix-a-worksheets)

Worksheet 1: PMP Development Team Skills Matrix
[82](#worksheet-1-pmp-development-team-skills-matrix)

Worksheet 2: PMP Development Workplan
[83](#worksheet-2-pmp-development-workplan)

Worksheet 3: Results Statement Assessment
[84](#worksheet-3-results-statement-assessment)

Worksheet 4: Results Framework Assessment
[85](#worksheet-4-results-framework-assessment)

Worksheet 5: Performance Indicator Quality Assessment
[86](#worksheet-5-performance-indicator-quality-assessment)

Worksheet 6: Performance Indicator Reference Sheet
[88](#worksheet-6-performance-indicator-reference-sheet)

Worksheet 7: Data Quality Assessment Checklist
[91](#worksheet-7-data-quality-assessment-checklist)

Worksheet 8: Summary Performance Data Table
[97](#worksheet-8-summary-performance-data-table)

Worksheet 9: Performance Management Task Schedule
[98](#worksheet-9-performance-management-task-schedule)

Worksheet 10: Evaluations and Special Studies Planning
[100](#worksheet-10-evaluations-and-special-studies-planning)

Worksheet 11: Evaluation Scope of Work (SOW) Planning
[101](#worksheet-11-evaluation-scope-of-work-sow-planning)

Appendix B: Helpful Hints [102](#appendix-b-helpful-hints)

Helpful Hint 1: Facilitating Group Discussions and Decision-Making
[103](#helpful-hint-1-facilitating-group-discussions-and-decision-making)

Helpful Hint 2: Indicators for Hard-to-Measure Results
[104](#helpful-hint-2-indicators-for-hard-to-measure-results)

Helpful Hint 3: Performance Indicator Brainstorming Session
[105](#helpful-hint-3-performance-indicator-brainstorming-session)

Helpful Hint 4: Rapid Low-Cost Data Collection Methods
[106](#helpful-hint-4-rapid-low-cost-data-collection-methods)

Helpful Hint 5: Information Systems for Data Collection and Performance
Monitoring
[108](#helpful-hint-5-information-systems-for-data-collection-and-performance-monitoring)

Helpful Hint 6: Key Features of Quality Data Sources
[110](#helpful-hint-6-key-features-of-quality-data-sources)

Helpful Hint 7: Tips to Minimize Bias
[112](#helpful-hint-7-tips-to-minimize-bias)

Helpful Hint 8: Tips for Communicating Performance Information in
Reports
[113](#helpful-hint-8-tips-for-communicating-performance-information-in-reports)

Helpful Hint 9: Questions to Guide Portfolio Reviews
[114](#helpful-hint-9-questions-to-guide-portfolio-reviews)

Helpful Hint 10: Maintaining Official SO Team Files
[116](#helpful-hint-10-maintaining-official-so-team-files)

Appendix C: Helpful Resources [117](#appendix-c-helpful-resources)

Resources available on the Internet
[117](#resources-available-on-the-internet)

USAID Economic and Social Data Service (ESDS)
[121](#usaid-economic-and-social-data-service-esds)

Research and Reference Services (R&RS)
[121](#research-and-reference-services-rrs)

USAID Library and Learning Resources Center
[121](#usaid-library-and-learning-resources-center)

Development Experience Clearinghouse
[122](#development-experience-clearinghouse)

Appendix D: ADS 201.3.7.6 and ADS Chapter 203
[123](#appendix-d-ads-201.3.7.6-and-ads-chapter-203)

#  

# Toolkit Quick Reference Guide

*Use this guide to quickly identify key worksheets, techniques, and
helpful hints.*

**[Part 1: Plan for PMP Development]{.underline}**

  ------------------------------- ---------------------------------------
  **Tasks**                       **Worksheets, Techniques, Helpful
                                  Hints**

  Assemble a PMP development team Worksheet 1: PMP Development Team
                                  Skills Matrix

  Collect and review background   
  materials                       

  Develop a workplan              Worksheet 2: PMP Development Workplan

  Conduct a team briefing meeting 
  ------------------------------- ---------------------------------------

**[Part 2: Develop PMP]{.underline}**

**Task 1: Review Results Framework**

+------------------------------+---------------------------------------+
| **Sub-tasks**                | **Worksheets, Techniques, Helpful     |
|                              | Hints**                               |
+------------------------------+---------------------------------------+
| Assess quality of results    | Worksheet 3: Results Statement        |
| statements                   | Assessment                            |
|                              |                                       |
|                              | Worksheet 4: Results Framework        |
|                              | Assessment                            |
|                              |                                       |
|                              | Technique: Assess Framework in a      |
|                              | Facilitated Session                   |
|                              |                                       |
|                              | Helpful Hint 1: Facilitating Group    |
|                              | Discussions and Decision-Making       |
+------------------------------+---------------------------------------+
| Validate logic               |                                       |
+------------------------------+---------------------------------------+
| Verify USAID\'s manageable   |                                       |
| interest                     |                                       |
+------------------------------+---------------------------------------+
| Ensure critical assumptions  |                                       |
| are identified               |                                       |
+------------------------------+---------------------------------------+

**Task 2: Develop Performance Indicators**

+------------------------------+---------------------------------------+
| **Sub-tasks**                | **Worksheets, Techniques, Helpful     |
|                              | Hints**                               |
+------------------------------+---------------------------------------+
| Develop list of potential    | Technique: Use Current Resources to   |
| indicators                   | Identify Potential Indicators         |
|                              |                                       |
|                              | Technique and Helpful Hint 2:         |
|                              | Indicators for Hard-to-Measure        |
|                              | Results                               |
+------------------------------+---------------------------------------+
| Assess potential Indicators  | Worksheet 5: Performance Indicator    |
|                              | Quality Assessment                    |
+------------------------------+---------------------------------------+
| Select best indicators       | Technique and Helpful Hint 3:         |
|                              | Performance Indicator Brainstorming   |
|                              | Session                               |
+------------------------------+---------------------------------------+
| Document indicators in the   | Worksheet 6: Performance Indicator    |
| PMP                          | Reference Sheet                       |
+------------------------------+---------------------------------------+

**Task 3: Identify Data Sources and Collection Methods**

+------------------------------+---------------------------------------+
| **Sub-tasks**                | **Worksheets, Techniques, Helpful     |
|                              | Hints**                               |
+------------------------------+---------------------------------------+
| Identify potential data      | Technique: Assess Results Framework   |
| sources                      | in a Facilitated Session (also        |
|                              | Helpful Hint 1)                       |
+------------------------------+---------------------------------------+
| Generate data collection     | Helpful Hint 4: Rapid Low-Cost Data   |
| options                      | Collection Methods                    |
+------------------------------+---------------------------------------+
| Select data collection       | Technique: Use Decision Chart to      |
| option                       | Select Best Data Collection Option    |
|                              |                                       |
|                              | Worksheet 6: Performance Indicator    |
|                              | Reference Sheet                       |
+------------------------------+---------------------------------------+
| Develop data collection      |                                       |
| tools                        |                                       |
+------------------------------+---------------------------------------+

**Task 4: Collect Baseline Data and Verify Quality**

+------------------------------+---------------------------------------+
| **Sub-tasks**                | **Worksheets, Techniques, Helpful     |
|                              | Hints**                               |
+------------------------------+---------------------------------------+
| Collect data                 | Technique: Storing Data in an         |
|                              | Information Database, and Helpful     |
|                              | Hint 5                                |
+------------------------------+---------------------------------------+
| Conduct a data quality       | Technique: Plan Data Quality          |
| assessment                   | Assessments                           |
|                              |                                       |
|                              | Worksheet 7: Data Quality Assessment  |
|                              | Checklist                             |
|                              |                                       |
|                              | Technique: Assess Data from Data      |
|                              | Sources, and Helpful Hint 6           |
|                              |                                       |
|                              | Helpful Hint 7: Tips to Minimize Bias |
|                              |                                       |
|                              | Worksheet 6: Performance Indicator    |
|                              | Reference Sheet                       |
+------------------------------+---------------------------------------+
| Build commitment to and      | Technique: Foster Organizational      |
| capacity for quality         | Commitment                            |
+------------------------------+---------------------------------------+

**Task 5: Establish Performance Targets**

+------------------------------+---------------------------------------+
| **Sub-tasks**                | **Worksheets, Techniques, Helpful     |
|                              | Hints**                               |
+------------------------------+---------------------------------------+
| Establish baselines          | Technique: How to Establish Baselines |
|                              | when Information is Inadequate        |
+------------------------------+---------------------------------------+
| Establish indicators         | Technique: Conduct a Target Setting   |
|                              | Meeting                               |
|                              |                                       |
|                              | Helpful Hint 1: Facilitating Group    |
|                              | Discussions and Decision-Making       |
|                              |                                       |
|                              | Technique: Approaches to Target       |
|                              | Setting                               |
+------------------------------+---------------------------------------+
| Input baselines and targets  | Worksheet 8: Performance Data Table   |
| into performance data table  |                                       |
+------------------------------+---------------------------------------+

**Task 6: Plan for Other Assessing and Learning Elements**

+------------------------------+---------------------------------------+
| **Sub-tasks**                | **Worksheets, Techniques, Helpful     |
|                              | Hints**                               |
+------------------------------+---------------------------------------+
| Plan for data analysis and   | Worksheet 9: Performance Management   |
| use                          | Task Schedule                         |
|                              |                                       |
|                              | Worksheet 6: Performance Indicator    |
|                              | Reference Sheet                       |
|                              |                                       |
|                              | Helpful Hint 8: Tips for              |
|                              | Communicating Performance Information |
|                              | in Reports                            |
|                              |                                       |
|                              | Technique: Chart Book Analysis        |
|                              | Presentation Approach                 |
+------------------------------+---------------------------------------+
| Plan for performance reviews | Technique: Portfolio Review           |
|                              | Approaches                            |
|                              |                                       |
|                              | Helpful Hint 9: Questions to Guide    |
|                              | Portfolio Reviews                     |
+------------------------------+---------------------------------------+
| Plan for evaluations and     | Technique: Planning for Evaluations   |
| special studies              | (also Helpful Hint 4)                 |
|                              |                                       |
|                              | Worksheet 10: Evaluations and Special |
|                              | Studies Planning                      |
|                              |                                       |
|                              | Worksheet 11: Evaluation Scope of     |
|                              | Work Planning                         |
+------------------------------+---------------------------------------+
| Plan for performance         | Technique: Plan for Annual Report     |
| reporting                    | Reporting                             |
+------------------------------+---------------------------------------+
| Plan for ongoing data        | Technique: On-going Data Quality      |
| quality assessments          | Assessments                           |
|                              |                                       |
|                              | Worksheet 7: Data Quality Assessment  |
|                              | Checklist                             |
+------------------------------+---------------------------------------+

**[Part 3: Use, Review and Revise PMP]{.underline}**

  ------------------------------- ---------------------------------------
  **Tasks**                       **Worksheets, Techniques, Helpful
                                  Hints**

  Implement PMP                   Helpful Hint 10: List of Official SO
                                  Team Files

  Review PMP                      Helpful Hint 9: Questions to Guide
                                  Portfolio Reviews

  Revise PMP                      
  ------------------------------- ---------------------------------------

# List of Acronyms

> ADS Automated Directives System
>
> CDIE USAID Center for Development Information & Evaluations
>
> CSP Country Strategic Plan
>
> CTO Cognizant Technical Officer
>
> DHS Demographic & Health Survey
>
> GAO U.S. General Accounting Office
>
> GIS Geographic Information System
>
> IR Intermediate Result
>
> M&E Monitoring and Evaluation
>
> MFR Managing for Results
>
> NGO Non-Governmental Organization
>
> OMB Office of Management and Budget
>
> PMP Performance Management Plan
>
> PPC USAID Bureau for Policy and Program Coordination
>
> PVO Private Voluntary Organization
>
> SO Strategic Objective
>
> TLA Three-Letter Acronym!

# Overview

The purpose of this Toolkit is to provide USAID staff and partners with
practical resources for a critical aspect of managing for results --
developing and implementing a ***performance management plan (PMP***).
The emphasis of this Toolkit is on the *how to* of developing a PMP
rather than a discussion of *what is* performance management, which is
contained in the official programming policies---the USAID Automated
Directives System Chapters 200-203.

The primary target audience for this Toolkit is the ***Strategic
Objective (SO) teams*** who are responsible for developing the PMP. The
objectives of this Toolkit are to help you (the team):

-   Understand the purpose of a PMP

-   Carry out preparations to develop a PMP

-   Review results statements

-   Develop performance indicators

-   Collect performance data

-   Verify performance data quality

-   Analyze, evaluate and use performance information

-   Implement and continuously improve the PMP

The Toolkit describes key performance management processes and provides
tools and techniques that can be used to implement them. It provides
examples to illustrate key points and identifies reference sources where
additional information can be sought. Lastly, the Toolkit emphasizes the
importance of *documenting* performance data and analysis. Documentation
helps you:

-   Ensure the availability of information you need to analyze and
    improve program performance

-   Tell your story with confidence in the information you provide

-   Explain your procedures to stakeholders who seek assurance that
    quality standards are being maintained in the collection and
    reporting of performance data.

## Introduction to the Performance Management Plan (PMP)

#### What is a PMP?

A PMP is a performance management tool used by an Operating Unit and
Strategic Objective (SO) team to help plan and manage the process of
assessing and reporting progress towards achieving a Strategic
Objective. It is a critical tool for *planning, managing, and
documenting* how performance data is collected and used. A PMP serves
to:

-   Define specific performance indicators for each SO and IR, determine
    baselines and set targets

-   Plan and manage the Annual Report data collection process to meet
    quality standards

-   Incorporate relevant data collection requirements into activities
    and obligation agreements

-   Plan potential related evaluative work to supplement Annual Report
    indicator data

-   Estimate costs related to data collection and plan how these will be
    financed

-   Communicate expectations to partner institutions responsible for
    producing the outputs intended to cause measurable changes in
    performance

A PMP contributes to the effectiveness of the performance monitoring
system by assuring that **comparable** data will be collected on a
**regular and timely** basis. Using the PMP to sufficiently document
indicator definitions, sources, and methods of data collection increases
the likelihood that you will collect comparable data over time - even
when key personnel change. PMPs also support reliable data collection by
documenting the frequency and schedule of data collection and assigning
responsibilities.

#### What is contained in a PMP? 

A PMP contains full documentation of the indicators used to track
progress toward the Strategic Objective, their data sources, the quality
of data available and responsibilities for collection and analysis of
the data. There is no standard PMP format, however, you are encouraged
to develop a comprehensive PMP that goes beyond the one-page matrix
often encountered. Your PMP should help the team establish systems to
monitor, evaluate, analyze, review, and report performance data. Agency
guidance identifies required and recommended PMP elements.

+-----------------------------------------------------------------------+
| **ADS Guidance on PMP Elements**                                      |
|                                                                       |
| ***Contents of a Preliminary PMP (ADS 201.3.7.6)***                   |
|                                                                       |
| -   Propose performance indicators for the Strategic Objective-level  |
|     result (with baseline data and ultimate targets).                 |
|                                                                       |
| -   If possible, include performance indicators for the Intermediate  |
|     Results (with baseline data and ultimate targets).                |
|                                                                       |
| ***Mandatory Elements of a Complete PMP (ADS 203.3.3.1)***            |
|                                                                       |
| -   At least one indicator to measure progress at the SO level with   |
|     baseline data and performance targets                             |
|                                                                       |
| -   At least one indicator to measure progress towards each IR with   |
|     baseline data and performance targets                             |
|                                                                       |
| ***Additional Contents of a Complete PMP (ADS 203.3.3.1)***           |
|                                                                       |
| -   Calendar of performance management tasks                          |
|                                                                       |
| -   Statement of all performance indicators that will be used to      |
|     assess progress over the life of the SO                           |
|                                                                       |
| -   Baseline values and performance targets for each indicator        |
|                                                                       |
| -   Specification of the data source and collection method            |
|                                                                       |
| -   Specification of the schedule for data collection                 |
|                                                                       |
| -   Description of known data limitations                             |
|                                                                       |
| -   Description of the data quality assessment procedures that will   |
|     be used to verify data quality                                    |
|                                                                       |
| NOTE: More specific guidance applies to indicators that are reported  |
| to Washington                                                         |
+-----------------------------------------------------------------------+

#### When is a PMP prepared? 

You should begin planning for performance management early in the
strategic planning process. ADS 201.3.7.6 provides that Operating Units
must include a preliminary PMP when submitting a Strategic Plan or new
Strategic Objective for approval. A complete PMP is required within one
year of approving a Strategic Plan or new Strategic Objective. The PMP
may be reviewed and approved by the Operating Unit director. You should
review and update your PMP at least annually as part of the Portfolio
Review and Annual Report preparation.

Usually, Operating Units and SO teams develop the PMP in three main
stages as strategic planning and implementation proceed. Figure 0-1
illustrates the relationship between PMP development and the strategic
planning process.

![](media/image2.png){width="3.9972222222222222in"
height="2.997916666666667in"}

## How to Use This Toolkit

The Toolkit is divided into three parts.

-   **Part 1** focuses on how to begin preparing for PMP development

-   **Part 2** focuses on how to develop a comprehensive PMP

-   **Part 3** focuses on how to implement and continually improve the
    PMP.

Each part includes information essential for establishing a system of
using performance information to make program decisions. These icons can
help you quickly navigate through the Toolkit:

![](media/image3.wmf){width="0.875in" height="0.7395833333333334in"}

**KEY DEFINITIONS:** Indicates key definitions for the relevant PMP
development

phase.

![](media/image4.wmf)

**WORKSHEET:** Indicates that a tool is available to help you document
PMP elements and analysis throughout the PMP development process.
Examples of completed worksheets are frequently presented in the text.
Blank copies of the worksheets can be found in the ***WORKSHEET***
appendix of the Toolkit.

![](media/image5.png)

**TECHNIQUE:** Indicates a technique (e.g., facilitating a brainstorming
session, questions to ask) that you might consider using in order to
complete a performance management process. Some of these techniques will
refer you to additional information in the ***HELPFUL HINTS*** appendix.

![](media/image6.png){width="1.1in" height="1.1in"}

**CONSIDER THIS:** Indicates questions or principles that you should
consider in developing the PMP.

![](media/image7.png)

**INFORMATION SERVICE:** Indicates a contracted service available to
USAID employees for procuring and analyzing data and information. More
information about these resources, and others, are contained in the
*RESOURCES* appendix.

The Toolkit also contains ***ADS EXCERPTS*** from relevant sections of
the ADS, lists of helpful resources, and practical examples. The tools,
techniques, and tips in this Toolkit provide a general approach to PMP
development and implementation. You can and should consider tailoring
Toolkit elements to meet the specific needs of the your program.

# Part 1: Plan for PMP Development

Developing a useful PMP requires the time and attention of the entire SO
team and many of your partners. Because it is a team effort, you will
save time and effort during the development process by first conducting
these four key preparation tasks:

## Task 1 -- Assemble a PMP development team

A team approach to PMP development will help facilitate a shared sense
of ownership among those who use the PMP, and bring creativity and
innovation to developing each PMP element. The first step in the process
is to designate a team leader. Designate leadership to one, or at most
two, team members. The team leader will ensure the coordination of team
meetings, collection and distribution of background materials,
facilitation of discussions, and documentation of the PMP.

Once the team leader is identified, assemble the PMP development team.
An effective team will have a balanced set of skills that include:

-   Knowledge of USAID "managing for results" approach

-   Experience in the relevant sector/sub-sector

-   Educational background/training in the relevant sector/sub-sector

-   In-depth knowledge of the target country and understanding of local
    conditions

-   General knowledge of USAID structures, processes and culture

-   Knowledge of performance measurement methodologies and best
    practices

-   Strong facilitation, analytical and report writing skills

In most cases, these skills will be available within the SO team. In
other cases, external help from another Operating Unit or contractor
will supplement the existing skill sets. When you bring in external
help, it is critical to maintain a team approach to PMP development.
This means that external team members help fill in gaps in the PMP
development team skill set, rather than duplicate skills that already
exist on the SO team. This will ensure that the best possible PMP is
produced at the conclusion of the exercise.

![](media/image8.wmf)

**WORKSHEET 1 - PMP Development Team Skills Matrix:** Use Worksheet 1 to
help assemble the PMP development team. Table 1-1 presents a sample
matrix to help you assess if you have assembled the team that you need.

**Table 1-1. PMP Development Teams Skills Matrix**

  ------------ ------------ ------------ -------------- ------------- ------------ ------------ ------------- ---------------
  **Name**     **Role**     **Knows      **Has sector   **Has sector  **Knows      **Knows      **Knows PM    **Has
                            USAID MFR    experience**   training or   local        USAID        methods and   facilitation,
                            approach**                  education**   conditions   structure,   best          analytical and
                                                                      in-depth**   processes,   practices**   report writing
                                                                                   culture**                  skills**

  1\. Jane     Team Leader                                                                               
  Smith                                                                                                       

  2\. Sara     Data                                                                                        
  Jones        collection                                                                                     

  3\. Fred     Data                                                                                       
  Wilson       collection                                                                                     

  4\. Kate     Data                                                                                       
  Thomas       analysis                                                                                       

  5\. T.       Data                                                                                        
  Consultant   analysis                                                                                       
  ------------ ------------ ------------ -------------- ------------- ------------ ------------ ------------- ---------------

## Task 2 -- Collect and review background materials

You may want to gather and review some of the materials listed in Table
1-2. Having these materials available from the beginning will save time
during the PMP development process.

+---------------+-----------------------------------+------------------+
| **Table 1-2.  |                                   |                  |
| Other         |                                   |                  |
| Resources for |                                   |                  |
| PMP           |                                   |                  |
| Preparation   |                                   |                  |
| (See Helpful  |                                   |                  |
| Resources     |                                   |                  |
| appendix for  |                                   |                  |
| resource      |                                   |                  |
| locations)**  |                                   |                  |
+---------------+-----------------------------------+------------------+
| **Agency      | **Operating Unit Specific         | **Background     |
| Guidance**    | Materials**                       | Materials**      |
+---------------+-----------------------------------+------------------+
| -   ADS       | -   Most recent Country Strategic | -   Lists of     |
|     Chapters  |     Plan (CSP)                    |     indicators   |
|     200-203   |                                   |     from Annual  |
|               | -   Most recent Annual Report     |     Report data  |
| -   TIPS      |                                   |     base or      |
|     Series    | -   Relevant diagnostic studies,  |     SO-relevant  |
|     (titles   |     assessments or evaluations    |     handbook     |
|     a         |     relating to the SO            |                  |
| bbreviated):\ |                                   | -   GAO, 1998,   |
|     6:        | -   Preliminary PMP or Draft PMP  |     The Results  |
|               |                                   |     Act -- An    |
|   Performance | -   Information/reports prepared  |     Evaluator's  |
|               |     by partners                   |     Guide to     |
|   Indicators\ |                                   |     Assessing    |
|     7: The    | -   Strategic Plan approval cable |     Agency       |
|     PMP\      |     from USAID/W                  |     Annual       |
|     8:        |                                   |     Performance  |
|               | -   Background information on     |     Plans        |
|   Performance |     likely data sources           |                  |
|     Targets\  |                                   | -   Harry Hatry, |
|     12:       | -   Other performance management  |     1999,        |
|     Indicator |     information, e.g., IG         |     Performance  |
|     and Data  |     reports, data quality         |     Measurement  |
|     Quality\  |     assessments, guidance         |     -- Getting   |
|     13:       |     specific to the Operating     |     Results      |
|     Results   |     Unit.                         |                  |
|     Framework |                                   |                  |
+---------------+-----------------------------------+------------------+

If your SO contains elements of policy reform or institutional capacity
building, you may want to include TIPS 14: Monitoring the Policy Reform
Process or TIPS 15: Building Institutional Capacity, with your
background materials. (RESOURCES provides their location).

![](media/image9.png)

> **INFORMATION SERVICES:**

-   **Economic and Social Data Service (ESDS):** ESDS staff specialize
    in selecting the most appropriate quantitative data for specific
    research purposes. Access ESDS via CDIE Online at
    [http://cdie.usaid.gov](http://cdie.usaid.gov/) (accessible only
    within the USAID firewall; click \'Statistics\' at the top of the
    homepage)

```{=html}
<!-- -->
```
-   **Research and Reference Services (R&RS):** R&RS staff help
    development practitioners clarify their information needs after
    which they identify, analyze and deliver appropriate information in
    a useful form. R&RS also manages the USAID Library and Learning
    Resources Center. Access R&RS via CDIE Online at
    [http://cdie.usaid.gov](http://cdie.usaid.gov/) (available only
    within the USAID firewall; click \'Research\' or \'Library\' at the
    top of the homepage)

## Task 3 -- Develop a workplan 

![](media/image10.wmf)

**WORKSHEET 2 -- PMP Development Workplan:** As you read through this
task, refer to Worksheet 2 to help assemble the PMP development
workplan. Project management software, such as MS Project, can also be
used to develop the PMP workplan.

A workplan will help guide the team through the PMP development process.
The workplan should document what needs to be done, due dates, and
persons responsible for achieving each task. It is usually prepared by
the team leader and one/two other team members. Developing a detailed
work plan involves the following steps:

-   **Identify the major PMP development tasks:** Often, the tasks of
    PMP development are defined by major deliverables that you will
    produce or major tasks that contribute to a deliverable. For
    example, major tasks may include:

-   Review Results Framework

-   Develop indicators and collection methods

-   Collect data and verify quality

-   Establish baselines and targets

-   Plan for other assessing and learning elements

-   Assemble draft PMP

-   Review and revise the PMP

-   **Identify the sub-tasks:** For each major task, identify the
    sub-tasks you will need to accomplish. For example, within \"Plan
    for other assessing and learning elements\" the sub-tasks may be:

-   Plan for data analysis and use

-   Plan for performance reviews,

-   Plan for evaluations and special studies, etc.

```{=html}
<!-- -->
```
-   **Estimate the time duration and the appropriate timing of each task
    or sub-task:** Determine how long each sub-task will take to
    complete and when the task should commence. Base the projected
    duration of that task on reasonable expectations, weighing the
    workload and other responsibilities of the team against the need to
    assemble the plan in a reasonable timeframe. Make sure to leave
    adequate time for management reviews and check to be sure that
    outside reviewers are available during the time you want them to
    review the document. Also plan for the time needed to handle
    administrative matters such as contracting for services, etc.\
    \
    Note, too, that in most cases developing a PMP from start to finish
    may not be possible over a period of consecutive days, and thus may
    have to be done in phases. For example, you may decide to proceed
    with reviewing the Results Framework and developing the performance
    indicators soon after the Strategic Plan has been approved. However,
    you may recognize that it would be better to collect baseline data
    and establish targets once a new contractor is in place in a couple
    of months. In this case the timing of these events should be
    reflected in the workplan.

```{=html}
<!-- -->
```
-   **Assign resources to each sub-task:** Identify which team members
    will work on each sub-task and any financial resources that will be
    needed to complete it.

-   **Validate the workplan with the SO team:** Usually the workplan is
    developed by the team leader and one/two other team members. At this
    point, review the workplan informally with all SO team members to
    verify that the assumptions regarding tasks, timing and
    responsibilities are reasonable. Revise the workplan as needed.

-   **Complete any contracting plans as needed:** Lastly, if you need to
    contract out for services to help develop the PMP, prepare a
    detailed scope of work specifying the final product, schedule, skill
    needs and expectations. Complete the contracting process in time for
    additional team members to be present at the Team Briefing Meeting
    (see Task 4).

## Task 4 -- Conduct team briefing meeting

Once the first three steps are completed, the team leader usually
conducts a team briefing meeting to get the team on the same page,
particularly in terms of the final product, the workplan, and assigned
responsibilities. Focus on explaining the team leader's expectations and
clarifying team members concerns and questions. Also take the
opportunity to introduce team members not familiar with one another. The
team leader should bring copies of the materials to be discussed at the
meeting. Figure 1-1 presents elements of a meeting agenda.

# Part 2: Develop the PMP

Part 2 of the Toolkit focuses on how to develop a comprehensive PMP.
Each of the six tasks below have a number of sub-tasks associated with
them. These will be explored in detail in the following pages.

Before discussing the PMP development process in detail, keep in mind
some guiding principles for PMP development.

![](media/image6.png){width="1.1in" height="1.1in"}

> **CONSIDER THIS -- Guiding Principles of PMP Development:**

-   **The PMP is the foundation for a sound performance management
    system.** A good PMP is a useful tool for management and
    organizational learning - it provides intelligence for
    decision-makers, and thus serves as a constant desk reference to
    guide the assessment of results. A good PMP is updated annually to
    ensure maximum use for decision-making. The PMP is NOT something
    developed only to satisfy Washington and then left to collect dust.

```{=html}
<!-- -->
```
-   **An effective monitoring system yields performance information that
    helps "tell your story" better.** Your ability to communicate the
    achievement of development results and share lessons learned is
    dependent on your ability to collect useful performance information.

-   **Performance indicators are the basis of the PMP.** Effective
    performance monitoring starts with indicators that are direct,
    objective, practical, and adequate. Indicators are useful for timely
    management decisions and credibly reflect the actual performance of
    USAID-sponsored activities.

-   **Performance monitoring is based on access to and use of data that
    is of reasonable quality given the reality of the situation**. Your
    team's management decisions should be based on data that is
    reasonably valid, reliable, and timely. Good performance monitoring
    systems include regular data quality assessments.

-   **A good PMP helps the SO team focus on what\'s important.** The PMP
    provides the conceptual framework around which the SO team
    prioritizes and carries out its work. A PMP helps clearly assign
    accountability for results. It also outlines the mechanisms through
    which these results are shared both internally with employees and
    externally with partners and other stakeholders.

-   **Effective leadership makes for a smoother process.** Every SO team
    member is responsible for the success of the SO. However, teams who
    work in Operating Units where the leaders of the unit (e.g., mission
    directors, SO team leaders, program officers) agree on the
    importance of a sound performance management system, and demonstrate
    its value by using it, will generally be more enthusiastic about
    participating in the process and using the information that results
    from it. Effective leadership also means creating a learning
    environment for results reporting that is seen by the staff as
    positive, not punitive.

![](media/image6.png){width="1.1in" height="1.1in"}

**CONSIDER THIS -- Involve Customers, Partners and Stakeholders:** Your
customers, stakeholders, and partners will play an important role in
performance management. As such, their participation deserves special
mention as a guiding principal. Where appropriate, you should:

-   Include stakeholders when developing Performance Management Plans
    and collecting, interpreting, and sharing information and experience

```{=html}
<!-- -->
```
-   Communicate Results Framework indicators to implementing partners
    and explain how their performance data feeds into the goals and
    objectives of the Operating Unit.

```{=html}
<!-- -->
```
-   Encourage implementing partners to use common definitions and
    descriptors of performance indicators

```{=html}
<!-- -->
```
-   Consider the special information needs of partners. Wherever
    feasible, integrate your performance monitoring and evaluation
    activities with similar processes of your partners

-   Help partners develop their own performance monitoring and
    evaluation capacity

-   Consider the financial and technical assistance resources needed to
    ensure stakeholder participation in performance monitoring and
    evaluation

## Task 1 - Review Results Framework

The purpose of this task *is not* to develop a Results Framework from
scratch. This would already have been done during the development of the
Strategic Plan. In preparation for a PMP, however, it is often useful to
conduct a quick review of Results Framework to validate the implied
cause-effect relationships and ensure that all results are within
USAID's manageable interest. The process of reviewing results statements
can be broken down into four sub-steps.

### ![](media/image11.wmf) 

### 

### 

### Key Definitions, ADS Guidance and Helpful Resources

> ![](media/image3.wmf){width="0.9in"
> height="0.9868055555555556in"}**KEY DEFINITIONS**: The following
> definitions are relevant to this PMP development task:

-   **Result:** A significant, intended, and measurable change in the
    condition of a customer, or a change in the host country,
    institutions or other entities that will affect the customer
    directly or indirectly. Results are typically broader than
    USAID-funded outputs, and required support from other donors and
    partners not within USAID's control.

-   **Results Framework:** A planning, communications and management
    tool, which conveys the development hypothesis implicit in the
    strategy and the cause-effect linkages between the Intermediate
    Results (IR) and the Strategic Objective (SO). A Results Framework
    includes the SO and the IRs necessary to achieve the SO, whether
    funded by USAID or its partners. It includes any critical
    assumptions that should hold for the development hypothesis to lead
    to achieving the relevant objective. Typically, it is laid out in
    graphic form supplemented by a narrative.

-   **Strategic Objective (SO):** The most ambitious result that a USAID
    Operating Unit, along with its partners, can materially affect, and
    for which it is willing to be held accountable within the time
    period of the Strategic Objective. SOs can be designed for an
    Operating Unit to provide analytic, technical, logistical, or other
    types of support to the SOs of other Operating Units (whether
    bilateral, multi-country, or global in nature)

```{=html}
<!-- -->
```
-   **Intermediate Result (IR):** An important result that is seen as an
    essential step to achieving a Strategic Objective. IRs are
    measurable results that may capture a number of discrete and more
    specific results.

-   **Development Hypothesis:** A narrative description of the specific
    causal linkages between IRs and a SO that are expected to lead to
    the achievement of the SO. They hypothesis is based on sound
    development theory, knowledge and experience within the context of a
    specific SO. Generally, the term refers to plausible linkages and
    not statistically accurate relationships

-   **Manageable Interest:** The concept of manageable interest
    recognizes that achievement of results requires joint action on the
    part of many other actors such as host country governments,
    institutions, other donors, civil society, and the private sector.
    When an objective is within USAID's manageable interest, it means we
    have reason to believe that our ability to influence, organize, and
    support others around commonly shared goals can lead to the
    achievement of desired results, and that the probability of success
    is high enough to warrant expending program and staff resources. A
    result is within an entity's manageable interest when there is
    sufficient reason to believe that its achievement can be
    significantly and critically influenced by interventions of that
    entity.

+-----------------------------------------------------------------------+
| **ADS Guidance for Results Frameworks (ADS 201.3.7.3)**               |
|                                                                       |
| A Results Framework should:                                           |
|                                                                       |
| -   Succinctly capture the key elements of the development hypothesis |
|     for achieving a SO (program intent and content)                   |
|                                                                       |
| -   Clearly state results at SO and IR levels                         |
|                                                                       |
| -   Provide sufficient detail and causal linkages to diagram the      |
|     development hypothesis                                            |
|                                                                       |
| -   Use results statements that are measurable and feasible during    |
|     the planned life of the SO given anticipated resource levels      |
|                                                                       |
| -   Propose a preliminary PMP                                         |
|                                                                       |
| -   Incorporate critical assumptions                                  |
|                                                                       |
| -   Identify any results to be achieved through other SOs that would  |
|     significantly support achievement of IRs in the Results           |
|     Framework.                                                        |
+-----------------------------------------------------------------------+

**Helpful Resources to Learn More about Reviewing Results Frameworks**

-   ADS Chapter 201 - Planning
    ([www.usaid.gov/pubs/ads/200/](http://www.usaid.gov/pubs/ads/200/))

-   TIPS13: Building a Results Framework
    ([www.dec.org/pdf_docs/PNACA947.pdf](http://www.dec.org/pdf_docs/PNACA947.pdf))

-   Handbook of Democracy and Governance Program Indicators
    ([www.dec.org/pdf_docs/PNACC390.pdf](http://www.dec.org/pdf_docs/PNACC390.pdf))

Ideally, all of the tasks below can be best accomplished through a
facilitated session of your entire SO team. The techniques and tools to
help guide you through this process are:

![](media/image12.png){width="1.1in" height="1.0in"}

**TECHNIQUE -- Assess Results Framework in a Facilitated Session:**
Conducting the quality assessment of a Results Framework in a
facilitated session of your entire team ensures that all team members
understand the logic and reasoning behind each result statement.
***Helpful Hint 1: Facilitating Group Discussions and Decision-Making***
provides some tips and techniques for facilitating group sessions.

![](media/image13.wmf)

> **WORKSHEETS 3 and 4-- Results Statements Assessment** and **Results
> Framework Assessment:** Refer to Worksheets 3 and 4 as you read
> through each of the tasks below. These worksheets can be useful for
> guiding the team through the facilitated discussion of each Results
> Framework element.

### 1.1 Assess quality of results statements 

Good performance indicators start with good results statements that
people can understand and agree on. Therefore, begin the PMP development
process by reviewing SOs and IRs to make sure that each individual
results statement is of good quality. Using Worksheet 3, you can
determine if your results statement is:

-   Measurable and objectively verifiable

-   Meaningful and realistic

```{=html}
<!-- -->
```
-   Focused on USAID's strategic commitments

```{=html}
<!-- -->
```
-   Customer or stakeholder driven

-   Can be materially affected by the Operating Unit and its partners
    (within manageable interest)

-   Statement of results -- not an activity, process or output

-   Uni-dimensional -- not a combination of results

![](media/image6.png){width="1.1in" height="1.1in"}

> **CONSIDER THIS -- Rules of Thumb for Results Statement Review**: Some
> rules of thumb to keep in mind when reviewing results statements are:

-   **Avoid overly broad statements***.* Which specific aspects of the
    result will program activities emphasize?

-   **State results as future completed actions.** The statement should
    describe the end state that is desired as result of the activity.

-   **Use strong action verbs.** This makes results easier to implement.
    Examples of strong action verbs are: constructed, eradicated, or
    reduced. Examples of weak action verbs are: enhanced, liaised, or
    coordinated.

-   **Be clear about what type of change is implied.** What is expected
    to change \-- a situation, a condition, the level of knowledge, an
    attitude, or a behavior? Is the expected change an absolute change,
    relative change, or no change?

-   **Identify more precisely the specific targets for change.** Who or
    what are the specific targets for the change? Is change expected to
    occur among individuals, families, groups, communities, regions?

-   **Study the activities and strategies directed at achieving
    change.** Is the expected relationship between activities and their
    intended results direct or indirect?

### 1.2 Validate logic 

##### Causality

The linkages up the Results Framework should be **causal** \--
achievement of one result is necessary for, and contributes to,
achievement of the other. The causal connection between two IRs or
between an IR and a SO in a Results Framework should also be **direct**.
You should not need to infer additional IRs to understand the linkage
between two results, or accept many or broad assumptions to move from a
"lower" result to a "higher" result or SO.

Figure 2-1 illustrates this requirement for causal linkages within the
Results Framework.

![](media/image14.wmf)

As you move up the framework, you should be able to answer the questions
"w*hy* does this matter?" and "*So what* if we do this?" The result that
emerges in response to those questions is the SO statement.

As you move down the framework, you should be able to answer the
question "*how* do we cause this effect?" This question does *not* refer
to the activities needed to achieve the result but to other Intermediate
Results needed to achieve the higher-level result.

##### 

A problem that occurs commonly in results frameworks is that they are
*not* set up with a causal logic. Instead, the relationship between the
results is set up to be categorical, definitional or chronological. The
next few paragraphs illustrate these three common logical flaws with
examples and shows how they can be rectified.

In the first example, below, what is the difference between the SO-level
result and the Intermediate Result? If your IR is a restatement or a
clarification of your SO, then your IR is probably acting as a
*definition* of another level. Instead, you want the lower level
result(s) to cause the achievement of the higher-level result.

In the second example, below, the Intermediate Results are
sub-categories of the SO-level result. As yourself how you would measure
each of the four results statements on the left side. For each result
statement, you would be measuring an aspect of "effective management",
but with a different kind of organization. If it sounds like the same
indicator (just disaggregated for each result), then the levels of your
results frameworks are probably *categories* of the same result.
Instead, you want the lower levels of the results framework to *cause*
the achievement of a higher-level result.

![](media/image15.png){width="4.319444444444445in"
height="3.2506944444444446in"}

The third example, below, does not make it clear what specific problems
or constraints the planners are trying to address to get to the key IR.
The two sub-IRs are important steps in the process of arriving at IR 1,
but they do not describe results that *cause* the IR 1. Although the
lower level results may be more linked to specific activities, be
careful that your results framework does not represent a *chronological
workplan* of activities (first do step one, then do step two, then...).
Instead, you want the lower levels of the results framework to *cause*
the achievement of a higher-level result.

![](media/image16.png){width="4.342361111111111in" height="3.2625in"}

##### USAID partner results 

As one moves across the IRs and sub-IRs of the framework, one should be
able to answer the question "*what else* is required to achieve the
above result?" The logic of the development hypothesis is strengthened
when the Intermediate Results of other USAID partners have been
considered. However, while it is essential to consider USAID partner
results, it is not necessary that all of the Intermediate Results be
reflected in the Results Framework itself. In fact, only the results
that are most relevant and critical to achieving the SO may be included.

However, since SO teams are encouraged to document USAID partner results
in the accompanying narrative to the Results Framework, you may want to
refer to this documentation if there are any questions as to whether all
of the critical results have been considered. In general, the more
thoroughly and specifically the contributing Intermediate Results are
determined, the stronger the logic of the hypothesis and the greater the
chances of being able to manage activities for the achievement of the
Strategic Objective.

### 1.3 Verify results are within USAID's manageable interest 

[]{.mark}

The ADS list of definitions (200.6.B) states that a result is within an
entity's manageable interest "when there is sufficient reason to believe
that its achievement can be significantly and critically influenced by
interventions of that entity." Even though USAID recognizes that
development outcomes require joint action on the part of many other
actors than itself (e.g., host country governments, other donors, civil
society), the Agency seeks to ensure that the objectives it sets for
itself are within its own manageable interest. Manageable interest gives
USAID reason to believe that its ability to influence, organize, and
support others around commonly shared goals can lead to the achievement
of desired results, and that the probability of success is high enough
to warrant expending program and staff resources.

Verifying that all results within an Operating Unit's Results Framework
are within the Agency's manageable interest holds managers accountable
for the results stated in the Results Framework. Good results statements
will reflect a realistic level of accountability. The SO is the highest
result that the Operating Unit, working with its partners, can expect to
materially affect and for which it is willing to be held accountable.

  ------------------------------------ ----------------------------------
  **Poor Examples**                    **Good Examples**

  SO: Broad-based sustainable economic SO: Increased employment in the
  growth                               formal, off-farm private sector

  SO: Reduced population growth        SO: Reduced fertility
  ------------------------------------ ----------------------------------

As earlier discussed, the causal connections between the IRs and SO
should also be reasonable.

+-----------------------------------+----------------------------------+
| **Poor Examples**                 | **Good Examples**                |
+-----------------------------------+----------------------------------+
| SO: Increased use of modern       | SO: Increased use of modern      |
| contraception                     | contraception                    |
|                                   |                                  |
| IR: Improved training of health   | IR: Increased availability of    |
| care providers                    | contraceptive services and       |
|                                   | commodities                      |
+-----------------------------------+----------------------------------+
| SO: Increased off-farm employment | SO: Increased-off farm           |
|                                   | employment                       |
| IR: Increased citizen's skills    |                                  |
| for private sector development    | IR: Increased number of formal   |
|                                   | private sector enterprises       |
+-----------------------------------+----------------------------------+

### 1.4 Ensure critical assumptions are identified

Every result involves risks (e.g., events, conditions, or decisions)
that could cause it to fail. Risks outside the direct control of the SO
team are called assumptions.

A *critical assumption* is defined as a general condition under which
the development hypothesis or strategy for achieving the objective will
hold true. Assumptions complete the "if/then" logic (describing the
necessary conditions between each level) by adding the "if/AND/then"
logic (describing the necessary and sufficient conditions between each
level). You can determine the assumptions by asking the question "what
conditions should exist in addition to my objectives in order to achieve
the next level?'

![](media/image6.png){width="1.1in" height="1.1in"}

**CONSIDER THIS -- Importance of Clarifying Critical Assumptions:** As
you identify critical assumptions that underpin your development
hypothesis, keep in mind why clarifying critical assumptions is
important to the PMP development process:

-   Identifies what is beyond the program/activity manager's control

-   Provides shared judgment of the probability of success

-   Promotes realistic program and activity design

-   Improves communication between the program/activity manager and the
    SO team

At the planning stage, assumptions help identify risks that can be
avoided by incorporating additional components into the Results
Framework itself. In activity execution, assumptions indicate the
factors that SO teams should anticipate, try to influence, and develop
contingency plans for in case things go wrong.

Do not confuse critical assumptions that are outside the control of
USAID and its partners, with results. Critical assumptions reflect
conditions likely to affect the achievement of results in the Results
Framework - such as the level of political commitment to women's
empowerment or the openness of export markets. A critical assumption
differs from an IR in that the IR represents a focused and discrete
outcome that specifically contributes to the achievement of the SO.

Be realistic when identifying critical assumptions and avoid defining
critical assumptions that have a comparatively low chance of holding
over the duration of the Strategic Plan. For example, an assumption such
as no outbreak of war is surely an important condition, however, it does
not help the design of the Results Framework. If a war were to occur, it
is probably self evident that the program would suffer.

Finally, recognize that critical **assumptions can be found at every
level** within the Results Framework. And remember that they should be
**continuously monitored** to ensure that development hypothesis is
built around valid assumptions.

## Task 2 -- Develop Performance Indicators

To manage for results, you need reliable and timely data. Performance
indicators specify the data that the SO team will collect in order to
measure program progress and compare actual results over time against
what was planned. Performance indicators can also be used to:

-   Orient and motivate operating unit staff toward achieving results

-   Better communicate USAID achievements to host country counterparts,
    other partners and customers

-   Clearly and consistently report results achieved to USAID
    stakeholders, including Congress, Office of Management and Budget,
    and citizens.

The process of developing performance indicators can be broken down into
four sub-steps.

### ![](media/image17.wmf){width="5.725694444444445in" height="1.5034722222222223in"}

### 

### Key Definitions, ADS Guidance and Helpful Resources

> ![](media/image3.wmf){width="0.9in"
> height="0.9868055555555556in"}**KEY DEFINITIONS**: The following
> definitions are relevant to this PMP task:

-   **Performance Indicator:** A particular characteristic or dimension
    used to measure intended changes defined by a Results Framework.
    Performance indicators are used to observe progress and to measure
    actual results compared to expected results. Performance indicators
    help to answer "how" and "if" an Operating Unit or SO team is
    progressing towards its objective, rather than "why" such progress
    is or is not being made. Performance indicators may measure
    performance at any level of a Results Framework.

```{=html}
<!-- -->
```
-   **Outcome:** A result sought by USAID. In ADS Chapters 200-203, the
    term "outcome" is equivalent to "result"

-   **Output:** A tangible, immediate and intended product or
    consequence of an activity within USAID's control. Examples of
    outputs include people fed, personnel trained, better technologies
    developed and new construction. Deliverables included in contracts
    will generally be considered outputs, as will tangible products and
    consequences of USAID grantees.

+-----------------------------------------------------------------------+
| **Characteristics of Good Performance Indicators (ADS203.3.4.2)**     |
|                                                                       |
| Good performance indicators generally exhibit the following           |
| characteristics:                                                      |
|                                                                       |
| -   **Direct:** An indicator should closely track the result it is    |
|     intended to measure. When direct indicators cannot be used        |
|     because of costs or other factors, a reasonable proxy indicator   |
|     maybe used.                                                       |
|                                                                       |
| -   **Objective:** Objective indicators are operationally precise and |
|     uni-dimensional. They should be unambiguous about what is being   |
|     measured and what data are being collected.                       |
|                                                                       |
| -   **Useful for Management:** Indicators should be useful for        |
|     management purposes at relevant levels of decision making         |
|                                                                       |
| -   **Practical:** An indicator is practical if data can be obtained  |
|     in a timely way and at reasonable cost.                           |
|                                                                       |
| -   **Attributable to USAID:** Performance indicators should measure  |
|     change that is clearly and reasonably attributable, at least in   |
|     part, to the efforts of USAID. That is, indicators should         |
|     credibly reflect the actual *performance* of the Strategic Plan.  |
|                                                                       |
| -   **Timely:** Performance data should be available when they are    |
|     needed to make decisions.                                         |
|                                                                       |
| -   **Adequate:** Taken as a group, a performance indicator and its   |
|     companion indicators should be the minimum necessary to ensure    |
|     that progress toward the given results is sufficiently captured.  |
+-----------------------------------------------------------------------+

**Helpful Resources to Learn More about Developing Performance
Indicators**

-   ADS Chapter 203,
    [www.usaid.gov/pubs/ads/200/200.pdf](http://www.usaid.gov/pubs/ads/200/200.pdf)

-   TIPS 6: Selecting Performance Indicators,
    [www.dec.org/pdf_docs/pnaby214.pdf](http://www.dec.org/pdf_docs/pnaby214.pdf)

-   TIPS 12: Guidelines for Indicator and Data Quality,
    [www.dec.org/pdf_docs/pnaca927.pdf](http://www.dec.org/pdf_docs/pnaca927.pdf)

-   TIPS 14: Monitoring the Policy Reform Process,
    [www.dec.org/pdf_docs/pnaca949.pdf](http://www.dec.org/pdf_docs/pnaca949.pdf)

-   TIPS 15: Measuring Institutional Capacity,
    [www.dec.org/pdf_docs/pnacn612.pdf](http://www.dec.org/pdf_docs/pnacn612.pdf)
    and
    [www.dec.org/pdf_docs/pnacg624.pdf](http://www.dec.org/pdf_docs/pnacg624.pdf)

-   Handbook of Democracy and Governance Program Indicators,
    [www.dec.org/pdf_docs/PNACC390.pdf](http://www.dec.org/pdf_docs/PNACC390.pdf)

### 2.1 Develop list of potential indicators

Each result usually has many possible indicators, some of which will be
more appropriate and useful than others. Start with a list of potential
indicators and then narrow down the list to a final set based on a set
of criteria.

![](media/image12.png){width="1.1in" height="1.0in"}**TECHNIQUE -- Use
Current Resources to Identify Potential Indicators:** Tap information
from some of these resources to help identify potential indicators.

-   Your portfolio of activities

-   USAID Annual Report Database
    ([www.dec.org/partners/ardb/](http://www.dec.org/partners/ardb/))

-   USAID sector expertise (Pillar and Regional Bureaus; other Operating
    Units)

-   Internal brainstorming by SO team

-   Experience of other Operating Units with similar indicators

-   External sector/regional experts

-   Handbooks of sector indicators

-   The Internet (for indicators used by other organizations)

The key to creating a useful list of potential indicators is to view the
desired result in all its aspects and from all perspectives. Another
best practice is to use a participatory approach in selecting
performance indicators. Collaborating closely with development partners,
host country counterparts, and customers at each step of the indicator
selection process has many benefits. It makes good sense to draw on the
experience of others and obtain their consensus throughout the process.

### 2.2 Assess potential indicators

Once the list of potential indicators is developed, assess each
potential indicator. The quality of a potential indicator can be
assessed to determine if it is:

-   Direct

-   Objective

-   Useful for Management

-   Practical

-   Attributable to USAID and its partners

-   Timely

```{=html}
<!-- -->
```
-   Adequate

![](media/image18.wmf)

**WORKSHEET 5 -- Performance Indicator Quality Assessment:** In the
following discussion, use Worksheet 5 to help document your assessment
of each potential indicator.

**DIRECT:** The performance indicator should closely track the result it
is intended to measure.

+-----------------------------------+----------------------------------+
| **Poor Example (Direct)**         | **Good Example (Direct)**        |
+-----------------------------------+----------------------------------+
| Result: Increased conservation of | Result: Increased transfer of    |
| natural habitats                  | environmentally sustainable      |
|                                   | farming practices                |
| Indicator: Number of park         |                                  |
| visitors                          | Indicator: Number of farmers     |
|                                   | using X number of specific       |
|                                   | environmentally sustainable      |
|                                   | practices                        |
+-----------------------------------+----------------------------------+

If direct indicators are not feasible, then use credible proxy measures.
**Proxy indicators** are an indirect measure of a given result.

+-----------------------------------+----------------------------------+
| **Poor Example (Proxy             | **Good Example (Proxy            |
| Indicator)**                      | Indicator)**                     |
+===================================+==================================+
| Result: Increased conservation of | Result: Increased transfer of    |
| natural habitats                  | environmentally sustainable      |
|                                   | farming practices                |
| Direct Indicator: Number of park  |                                  |
| visitors                          | Direct Indicator: Percent of     |
|                                   | farmers using X number of        |
| Proxy Indicator: Percent of park  | specific environmentally         |
| costs met from private sources    | sustainable practices            |
|                                   |                                  |
|                                   | Proxy Indicator: Number of       |
|                                   | farmers *trained* to use X       |
|                                   | number of specific               |
|                                   | environmentally sustainable      |
|                                   | practices.                       |
+-----------------------------------+----------------------------------+

**OBJECTIVE:** Performance indicators should be unambiguous about what
is being measured. Thus, it should be uni-dimensional (should measure
only *one* phenomenon at a time) and be precisely defined in the PMP.

+-----------------------------------+-----------------------------------+
| **Example (Precisely Defined)**   | **Example (Uni-dimensional)**     |
+-----------------------------------+-----------------------------------+
| Result: Improved performance of   | Result: Improved access to basic  |
| export firms                      | education                         |
|                                   |                                   |
| Precise Indicator: % of export    | Uni-dimensional Indicator:        |
| firms experiencing an annual      | Primary school enrollment         |
| increase in revenues of at least  |                                   |
| 5%                                | Multi-dimensional Indicator:      |
|                                   | Primary school enrollment and     |
| Imprecise Indicator: Number of    | literacy rates                    |
| successful export firms           |                                   |
+-----------------------------------+-----------------------------------+

**USEFUL FOR MANAGEMENT:** Performance indicators should be useful for
the relevant level of decision-making (at the SO team, Operating Unit or
Agency level). Avoid collecting and reporting information that is not
used to support program management decisions. As discussed in ADS
203.3.2, Operating Units usually have varying information needs for
decision-making depending on where the Operating Unit is in implanting
the SO.

![](media/image19.png){width="4.0125in" height="2.997916666666667in"}

**PRACTICAL:** Operating Units should select performance indicators for
which data can be obtained at reasonable cost and in a timely fashion.
Data that are very costly to procure are of limited use. Data collection
costs, in terms of both human and financial resources, is an important
consideration. In general, the cost of collecting data for an indicator
should not exceed the management utility of the data. A good rule of
thumb is that costs to an Operating Unit for performance monitoring and
evaluations should normally range between three to ten percent of the
total budget for the Strategic Objective (see ADS 203.3.2.2).

**ATTRIBUTABLE TO USAID AND ITS DEVELOPMENT PARTNERS:** Performance
indicators included in the PMP should measure changes that are clearly
and reasonably attributable, at least in part, to USAID efforts.
Attribution exists when the outputs produced by USAID financed
activities have a logical and causal effect on the results being
measured by the performance indicator. Attribution is based upon a solid
and credible development hypothesis that is reflected in the Results
Framework, combined with a strong causal link between outputs of
activities and the Intermediate Results measured.

+-----------------------------------------------------------------------+
| **Indicators Attributable to USAID**                                  |
+-----------------------------------------------------------------------+
| A simple way to assess attribution is to ask the question:            |
|                                                                       |
| -   If there had been no USAID activity, would the result have been   |
|     different?                                                        |
|                                                                       |
| If the answer is "no," then there is likely an attribution issue and  |
| a more suitable indicator should be sought.                           |
+-----------------------------------------------------------------------+

**TIMELY:** Performance information should be available when it is
needed to make management decisions.

The necessary timeliness of the data really depends upon the nature of
the decision to be made though experience suggests that information for
managing activities should be available on a quarterly basis.

In some cases---humanitarian crises, for example---daily information may
be required. For most routine SO team level decisions, however, data
should be regularly available from performance monitoring systems. Even
data that are available on an annual basis may not be as useful for
addressing routine management issues as data that are available more
frequently.

Data that are collected infrequently (every 2-5 years), or with a
substantial lag time (longer than a year), may be useful for tracking
long-term trends and confirming the accuracy of lower-level data. Such
indicators should be supplemented with other performance indicators
(direct or proxy), which are able to reflect program performance on a
more frequent basis.

**ADEQUATE:** The PMP should contain only as many indicators as are
necessary and cost effective for management and reporting purposes.

Strike a balance between having too many indicators, which can increase
the cost of collecting, and analyzing the data, and too few indicators,
which could be insufficient to assess progress. The general rule of
thumb is two to three indicators per result, but this may differ
depending on the:

-   Complexity of the result being measured

```{=html}
<!-- -->
```
-   Level of resources available for monitoring performance

-   Amount of information needed to make reasonably confident decisions

+-----------------------------------+----------------------------------+
| **Poor Example (Adequate)**       | **Good Example (Adequate)**      |
+-----------------------------------+----------------------------------+
| Result: Increased use of child    | Result: Increased use of child   |
| survival services                 | survival services                |
|                                   |                                  |
| Indicator: DPT3 vaccination rate  | Indicator: DPT3 vaccination rate |
|                                   |                                  |
|                                   | Indicator: Oral rehydration      |
|                                   | therapy use rate                 |
|                                   |                                  |
|                                   | Indicator: Acute respiratory     |
|                                   | infection case management rate   |
+-----------------------------------+----------------------------------+

![](media/image6.png){width="1.1in" height="1.1in"}**CONSIDER THIS:
Reflecting Gender Considerations in Performance Indicators**

Men and women have different access to development programs and are
affected differently by USAID activities. USAID seeks to understand
these differences, to improve the efficiency and overall impact of its
programs, and to ensure that both women and men have equitable access to
development activities and their benefits.

Agency guidance (ADS 203.3.4.3) provides that performance management
systems and evaluations at the SO and IR levels SHOULD include
gender-sensitive indicators and sex-disaggregated data when the
technical analyses supporting the SO, the IRs or the activities
demonstrate that:

-   The activities or their anticipated results involve or affect women
    and men differently

-   This difference is potentially significant for managing towards
    sustainable program impact.

+-----------------------------------+----------------------------------+
| **Poor Example (Disaggregated)**  | **Good Example (Disaggregated)** |
+-----------------------------------+----------------------------------+
| SO: Increase foreign exchange     | SO: Increased agricultural       |
| revenues                          | production                       |
|                                   |                                  |
| IR: Increased tourism receipts    | IR: Increased adoption of        |
|                                   | improved production technologies |
| -   Number of male tourists       |                                  |
|                                   | -   Number or % of male-headed   |
| -   Number of female tourists     |     farm households adopting     |
|                                   |     improved technology          |
|                                   |                                  |
|                                   | -   Number or % of female-headed |
|                                   |     farm household adopting      |
|                                   |     improved technologies        |
+-----------------------------------+----------------------------------+

If it is too difficult to track and report sex-disaggregated data, the
Agency encourages Operating Units to use performance indicators that may
assess gender impact indirectly.

Operating units should also consider gender when determining how data
will be collected. For example, using only men to collect data may not
yield an accurate data set if societal norms restrict social relations
between the sexes.

Table 2-1 demonstrates how indicator data can be disaggregated.

**Table 2-1. Data Disaggregation and Analysis, by Indicator**

+-----------------------------------+---+---+---+---+--------------------+
| **Performance Indicator:**        | > |   |   |   | **Additional       |
|                                   |   |   |   |   | Analysis by:**     |
|                                   | * |   |   |   |                    |
|                                   | * |   |   |   |                    |
|                                   | A |   |   |   |                    |
|                                   | n |   |   |   |                    |
|                                   | a |   |   |   |                    |
|                                   | l |   |   |   |                    |
|                                   | y |   |   |   |                    |
|                                   | z |   |   |   |                    |
|                                   | e |   |   |   |                    |
|                                   | > |   |   |   |                    |
|                                   |   |   |   |   |                    |
|                                   | b |   |   |   |                    |
|                                   | y |   |   |   |                    |
|                                   | : |   |   |   |                    |
|                                   | * |   |   |   |                    |
|                                   | * |   |   |   |                    |
+-----------------------------------+---+---+---+---+--------------------+
|                                   | > | > | > | > |                    |
|                                   |   |   |   |   |                    |
|                                   | * | * | * | * |                    |
|                                   | * | * | * | * |                    |
|                                   | A | A | G | Y |                    |
|                                   | g | c | e | o |                    |
|                                   | g | t | n | u |                    |
|                                   | r | i | d | t |                    |
|                                   | e | v | e | h |                    |
|                                   | g | i | r | > |                    |
|                                   | a | t | * |   |                    |
|                                   | t | y | * | / |                    |
|                                   | e | * |   | > |                    |
|                                   | * | * |   |   |                    |
|                                   | * |   |   | A |                    |
|                                   |   |   |   | d |                    |
|                                   |   |   |   | u |                    |
|                                   |   |   |   | l |                    |
|                                   |   |   |   | t |                    |
|                                   |   |   |   | * |                    |
|                                   |   |   |   | * |                    |
+-----------------------------------+---+---+---+---+--------------------+
| ***IR 3.1.1 Vulnerable            |   |   |   |   |                    |
| Communities Better Able to Meet   |   |   |   |   |                    |
| Their Own Needs***                |   |   |   |   |                    |
+-----------------------------------+---+---+---+---+--------------------+
| \- Value of external              | * | * |   |   | Cash, Material,    |
| contributions to community        | * | * |   |   | Labor, Source      |
| development projects, by source   |  |  |   |   |                    |
|                                   | * | * |   |   |                    |
|                                   | * | * |   |   |                    |
+-----------------------------------+---+---+---+---+--------------------+
| \- Number of community            | * | * |   |   | Project Type,      |
| development projects completed    | * | * |   |   | Region             |
|                                   |  |  |   |   |                    |
|                                   | * | * |   |   |                    |
|                                   | * | * |   |   |                    |
+-----------------------------------+---+---+---+---+--------------------+
| \- Number of direct beneficiaries | * | * | * | * | Intervention Type  |
| under IR 3.1.1                    | * | * | * | * |                    |
|                                   |  |  |  |  |                    |
|                                   | * | * | * | * |                    |
|                                   | * | * | * | * |                    |
+-----------------------------------+---+---+---+---+--------------------+

![](media/image6.png){width="1.1in" height="1.1in"}

**CONSIDER THIS -- Quantitative versus Qualitative Indicators:** The
concept of quantitative versus qualitative indicators has been a subject
of frequent discussion over the past few years. New Agency guidance
indicates a shift away from the approach that indicators should be
quantitative rather than qualitative. Because quantitative indicators
are numerical (e.g., number or percentage of dollar value, tonnage)
versus the descriptive, qualitative indicators (e.g., description of the
status of an intended result, analysis of documents, documented
observations, representative case descriptions), their numerical
precision tends to lead to more agreement on the interpretation of
results. However, qualitative indicators can supplement the numbers and
percentages with a richness of information that brings a program's
results to life.

Agency guidance (ADS 203.3.4.1) states that indicators may be
quantitative or qualitative. You are expected to select the type of
indicator that is most appropriate for the result being measured. If a
qualitative indicator is determined to be most appropriate, you should:

-   Clearly define each term used in the measure

-   Make sure to document all definitions

-   Find other ways (such as using rating scales) to minimize
    subjectivity

+-----------------------------------------------------------------------+
| **Quantitative vs. Qualitative Indicators**                           |
+-----------------------------------------------------------------------+
| When selecting indicators, ask yourself:                              |
|                                                                       |
| -   Can we get meaningful information by using quantitative           |
|     indicators?                                                       |
|                                                                       |
| -   Can we get objective, convincing information by using qualitative |
|     indicators?                                                       |
|                                                                       |
| -   Can we quantify our qualitative indicators without losing         |
|     important information?                                            |
|                                                                       |
| ```{=html}                                                            |
| <!-- -->                                                              |
| ```                                                                   |
| -   Do we need a mix of the two?                                      |
+-----------------------------------------------------------------------+

> ![](media/image12.png){width="1.1in" height="1.0in"}**TECHNIQUE -- Use
> Scales, Indexes, and Scorecards for Hard-to-Measure Results: *Helpful
> Hint 2: Indicators for Hard-to-Measure Results*** describes several
> methods that can be used to develop indicators that quantify complex
> results. The methods are: rating scales, milestone scales, indexes,
> and scorecards. These tools help to introduce a measure of objectivity
> to inherently subjective measures. When using scales, indexes, and
> scorecards, keep in mind validity and reliability. Figure 2.3 presents
> an example of rating system that converts a qualitative assessment to
> a quantitative indicator.

+-----------------------------------------------------------------------+
| **Figure 2-3. Transforming Qualitative Data into Quantitative         |
| Performance Measures**                                                |
|                                                                       |
| To measure an IR that emphasizes improvements in quality of maternal  |
| and child health services, USAID/Yemen devised a scale that           |
| transforms qualitative information about services into a rating       |
| system against which targets can be set:                              |
|                                                                       |
| 0 points = Service not offered                                        |
|                                                                       |
| 1 point = Offers routine antenatal care                               |
|                                                                       |
| 1 point = Offers recognition and appropriate management of high risk  |
| pregnancies                                                           |
|                                                                       |
| 1 point = Offers routine deliveries                                   |
|                                                                       |
| 1 point = Offers appropriate management of complicated deliveries     |
|                                                                       |
| 1 point = Offers post-partum care                                     |
|                                                                       |
| 1 point = Offers neonatal care                                        |
|                                                                       |
| Score: [Total actual service delivery points]{.underline}             |
|                                                                       |
| Total possible service delivery points                                |
|                                                                       |
| *(Adapted from TIPS 8 -- Establishing Performance Targets)*           |
+-----------------------------------------------------------------------+

Another interesting example of how to document progress towards results
is the approach used by USAID/Mozambique in the early 1990\'s. This
approach is presented in Figure 2.4.

Lastly, TIPS 14: Monitoring the Policy Reform Process
([www.dec.org/pdf_docs/pnaca949.pdf](http://www.dec.org/pdf_docs/pnaca949.pdf))
and TIPS 15: Measuring Institutional Capacity
([www.dec.org/pdf_docs/pnacn612.pdf](http://www.dec.org/pdf_docs/pnacn612.pdf))
provide detailed guidance for selecting indicators that measure progress
in the areas of policy reform and institutional capacity building.

+-----------------------------------------------------------------------+
| **Figure 2.4 Performance Monitoring Challenge: War-to-Peace           |
| Transition**                                                          |
|                                                                       |
| **USAID/Mozambique, 1992-1995**                                       |
|                                                                       |
| **Our Problem**                                                       |
|                                                                       |
| How to monitor an array of people-level changes across a vast         |
| territory, in the absence of transport and communications             |
| infrastructure, data sources, and security?                           |
|                                                                       |
| **Our Solution**                                                      |
|                                                                       |
| -   **Select a small number of sites** that can be visited safely     |
|                                                                       |
| (we chose six providing a range of "baseline" conditions:             |
| north/center/south; isolated/accessible; more or less affected by     |
| war/drought)                                                          |
|                                                                       |
| -   **Mobilize existing staff** to form a small team for each site    |
|     that crosses all technical/ sectoral lines; schedule regular      |
|     visits; arrange logistics so trips are short and easy as possible |
|                                                                       |
| > (3-person teams worked well; only 2 of the 3 ever needed to travel  |
| > at one time; Americans were balanced with FSN/TCN staff; tried not  |
| > to put two agriculture or two health staff on the same team;        |
| > quarterly visits worked best)                                       |
|                                                                       |
| -   **Develop a simple site visit guide**, covering all topics of     |
|     interest, to be used systematically by teams visiting all sites   |
|     to "tell the story"                                               |
|                                                                       |
| > (ours included (i) questions to ask a key informant about the       |
| > overall situation in the site, (ii) questions to ask a "typical"    |
| > person about his own circumstances/perceptions, and (iii)           |
| > observations for the monitors to make during each visit; together   |
| > these addressed food security, health and nutrition, crops,         |
| > access/transport, water, resettlement of displaced persons,         |
| > perceptions of security, openness of political activity, schools,   |
| > market development, land mines, demobilization/reintegration of     |
| > combatants, intention to vote, etc.)                                |
|                                                                       |
| -   **Take lots of photographs** \-- gave each team an easy-to-use    |
|     camera and lots of film/data disks; gave monitors ideas of what   |
|     to photograph - encouraged them not to be shy; developed          |
|     film/downloaded images and captioned them immediately after the   |
|     trip; used pictures in as many ways as possible                   |
|                                                                       |
| ("before and after" times series proved extremely important to our    |
| tracking, e.g. in one site a crossroads sign photographed quarterly   |
| was isolated and overgrown at the first visit but the center of a     |
| bustling truck/bus stop 18 months later, and our photographs showed   |
| every phase in this transformation; anything that strikes the         |
| monitor's eye should be snapped, our high-impact images included      |
| beaming women showing off voter registration cards, food-for-work     |
| crews clearing 15-year-old trees from the middle of long-disused      |
| roads, etc.)                                                          |
|                                                                       |
| -   **Insist on prompt reports** on each visit, but allow lots of     |
|     latitude for report format                                        |
|                                                                       |
| > (ours ranged from 12-page narratives to bulleted phrases in an      |
| > email, but all were useful; the key is to get the report within a   |
| > day of the team's return and circulate it widely; encourage         |
| > monitors to report statistics when they find them, but also to      |
| > report and use qualitative information, since "the plural of        |
| > anecdote is data")                                                  |
|                                                                       |
| -   **Organize occasional "all-Mission" meetings** to talk about the  |
|     trends and implications of what the teams are reporting from the  |
|     different sites                                                   |
|                                                                       |
| (such sessions proved crucial as the transition period advanced and   |
| decisions needed to be made about moving from emergency relief        |
| approaches to "development" approaches; they also gave us a head      |
| start on developing our post-transition Strategic Plan)               |
|                                                                       |
| The photographs and qualitative information we collected proved       |
| useful well beyond our Mission-level program management and results   |
| reporting. CDIE and other Agency units used our photographs and       |
| anecdotes to illustrate reports, evaluations, etc. We've now          |
| digitized the 1400-images and created a set of three CDs, retiring    |
| the photos while still accessing the images easily.                   |
|                                                                       |
| It didn't work perfectly. Some monitoring teams fell apart at various |
| points, especially as staff changed. Their performance varied         |
| considerably over the three years. It was difficult to shift gears    |
| from this approach to the formal "statistical" PMPs as we began our   |
| post-transition Strategic Plan in 1996. An investment in costly       |
| household surveys was needed to establish baselines. But overall this |
| proved to be a richly rewarding approach to monitoring.               |
|                                                                       |
| Prepared 3 October 2000; for more information contact Juliet Born at  |
| USAID/Mozambique (juborn@usaid.gov)                                   |
+-----------------------------------------------------------------------+

###  2.3 Select best indicators

The next step is to narrow the list to the final indicators that will be
used in the performance monitoring system. They should be the optimum
set that meets the need for management-useful information at a
reasonable cost.

> ![](media/image12.png){width="1.1in" height="1.0in"}**TECHNIQUE --
> Select Best Indicators in a Brainstorming Session:** Involve all SO
> team members in selecting indicators using a facilitated approach.
> ***Helpful Hint 3: Performance Indicator Brainstorming Session*** has
> tips on facilitating the performance indicator selection session. Use
> your completed copies of Worksheet 4 to guide the team's discussions.
> Continue to use a collaborative approach when selecting the best
> indicators. Obtain input from all members of the SO team, your
> partners, sector experts and data collection experts.

Be selective. Remember the costs associated with data collection and
analysis. Limit the number of indicators used to track each objective or
result to a few (two or three). Select only those that represent the
most basic and important dimensions of project aims.

If you find that a performance indicator does not meet all the criteria
earlier discussed, you may revisit the indicator and revise it to meet
the criteria. Otherwise, you will need to identify alternative
indicators.

### 2.4 Document indicators in the PMP

Proper documentation will facilitate the maintenance of quality
performance indicators and data.

![](media/image20.wmf)

**WORKSHEET 6 -- Performance Indicator Reference Sheet:** Use Worksheet
6 to complete the SO/IR/Indicator section and the \"Description\"
section. Note that Worksheet 6 behind the "Worksheets" tab, also
includes detailed instructions for filling out the indicator reference
sheet. By the end of Part 2 you will have completed the remaining
sections of the worksheet. Table 2-2 presents an example of a completed
performance indicator reference sheet.

+----------------+-----------------+----------------+-----------------+
| **Table 2-2:   |                 |                |                 |
| Example of a   |                 |                |                 |
| completed      |                 |                |                 |
| Performance    |                 |                |                 |
| Indicator      |                 |                |                 |
| Reference      |                 |                |                 |
| Sheet          |                 |                |                 |
| (Worksheet     |                 |                |                 |
| 6)**           |                 |                |                 |
+----------------+-----------------+----------------+-----------------+
| **Strategic    |                 |                |                 |
| Objective 5:** |                 |                |                 |
| Increased      |                 |                |                 |
| Opportunities  |                 |                |                 |
| for Domestic   |                 |                |                 |
| and Foreign    |                 |                |                 |
| Investment     |                 |                |                 |
+----------------+-----------------+----------------+-----------------+
| **Intermediate |                 |                |                 |
| Result 5.2:**  |                 |                |                 |
| Strengthened   |                 |                |                 |
| capacity of    |                 |                |                 |
| selected       |                 |                |                 |
| institutions   |                 |                |                 |
| to foster      |                 |                |                 |
| private        |                 |                |                 |
| enterprise     |                 |                |                 |
+----------------+-----------------+----------------+-----------------+
| **Indicator    |                 |                |                 |
| 5.2c:**        |                 |                |                 |
| Revenue        |                 |                |                 |
| generated by   |                 |                |                 |
| fee-based      |                 |                |                 |
| services at    |                 |                |                 |
| targeted       |                 |                |                 |
| associations   |                 |                |                 |
+----------------+-----------------+----------------+-----------------+
| *              |                 |                |                 |
| *DESCRIPTION** |                 |                |                 |
+----------------+-----------------+----------------+-----------------+
| **Precise      |                 |                |                 |
| Definitions:** |                 |                |                 |
|                |                 |                |                 |
| *Revenue*      |                 |                |                 |
| means income   |                 |                |                 |
| from fee-based |                 |                |                 |
| services.      |                 |                |                 |
| General        |                 |                |                 |
| membership     |                 |                |                 |
| fees cannot be |                 |                |                 |
| counted.       |                 |                |                 |
| *Fee-based     |                 |                |                 |
| services* mean |                 |                |                 |
| any training,  |                 |                |                 |
| workshop,      |                 |                |                 |
| publication,   |                 |                |                 |
| or internet    |                 |                |                 |
| based service  |                 |                |                 |
| that is        |                 |                |                 |
| offered to the |                 |                |                 |
| association's  |                 |                |                 |
| entire         |                 |                |                 |
| membership (or |                 |                |                 |
| segments of    |                 |                |                 |
| its            |                 |                |                 |
| membership)    |                 |                |                 |
| and for which  |                 |                |                 |
| a fee is       |                 |                |                 |
| levied.        |                 |                |                 |
| *Targeted      |                 |                |                 |
| associations*  |                 |                |                 |
| are AAA, BBB,  |                 |                |                 |
| and CCC.       |                 |                |                 |
| *Members* can  |                 |                |                 |
| be individuals |                 |                |                 |
| and/or         |                 |                |                 |
| associations   |                 |                |                 |
| (and their     |                 |                |                 |
| members) that  |                 |                |                 |
| are members of |                 |                |                 |
| the targeted   |                 |                |                 |
| association.   |                 |                |                 |
+----------------+-----------------+----------------+-----------------+
| **Unit of      |                 |                |                 |
| Measure:**     |                 |                |                 |
| peso           |                 |                |                 |
+----------------+-----------------+----------------+-----------------+
| *              |                 |                |                 |
| *Disaggregated |                 |                |                 |
| by:** Name of  |                 |                |                 |
| each           |                 |                |                 |
| association    |                 |                |                 |
+----------------+-----------------+----------------+-----------------+
| **Justificat   |                 |                |                 |
| ion/Management |                 |                |                 |
| Utility:** An  |                 |                |                 |
| increase in    |                 |                |                 |
| revenues from  |                 |                |                 |
| services to    |                 |                |                 |
| members is a   |                 |                |                 |
| direct measure |                 |                |                 |
| of improved    |                 |                |                 |
| capability of  |                 |                |                 |
| the            |                 |                |                 |
| association to |                 |                |                 |
| offer services |                 |                |                 |
| that are       |                 |                |                 |
| valued         |                 |                |                 |
| (demonstrated  |                 |                |                 |
| by willingness |                 |                |                 |
| of members to  |                 |                |                 |
| pay for the    |                 |                |                 |
| service) by    |                 |                |                 |
| its            |                 |                |                 |
| membership.    |                 |                |                 |
+----------------+-----------------+----------------+-----------------+
| **PLAN FOR     |                 |                |                 |
| DATA           |                 |                |                 |
| ACQUISITION BY |                 |                |                 |
| USAID**        |                 |                |                 |
+----------------+-----------------+----------------+-----------------+
| **Data         |                 |                |                 |
| Collection     |                 |                |                 |
| Method:**      |                 |                |                 |
| Implementing   |                 |                |                 |
| partner        |                 |                |                 |
| conducts       |                 |                |                 |
| *document      |                 |                |                 |
| review* of     |                 |                |                 |
| internal       |                 |                |                 |
| records of     |                 |                |                 |
| targeted       |                 |                |                 |
| associations   |                 |                |                 |
+----------------+-----------------+----------------+-----------------+
| **Data         |                 |                |                 |
| Source(s):**   |                 |                |                 |
| Implementing   |                 |                |                 |
| partners XXX   |                 |                |                 |
| and YYY        |                 |                |                 |
+----------------+-----------------+----------------+-----------------+
| **Method of    |                 |                |                 |
| Acquisition by |                 |                |                 |
| USAID:**       |                 |                |                 |
| Semi-annual    |                 |                |                 |
| monitoring     |                 |                |                 |
| reports from   |                 |                |                 |
| implementing   |                 |                |                 |
| partners       |                 |                |                 |
| submitted to   |                 |                |                 |
| USAID activity |                 |                |                 |
| manager, with  |                 |                |                 |
| the above      |                 |                |                 |
| information,   |                 |                |                 |
| brief          |                 |                |                 |
| analysis, and  |                 |                |                 |
| detailed back  |                 |                |                 |
| up.            |                 |                |                 |
+----------------+-----------------+----------------+-----------------+
| **Frequency &  |                 |                |                 |
| Timing of Data |                 |                |                 |
| Acquisition:** |                 |                |                 |
| Semi-annually, |                 |                |                 |
| January 31 and |                 |                |                 |
| July 31.       |                 |                |                 |
+----------------+-----------------+----------------+-----------------+
| **Estimated    |                 |                |                 |
| Cost of Data   |                 |                |                 |
| Acquisition:** |                 |                |                 |
| Low cost --    |                 |                |                 |
| part of data   |                 |                |                 |
| collection by  |                 |                |                 |
| targeted       |                 |                |                 |
| associations & |                 |                |                 |
| implementing   |                 |                |                 |
| partners       |                 |                |                 |
+----------------+-----------------+----------------+-----------------+
| *              |                 |                |                 |
| *Individual(s) |                 |                |                 |
| responsible at |                 |                |                 |
| USAID:** Jane  |                 |                |                 |
| Doe, Activity  |                 |                |                 |
| Manager        |                 |                |                 |
+----------------+-----------------+----------------+-----------------+
| **Individual   |                 |                |                 |
| responsible    |                 |                |                 |
| for providing  |                 |                |                 |
| data to        |                 |                |                 |
| USAID:** Raj   |                 |                |                 |
| Patel, Project |                 |                |                 |
| M&E specialist |                 |                |                 |
+----------------+-----------------+----------------+-----------------+
| **Location of  |                 |                |                 |
| data           |                 |                |                 |
| storage:** MS  |                 |                |                 |
| Excel File     |                 |                |                 |
| (S:\\SO2\\EG   |                 |                |                 |
| PMP\\EG        |                 |                |                 |
| Performance    |                 |                |                 |
| data           |                 |                |                 |
| table.xls)     |                 |                |                 |
+----------------+-----------------+----------------+-----------------+
| **DATA QUALITY |                 |                |                 |
| ISSUES**       |                 |                |                 |
+----------------+-----------------+----------------+-----------------+
| **Date of      |                 |                |                 |
| Initial Data   |                 |                |                 |
| Quality        |                 |                |                 |
| Assessment:**  |                 |                |                 |
| See Assessment |                 |                |                 |
| Report "Data   |                 |                |                 |
| Quality        |                 |                |                 |
| Assessments    |                 |                |                 |
| for Annual     |                 |                |                 |
| Report         |                 |                |                 |
| Indicators,"   |                 |                |                 |
| for details.   |                 |                |                 |
+----------------+-----------------+----------------+-----------------+
| **Known Data   |                 |                |                 |
| Limitations    |                 |                |                 |
| and            |                 |                |                 |
| Significance   |                 |                |                 |
| (if any):**    |                 |                |                 |
| For            |                 |                |                 |
| association    |                 |                |                 |
| BBB, initial   |                 |                |                 |
| data quality   |                 |                |                 |
| assessment     |                 |                |                 |
| revealed that  |                 |                |                 |
| revenue from   |                 |                |                 |
| services had   |                 |                |                 |
| been           |                 |                |                 |
| erroneously    |                 |                |                 |
| included with  |                 |                |                 |
| fee revenue in |                 |                |                 |
| the baseline   |                 |                |                 |
| data.          |                 |                |                 |
| Transcription  |                 |                |                 |
| error was not  |                 |                |                 |
| material but   |                 |                |                 |
| was corrected  |                 |                |                 |
| immediately.   |                 |                |                 |
+----------------+-----------------+----------------+-----------------+
| **Actions      |                 |                |                 |
| Taken or       |                 |                |                 |
| Planned to     |                 |                |                 |
| Address Data   |                 |                |                 |
| Limitations:** |                 |                |                 |
| Error in data  |                 |                |                 |
| was corrected. |                 |                |                 |
+----------------+-----------------+----------------+-----------------+
| **Date of      |                 |                |                 |
| Future Data    |                 |                |                 |
| Quality        |                 |                |                 |
| Assessments:** |                 |                |                 |
| Q2 FY 2003     |                 |                |                 |
+----------------+-----------------+----------------+-----------------+
| **Procedures   |                 |                |                 |
| for Future     |                 |                |                 |
| Data Quality   |                 |                |                 |
| Assessments:** |                 |                |                 |
| Incorporate    |                 |                |                 |
| into normal    |                 |                |                 |
| activity       |                 |                |                 |
| monitoring;    |                 |                |                 |
| schedule with  |                 |                |                 |
| activity       |                 |                |                 |
| monitoring     |                 |                |                 |
| field visit;   |                 |                |                 |
| review partner |                 |                |                 |
| back-up data;  |                 |                |                 |
| interview      |                 |                |                 |
| responsible    |                 |                |                 |
| individuals in |                 |                |                 |
| targeted       |                 |                |                 |
| associations.  |                 |                |                 |
+----------------+-----------------+----------------+-----------------+
| **PLAN FOR     |                 |                |                 |
| DATA ANALYSIS, |                 |                |                 |
| REVIEW, &      |                 |                |                 |
| REPORTING**    |                 |                |                 |
+----------------+-----------------+----------------+-----------------+
| **Data         |                 |                |                 |
| Analysis:**    |                 |                |                 |
| Summary data   |                 |                |                 |
| for all        |                 |                |                 |
| associations   |                 |                |                 |
| analyzed and   |                 |                |                 |
| summarized by  |                 |                |                 |
| USAID          |                 |                |                 |
+----------------+-----------------+----------------+-----------------+
| **Presentation |                 |                |                 |
| of Data:** Bar |                 |                |                 |
| or line graphs |                 |                |                 |
| showing        |                 |                |                 |
| targets and    |                 |                |                 |
| actuals as     |                 |                |                 |
| well as        |                 |                |                 |
| disaggregation |                 |                |                 |
| as noted       |                 |                |                 |
| above.         |                 |                |                 |
+----------------+-----------------+----------------+-----------------+
| **Review of    |                 |                |                 |
| Data:** Annual |                 |                |                 |
| review of data |                 |                |                 |
| by EG SO team  |                 |                |                 |
| during         |                 |                |                 |
| Portfolio      |                 |                |                 |
| Review; annual |                 |                |                 |
| review by      |                 |                |                 |
| .....(insert   |                 |                |                 |
| name of        |                 |                |                 |
| activity       |                 |                |                 |
| manager) with  |                 |                |                 |
| target         |                 |                |                 |
| associations.  |                 |                |                 |
+----------------+-----------------+----------------+-----------------+
| **Reporting of |                 |                |                 |
| Data:** Annual |                 |                |                 |
| Report data    |                 |                |                 |
| tables and     |                 |                |                 |
| narrative;     |                 |                |                 |
| Activity       |                 |                |                 |
| Implementation |                 |                |                 |
| Reviews        |                 |                |                 |
| (AIRs), Annual |                 |                |                 |
| Portfolio      |                 |                |                 |
| Review.        |                 |                |                 |
+----------------+-----------------+----------------+-----------------+
| **OTHER        |                 |                |                 |
| NOTES**        |                 |                |                 |
+----------------+-----------------+----------------+-----------------+
| **Note on      |                 |                |                 |
| Baseli         |                 |                |                 |
| nes/Targets:** |                 |                |                 |
| Original       |                 |                |                 |
| baseline       |                 |                |                 |
| collected was  |                 |                |                 |
| not for entire |                 |                |                 |
| CY2000.        |                 |                |                 |
| Recommend      |                 |                |                 |
| baseline data  |                 |                |                 |
| be collected   |                 |                |                 |
| in Jan 2001    |                 |                |                 |
| and replace    |                 |                |                 |
| data collected |                 |                |                 |
| in Nov 2000.   |                 |                |                 |
| (Note: Done in |                 |                |                 |
| Jan 01).       |                 |                |                 |
| Targets will   |                 |                |                 |
| need to be set |                 |                |                 |
| with partners. |                 |                |                 |
|                |                 |                |                 |
| **Other        |                 |                |                 |
| Notes:** None. |                 |                |                 |
+----------------+-----------------+----------------+-----------------+
| **PERFORMANCE  |                 |                |                 |
| INDICATOR      |                 |                |                 |
| VALUES**       |                 |                |                 |
+----------------+-----------------+----------------+-----------------+
| **Year**       | **Target**      | **Actual**     | **Notes**       |
+----------------+-----------------+----------------+-----------------+
| **2000**       |                 | Peso 400,000   | Covers full CY  |
|                |                 |                | 00. Collected   |
|                |                 |                | Jan 01.         |
+----------------+-----------------+----------------+-----------------+
| **2001**       | Peso 600,000    | Peso 565,000   |                 |
+----------------+-----------------+----------------+-----------------+
| **2002**       | Peso 1,000,000  | Peso 980,000   |                 |
+----------------+-----------------+----------------+-----------------+
| **THIS SHEET   |                 |                |                 |
| LAST UPDATED   |                 |                |                 |
| ON:** 4/08/03  |                 |                |                 |
+----------------+-----------------+----------------+-----------------+

##  Task 3 -- Identify Data Source and Collection Method

Data can be gathered and collected from a variety of sources using a
variety of methods. Some methods are hands-on and highly participatory,
while others are more exclusive and rely on the opinion of one or two
specialists. In most cases, it is best to use more than one data
collection method per SO. The process of identifying quality data
sources and developing data collection methods can be broken down into
four sub-steps (see Task 3).

![](media/image21.wmf){width="5.996527777777778in"
height="1.4423611111111112in"}

### ADS Guidance and Helpful Resources

+-----------------------------------------------------------------------+
| **Good Practices in Data Collection (ADS 203.3.5.3c)**                |
|                                                                       |
| -   Data are collected using methods to address and minimize sampling |
|     and non-sampling errors                                           |
|                                                                       |
| -   Written procedures are in place for data collection               |
|                                                                       |
| -   Data are collected by qualified personnel, and personnel are      |
|     properly supervised                                               |
|                                                                       |
| -   Data are collected using a consistent collection process from     |
|     year to year                                                      |
|                                                                       |
| -   Safeguards are in place to prevent unauthorized changes to the    |
|     data                                                              |
|                                                                       |
| -   Source documents are maintained and readily available             |
|                                                                       |
| -   Duplicate data are detected                                       |
+-----------------------------------------------------------------------+

**Helpful Resources to Learn More about Data Collection**

-   ADS Chapter 203,
    [www.usaid.gov/pubs/ads/200/203.pdf](http://www.usaid.gov/pubs/ads/200/203.pdf)

```{=html}
<!-- -->
```
-   TIPS 1: Conducting a Participatory Evaluation,
    [www.dec.org/pdf_docs/pnabs539.pdf](http://www.dec.org/pdf_docs/pnabs539.pdf)

```{=html}
<!-- -->
```
-   TIPS 2: Conducting Key Informant Interviews,
    [www.dec.org/pdf_docs/pnabs541.pdf](http://www.dec.org/pdf_docs/pnabs541.pdf)

```{=html}
<!-- -->
```
-   TIPS 3: Preparing an Evaluation Scope of Work,
    [www.dec.org/pdf_docs/pnaby207.pdf](http://www.dec.org/pdf_docs/pnaby207.pdf)

-   TIPS 4: Using Direct Observation Techniques,
    [www.dec.org/pdf_docs/pnaby208.pdf](http://www.dec.org/pdf_docs/pnaby208.pdf)

```{=html}
<!-- -->
```
-   TIPS 5: Using Rapid Appraisal Methods,
    [www.dec.org/pdf_docs/pnaby209.pdf](http://www.dec.org/pdf_docs/pnaby209.pdf)

-   TIPS 10: Conducting Focus Group Interviews,
    [www.dec.org/pdf_docs/pnaby233.pdf](http://www.dec.org/pdf_docs/pnaby233.pdf)

```{=html}
<!-- -->
```
-   TIPS 11: The Role of Evaluation in USAID,
    [www.dec.org/pdf_docs/pnaby239.pdf](http://www.dec.org/pdf_docs/pnaby239.pdf)

-   TIPS 12: Guidelines for Indicator and Data Quality,
    [www.dec.org/pdf_docs/pnaca927.pdf](http://www.dec.org/pdf_docs/pnaca927.pdf)

```{=html}
<!-- -->
```
-   Department of Energy, "The Performance-Based Management Handbook,
    Volume 4: Collecting Data to Assess Performance",
    [www.orau.gov/pbm/pbmhandbook/pbmhandbook.html](http://www.orau.gov/pbm/pbmhandbook/pbmhandbook.html)

-   Kumar, Krishna, "Rapid, Low-Cost Data Collection Methods for USAID,"
    December 1987,
    [www.dec.org/pdf_docs/PNAAL100.pdf](http://www.dec.org/pdf_docs/PNAAL100.pdf)

-   CDIE Resource Book on Strategic Planning and Performance Monitoring
    Under Re-engineering, "Common Problems/Issues with Using Secondary
    Data," April 1997, \[DEXS Document #PN-ACH-632\]

### 3.1 Identify potential data sources

For each selected performance indicator, SO teams should explore what
data sources are available (or might be available if the indicators are
conceptualized in different ways). Only indicators for which it is
feasible to collect data in a given country should be used.

![](media/image12.png){width="1.1in" height="1.0in"}**TECHNIQUE --
Identify Potential Data Sources in a Facilitated Session:** Facilitate a
brainstorming session (see ***Helpful Hint 1: Facilitating Group
Discussions and Decision-Making)*** or hold individual discussions with
the following resources in order to help identify potential data
sources:

-   **USAID:** Use primary data collected by the SO team or through
    independent entities contracted for this purpose.

```{=html}
<!-- -->
```
-   **Implementing partners:** Data often come from management
    information such as periodic reports, service statistics, etc.

-   **Secondary sources**: Includes government ministries, the United
    Nations, and international agencies, and are usually not under USAID
    control. This means that USAID does not have the right to audit the
    data or investigate data quality in depth.

![](media/image22.png)

> **INFORMATION SERVICES:** The following services may be useful for
> identifying potential secondary sources of data.

-   **Economic and Social Data Service (ESDS):** ESDS staff specialize
    in selecting the most appropriate quantitative data for specific
    research purposes. Access ESDS via CDIE Online at
    [http://cdie.usaid.gov](http://cdie.usaid.gov/) (available only
    within the USAID firewall; click \'Statistics\' at the top of the
    homepage)

```{=html}
<!-- -->
```
-   **Research and Reference Services (R&RS):** R&RS staff help
    development practitioners clarify their information needs after
    which they identify, analyze and deliver appropriate information in
    a useful form. R&RS also manages the USAID Library and Learning
    Resources Center. Access R&RS via CDIE Online at
    [http://cdie.usaid.gov](http://cdie.usaid.gov/) (available only
    within the USAID firewall; click \'Research\' or \'Library\' at the
    top of the homepage)

SO teams often rely on data collected by implementing partners and
secondary sources. Figure 2-5 illustrates these three sources of
performance data and their relationships to cost and USAID control over
quality.

![](media/image23.png){width="3.9972222222222222in"
height="2.997916666666667in"}

Determining appropriate potential sources of data will require
conversations with people knowledgeable about various data sources
(partners, government statistical or service agencies, public opinion
survey organizations, university social science research centers, etc.).
These contacts will help you to understand:

-   What data are already being collected

-   Whether existing data would be appropriate for a candidate indicator

-   Whether the candidate indicators are relevant and feasible for the
    situation

-   What alternatives may work

Grantee and contractor programs often also include data collection to
monitor their activities, which may provide potential data sources for
the result's indicators. If there are no feasible or reliable sources
available, then consider proxy indicators for which good data will be
available.

### 3.2 Generate data collection options

There are a number of data collection methods available. Some of the
most commonly used methods are known as **Rapid Low-Cost Data Collection
Methods**. These methods are often the preferred choice for SO teams
when cost and timeliness are important. These methods are usually
conducted by third party interviewers and/or observers who are skilled
in conducting and facilitating various types of interviews and meetings.
These methods include:

-   **Focus group interviews:** Small-group, facilitated session
    designed to quickly gather in-depth information while offering
    stakeholders a forum for direct participation.

-   **Key informant interviews:** In-depth discussions with person(s)
    who are knowledgeable on a specific topic.

-   **Community interviews:** Meetings conducted on a specific topic
    that are open to all members of a village/community.

-   **Direct observations:** Intensive and systematic observations of a
    phenomenon or process in its natural setting. May also include
    interviews with key informants.

-   **Informal surveys:** Informal surveys differ from formal, or
    sample, surveys in that they: focus on few variables, use a small
    sample size, use non-probability sampling, and thus typically permit
    more flexibility to interviewers in the field.

***Helpful Hint 4: Rapid Low-Cost Data Collection Methods*** provides
some additional insights into these data collection methods.

Other data collection methods you may want to consider are:

-   **File review:** Reviewing data that has been previously collected
    and is present in the program files, or other program documentation.
    This type of review offers a relatively quick method to discover
    what data has already been collected with an eye toward minimizing
    the need for additional data collection and the costs associated
    with that data collection effort.

-   **Case study:** A research method that uses extensive description
    and analysis of a complex situation studied in its context to answer
    questions about the efficiency and effectiveness of current
    programs.

-   **Content analysis:** Refers to the codification and analysis of
    qualitative data. By coding and classifying qualitative information,
    this method attempts to develop an understanding of the larger
    volumes of qualitative analysis.

```{=html}
<!-- -->
```
-   **Peer review/expert panel:** Involves review and assessment of
    program results by those with expertise in the field.

```{=html}
<!-- -->
```
-   **Survey:** Provides a rigorous and detailed sample survey method of
    gathering information from stakeholders and others by directly
    questioning them.

### 

### 3.3 Select data collection option

The best data collection systems are designed to be as simple as
possible -- not too time-consuming, not unreasonably costly, but able to
provide you with good information at a frequency that meets your
management needs.

Therefore, take **practicality** into account when selecting a data
collection tool. Consider the level of effort and resources required to
develop the data collection tool and analyze the data. Also think about
how often and at what point during the management cycle the data will be
available for use, and the conditions in the country environment in
which you operate.

For example, if data of adequate quality are already collected routinely
by a secondary source, costs may be minimal. If primary data have to be
collected at your expense, costs will be higher -- how much higher will
depend on the scope, method, and frequency of the data collection. A
survey, for example, may cost several hundred thousand dollars, whereas
a rapid appraisal would be much less expensive. Table 2.3 lists some of
the factors and related questions to consider in selecting an
appropriate method.

**Table 2.3 Factors to Consider in Selecting a Data Collection Method**

  ------------- ----------------------------------------------------------
  **Factor**    **Questions to Consider**

  Cost          What is a reasonable cost for the team to incur for
                collecting the data? Some low-cost data collection methods
                limit the type of information that can be collected

  Speed         How much time is available and reasonable for data
                collection and processing? How will shorter collection
                times impact other data characteristics - accuracy/level
                of detail?

  Geographic    What is the geographic area impacted by the program? How
  Diversity     can data be effectively collected in hard-to-reach or
                widely-dispersed geographic areas?

  Demographic   How much diversity is present in the target audience
  Diversity     (e.g., income, size of organization, ethnicity)? A diverse
                population whose target audience is non-homogeneous on one
                or more factors may require a bigger sample size to
                capture impact accurately.

  Level of      How accurate should the data be? How accurate are the
  Accuracy      local government statistics? How do you balance level of
                accuracy against the cost of collecting data?

  Reliability   Can comparable data be collected using this same method in
                the future?

  Frequency     How often are the data to be collected? How does this
                impact data collection in terms of staff/partner resources
                and costs associated with collecting the data?
  ------------- ----------------------------------------------------------

![](media/image6.png){width="1.1in" height="1.1in"}

> **CONSIDER THIS -- Limitations to Collecting Data:** Your ability to
> use certain data collection methods will vary by:

-   Data collection capacity and tradition in the host country

-   Access to government information

-   Local government unit capacity

-   Capacity of implementing partners, think tanks and academic
    institutions

-   Public attitudes toward social data and surveys

-   Available data documentation

-   Sector and sub-sector

```{=html}
<!-- -->
```
-   USAID resources

-   Confidentiality and requirements such as parental consent to survey
    their children. Such requirements can add considerably to the effort
    required to obtain the data

![](media/image12.png){width="1.1in" height="1.0in"}

**TECHNIQUE -- Use a Decision Chart to Select Best Data Collection
Option:** Figure 2-6 presents a sample decision flow that SO teams can
also use to guide selection of data collection methods for indicators.

![](media/image24.wmf)**WORKSHEET 6 -- Performance Indicator Reference
Sheet:** Once you\'ve completed Tasks 3.1-3.3, refer to the copies of
Worksheet 6 that you completed in Task 2. Update the section called
\"Plan for Data Acquisition by USAID.\"

### ![](media/image25.wmf) 

### 3.4 Develop data collection tools

Once data collection methods are chosen, you may need to develop tools
to collect the data. Table 2-4 presents some guidelines for developing
and using several of these tools.

**Table 2-4. Guidelines for Developing and Using Data Collection Tools**

+--------------+-------------------------------------------------------+
| **Collection | **Guidelines**                                        |
| Method**     |                                                       |
+--------------+-------------------------------------------------------+
| **Rapid      | -   Define the problem and formulate the research     |
| Low-Cost     |     question                                          |
| (focus       |                                                       |
| groups,      | -   Identify the sample population for the study      |
| community    |                                                       |
| interviews,  | -   Carefully choose a facilitator                    |
| informal     |                                                       |
| surveys,     | -   Generate and pre-test the interview guide         |
| etc)**       |                                                       |
|              | -   Recruit the sample                                |
|              |                                                       |
|              | -   Conduct the interviews, meetings, focus groups,   |
|              |     survey, etc.                                      |
|              |                                                       |
|              | -   Analyze data and share the results with           |
|              |     stakeholders                                      |
+--------------+-------------------------------------------------------+
| **Case       | -   Define the problem and formulate the scope and    |
| study**      |     objective of the query with specific attention    |
|              |     toward the nature and context of subject          |
|              |                                                       |
|              | -   Identify samples to be used in the study. They    |
|              |     should address the representational needs of the  |
|              |     range of data being evaluated and show the        |
|              |     relevance of the study                            |
|              |                                                       |
|              | -   Select the type of case most appropriate to the   |
|              |     needs of the program                              |
|              |                                                       |
|              | -   Collect the data to be analyzed through a         |
|              |     combination of sources                            |
|              |                                                       |
|              | -   Analyze the data, accounting for rival            |
|              |     explanations, reproduction of findings, internal  |
|              |     validity, plausibility, ability to generalize,    |
|              |     and overall coherence                             |
|              |                                                       |
|              | -   Evaluate the results regarding ability to         |
|              |     generalize and internal data validity             |
|              |                                                       |
|              | -   Write the report and share the findings           |
+--------------+-------------------------------------------------------+
| **Content    | -   Determine the data source                         |
| analysis**   |                                                       |
|              | -   Establish the coding categories and code the text |
|              |                                                       |
|              | -   Analyze category frequencies, correlation, and    |
|              |     patterns                                          |
|              |                                                       |
|              | -   Write the report                                  |
+--------------+-------------------------------------------------------+
| **Peer       | -   Use peer review in conjunction with other         |
| review/**    |     evaluation techniques                             |
|              |                                                       |
| **expert     | -   Use peer review for research and development      |
| panel        |     activities that are public domain                 |
| evaluation** |                                                       |
|              | -   Peers should be readily identifiable              |
|              |                                                       |
|              | -   Avoid internal peers                              |
|              |                                                       |
|              | -   Guard against dysfunctional group dynamics        |
|              |                                                       |
|              | -   If scales are used, test the validity and         |
|              |     reliability of those scales                       |
|              |                                                       |
|              | -   Provide a bias statement for reviewers            |
+--------------+-------------------------------------------------------+
| **File       | -   Review authorizing legislation, congressional     |
| review in    |     testimony, and comments from legislators          |
| evaluation** |                                                       |
|              | -   Review documents related to the regulatory        |
|              |     implementation of the legislation                 |
|              |                                                       |
|              | -   Review budget documents, administrative           |
|              |     documents, and meeting minutes                    |
|              |                                                       |
|              | -   Review program participant data collected as part |
|              |     of their interaction with the program             |
+--------------+-------------------------------------------------------+
| **Surveys**  | -   Define the areas of evaluation and develop        |
|              |     applicable questions                              |
|              |                                                       |
|              | -   Establish a survey plan                           |
|              |                                                       |
|              | -   Develop a sampling protocol that includes a well  |
|              |     thought out method of data collection, sampling   |
|              |     techniques and method of analysis                 |
|              |                                                       |
|              | -   Develop the questionnaire                         |
|              |                                                       |
|              | -   Field test the questionnaire, individual          |
|              |     questions and the time it takes to administer the |
|              |     test                                              |
|              |                                                       |
|              | -   Distribute the questionnaire to respondents with  |
|              |     a return date.                                    |
|              |                                                       |
|              | -   Provide a follow-up contact with non-respondents  |
|              |                                                       |
|              | -   Analyze data and share the results with           |
|              |     stakeholders                                      |
+--------------+-------------------------------------------------------+

##  Task 4 -- Collect Baseline Data and Verify Quality

In order to manage for results, you need to gather and analyze data that
meets the Agency's data quality standards. Poor quality data can lead to
incorrect inferences, e.g., USAID interventions had a given impact when
they did not or vice versa. You should take steps to understand the
appropriateness and use of different kinds of data collected, understand
data limitations, correct these limitations where cost effective, and
learn to manage for results when data are known to be imperfect.

In addition, the public is carefully scrutinizing the performance of
government agencies. With the advent of the Government Performance
Results Act and other government requirements, agencies are moving from
accountability for inputs to accountability for results. The public,
Congress, and OMB are increasingly taking a more "results oriented" and
\"cost-effective\" look at agency programs. In an era of shrinking
Federal budgets, demonstrating effective performance and sustainable
program impacts helps justify programs and their costs.

Knowing that demonstrating performance rests on the quality of
performance data, you can act effectively to improve activity design and
performance and revise strategies appropriately. The process of
verifying performance data quality can be broken down into three
sub-steps.

### ![](media/image26.wmf)

### 

### Key Definitions, ADS Guidance and Helpful Resources

> ![](media/image3.wmf){width="0.9in"
> height="0.9868055555555556in"}**KEY DEFINITIONS**: The following
> definitions are relevant to this task:

-   **Performance Baseline:** The value of a performance indicator
    before the implementation of a USAID-supported activity that
    contributes to the achievement of the relevant result.

-   **Verification:** Checking performance data against data quality
    standards to reduce the risk of using data that contain significant
    errors.

```{=html}
<!-- -->
```
-   **Bias:** Refers to the likelihood that data collected may reflect
    only a portion of the spectrum of relevant opinion. Bias often
    occurs as the result of the collection of an incomplete or
    inaccurately weighted sample of data.

-   **Significant error (including bias):** An error that affects
    conclusions about the extent to which performance goals have been
    achieved.

-   **Measurement error:** Results primarily from weaknesses in design
    of a data collection instrument; inadequate controls for biases in
    responses or reporting; or inadequately trained or supervised
    enumerators.

+-----------------------------------------------------------------------+
| **ADS Requirements for Data Quality (ADS 203.3.5.1)**                 |
|                                                                       |
| Performance data should be as complete, accurate, and consistent as   |
| management needs and resources permit. To be useful in managing for   |
| results and credible for reporting, performance data should meet      |
| reasonable standards of validity, integrity, precision, reliability,  |
| timeliness.                                                           |
|                                                                       |
| -   **Validity.** Data should clearly and adequately represent the    |
|     intended result. While proxy data may be used, the Operating Unit |
|     should consider how well the data measure the intended result.    |
|     Another key issue is whether data reflect a bias such as          |
|     interviewer bias, unrepresentative sampling, or transcription     |
|     bias.                                                             |
|                                                                       |
| -   **Integrity.** Data that are collected, analyzed, and reported    |
|     should have established mechanisms in place to reduce the         |
|     possibility that they are intentionally manipulated for political |
|     or personal reasons. Data integrity is at greatest risk of being  |
|     compromised during collection and analysis.                       |
|                                                                       |
| -   **Precision.** Data should be sufficiently precise to present a   |
|     fair picture of performance and enable management decision-making |
|     at the appropriate levels. One key issue is whether data are at   |
|     an appropriate level of detail to influence related management    |
|     decisions. A second key issue is what margin of error (the amount |
|     of variation normally expected from a given data collection       |
|     process) is acceptable given the management decisions likely to   |
|     be affected. In all cases, the margin of error should be less     |
|     than the intended change; if the margin of error is 10 percent    |
|     and the data show a change of 5 percent, the Operating Unit will  |
|     have difficulty determining whether the change was due to the     |
|     USAID activity or due to variation in the data collection         |
|     process. Operating Units should be aware that improving the       |
|     precision of data usually increases the cost of collection and    |
|     analysis.                                                         |
|                                                                       |
| -   **Reliability.** Data should reflect stable and consistent data   |
|     collection processes and analysis methods from over time. The key |
|     issue is whether analysts and managers would come to the same     |
|     conclusions if the data collection and analysis process were      |
|     repeated. Operating Units should be confident that progress       |
|     toward performance targets reflects real changes rather than      |
|     variations in data collection methods. When data collection and   |
|     analysis methods change, the PMP should be updated.               |
|                                                                       |
| -   **Timeliness.** Data should be timely enough to influence         |
|     management decision-making at the appropriate levels. One key     |
|     issue is whether the data are available frequently enough to      |
|     influence the appropriate level of management decisions. A second |
|     key issue is whether data are current enough when they are        |
|     available.                                                        |
+-----------------------------------------------------------------------+

**Helpful Resources to Learn More about Data Quality**

-   ADS Chapter 203 -- Assessing and Learning,
    [www.usaid.gov/pubs/ads/200/203.pdf](http://www.usaid.gov/pubs/ads/200/203.pdf)

-   TIPS 12: Guidelines for Indicator and Data Quality,
    [www.dec.org/pdf_docs/pnaca927.pdf](http://www.dec.org/pdf_docs/pnaca927.pdf)

-   Handbook of Democracy and Governance Program Indicators,
    [www.dec.org/pdf_docs/PNACC390.pdf](http://www.dec.org/pdf_docs/PNACC390.pdf)

-   U.S. General Accounting Office, "The Results Act: An Evaluator's
    Guide to Assessing Agency Performance Plans",
    [www.whitehouse.gov/OMB/mgmt-gpra/gplaw2m.html](http://www.whitehouse.gov/OMB/mgmt-gpra/gplaw2m.html)

-   U.S. General Accounting Office, "Performance Plans: Selected
    Approaches for Verification and Validation of Agency Performance
    Information"

-   U.S. General Accounting Office, Standards for Internal Controls,
    [www.gao.gov/](http://www.gao.gov/)

### 4.1 Collect baseline data

Completing Task 3 led your team through the process of determining what
data to collect and the method and tools that should be used to collect
data. Now is the time to execute your data collection plan and begin
collecting the initial data for each indicator.

The baseline measure establishes the reference point for the start of
the program period. In some cases, planners may want to go back several
years to correctly portray the context in which progress will be made.
It is preferable if the baseline immediately precedes the start of a new
Strategic Plan because we are trying to gauge the progress of a
particular Strategic Plan. It will not always be possible to secure
baseline data for the chosen year. In that instance, the baseline may be
the most recent past year for which the relevant information exists or
can be acquired.

![](media/image6.png){width="1.0in" height="1.0in"}

> **CONSIDER THIS -- Examine the Performance Trend line**: When
> selecting a baseline year or years, examine the trend line of past
> performance. There could be unexpected spikes or dips in the trend and
> a year in which one or the other occurs would be a poor year to select
> as the baseline year.
>
> ![](media/image12.png){width="1.1in" height="1.0in"}**TECHNIQUE -- How
> to Establish Baseline When Information is Inadequate**: Where baseline
> information is inadequate, many USAID Operating Units initiate a data
> collection effort as soon as their Strategic Plan is approved and the
> performance indicators they will use to judge progress are selected.
> The first set of data collected on these indicators becomes the formal
> baseline against which targets are set and future progress is
> assessed. For people-specific indicators, baselines should
> disaggregate data by gender and/or other relevant customer groups.

As you start collecting data, you may find that you\'re amassing a pile
of forms, papers and other documents that should be organized and
compiled before any real analysis can begin. When this scenario is
multiplied by all the SO teams in an Operating Unit, many units begin to
consider the use of an information database as a more effective way to
store and analyze data. The technique below addresses the database
issue.

> ![](media/image12.png){width="1.1in" height="1.0in"}**TECHNIQUE --
> Storing Data in an Information Database:** Traditional data collection
> involves manually collecting performance data on forms, looking for
> trends in the data, and summarizing the results in a printed
> management report. However, this traditional method of reporting is
> rapidly being replaced by automated software systems that rely on a
> computer's processing power to help analyze, process, and communicate
> information in real-time and in both visual and printed form. As data
> is collected, you may want to use an information database to store
> performance data.

Before jumping into a discussion of the pros and cons of certain
databases, it may be useful to figure out if a database is needed by
asking some basic questions:

-   How many people will need to work with the database at the same
    time? Are they in the same building and/or are they on the same
    computer system?

-   Does the data currently exist in any system?

-   Does data exist in well-organized hard-copy reports?

-   If the data does not exist in a centralized location, designing a
    database is only half the job; the other half is determining the
    process for identifying and entering data into the database. *NOTE:*
    Many database projects fail not because of the design of the
    database but because the organization using it has no established
    process for putting information into it.

The sophistication of the data collection system should be matched to
the needs of the Operating Unit. The investment in technology should
return benefits to the organization that exceeds the costs. These
benefits will typically accrue in terms of improved information
accuracy, security, accessibility, timeliness, and cost-effectiveness.

***Helpful Hint 5: Information Systems for Data Collection and
Performance Monitoring*** provides additional information on data
collection and performance monitoring systems.

### 4.2 Conduct a data quality assessment

You need to know if you can trust your data to use it for making
decisions. Performance data should be as complete, accurate and
consistent as management needs and resources permit. A data quality
assessment of each selected performance indicator will help you validate
the usefulness of your data. ADS 203.3.6.6 states that when conducting
these assessments, you should:

-   Verify that data are of reasonable quality based on the five data
    quality standards provided in ADS 203.3.5.1

-   Review data collection, maintenance, and processing procedures to
    ensure consistent application. Take steps to address identified data
    quality limitations.

-   Retain documentation (whether as simple as a memoranda or a detailed
    report).

-   Disclose limitations for data reported in some sections of the
    Annual Report (see ADS 203.3.8.3)

Recognize that no data are perfect, but they should be good enough to
document performance and support decision-making. Use your professional
judgment, and back it up by documenting your decisions and supporting
information. Judgments about sufficient quality levels should reflect:

-   Uses of the data

-   Management judgments about what level of quality is needed for
    confident decisions.

-   Technical assessments about what degree of quality is possible
    (e.g., professional judgment about acceptable levels of error;
    implications of data collection methodology, sampling strategy)

```{=html}
<!-- -->
```
-   Practical considerations. Trade-off between accuracy and cost or
    between dimensions of quality (e.g., timeliness and completeness).

![](media/image12.png){width="1.1in" height="1.0in"}**TECHNIQUE -- Plan
Data Quality Assessments:** A practical approach to planning data
quality assessments includes the following:

-   Develop and implement in overall data quality assurance plan that
    includes initial data quality assessment periodic quality reviews,
    partner and USAID capacity development

-   Maintain written policies and procedures for data collection,
    maintenance, and process

-   Maintain an audit trail -- document the assessment, decisions
    concerning data quality problems, and the steps taken to address
    them

-   Decide who should be involved in the data quality assessment (SO
    team members, partners, sector specialists, specialists in data
    quality)

-   Decide when data quality assessments will be done

![](media/image27.wmf)

**WORKSHEET 7: Data Quality Assessment Checklist:** Table 2-5 identifies
five criteria for assessing the performance data quality. These five
criteria are covered in more detail in Worksheet 7, which you can use to
assess the quality of your selected performance data:

+---------+------------------+-----------------------------------------+
| **Table |                  |                                         |
| 2-5.    |                  |                                         |
| Key     |                  |                                         |
| C       |                  |                                         |
| riteria |                  |                                         |
| for     |                  |                                         |
| As      |                  |                                         |
| sessing |                  |                                         |
| Perf    |                  |                                         |
| ormance |                  |                                         |
| Data    |                  |                                         |
| Qu      |                  |                                         |
| ality** |                  |                                         |
+---------+------------------+-----------------------------------------+
| **Cri   | **Answers the    | **Affected by:**                        |
| teria** | question:**      |                                         |
+---------+------------------+-----------------------------------------+
| **Val   | -   Do data      | **Measurement error.** Can result from  |
| idity** |     clearly and  | weak design of data collection          |
|         |     directly     | instrument, inadequate control for      |
|         |     measure what | biases in responses or reporting, or    |
|         |     we intend to | inadequately trained or supervised      |
|         |     measure?     | enumerators.                            |
|         |                  |                                         |
|         |                  | **Sampling error.** Sample may not be   |
|         |                  | representative, too small for           |
|         |                  | statistical extrapolation or contain    |
|         |                  | sample units based on supposition       |
|         |                  | rather than statistical representation. |
|         |                  |                                         |
|         |                  | **Transcription error.** Data entries   |
|         |                  | may occur when transcribing data from   |
|         |                  | one source to another. Formulae should  |
|         |                  | be applied consistently, and final      |
|         |                  | numbers reported accurately.            |
+---------+------------------+-----------------------------------------+
| *       | -   Would you    | **Changes in the data collection        |
| *Reliab |     come to the  | process.** Ensuring that data are       |
| ility** |     same         | reliable requires that the collection   |
|         |     findings if  | process be consistent from year to      |
|         |     the data     | year.                                   |
|         |     collection   |                                         |
|         |     and          |                                         |
|         |     analytical   |                                         |
|         |     process were |                                         |
|         |     repeated?    |                                         |
+---------+------------------+-----------------------------------------+
| **Timel | -   Are data     | **Frequency.** Performance data are     |
| iness** |     available    | available on a frequent enough basis to |
|         |     frequently   | regularly inform program management     |
|         |     enough to    | decisions                               |
|         |     inform?      |                                         |
|         |                  | **Currency.** Data are sufficiently up  |
|         | -   Are data     | to date to guide decision-making (e.g., |
|         |     current      | quarterly). Data collected infrequently |
|         |     enough when  | (every 2-5 years), or with a            |
|         |     they are     | substantial lag time (\>1 year), can    |
|         |     available?   | help track long-term trends and confirm |
|         |                  | lower level data accuracy.              |
+---------+------------------+-----------------------------------------+
| **Prec  | -   Are data at  | **Acceptable margin of error.** The     |
| ision** |     the          | expected change being measured should   |
|         |     appropriate  | be greater than the margin of error     |
|         |     level of     |                                         |
|         |     detail?      |                                         |
|         |                  |                                         |
|         | -   What margin  |                                         |
|         |     of error is  |                                         |
|         |     acceptable   |                                         |
|         |     given the    |                                         |
|         |     management   |                                         |
|         |     decisions to |                                         |
|         |     be affected? |                                         |
+---------+------------------+-----------------------------------------+
| **Inte  | -   Are          | **Risk.** Data is at greatest risk      |
| grity** |     mechanisms   | during data collection and analysis     |
|         |     in place to  |                                         |
|         |     reduce the   | **Objectivity and independence.**       |
|         |     possibility  | Needed in key data collection,          |
|         |     that data    | management, and assessment procedures.  |
|         |     are          |                                         |
|         |     manipulated  | **Confidence in data.** Need for        |
|         |     for          | confidence in data from secondary       |
|         |     political or | sources. May require an independent     |
|         |     personal     | review of secondary source data.        |
|         |     reasons?     |                                         |
+---------+------------------+-----------------------------------------+

Before you begin assessing all of the data, take into consideration the
source of the data, and the impact this might have on the assessment
process.

> ![](media/image12.png){width="1.1in" height="1.0in"}**TECHNIQUE --
> Assess Data from Different Sources:** The rigor to which a data
> quality assessment is applied to a data source (i.e., USAID,
> implementing partner, secondary source) will differ for each source.
> The goal to assessing data from implementing partners and secondary
> sources is for you to be aware of data strengths and weaknesses and
> the extent to which data can be trusted when making management
> decisions and reporting. Table 2-6 presents a practical approach for
> assessing data from different data sources. ***Helpful Hint 6: Key
> Features of Quality Data Sources*** provides some additional insight.

+------------+---------------------------------------------------------+
| **Table    |                                                         |
| 2-6. Tips  |                                                         |
| for        |                                                         |
| Assessing  |                                                         |
| Data from  |                                                         |
| Different  |                                                         |
| Sources**  |                                                         |
+------------+---------------------------------------------------------+
| **Data     | **Tips for Conducting a Data Quality Assessment (DQA)** |
| Source**   |                                                         |
+------------+---------------------------------------------------------+
| **USAID**  | -   Focus the DQA on the written procedures and         |
|            |     training for sampling, data collection,             |
|            |     transcription and analysis and determine if they    |
|            |     are rigorous enough to produce data that meet the   |
|            |     five data quality standards. Specifically focus on  |
|            |     whether:                                            |
|            |                                                         |
|            | -   Written procedures are in place for data collection |
|            |                                                         |
|            | -   The data collection process is consistent from year |
|            |     to year                                             |
|            |                                                         |
|            | -   Data are collected using methods to address and     |
|            |     minimize sampling and non-sampling errors           |
|            |                                                         |
|            | -   Data are collected by qualified personnel and       |
|            |     personnel are properly supervised                   |
|            |                                                         |
|            | -   Duplicate data are detected                         |
|            |                                                         |
|            | -   Safeguards are in place too prevent unauthorized    |
|            |     changes to the data; and                            |
|            |                                                         |
|            | -   Source documents are maintained and readily         |
|            |     available                                           |
|            |                                                         |
|            | -   Ensure that data quality requirements are written   |
|            |     into any solicitations (SOW, RFP, RFA) for data     |
|            |     collection                                          |
|            |                                                         |
|            | -   Conduct spot checks to ensure that quality          |
|            |     assurance mechanisms are being used                 |
+------------+---------------------------------------------------------+
| **Im       | -   Periodically sample and review data for             |
| plementing |     completeness, accuracy and consistency. This        |
| partner**  |     includes making sure that the indicator definitions |
|            |     being used by the partner are consistent with the   |
|            |     definitions contained in your PMP.                  |
|            |                                                         |
|            | -   Review partner reports to determine if they are     |
|            |     sufficiently consistent to be reliable              |
|            |                                                         |
|            | -   Conduct field visits to compare central office      |
|            |     records with field site records; try to visit a     |
|            |     broad range of sites.                               |
|            |                                                         |
|            | -   Audit financial information when performance        |
|            |     indicators use financial information                |
+------------+---------------------------------------------------------+
| *          | -   Periodically sample and review data for             |
| *Secondary |     completeness, accuracy and consistency              |
| source**   |                                                         |
|            | -   Review the data with other development partners to  |
|            |     gain an appreciation of accuracy and credibility    |
|            |                                                         |
|            | -   Request a briefing to gain a better understanding   |
|            |     of the data collection and analysis process. This   |
|            |     should help you determine whether the data can be   |
|            |     trusted or not.                                     |
|            |                                                         |
|            | *Some special considerations:*                          |
|            |                                                         |
|            | -   USAID does not have the right to audit or           |
|            |     investigate data quality in depth                   |
|            |                                                         |
|            | -   If USAID provides technical assistance to a         |
|            |     government ministry to improve data collection and  |
|            |     analysis, you maybe in a very good position to      |
|            |     assess the quality of the data                      |
+------------+---------------------------------------------------------+

While conducting the assessment, you may find evidence that points to
bias in the data collection process that is affecting the quality of the
data. Bias can come in many forms.

> ![](media/image6.png){width="1.1in" height="1.1in"}**CONSIDER THIS --
> Types of Data Bias**:

-   **Interviewer bias:** Interviewers who gather program performance
    data through surveys or focus groups (for example) with program
    beneficiaries may inject their own bias, either intentionally or
    not, into the way they conduct the interviews. Likewise, different
    interviewers may not ask questions in the same way.

-   **Instrument or measurement bias:** Instruments can be biased, for
    example, if a different instrument is used for the beneficiaries and
    for the control group. Also, instruments may be written in a way
    that sways people to give one response over another.

-   **Response bias:** For example, if a sufficiently large group of
    beneficiaries who share common characteristics or opinions choose
    not to answer a survey question, they can bias the results by not
    having their responses included.

-   **Recall bias:** For instance, respondents to data collection
    instruments are asked to respond to questions that require them to
    think back and comment on conditions they lived in several years
    ago. The passing of time may lead people to recall the conditions
    differently from the reality of the time.

-   **Time or seasonal bias:** Some data may be biased if they are
    collected at different times of the day or different seasons of the
    year.

> ***Helpful Hint 7: Tips to Minimize Bias*** presents some helpful
> hints to reduce bias.

The assessment process is not over when the worksheet has been
completed. The results may indicate that you need to develop and
implement a plan that will improve the quality of the data and your
ability to use it for performance management. The steps in your plan may
include:

-   Ensure transparency -- report data limitations and their
    implications for assessing performance

-   Improve an indicator by using another source or new methods of
    measurement

-   Adjust, supplement, or replace problematic data

```{=html}
<!-- -->
```
-   Triangulate -- use multiple data sources with offsetting strengths
    and limitations.

### 4.3 Build commitment to and capacity for quality

Management needs to create a climate that encourages coordination,
resource allocation, and attention to data quality issues that enable
improvements in data quality. The following techniques may help
encourage organizational commitment. Figure 2-7 describes the data
quality assessment used by the USAID/Bangladesh mission.

> ![](media/image12.png){width="1.1in" height="1.0in"}**TECHNIQUE --
> Foster Organizational Commitment:** Some approaches that can be used
> to foster organizational commitment to and capacity for quality
> include:

-   Have Agency executives provide confidence that they value and use
    good quality data by communicating its importance, making data
    quality an organizational goal, creating a climate of managing for
    results, and providing technical and financial support.

-   Review organizational capacities and procedures for data collection
    and use. Review the designation of responsibilities for integrating
    and coordinating data; sufficient staff and expertise to fulfill
    responsibilities, appropriate hardware and software, and resources
    for building, upgrading, and maintaining data systems.

-   Assign clear responsibilities for data creation and maintenance,
    training and supervision of those who enter data, transferring data
    from initial to final formats, and appropriately analyzing and
    reporting performance measures.

-   Adopt audit and independence review mechanisms that encourage
    objectivity and independence in collecting and managing data

-   Provide responsible staff with training and guidance for needed
    skills and knowledge in performance management

-   Share Agency data quality standards with partners (including need
    for baselines, targets, and disaggregated data)

```{=html}
<!-- -->
```
-   Support partners in the development of written activity-level PMPs

+-----------------------------------------------------------------------+
| **Figure 2-7. Data Quality Improvement in Action: USAID/Bangladesh    |
| Democracy Partnership**                                               |
+=======================================================================+
| **Data Quality Challenges:**                                          |
|                                                                       |
| -   Indicator definitions, measurement criteria and data collection   |
|     methods were not uniformly understood                             |
|                                                                       |
| -   Data collection process was time consuming, decreasing time       |
|     available to implement program activities                         |
|                                                                       |
| -   Data collection and use were not well matched                     |
|                                                                       |
| -   Insufficient clarity concerning roles and responsibilities among  |
|     the Partners for data design, collection, analysis and reporting  |
|                                                                       |
| -   Defining target populations and maintaining baseline values for   |
|     indicators had become problematic                                 |
|                                                                       |
| -   Qualitative data did not complement the other data provided       |
|                                                                       |
| -   Intended comprehensive picture was not clearly drawn              |
|                                                                       |
| **Actions Taken:**                                                    |
|                                                                       |
| -   Conducted a data quality review in a fully collaborative fashion  |
|                                                                       |
| > \--Active participation by the implementing NGOs in each critical   |
| > step along the way                                                  |
| >                                                                     |
| > \--Three complementary forums: meetings of three partner            |
| > institutions, consultative sessions with partner organizations and  |
| > implementing NGOs, plenary workshops                                |
| >                                                                     |
| > \--Decisions were obtained by consensus at these sessions           |
|                                                                       |
| -   Took a fresh look at the Results Framework, including indicators  |
|                                                                       |
| > \--Clarified indicator definitions, approaches to data collection   |
| > for these indicators and roles of the various partners and the NGOs |
| > in data collection                                                  |
|                                                                       |
| -   Drafted a new Performance Management Plan to communicate          |
|     adjustments                                                       |
|                                                                       |
| ```{=html}                                                            |
| <!-- -->                                                              |
| ```                                                                   |
| -   Made improvements in clarity of measurement                       |
|                                                                       |
| > \--Efficiency of data collection and usefulness of performance      |
| > reporting were key priorities                                       |
|                                                                       |
| -   Developed a "Data Collection and Reporting Funnel"                |
|                                                                       |
| > \--Information needs begin with the grassroots NGOs (broad          |
| > array/relatively frequent)                                          |
| >                                                                     |
| > \--Subset needed by TAF to synthesize results of NGO efforts and to |
| > report periodically to USAID/Bangladesh                             |
| >                                                                     |
| > \--Subset needed by USAID/Bangladesh to meet its own program        |
| > management and reporting needs                                      |
| >                                                                     |
| > \--Helped to clarify the commonalties and differences in data needs |
| > among various data users                                            |
|                                                                       |
| **Lessons Learned:**                                                  |
|                                                                       |
| -   Plenty of experience with largely uninformative numerical data    |
|                                                                       |
| ```{=html}                                                            |
| <!-- -->                                                              |
| ```                                                                   |
| -   High level of need for NGOs to report on the substance of their   |
|     activities and accomplishments                                    |
|                                                                       |
| ```{=html}                                                            |
| <!-- -->                                                              |
| ```                                                                   |
| -   A mix of data collection approaches: formal surveys, community    |
|     interviews and NGO-based counts accompanied by interpretive       |
|     narratives.                                                       |
|                                                                       |
| ```{=html}                                                            |
| <!-- -->                                                              |
| ```                                                                   |
| -   Performance monitoring system                                     |
|                                                                       |
| > \--Training plan for NGOs in data collection and reporting          |
| >                                                                     |
| > \--Data collection instruments have been designed and are being     |
| > pre-tested in the field                                             |
|                                                                       |
| -   Bangladesh consulting firm has been contracted to collect,        |
|     analyze and report data for a few selected indicators through     |
|     probability surveys                                               |
|                                                                       |
| ```{=html}                                                            |
| <!-- -->                                                              |
| ```                                                                   |
| -   The demand for data use should determine the frequency of         |
|     reporting                                                         |
|                                                                       |
| ```{=html}                                                            |
| <!-- -->                                                              |
| ```                                                                   |
| -   The more "internal" the expected use of the data, the more        |
|     frequently we expect reporting to take place                      |
|                                                                       |
| ```{=html}                                                            |
| <!-- -->                                                              |
| ```                                                                   |
| -   Data collection activities are the minimum necessary to support   |
|     managing for results at all levels                                |
|                                                                       |
| > \--Annual for USAID/W, via the Annual Report                        |
| >                                                                     |
| > \--DP has needs for more frequent reporting (The Asia Foundation    |
| > needs NGO reports on a semi-annual basis for its program management |
| > and reporting purposes.)                                            |
|                                                                       |
| -   NGOs are likely to analyze their own data on approximately a      |
|     monthly basis                                                     |
+-----------------------------------------------------------------------+

##  Task 5 -- Establish Performance Targets

![](media/image28.wmf)Baseline data and performance targets are critical
to managing for results because they are key reference points for
assessing program performance. Baseline data establishes a reference
point for the start of the program period. When the team establishes
performance targets, it commits itself to specific intended results to
be achieved within explicit timeframes. Each year, the team assesses its
performance by comparing actual results against these targets. The
process of establishing baselines and targets can be broken down into
two sub-steps.

### Key Definitions, ADS Guidance and Helpful Resources

![](media/image3.wmf){width="0.9in" height="0.9868055555555556in"}

> **KEY DEFINITIONS**: The following definitions are relevant to this
> PMP task:

-   **Performance Targets:** Specific, planned level of result to be
    achieved within an explicit time frame.

-   **Interim Target:** Targets set for years in between the baseline
    and final target year (e.g., for years in which change is expected
    and data collection is possible).

-   **Ultimate, or Final, Target:** The planned value of a performance
    indicator at the end of the planning period. For SOs, final targets
    are often set at five to eight years away. For IRs, final targets
    are usually set three to five years away.

**Helpful Resources to Learn More about Establishing Targets**

-   TIPS 8, "Establishing Performance Targets",
    [www.dec.org/pdf_docs/pnaby226.pdf](http://www.dec.org/pdf_docs/pnaby226.pdf)

-   Handbook of Democracy and Governance Program Indicators,
    [www.dec.org/pdf_docs/PNACC390.pdf](http://www.dec.org/pdf_docs/PNACC390.pdf)

### 5.1 Establish indicator targets

Once performance indicators have been developed and baseline data
collected, establish final (usually end of SO date) and interim (usually
annual) performance targets. Targets should be optimistic, but
realistic. A common practice is to set targets that will force you to
"stretch" to exceed your past performance. However, special care should
be taken not to set the target outside of reasonable expectations.
Setting a target too high, or allowing zero tolerance for human error,
undermines morale and makes targets appear unattainable. Instead, set
targets that excite team members' and partners' interest and elicit
commitment.

> ![](media/image12.png){width="1.1in" height="1.0in"}**TECHNIQUE --
> Conduct a Target Setting Meeting:** Conduct a target setting meeting
> to identify potential performance targets. ***Helpful Hint 1:
> Facilitating Group Discussions and Decision-Making*** has tips on
> facilitating decision-making meetings. Have at least one target
> setting session for each indicator. Involve your implementing partners
> in the meetings, whenever possible. Collaborating with others who are
> knowledgeable about the local situation and about reasonable
> expectations for accomplishments is key to target setting. Other USAID
> Operating Units, other development agencies, host country
> counterparts, partners, customers and experts can all be invaluable in
> helping to determine the progress that might be expected.

![](media/image12.png){width="1.1in" height="0.8923611111111112in"}

**TECHNIQUE -- Approaches to Target Setting:** Determining appropriate
targets for each indicator can be accomplished in several ways. Much
will depend on the information available or readily gathered. Target
setting approaches include:

-   **Project future trend, then add the "value added" by USAID
    activities.** This approach involves estimating the future trend
    without USAID's program, and then adding whatever gains can be
    expected as a result of USAID's efforts. Projecting the future can
    be very difficult, but can be made somewhat easier if historical
    data are available to establish a trend line.

-   **Establish a final performance target for the end of the planning
    period, then plan progress from the baseline level.** This approach
    involves deciding on the program's performance target for the final
    year, and then defining a path of progress for the years in between.
    Final targets may be based on benchmarking techniques or on
    judgments of experts, program staff, customers or partners.

-   **Set annual performance targets.** This approach is based on
    judgments about what can be achieved each year, instead of starting
    with a final performance level and working backwards.

```{=html}
<!-- -->
```
-   **Benchmarking.** Look at other organizations or institutions that
    use the same types of indicators to demonstrate progress and set
    targets accordingly. For example, if you are tracking the number of
    days for an institution to register new enterprises, research the
    length of time it takes for other countries and use those data
    points as benchmarks for setting your indicator targets.

> ![](media/image6.png){width="1.1in" height="1.1in"}**CONSIDER THIS --
> Principles of Target Setting:** As you apply the target setting
> approaches described above, keep in mind some basic principles for
> setting targets.

-   Think about what the trend has been in the past for any given
    indicator

```{=html}
<!-- -->
```
-   Consider parallel experience from other countries

```{=html}
<!-- -->
```
-   Think through *when* program activities will have an impact on
    indicator values

-   Think about external conditions which may affect indicator values
    over time

-   Consider setting a *target range* rather than a single numerical
    target

-   Consider how clearly the target or the actual will communicate and
    how the trend line will move when deciding on an indicator's unit of
    measurement

-   When indicators are disaggregated, targets should be disaggregated
    as well

### 5.2 Input baselines and targets into performance data tables

![](media/image29.wmf)

> **WORKSHEET 6 -- Performance Indicator Reference Sheet:** Input
> baseline data and targets into the data table in each indicator
> reference sheet.
>
> **WORKSHEET 8 -- Summary Performance Data Table:** You may also want
> to use Worksheet 8 to consolidate all your performance data into a
> summary table. Table 2-7 is an example of a completed summary data
> table.

**Table 2-7. Example of Worksheet 8: Summary Performance Data Table**

<table>
<colgroup>
<col style="width: 5%" />
<col style="width: 10%" />
<col style="width: 9%" />
<col style="width: 9%" />
<col style="width: 11%" />
<col style="width: 6%" />
<col style="width: 6%" />
<col style="width: 5%" />
<col style="width: 5%" />
<col style="width: 6%" />
<col style="width: 6%" />
<col style="width: 6%" />
<col style="width: 8%" />
</colgroup>
<tbody>
<tr class="odd">
<td><strong>SO or IR</strong></td>
<td><strong>Results Statement</strong></td>
<td><strong>Indicator</strong></td>
<td><strong>Unit of Measure</strong></td>
<td><strong>Disaggregation</strong></td>
<td><strong>Baseline Year</strong></td>
<td><strong>Baseline Value</strong></td>
<td><strong>2001 Target</strong></td>
<td><strong>2001 Actual</strong></td>
<td><strong>2002 Target</strong></td>
<td><strong>2002 Actual</strong></td>
<td><strong>2003 Target</strong></td>
<td><strong>2003 Actual</strong></td>
</tr>
<tr class="even">
<td><p>IR</p>
<p>3.1.1</p></td>
<td>Vulnerable communities better able to meet own needs</td>
<td>Number of community groups organized</td>
<td># groups</td>
<td></td>
<td>May-Nov 2000</td>
<td>0</td>
<td>12</td>
<td></td>
<td>36</td>
<td></td>
<td>48</td>
<td></td>
</tr>
<tr class="odd">
<td></td>
<td></td>
<td>Value and percent of community contributions to community
projects</td>
<td><ol type="a">
<li><p>Dollars</p></li>
<li><p>% per project</p></li>
</ol></td>
<td>Source of contribution</td>
<td>May-Nov 2000</td>
<td><p>a) 0</p>
<p>b) 0</p></td>
<td><p>a) $5K</p>
<p>b) 5%</p></td>
<td><p>a)</p>
<p>b)</p></td>
<td><p>a) $20K</p>
<p>b) 10%</p></td>
<td><p>a)</p>
<p>b)</p></td>
<td><p>a) $40K</p>
<p>b) 20%</p></td>
<td><p>a)</p>
<p>b)</p></td>
</tr>
<tr class="even">
<td><p>IR</p>
<p>3.1.1.1</p></td>
<td>Increased access to economic opportunities and support services</td>
<td>Number of loans disbursed</td>
<td># loans</td>
<td><p>Recipients:</p>
<p>Male</p>
<p>Female</p></td>
<td>May-Nov 2000</td>
<td><p>M 15</p>
<p>F 10</p></td>
<td><p>M 25</p>
<p>F 20</p></td>
<td><p>M</p>
<p>F</p></td>
<td><p>M 30</p>
<p>F 30</p></td>
<td><p>M</p>
<p>F</p></td>
<td><p>M 35</p>
<p>F 40</p></td>
<td><p>M</p>
<p>F</p></td>
</tr>
<tr class="odd">
<td></td>
<td></td>
<td>Number of jobs created</td>
<td># jobs</td>
<td><p>Male</p>
<p>Female</p></td>
<td>May-Nov 2000</td>
<td><p>M 75</p>
<p>F 50</p></td>
<td><p>M 125</p>
<p>F 100</p></td>
<td><p>M</p>
<p>F</p></td>
<td><p>M 150</p>
<p>F 150</p></td>
<td><p>M</p>
<p>F</p></td>
<td><p>M 175</p>
<p>F 200</p></td>
<td><p>M</p>
<p>F</p></td>
</tr>
<tr class="even">
<td><p>IR</p>
<p>3.1.1.2</p></td>
<td>Communities organized to address self-defined needs</td>
<td>Number of community development projects developed</td>
<td># projects</td>
<td>Project Type</td>
<td>May-Nov 2000</td>
<td>0</td>
<td>6</td>
<td></td>
<td>18</td>
<td></td>
<td>24</td>
<td></td>
</tr>
</tbody>
</table>

## Task 6 -- Plan for Other Assessing and Learning Activities

Assessing and learning is the process of systematically obtaining useful
feedback and applying it to move programs forward and report progress to
others. Therefore, think about supplementing performance monitoring with
planned evaluations, special studies, and other formal and informal
information sources as needed. This comprehensive approach to planning
for assessment and learning will yield useful performance information
that will help you:

-   Make informed management decisions regarding the best use of
    resources to achieve desired objectives over time

-   Improve the performance, effectiveness, and design of existing
    development assistance

-   Document findings on the impact of development assistance

![](media/image30.wmf)The process of planning for these other elements
can be broken down into five sub-tasks.

### 

### Key Definitions, ADS Guidance and Helpful Resources

![](media/image3.wmf){width="0.9in" height="0.9868055555555556in"}

> **KEY DEFINITIONS**: The following definitions are relevant to this
> PMP development task:

-   **Evaluation:** A relatively structured, analytical effort
    undertaken to answer specific program management questions. An
    evaluation can provide a systematic way to gain insights and reach
    judgments about the effectiveness of specific activities, validity
    of a development hypothesis, utility of performance monitoring
    efforts, or impact of other changes in the development setting on
    achievement of results.

```{=html}
<!-- -->
```
-   **Portfolio Review:** A required periodic review of all aspects of
    an Operating Unit or SO team's programs. It focuses on both
    strategic and operational issues and examines the robustness of the
    underlying development hypothesis and the impact of activities on
    results. It is often held in preparation for submission of the
    Annual Report.

-   **Intensive Program Review:** A program review that is required at
    least once every three years. Intensive program reviews are
    conducted by regional and pillar bureaus for each operating
    unit/program under its responsibility. The purpose of the review is
    to provide an opportunity for Washington offices to examine planned
    and actual progress toward results set forth in the Results
    Framework and Performance Management Plan and to review future
    resource requirements for each SO.

-   **Annual Report:** An annual document produced by each Operating
    Unit and submitted to the responsible Bureau to report on past
    performance, future resources needed, and data needed for
    Agency-wide management, budget decisions, and external reporting.
    Annual Reports began in 2001 and replaced the Results Review and
    Resource Request (R4).

+-----------------------------------------------------------------------+
| **Keys to Effective Performance Management (ADS 203.3.2.1)**          |
|                                                                       |
| -   **Actively Plan for Performance Management:** Operating Units     |
|     should plan for performance management while developing Strategic |
|     Objectives and designing activities. Starting early is key        |
|     because assembling the various elements of the system takes time. |
|     When trying to develop preliminary Performance Management Plans   |
|     (PMP), some Operating Units may discover that the proposed        |
|     Strategic Objectives or Results Frameworks need revision, which   |
|     may require additional time.                                      |
|                                                                       |
| -   **Make Decisions on the Basis of Performance Data:** Operating    |
|     Units should use performance information systematically to assess |
|     progress in achieving results and to make management decisions.   |
|     In terms of the supply of performance information, it is          |
|     important that information be available when required for         |
|     decision-making. On the demand side, decision-makers at all       |
|     levels should use performance information to influence            |
|     decision-making processes and encourage partners to do likewise.  |
|                                                                       |
| -   **Seek** **participation:** Operating Units can strengthen        |
|     performance management by involving customers, partners,          |
|     stakeholders, and other USAID and USG entities in the process of  |
|     performance management. Operating Units can promote participation |
|     by (1) including customers, partners, and stakeholders when       |
|     developing PMPs; (2) including partners when collecting,          |
|     interpreting, and sharing performance information and             |
|     experience; (3) integrating USAID performance management efforts  |
|     with similar processes of partners; and (4) assisting partners in |
|     developing their own performance management and evaluation        |
|     capacity.                                                         |
|                                                                       |
| -   **Streamline the process:** Operating Units should only collect   |
|     and report on the information that is most directly useful for    |
|     performance management. More information is not necessarily       |
|     better because it markedly increases the management burden and    |
|     cost to collect and analyze. Operating Units should also seek to  |
|     align their performance information needs with those of their     |
|     partners, thereby lessening the reporting burden for partner      |
|     organizations. Examples include jointly defining a critical set   |
|     of performance indicators or incorporating data collection        |
|     directly into assistance and acquisition mechanisms. Operating    |
|     Units should ensure that reporting requirements are included in   |
|     acquisition and assistance instruments, and that partner          |
|     reporting schedules provide information at the appropriate times  |
|     for Agency reporting.                                             |
|                                                                       |
| -   **Be transparent:** Operating Units should share information      |
|     widely and be candid in reporting. Transparency involves (1)      |
|     communicating any limitations in data quality so that             |
|     achievements can be honestly assessed; (2) conveying clearly and  |
|     accurately the problems that impede progress and steps that are   |
|     being taken to address them; and (3) avoiding the appearance of   |
|     claiming jointly-achieved results as solely USAID results.        |
+-----------------------------------------------------------------------+

**Helpful Resources to Learn More about Analysis and Evaluation**

-   ADS Chapter 203,
    [www.usaid.gov/pubs/ads/200/203.pdf](http://www.usaid.gov/pubs/ads/200/203.pdf)

-   TIPS 1: Conducting a Participatory Evaluation,
    [www.dec.org/pdf_docs/pnabs539.pdf](http://www.dec.org/pdf_docs/pnabs539.pdf)

```{=html}
<!-- -->
```
-   TIPS 2: Conducting Key Informant Interviews,
    [www.dec.org/pdf_docs/pnabs541.pdf](http://www.dec.org/pdf_docs/pnabs541.pdf)

```{=html}
<!-- -->
```
-   TIPS 3: Preparing an Evaluation Scope of Work,
    [www.dec.org/pdf_docs/pnaby207.pdf](http://www.dec.org/pdf_docs/pnaby207.pdf)

-   TIPS 4: Using Direct Observation Techniques,
    [www.dec.org/pdf_docs/pnaby208.pdf](http://www.dec.org/pdf_docs/pnaby208.pdf)

```{=html}
<!-- -->
```
-   TIPS 5: Using Rapid Appraisal Methods,
    [www.dec.org/pdf_docs/pnaby209.pdf](http://www.dec.org/pdf_docs/pnaby209.pdf)

-   TIPS 10: Conducting Focus Group Interviews,
    [www.dec.org/pdf_docs/pnaby233.pdf](http://www.dec.org/pdf_docs/pnaby233.pdf)

```{=html}
<!-- -->
```
-   TIPS 11: The Role of Evaluation in USAID,
    [www.dec.org/pdf_docs/pnaby239.pdf](http://www.dec.org/pdf_docs/pnaby239.pdf)

Figure 2-8 presents a summary of the assessing and learning practices
for USAID/Mali.

+-----------------------------------------------------------------------+
| **Figure 2-8. Assessing and Learning Put into Practice: USAID/Mali's  |
| Experience**                                                          |
|                                                                       |
| USAID/Mali has developed an extensive system for measuring            |
| programmatic performance and for monitoring/evaluating that           |
| performance. The following is a summary of the key documents that are |
| generated and the mechanisms through which data are collected and     |
| analyzed.                                                             |
|                                                                       |
| **Key Documents:**                                                    |
|                                                                       |
| -   **Results Frameworks and Performance Indicator Table:** A         |
|     separate table for each SO serves as the core instruments for     |
|     documenting performance.                                          |
|                                                                       |
| -   **System of Performance Measurement:** A data set which contains  |
|     all the detailed information that goes into the Results           |
|     Frameworks and Performance Indicators tables. The SP contains     |
|     additional "lower level" results data on individual activities,   |
|     generates various reports, and tabulates information through a    |
|     customized ACCESS-based management information system.            |
|                                                                       |
| -   **Annual Report:** An Operating Unit's major reporting document,  |
|     which also can provide information for the program reviews.       |
|                                                                       |
| -   **Policy Agenda Matrix:** Consolidates, in summary form, all the  |
|     major policy agenda items being tackled through SO programs and   |
|     quantifies progress to date.                                      |
|                                                                       |
| -   **Contractor and PVO progress reports:** Contain basic data and   |
|     performance indicators submitted by contractors and are           |
|     incorporated into the system and Results Frameworks.              |
|                                                                       |
| -   **Government of Mali (GM) national statistics:** Statistics       |
|     collected and compiled by GM are used to measure progress         |
|                                                                       |
| -   **Geographic Information System (GIS):** An Arc/View software     |
|     used to show the geographic location of different                 |
|     activities/facilities of interest.                                |
|                                                                       |
| **Key Monitoring and Evaluation Mechanisms:**                         |
|                                                                       |
| -   **Contractor, PVO and GM data collection systems:** USAID/Mali    |
|     specifies appropriate measures of performance, benchmarks/targets |
|     for achievement at various stages or dates, and the system for    |
|     reporting accomplishments with contractors and PVO partners.      |
|                                                                       |
| -   **Surveys and field visits:** Numerous surveys/studies are done   |
|     annually to either collect or verify information provided by      |
|     partners                                                          |
|                                                                       |
| -   **Program Implementation Reviews (PIR):** Bi-annual sessions used |
|     to review progress activity-by-activity, not just overall SO or   |
|     program.                                                          |
|                                                                       |
| -   **PVO Continuation Application Reviews:** Formal, annual reviews  |
|     held with partner PVOs to assess progress, plans for the next     |
|     phase, and financial requirements                                 |
|                                                                       |
| -   **Retreats for the Operating Unit or partners:** Use retreats as  |
|     a mechanism to look at overall performance -- for both staff of   |
|     the Operating Unit and its partners                               |
+-----------------------------------------------------------------------+

> ![](media/image6.png){width="1.1in" height="1.1in"}**CONSIDER THIS --
> Budget for Performance Monitoring:** As you plan to assess and use
> performance data, consider the costs of collecting, analyzing and
> using that data. Sufficient funding and personnel resources should be
> made available for performance monitoring work. ADS 203.3.2.2 provides
> that three to ten percent of total program resources should be
> allocated in most cases, though factors unique to each activity or
> strategic element will influence this decision. Strive for
> cost-effective performance monitoring. If anticipated costs appear
> prohibitive, consider:

-   Modifying performance indicators to permit less expensive approaches
    to regular data collection

-   Modifying the approach/design of evaluative activities, considering
    rapid, low cost alternatives

-   Modifying the relevant Strategic Objective or Intermediate Result,
    since it is not possible otherwise to judge progress at reasonable
    costs

![](media/image31.wmf)

**WORKSHEET 9 -- Performance Management Task Schedule:** As you go
through Task 6, use Worksheet 9 to schedule all your team\'s assessing
and learning activities---data acquisition and analysis, Portfolio
Reviews performance reporting, data quality assessment, evaluation
plans, etc.

> Table 2-8 is an excerpt from a performance management task schedule
> completed for one USAID Operating Unit in the field.

**Table 2-8. Example of Worksheet 9: Performance Management Task
Schedule**

  -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
  **PERFORMANCE       **FY                             **FY                             **FY                             **FY2007**                           **Notes**
  MANAGEMENT TASKS**  2004**                           2005**                           2006**                                                                
  ------------------- -------- ------- ------- ------- -------- ------- ------- ------- -------- ------- ------- ------- ------------ ------- ------- ------- -------------------
                      Q1       Q2      Q1      Q2      Q3       Q4      Q3      Q4      Q1       Q2      Q3      Q4      Q1           Q2      Q3      Q4      

  **COLLECT                                                                                                                                                   
  CONTEXT-LEVEL                                                                                                                                               
  DATA**                                                                                                                                                      

  Gather data on                               **X**                            **X**                            **X**                                **X**   Before Annual
  macroeconomic                                                                                                                                               Report
  situation                                                                                                                                                   
  (Inflation rates,                                                                                                                                           
  poverty, exchange                                                                                                                                           
  rates)                                                                                                                                                      

  Summarize IMF                                **X**                            **X**                            **X**                                **X**   Before Annual
  country report                                                                                                                                              Report

  **COLLECT SO-LEVEL                                                                                                                                          
  PERFORMANCE DATA**                                                                                                                                          

  SO Indicator #1:    **X**    **X**   **X**   **X**   **X**    **X**   **X**   **X**   **X**    **X**   **X**   **X**   **X**        **X**   **X**   **X**   
  Number of                                                                                                                                                   
  micro-loans by                                                                                                                                              
  USAID-assisted                                                                                                                                              
  micro-finance                                                                                                                                               
  intermediaries                                                                                                                                              

  SO indicator #2:                                              **X**           **X**                            **X**                                **X**   
  Loan approval rate                                                                                                                                          
  for USAID-assisted                                                                                                                                          
  micro-finance                                                                                                                                               
  intermediaries                                                                                                                                              

  **COLLECT IR-LEVEL                                                                                                                                          
  PERFORMANCE DATA**                                                                                                                                          

  Indicator #1.1:              **X**           **X**                                             **X**                                **X**                   
  Milestone score                                                                                                                                             
  rating of progress                                                                                                                                          
  reengineering                                                                                                                                               
  selected                                                                                                                                                    
  administrative                                                                                                                                              
  processes                                                                                                                                                   

  Indicator #3.2:     **X**            **X**           **X**            **X**           **X**            **X**           **X**                **X**           
  Number of loan                                                                                                                                              
  applications in                                                                                                                                             
  region XYZ                                                                                                                                                  

  Etc.                                                                                                                                                        

  Etc.                                                                                                                                                        

  **COLLECT                                                                                                                                                   
  ACTIVITY-LEVEL                                                                                                                                              
  DATA**                                                                                                                                                      

  Gather partner      **X**    **X**   **X**   **X**   **X**    **X**   **X**                                                                                 Project with
  progress reports                                                                                                                                            Partner XXX
  from Partner XXX                                                                                                                                            currently ends in
                                                                                                                                                              third year

  Gather partner      **X**    **X**   **X**   **X**   **X**    **X**   **X**   **X**   **X**    **X**   **X**   **X**   **X**        **X**   **X**   **X**   
  progress reports                                                                                                                                            
  from Partner YYY                                                                                                                                            

  Gather partner                               **X**   **X**    **X**   **X**   **X**   **X**    **X**   **X**   **X**   **X**        **X**   **X**   **X**   Project with
  progress reports                                                                                                                                            Partner ZZZ
  from Partner ZZZ                                                                                                                                            scheduled to begin
                                                                                                                                                              in second year

  **CONDUCT                                                                                                                                                   
  EVALUATIONS &                                                                                                                                               
  SPECIAL STUDIES**                                                                                                                                           

  Report on                    **X**                                    **X**                                                                                 
  administrative                                                                                                                                              
  reengineering at                                                                                                                                            
  ABC government                                                                                                                                              
  agency                                                                                                                                                      

  Mid-strategy review                                                                                    **X**   **X**                                        Specific
                                                                                                                                                              assessments TBD

  Estimate increased                                                                                                                  **X**   **X**           
  investment                                                                                                                                                  
  attributable to                                                                                                                                             
  reengineering of                                                                                                                                            
  key business                                                                                                                                                
  processes                                                                                                                                                   

  **REVIEW                                                                                                                                                    
  PERFORMANCE                                                                                                                                                 
  INFORMATION**                                                                                                                                               

  Review partner      **X**    **X**   **X**   **X**   **X**    **X**   **X**   **X**   **X**    **X**   **X**   **X**   **X**        **X**   **X**   **X**   Informal review of
  activity progress                                                                                                                                           partner reports
  reports                                                                                                                                                     

  Portfolio Review                     **X**                            **X**                            **X**                                **X**           Operational-level
  with Mission                                                                                                                                                review
  Director                                                                                                                                                    

  Review in           **X**                            **X**                            **X**                            **X**                                
  preparation for                                                                                                                                             
  Annual Report                                                                                                                                               

  **REPORT                                                                                                                                                    
  PERFORMANCE                                                                                                                                                 
  RESULTS**                                                                                                                                                   

  Submit Annual                **X**                            **X**                            **X**                                **X**                   
  Report                                                                                                                                                      

  **ASSESS DATA                                                                                                                                               
  QUALITY**                                                                                                                                                   

  Asses data from                                                       **X**                                                                                 Coincides with
  partner XXX                                                                                                                                                 assessment of ABC
                                                                                                                                                              government agency

  Asses data from                              **X**                            **X**                            **X**                                **X**   IR 3 leader
  partner YYY                                                                                                                                                 responsible

  Asses data from                                                               **X**                                                                 **X**   IR 2 leader
  partner ZZZ                                                                                                                                                 responsible

  **REVIEW & UPDATE                                                                                                                                           
  PMP**                                                                                                                                                       

  IR Managers update  **X**    **X**   **X**   **X**   **X**    **X**   **X**   **X**   **X**    **X**   **X**   **X**   **X**        **X**   **X**   **X**   
  indicator sheets on                                                                                                                                         
  shared drive                                                                                                                                                

  Complete PMP in                              **X**                            **X**                            **X**                                **X**   
  binder printed and                                                                                                                                          
  updated                                                                                                                                                     
  -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

### 6.1 Plan for data analysis and use

Everyone needs information, but not everyone knows what to do with raw
data. Data should be processed and synthesized before reporting and
using. Sound analysis of performance data will provide you with useful
information about what happened (against expected results) and why
progress is or is not on track. Properly planning how performance data
will be analyzed, used and presented is at the heart of performance
management. To plan for this, ask these key questions:

-   How will the data be analyzed?

-   Who will be involved in the analysis?

-   Who will use the data and for what purpose?

-   How will the data be presented?

Table 2-9 presents some approaches to apply in planning for performance
data analysis and use:

+------------+--------------------------------+-----------------------+
| **Table    |                                |                       |
| 2-9. Data  |                                |                       |
| Analysis   |                                |                       |
| Ap         |                                |                       |
| proaches** |                                |                       |
+------------+--------------------------------+-----------------------+
| **Analysis | **Analysis Technique**         | **Questions to        |
| Approach** |                                | Consider**            |
+------------+--------------------------------+-----------------------+
| **Analyze  | -   Compare actual performance | -   Did we meet our   |
| data for a |     against targets            |     targets? Why or   |
| single     |                                |     why not?          |
| result**   | -   Compare current            |                       |
|            |     performance to prior year  | -   How does this     |
|            |                                |     period's          |
|            | -   Compare current            |     performance       |
|            |     performance to baseline    |     compare to last   |
|            |                                |     period? Are we on |
|            | -   Analyze trends in          |     track for our     |
|            |     performance                |     ultimate target?  |
|            |                                |                       |
|            |                                | -   Did our critical  |
|            |                                |     assumptions hold  |
|            |                                |     during the        |
|            |                                |     performance       |
|            |                                |     period?           |
|            |                                |                       |
|            |                                | -   What happened     |
|            |                                |     that we did not   |
|            |                                |     expect?           |
|            |                                |                       |
|            |                                | -   What improvements |
|            |                                |     are needed?       |
|            |                                |                       |
|            |                                | -   Are new results   |
|            |                                |     statements,       |
|            |                                |     indicators, or    |
|            |                                |     targets needed?   |
+------------+--------------------------------+-----------------------+
| **Analyze  | -   Examine performance of     |                       |
| data       |     lower results in relation  |                       |
| across the |     to higher results          |                       |
| Results    |                                |                       |
| F          | -   Examine data from critical |                       |
| ramework** |     assumptions to help        |                       |
|            |     interpret results          |                       |
+------------+--------------------------------+-----------------------+
| **Analyze  | -   Examine timing of results  |                       |
| the        |     in relation to timing of   |                       |
| co         |     USAID program efforts      |                       |
| ntribution |                                |                       |
| of USAID's | -   Compare movement in        |                       |
| activities |     results trends to movement |                       |
| to the     |     in level of USAID program  |                       |
| a          |     efforts                    |                       |
| chievement |                                |                       |
| of         | -   Compare performance to     |                       |
| results**  |     control group in similar   |                       |
|            |     environment                |                       |
+------------+--------------------------------+-----------------------+

> ![](media/image32.wmf)
>
> **WORKSHEET 6 -- Performance Indicator Reference Sheet:** Once you
> have determined your data analysis approach, refer to your completed
> indicator worksheets and input your plan for each indicator.

The key to presenting data analysis is to tell a compelling story. Be
candid. Users of performance information will want to know how you plan
to address performance problems and limitations. Visual displays such as
tables, boxes, and figures can condense information, present it in a
clear format, and highlight underlying relationships and trends. This
helps communicate findings to decision-makers more clearly and quickly.
***Helpful Hint 8: Tips for Communicating Performance Information in
Reports*** will provide you with some guidelines in presenting data
analysis in report format.

> ![](media/image12.png){width="1.1in" height="1.0in"}**TECHNIQUE --
> Chart Book Analysis Presentation Approach**: Figure 2-9 provides an
> example of a creative storyboard technique for presenting performance
> data analysis. The slides in the presentation are designed in such a
> way that the audience can clearly understand how the data are
> supporting the program results. The presenter is telling a story that
> is easy to follow without getting lost in all the data contained in
> the graphics. Presentations that simply show one complicated graphic
> after another, with little/no text to make the points that should be
> taken from the analysis, have less impact on the audience. The same
> can also be said about the opposite approach -- avoid reports with
> overly long text and too few graphic presentations that could help the
> reader easily understand program results.
>
> ![](media/image33.wmf)

### 6.2 Plan for performance reviews 

ADS Chapter 203 requires two types of performance reviews of Operating
Unit performance -- the *Portfolio Review* which as an annual exercise
conducted by the operating unit and the *Intensive Program Review* which
is a triennial exercise led by the responsible bureau.

**Portfolio Reviews**: The portfolio review is a required annual review
conducted by the operating unit. Its purpose is to systematically
analyze the progress of each SO examining both strategic and operational
issues. It is intended to bring together various expertise and points of
view to determine whether a program is "on track" or if new actions are
needed to improve the chance of achieving results. Portfolio reviews
should lead to management decisions about program implementation and
feedback into planning and achieving processes. Specific guidance on
conducting Portfolio Reviews can be found in ADS 203.3.7. The guidance
states that the structure, process and timing of a Portfolio Review is
left up to you, but requires that, during the course of the fiscal year,
you review:

-   Progress towards the achievement of the SO during the past year and
    expectations regarding future results achievement

-   Evidence that outputs of activities are adequately supporting the
    relevant IRs and ultimately contributing to the achievement of the
    SO

-   Adequacy of inputs for producing planned outputs

-   Adequacy of performance indicators selected in the PMP

-   Status and timeliness of input mobilization efforts

-   Progress on the annual procurement plan

-   Status of critical assumptions and causal relationships defined in
    the Results Framework along with the related implications for
    performance

-   Status of cross-cutting themes and/or synergies between Strategic
    Objectives

-   Status of related partner efforts that contribute to the achievement
    of IRs and SOs

-   Status of the Operating Unit\'s management agreement and need for
    any changes to the approved Strategic Plan

-   Pipeline levels and future resource requirements

-   SO team effectiveness and adequacy of staffing

-   Vulnerability issues, related corrective efforts and their costs.

You should use information from the reviews to identify and analyze the
implications for the achievement of results. When you identify
significant deficiencies or problems, you may need to alter, increase,
or discontinue activities, or rethink the logic behind the original
expectations.

> ![](media/image12.png){width="1.1in" height="1.0in"}**TECHNIQUE --
> Portfolio Review Approach:** There is no one prescribed structure or
> process for conducting Portfolio Reviews. One of the most common
> approaches is for designated staff to analyze a variety of
> program-related information and prepare issues for discussion in a
> larger group forum that may include SO team members, other members of
> the Operating Unit, and partners. Operating Units may choose to define
> standard procedures that are judged useful for their programs. Many
> units will find it particularly useful to conduct a Portfolio Review
> as part of the preparation process for Annual Reports. Some operating
> units choose to conduct the portfolio review in two parts with each
> part having a different focus. For example, one part might be focused
> on program results such as progress towards SOs, status of
> cross-cutting themes and status of critical assumptions while the
> other is focused on internal mission management issues such as
> procurement, pipeline and staffing.
>
> ![](media/image34.wmf)
>
> **HELPFUL HINT 9: Questions to Guide Portfolio Reviews:** Use Helpful
> Hint 9 to help plan and carry out Portfolio Reviews. The questions
> listed will help you to address key issues that affect the management
> of your portfolio. These issues can be categorized into three broad
> areas as follows:

-   *Strategy and activity issues*: results, outputs, inputs,
    development hypothesis, critical assumptions, non-USAID
    circumstances, and interface between tactics and strategy

-   *Process issues*: indicators and targets, evaluations, teamwork, and
    customer/partner perceptions

-   *Vulnerability issues*: financial vulnerability, other
    vulnerability, and audit readiness

**Intensive Program Reviews:** Intensive program reviews are mandatory
reviews conducted every three years by pillar or regional bureaus of
each operating unit or program. The purpose of the review is to provide
Washington Offices the opportunity to examine thoroughly how each
program is proceeding relative to the results framework and performance
management plan for each SO, and to review resource requirements.

Procedures for conducting an intensive program review is left up to each
Bureau but the process must include other bureaus and offices.

As part of your performance management planning effort, you should
consult with your Bureau about if and when an intensive program review
of your OU will be conducted, so that you can schedule the review
appropriately.

### 6.3 Plan for evaluations and special studies

Evaluation is a relatively structured analytical effort undertaken
selectively to answer specific management questions regarding
USAID-funded assistance programs or activities. Evaluation is also a
management tool that plays a vital role in Agency decision-making,
accountability reporting, and learning. It is an important source of
information about the performance of USAID activities, programs and
strategies.

Determine in advance if special studies can be identified (e.g., DHS,
intermittent surveys) or if evaluation issues can be predicted. Plan to
determine, at the time of the Portfolio Review, if there are performance
deviations (positive or negative) that show need for evaluations or
studies. Near the end of the SO life, plan to determine if something
happened that requires a study to better document the results.

> ![](media/image35.png)**INFORMATION SERVICES:**
>
> **Research and Reference Services (R&RS):** R&RS staff can help you
> determine if other evaluations or special studies have been conducted
> on similar topics. Access R&RS via CDIE Online at
> [http://cdie.usaid.gov](http://cdie.usaid.gov/) (available only within
> the USAID firewall; click \'Research\' or \'Library\' at the top of
> the homepage)
>
> ![](media/image6.png){width="1.1in" height="1.1in"}
>
> **CONSIDER THIS -- Situations that Prompt a Need for Evaluation**: To
> complement ongoing performance monitoring, consider planning an
> evaluation when there is a distinct and clear management need, as in
> the following situations:

-   A key management decision should be made and there is inadequate
    information

-   Performance information indicates an unexpected result (positive or
    negative) that should be explained

-   Customer, partner, or other informed feedback suggests that there
    are implementation problems, unmet needs, or unintended consequences
    or impacts

-   Issues of sustainability, cost-effectiveness, or relevance arise

-   Validity of Results Framework hypotheses or critical assumptions is
    questioned

-   Periodic Portfolio Reviews have identified key questions that need
    to be answered or on which consensus should be developed

-   Extracting lessons is important for the benefit of other Operating
    Units or future programming

![](media/image12.png){width="1.1in" height="1.0in"}

**TECHNIQUE -- Planning for Evaluations:** Take into account the
following key steps and questions, as listed in Table 2-11, when
planning evaluations and special studies.

+-----------------------------------+-----------------------------------+
| **Table 2-11. Key Steps and       |                                   |
| Questions for Planning            |                                   |
| Evaluations**                     |                                   |
+-----------------------------------+-----------------------------------+
| **KEY STEPS**                     | **KEY QUESTIONS**                 |
+-----------------------------------+-----------------------------------+
| -   Decide if and when to         | -   Who is likely to need         |
|     evaluate                      |     information from or about the |
|                                   |     program?                      |
| -   Clarify the evaluation        |                                   |
|     purpose                       | -   What do they need to know?    |
|                                   |                                   |
| -   Use R&RS for research support | -   How would they use the        |
|                                   |     information if they had it?   |
| -   Identify the research         |                                   |
|     questions                     | -   When do they need it?         |
|                                   |                                   |
| -   Select appropriate evaluation | -   How accurate should it be?    |
|     methods                       |                                   |
|                                   | -   When and how should the data  |
| -   Plan for data collection and  |     be collected and analyzed?    |
|     analysis                      |                                   |
|                                   | -   Who is responsible for data   |
| -   Form an evaluation team       |     collection and analysis?      |
|                                   |                                   |
| -   Plan procedures (e.g.,        |                                   |
|     schedule, logistics,          |                                   |
|     reporting needs, budget)      |                                   |
+-----------------------------------+-----------------------------------+

If an evaluation is likely, plan to select an evaluation method from
among several options. Table 2-12 presents some evaluation method
options. ***Helpful Hint 4: Rapid Low-Cost Data Collection Methods***
provides supplemental information on the appropriateness, advantages,
and limitations of rapid appraisal techniques.

**Table 2-12. Evaluation Methods and Key Considerations**

+-----------------------------------------------------------------------+
| **EVALUATION METHOD**                                                 |
+-----------------------------------------------------------------------+
| -   Short workshops to reflect on whether the development hypothesis  |
|     is valid                                                          |
|                                                                       |
| -   Community interview or customer focus groups                      |
|                                                                       |
| -   Large scale surveys                                               |
|                                                                       |
| -   Rapid appraisal or participatory techniques                       |
|                                                                       |
| -   Traditional, formal impact evaluations                            |
+-----------------------------------------------------------------------+
| **KEY CONSIDERATIONS**                                                |
+-----------------------------------------------------------------------+
| -   Nature of the information, analysis, or feedback needed           |
|                                                                       |
| -   Trade-off between quality/validity and cost                       |
|                                                                       |
| -   Cultural considerations                                           |
|                                                                       |
| -   Time-frame of the management need for information                 |
|                                                                       |
| -   Time and resources available                                      |
|                                                                       |
| -   Level of accuracy required                                        |
+-----------------------------------------------------------------------+

![](media/image6.png){width="1.0in" height="1.0in"}

> **CONSIDER THIS -- Special Studies:** In some cases, you may need to
> conduct special studies that go beyond the scope of program monitoring
> or evaluation. One example is the Demographic and Health Surveys (DHS)
> that are conducted approximately every five years to inform health
> sector programming.
>
> As discussed in ADS 203.3.6.1, an important role of special studies is
> to examine whether the needs of vulnerable groups are being met.
> Special studies need to be designed with the same care as evaluations.
> When collecting data for analysis, take particular care to design and
> implement the study in order to minimize error and ensure good data
> quality.
>
> ![](media/image36.wmf)
>
> **WORKSHEET 10 -- Evaluations and Special Studies Planning:** Use
> Worksheet 10 to help document the possible evaluations and special
> studies identified to complement performance monitoring. This
> worksheet will help you identify the subject, timing, and any special
> research considerations. Table 2-13 presents an example of how an
> evaluation or special study may be planned.

**Table 2-13. Evaluations and Special Studies Planning**

+-------------------+-------------+------------------------------------+
| *                 | **When**    | **Key Research Question(s)**       |
| *Evaluation/Study |             |                                    |
| Subject**         |             |                                    |
+-------------------+-------------+------------------------------------+
| **Reproductive    | On-going    | What is the status of reproductive |
| Health Survey**   |             | health in the country?             |
|                   |             |                                    |
|                   |             | What is the nature of citizen's    |
|                   |             | knowledge, attitudes, behaviors,   |
|                   |             | and practices?                     |
+-------------------+-------------+------------------------------------+
| **Community       | April --    | Which communities should be        |
| Assessment for    | May, 2000   | targeted?                          |
| the Social        |             |                                    |
| Development       |             | How should funds be programmed?    |
| Fund**            |             |                                    |
+-------------------+-------------+------------------------------------+
| **World Vision    | April --    | How effective and efficient has    |
| Feeding Program** | May, 2000   | the program been?                  |
|                   |             |                                    |
|                   |             | What are the opportunities for     |
|                   |             | improvement?                       |
+-------------------+-------------+------------------------------------+
| **Community       | April --    | What are the merits of various     |
| Development       | May, 2000   | approaches?                        |
| Approaches**      |             |                                    |
|                   |             | What lessons can be learned from   |
|                   |             | community development efforts in   |
|                   |             | the country?                       |
|                   |             |                                    |
|                   |             | How can sustainability be ensured? |
+-------------------+-------------+------------------------------------+
| **Mission's       | June --     | Should USAID continue to fund      |
| Value-Added in    | Sept, 2000  | these programs?                    |
| the Health        |             |                                    |
| Sector**          |             |                                    |
+-------------------+-------------+------------------------------------+

Although the timing of the PMP development process may be such that the
evaluation will not take place in the immediate future, you may want to
consider the types of questions found in Worksheet 11 when the time
comes to develop the evaluation scope of work.

> ![](media/image37.wmf)
>
> **WORKSHEET 11 -- Evaluation Scope of Work Planning:** Use Worksheet
> 11 to help develop the scope of work for planned evaluations. An
> evaluation scope of work (SOW) is a plan for conducting evaluations or
> special studies. It conveys clear directions to the evaluation team. A
> good SOW usually:

-   Identifies the activity, Intermediate Results, or Strategic Plan to
    be evaluated

-   Provides a brief background on implementation

-   Identifies existing performance information sources

-   States the purpose, audience and use of the evaluation

-   Identifies the evaluation method to answer the questions

-   Clarifies the evaluation questions

-   Identifies the evaluation method(s) to answer the questions

-   Discusses evaluation team composition and participation of
    customers, partners and stakeholders

-   Covers procedures such as schedule and logistics

-   Clarifies requirements for reporting and dissemination

-   Includes a budget

### 6.4 Plan for performance reporting

To enhance learning opportunities within the Agency and among partners
and other stakeholders, plan to report and share progress toward
expected results. Base your reporting upon quantitative and qualitative
performance information gathered through your performance monitoring
systems, evaluations, and other relevant sources. Make every effort to
be open and direct and to share both successes and failures.

To support the Agency in its learning processes, share copies of annual
performance reports, evaluation reports, and other useful information
that may be of broad interest to other teams and units. Furthermore,
copies of these reports are required to be sent to the DEC in electronic
format for uploading onto their web site (see Resources appendix).

The Annual Report will serve as the primary document for reporting
performance information. You may also want to develop your own internal
reports to guide management decisions. For example, consider a periodic
*Activity Manager's Monitoring Report*, which can be revised to be more
clearly linked with performance indicators at the SO and IR levels.

![](media/image12.png){width="1.1in" height="1.0in"}

**TECHNIQUE -- Plan for Annual Reports:** At the end of each calendar
year, Operating Units must submit an Annual Report to Washington that
complies with the annual guidance cable. The annual Portfolio Review can
provide much of the analytical basis for this report. In the Annual
Report, plan to discuss:

-   Progress towards the achievement of the SO over the past fiscal
    year, as well as expectations for future results

-   Evidence that activities are supporting the relevant Intermediate
    Result(s), and ultimately contributing to the achievement of the SO

-   Status of critical assumptions (i.e. whether they continue to hold)
    and causal relationships defined in the Results Framework, and the
    related implications for performance towards the SO and IRs

-   Future resource requirements

> ![](media/image38.wmf)

-   **HELPFUL HINT 8 -- Tips for Communicating Performance Information
    in Reports:** Use Helpful Hint 8 to plan for more effectively
    reporting performance information. Such reports include Annual
    Reports, but may also include other reports internal or external to
    the Operating Unit or Agency.

**6.5 Plan for on-going data quality assessments**

Do not stop reviewing data quality once the performance data is reported
in the Annual Report. Plan to regularly review data quality to ensure
that it continues to support the needs of the SO and IRs in performance
monitoring.

![](media/image12.png){width="1.1in" height="1.0in"}

> **TECHNIQUE -- On-going Data Quality Assurance:** Over the course of
> implementing a Strategic Plan, prepare to:

-   Build data quality assessment into normal work processes, including
    ongoing reviews and site visits

-   Use software checks and edits of data on computer systems and review
    their implementation

-   Use feedback from data users and other stakeholders

-   Compare performance information with other sources of similar data
    or program evaluation

-   Obtain verification by independent parties, including other donors
    and the Office of the Inspector General

For each indicator reported in the Annual Report, reassess data quality
as necessary, but at intervals of no greater than three years. These
assessments will ensure that performance information is sufficiently
complete, accurate, and consistent. Conduct these assessments consistent
with Agency and external guidance. In particular:

-   Verify and validate performance information to ensure that data are
    of reasonable quality

-   Review data collection, maintenance, and processing procedures to
    ensure that they are consistently applied and continue to be
    adequate

```{=html}
<!-- -->
```
-   Document this assessment in the Annual Report, and keep a complete
    report on file

> ![](media/image39.wmf)
>
> **WORKSHEET 7 -- Data Quality Assessment Checklist:** As you did
> during the initial data quality assessment, use Worksheet 7 to conduct
> periodic or on-going assessments of reported performance data.
>
> ![](media/image12.png){width="1.0in"
> height="0.9090277777777778in"}**TECHNIQUE -- Set up a "data quality
> file":** A good way to maintain adequate documentation of data quality
> and assessment is to set up a simple data quality file. Use this file
> to store copies of data collection instruments, source documents, raw
> figures or worksheets used to calculate indicators, data quality
> assessment memos and reports, etc. Refer to ***Helpful Hint 10:
> Maintaining Official SO Team Files*** for additional ideas.
>
> ![](media/image6.png){width="1.1in" height="1.1in"}**CONSIDER THIS --
> Collection of Other Useful Data:** In addition to collecting data on
> *performance*, consider seeking out other useful data to track
> goal-level progress or illustrate the development context or
> environment, status of critical assumptions, and validity of the
> development hypothesis.

-   **Aspirations, or "goal-level" indicators:** An SO team may wish to
    track progress toward results that are beyond USAID's manageable
    interest, or results that are of a higher level than the Strategic
    Objective. For example, a hypothetical USAID economic growth program
    may aspire to achieve significant macro or national-level results
    within, say, twenty years. However, within the five year timeframe
    of the current Strategic Plan, such results may be beyond what the
    Operating Unit and its partners are willing to be held accountable
    for achieving. In this case, the SO team may choose to informally
    track such *aspirational indicators* in order to keep the team aware
    of progress toward long-term, sustainable development.

-   **Development context/environment:** Information about *trends*
    within the larger context of the current environment may be useful
    in understanding the progress of the SO. Many factors affect
    progress, not the least of which are the national social, health,
    and economic conditions. Consider taking these factors into account
    when analyzing and reporting data on performance indicators.
    Examples of *context indicators* relevant to the health sector, for
    instance, include poverty rates (aggregate and by region) and
    mortality rates (maternal, infant, child, and gross). These and
    other relevant macro or research-driven statistics can be tracked
    *informally*. Several data sources may be helpful in this larger
    analysis, including studies by UNICEF, the World Bank, national
    health and economic statistics, and reviews of current social,
    business, and government trends.

-   **Status of critical assumptions:** Gather data *informally* to test
    critical assumptions. Data sources may include studies, reports,
    conferences, and other communications from government institutions,
    other donors, NGOs and PVOs, and other key stakeholders. Review
    critical assumptions regularly and as part of the annual Portfolio
    Review to determine whether they continue to hold and their
    implications for adjusting the approach.

-   **Validity of the development hypothesis:** Validate and reassess
    the development hypothesis as activities progress and the
    environment evolves. Key tools for this assessment can include:
    empirical evidence of the development context and validity of
    critical assumptions, evaluations, special studies, and other
    information sources to include reports of activity impact.

![](media/image40.png)

> **INFORMATION SERVICES:**

-   **Economic and Social Data Service (ESDS):** ESDS can help identify
    and deliver relevant contextual data. Access ESDS via CDIE Online at
    [http://cdie.usaid.gov](http://cdie.usaid.gov/) (available only
    within the USAID firewall; click \'Statistics\' at the top of the
    homepage)

```{=html}
<!-- -->
```
-   **Research and Reference Services (R&RS):** R&RS staff can help
    identify, analyze and deliver relevant contextual data, evaluations,
    studies, reports, conferences, and other communications from
    government institutions, other donors, NGOs and PVOs, and other key
    stakeholders. Access R&RS via CDIE Online at
    [http://cdie.usaid.gov](http://cdie.usaid.gov/) (available only
    within the USAID firewall; click \'Research\' or \'Library\' at the
    top of the homepage)

## Conclusion

Congratulations! By completing Tasks in Parts 1 and 2 of the Toolkit,
you have addressed all of the required and recommended elements listed
in ADS 203.3.3 and referenced in the Overview of this Toolkit. If you
were to compile three of the Toolkit worksheets into your PMP, you will
have documented all of the required and most of the recommended PMP
elements. Table 2.14 shows the elements covered in each of the Toolkit
worksheets.

+------------------------------------+----------+----------+----------+
| **Table 2.14 PMP Elements By       | **W      | **W      | **W      |
| Toolkit Worksheet**                | orksheet | orksheet | orksheet |
|                                    | 6:       | 8:       | 9:       |
|                                    | Per      | Per      | Per      |
|                                    | formance | formance | formance |
|                                    | I        | Data     | Ma       |
|                                    | ndicator | Table**  | nagement |
|                                    | R        |          | Task     |
|                                    | eference |          | Sc       |
|                                    | Sheet**  |          | hedule** |
+------------------------------------+----------+----------+----------+
| ***PMP must include:***            |          |          |          |
+------------------------------------+----------+----------+----------+
| -   At least one performance       | **√**    | **√**    |          |
|     indicator at the SO level,     |          |          |          |
|     with baseline and ultimate     |          |          |          |
|     target levels                  |          |          |          |
+------------------------------------+----------+----------+----------+
| -   At least one performance       | **√**    | **√**    |          |
|     indicators for each IR, with   |          |          |          |
|     baseline and ultimate target   |          |          |          |
|     levels                         |          |          |          |
+------------------------------------+----------+----------+----------+
| ***PMP should include:***          |          |          |          |
+------------------------------------+----------+----------+----------+
| -   Calendar of performance        |          |          | **√**    |
|     management tasks               |          |          |          |
+------------------------------------+----------+----------+----------+
| -   Statement of the set of        | **√**    |          |          |
|     performance indicators to be   |          |          |          |
|     used over the life of the SO   |          |          |          |
+------------------------------------+----------+----------+----------+
| -   Baseline values and targeted   | **√**    | **√**    |          |
|     values for *all* indicators in |          |          |          |
|     the PMP                        |          |          |          |
+------------------------------------+----------+----------+----------+
| -   Source of data and method for  | **√**    |          |          |
|     data collection                |          |          |          |
+------------------------------------+----------+----------+----------+
| -   Schedule for data collection   |          |          | **√**    |
+------------------------------------+----------+----------+----------+
| -   Known data limitations         | **√**    |          |          |
+------------------------------------+----------+----------+----------+
| -   Data quality assessment        | **√**    |          |          |
|     procedures                     |          |          |          |
+------------------------------------+----------+----------+----------+
| -   Baselines and targets by       | **√**    | **√**    |          |
|     indicator                      |          |          |          |
+------------------------------------+----------+----------+----------+
| ***PMP may include:***             |          |          |          |
+------------------------------------+----------+----------+----------+
| -   Description of plans for data  | **√**    |          |          |
|     analysis, report, review and   |          |          |          |
|     use                            |          |          |          |
+------------------------------------+----------+----------+----------+
| -   Estimated costs of collecting, | **√**    |          |          |
|     analyzing and reporting        |          |          |          |
|     performance data               |          |          |          |
+------------------------------------+----------+----------+----------+
| -   Possible evaluation efforts    |          |          | **√**    |
|     identified to complement the   |          |          |          |
|     performance management effort  |          |          |          |
|     and circumstances that require |          |          |          |
|     evaluations or other special   |          |          |          |
|     studies                        |          |          |          |
+------------------------------------+----------+----------+----------+
| -   Plans for monitoring the       | **√**    |          |          |
|     underlying development         |          |          |          |
|     hypothesis, critical           |          |          |          |
|     assumptions and context        |          |          |          |
|     affecting the Results          |          |          |          |
|     Framework                      |          |          |          |
+------------------------------------+----------+----------+----------+

While ADS Chapter 203 does contain a list of PMP requirements, it does
not go so far as to prescribe the contents or organization of a PMP. In
general, these are left to the discretion of the SO team and the
approval of the Operating Unit Director. Refer to ***Helpful Hint 10:
Maintaining Official SO Team Files*** for additional information.

Once you have assembled the initial PMP, communicate the assembled plan
to all relevant stakeholders and begin implementation.

# Part 3: Use, Review and Revise the PMP

Once developed, your Performance Management Plan serves as a living
document that you can use to manage your program for results. As
emphasized at the beginning of this Toolkit, one of the key guiding
principles of the PMP is:

> ***The PMP is the foundation for a sound performance management
> system.** A good PMP is a useful tool for management and
> organizational learning - it provides intelligence for
> decision-makers, and thus serves as a constant desk reference to guide
> the assessment of results. A good PMP is updated annually to ensure
> maximum use for decision-making. The PMP should NOT be developed only
> to satisfy Washington reporting needs and then left to collect dust.*

Use the PMP continuously to make informed management decisions, improve
tactics and organizational processes, identify performance gaps, and set
goals for improvements.

> ![](media/image6.png){width="1.1in" height="1.1in"}
>
> **CONSIDER THIS -- Create a Results-Oriented Culture**: *Planning* for
> performance data analysis and use will not, by itself, create a
> result-oriented culture. If you do not actually *use* the data, no one
> will take performance measurement seriously. SO teams and operating
> unit leadership should strive to:

-   Encourage and emphasize activities that contribute to results

-   Continually assess activities to improve performance

-   Use a mix of measures -- consider effectiveness and efficiency

-   Display current performance information and targets in the work
    environment

-   Hold individuals and teams responsible for managing for results --
    link SO performance to staff work objectives and performance ratings

## Task 1 -- Use the PMP

Once you have developed your PMP, you must start using it in order to
derive the benefits that the PMP can provide. This involves executing
tasks such the following that you laid out in your PMP:

-   Collect performance data

-   Review performance data and implementing partner reports

-   Conduct evaluations and specials studies

-   Conduct performance reviews (portfolio reviews and intensive program
    reviews)

-   Report performance results

-   Conduct data quality assessments

Remember, any data that is reported to Washington must have had a data
quality assessment within the last three years.

Implementing the tasks mentioned above will allow you to:

-   Determine whether your development hypothesis is valid

-   Determine whether the critical assumptions continue to hold

```{=html}
<!-- -->
```
-   Make informed decisions on whether to abandon or modify Agency
    programs, Strategic Objectives, or activities that are not achieving
    intended results

-   Plan new programs, Strategic Objectives, and/or activities

-   Communicate progress to USAID/Washington, the U.S Congress and other
    stakeholders

As you begin the implementation process, you may also want to ensure
that the team is maintaining adequate documentation to support the
performance management process***. Helpful Hint 10: Maintaining Official
SO Team Files*** provides guidance in this area.

## Task 2 -- Review the PMP

Plan to review and revise the PMP at least annually and more often if
necessary. This can be done during the Portfolio Review or Annual Report
preparation. Consider the following questions:

-   Are our indicators working?

-   Are we getting the information that we need?

-   How can we improve the PMP?

During the different phases of implementing a Strategic Plan,
occasionally take a critical look at performance indicators and data
sources to make sure the indicators are still measuring what they were
intended to measure and that data are being collected. Include an
assessment of all performance indicators (at both SO and IR levels) and
cover each data source.

Through the review of PMP data, you may find that you need make changes
to your PMP. For example you may realize that you need to disaggregate
data differently or that you need to collect data more or less
frequently. Or, you may find that you need to understand a particular
trend in the data in more depth, possibly through an evaluation. If you
need to conduct an evaluation, this will be a new performance management
task that you will have to add to your calendar of tasks and plan for at
the appropriate time. In any event, your review of the PMP may result in
your having to update your PMP to accurately reflect current needs.

## 

## Task 3 -- Revise the PMP

If upon review, PMP elements such as indicator definition or responsible
individual need to be changed, the PMP should be updated to reflect
those changes. More significant changes such as changing, adding or
dropping an indicator should only be done if there is a compelling need.
When considering a change like this, Operating Units need to weigh the
advantages of making the change against problems that could occur as a
result of the change. For example, data collected before an indicator or
data collection method was changed may not be comparable to data
collected after the change.

Usually, operating units have the authority to approve changes to
performance indicators without Bureau or Agency approval. If, however,
the change in the indicator reflects a "significant" modification to a
previously approved SO, then approval to change the indicator has to be
sought. ADS 203.3.4.7 provides detailed guidance on changing indicators.

In addition to updating indicator reference sheets, also remember to
update your performance management task schedule. This calendar of tasks
is likely to change regularly based on evolving circumstances.
Maintaining an up-to-date calendar will be critical to ensuring
effective performance management.

When you make any change to your PMP (particularly to indicator
reference sheets), make sure that the rationale for the change is
documented. This is critical if the PMP is to be a useful management
tool to all who use it, including those who may succeed you in your job.
In addition, this prepares the team to answer questions from other
stakeholders who want to know why changes were made and to what degree
performance indicators and data were reviewed.

# Appendix A: Worksheets

  ----------------------------------------------------------- -----------
  Worksheet 1: PMP Development Team Skills Matrix             

  Worksheet 2: PMP Development Workplan                       

  Worksheet 3: Results Statement Assessment                   

  Worksheet 4: Results Framework Assessment                   

  Worksheet 5: Performance Indicator Quality Assessment       

  Worksheet 6: Performance Indicator Reference Sheet          

  Worksheet 7: Data Quality Assessment Checklist              

  Worksheet 8: Summary Performance Data Table                 

  Worksheet 9: Performance Management Task Schedule           

  Worksheet 10: Evaluations and Special Studies Planning      

  Worksheet 11: Evaluation Scope of Work (SOW) Planning       
  ----------------------------------------------------------- -----------

## Worksheet 1: PMP Development Team Skills Matrix 

*Use this worksheet to list all of the proposed team members of the PMP
development team and their proposed roles. Check-mark each skill that
the team members have. Use this worksheet to ensure that you have a good
cross-section of skills represented on the team. An example of the
completed matrix can be found in Part 1, Task 1 of the Toolkit.*

+-------------+----------+----+-----+-----+-----+-----+-----+-----+
| > **Name**  | >        | >  | >   | >   | > * | > * | > * | >   |
|             | **Role** |  * |  ** |  ** | *Kn | *Kn | *Kn |  ** |
|             |          | *K | Has | Has | ows | ows | ows | Has |
|             |          | no | >   | >   | >   | >   | >   | > f |
|             |          | ws | sec | sec |  lo |  US |  PM | aci |
|             |          | >  | tor | tor | cal | AID | > m | lit |
|             |          |  U | >   | >   | > c | > s | eth | ati |
|             |          | SA | exp |  tr | ond | tru | ods | on, |
|             |          | ID | eri | ain | iti | ctu | >   | > a |
|             |          | >  | enc | ing | ons | re, | and | nal |
|             |          |  M | e** | >   | > i | > p | > b | yti |
|             |          | FR |     |  or | n-d | roc | est | cal |
|             |          | >  |     | >   | ept | ess | >   | >   |
|             |          | ap |     |  ed | h** | es, |  pr | and |
|             |          | pr |     | uca |     | >   | act | >   |
|             |          | oa |     | tio |     | cul | ice | rep |
|             |          | ch |     | n** |     | tur | s** | ort |
|             |          | ** |     |     |     | e** |     | > w |
|             |          |    |     |     |     |     |     | rit |
|             |          |    |     |     |     |     |     | ing |
|             |          |    |     |     |     |     |     | >   |
|             |          |    |     |     |     |     |     |  sk |
|             |          |    |     |     |     |     |     | ill |
|             |          |    |     |     |     |     |     | s** |
+-------------+----------+----+-----+-----+-----+-----+-----+-----+
| 1\.         |          |    |     |     |     |     |     |     |
+-------------+----------+----+-----+-----+-----+-----+-----+-----+
| 2\.         |          |    |     |     |     |     |     |     |
+-------------+----------+----+-----+-----+-----+-----+-----+-----+
| 3\.         |          |    |     |     |     |     |     |     |
+-------------+----------+----+-----+-----+-----+-----+-----+-----+
| 4\.         |          |    |     |     |     |     |     |     |
+-------------+----------+----+-----+-----+-----+-----+-----+-----+
| 5\.         |          |    |     |     |     |     |     |     |
+-------------+----------+----+-----+-----+-----+-----+-----+-----+

## Worksheet 2: PMP Development Workplan 

*Use this worksheet to list all of the major tasks and sub-tasks needed
to prepare the PMP. Expand the worksheet by including additional rows in
the table as needed. Another approach would be to use Microsoft Project
to develop the workplan if someone on the team is familiar with it.*

  ----------------------- ----------- ----------- ----------- -----------
  **DESCRIPTION**         **START     **END       **LEVEL OF  **STAFF**
                          DATE**      DATE**      EFFORT**    

  **Phase 1:**                                                

  **Task 1:**                                                 

  **Sub-task 1:**                                             

  **Sub-task 2:**                                             

  **Task 2:**                                                 

  **Sub-task 1:**                                             

  **Sub-task 2:**                                             

  **Phase 2:**                                                

  **Task 1:**                                                 

  **Sub-task 1:**                                             

  **Sub-task 2:**                                             

  **Task 2:**                                                 

  **Sub-task 1:**                                             

  **Sub-task 2:**                                             
  ----------------------- ----------- ----------- ----------- -----------

##  Worksheet 3: Results Statement Assessment

**Sector:**

**Strategic Objective:**

**Results Statement (Name/Number):**

+-----------------------------+-----+-----+-----+---------------------+
| **CRITERIA FOR ASSESSING    | *   | **N | **  | ####### COMMENTS    |
| THE RESULTS STATEMENT**     | *Ye | o** | Un- |                     |
|                             | s** |     | sur |                     |
|                             |     |     | e** |                     |
+-----------------------------+-----+-----+-----+---------------------+
| Is the results statement    |    |    |    |                     |
| **MEASURABLE?**             |     |     |     |                     |
+-----------------------------+-----+-----+-----+---------------------+
| Is the results statement    |    |    |    |                     |
| **MEANINGFUL?**             |     |     |     |                     |
+-----------------------------+-----+-----+-----+---------------------+
| Is the results statement    |    |    |    |                     |
| **REALISTIC?**              |     |     |     |                     |
+-----------------------------+-----+-----+-----+---------------------+
| Is the results statement    |    |    |    |                     |
| focused on **USAID          |     |     |     |                     |
| STRATEGIC COMMITMENTS?**    |     |     |     |                     |
+-----------------------------+-----+-----+-----+---------------------+
| Is the results statement    |    |    |    |                     |
| **CUSTOMER or STAKEHOLDER   |     |     |     |                     |
| DRIVEN?**                   |     |     |     |                     |
+-----------------------------+-----+-----+-----+---------------------+
| Is the results statement    |    |    |    |                     |
| within the **MANAGEABLE     |     |     |     |                     |
| INTEREST** of the Operating |     |     |     |                     |
| Unit and its development    |     |     |     |                     |
| partners?                   |     |     |     |                     |
+-----------------------------+-----+-----+-----+---------------------+
| Is the results statement    |    |    |    |                     |
| focused on **RESULTS** or   |     |     |     |                     |
| outcomes of activities      |     |     |     |                     |
| (such as impact, quality,   |     |     |     |                     |
| cost/efficiency,            |     |     |     |                     |
| timeliness) rather than a   |     |     |     |                     |
| description of activities   |     |     |     |                     |
| themselves?                 |     |     |     |                     |
+-----------------------------+-----+-----+-----+---------------------+
| Is the statement            |    |    |    |                     |
| **UNI-DIMENSIONAL**         |     |     |     |                     |
| (focused on one result      |     |     |     |                     |
| rather than a combination   |     |     |     |                     |
| of results)?                |     |     |     |                     |
+-----------------------------+-----+-----+-----+---------------------+

+-----------------------------------+-----------------------------------+
| **OTHER COMMENTS:**               | **RECOMMENDATION:**               |
|                                   |                                   |
|                                   | \_\_\_ Accept results statement   |
|                                   |                                   |
|                                   | \_\_\_ Revise results statement   |
|                                   | and then accept                   |
|                                   |                                   |
|                                   | \_\_\_ Reject results statement   |
+-----------------------------------+-----------------------------------+

## Worksheet 4: Results Framework Assessment 

Sector:

Name of Strategic Objective:

  ------------------------------ --------- -------- ------------- ---------------------
  **CRITERIA FOR ASSESSING THE   **Yes**   **No**   **Un-sure**   **COMMENTS**
  RESULTS FRAMEWORK**                                             

  **CAUSAL LINKAGE:** At each                                  
  level of the results                                            
  framework, does achievement of                                  
  one result cause the                                            
  achievement of the other? Is                                    
  the linkage direct?                                             

  **CONTRIBUTIONS OF USAID                                     
  PARTNERS:** At each level of                                    
  the results framework, have                                     
  activities been identified                                      
  (regardless of whether they                                     
  will be conducted by USAID or                                   
  its partners) to cause the                                      
  result at the next level?                                       
  \[Note: not all results from                                    
  USAID partners need to be                                       
  identified in the framework                                     
  but there may at least be                                       
  mention of them in the                                          
  narrative that accompanies the                                  
  framework.\]                                                    

  **MANAGEABLE INTEREST (A):**                                 
  Is the SO level result one                                      
  that the team, working with                                     
  its partners, can materially                                    
  affect?                                                         

  **MANAGEABLE INTEREST (B):**                                 
  Is the team willing to be held                                  
  accountable for all results                                     
  within the results framework,                                   
  including the SO level result?                                  

  **CRITICAL ASSUMPTIONS**: Have                               
  all the critical assumptions                                    
  been identified at each level                                   
  of the results framework?                                       
  ------------------------------ --------- -------- ------------- ---------------------

+-----------------------------------+-----------------------------------+
| **OTHER COMMENTS:**               | **RECOMMENDATION:**               |
|                                   |                                   |
|                                   | \_\_\_ Accept results framework   |
|                                   |                                   |
|                                   | \_\_\_ Revise results framework   |
|                                   | and then accept                   |
|                                   |                                   |
|                                   | \_\_\_ Reject results framework   |
+-----------------------------------+-----------------------------------+

*NOTE: Refer to **TIPS 13: Building a Results Framework** for additional
information and examples of quality results frameworks.*

## Worksheet 5: Performance Indicator Quality Assessment

**Name of Indicator:**

**Name of Relevant Result:**

+-----------------------------------------+----------------------------+
| **CRITERIA**                            | **COMMENTS**               |
+=========================================+============================+
| ***Is the indicator DIRECT?***          |                            |
|                                         |                            |
|  Does it closely measure the result it |                            |
| is intended to measure?                 |                            |
|                                         |                            |
|  Is it grounded in theory and          |                            |
| practice?                               |                            |
|                                         |                            |
|  Does it represent an acceptable       |                            |
| measure to both proponents and          |                            |
| skeptics?                               |                            |
|                                         |                            |
|  If it is a proxy, is it as directly   |                            |
| related to the relevant result as       |                            |
| possible?                               |                            |
+-----------------------------------------+----------------------------+
| ***Is the indicator OBJECTIVE?***       |                            |
|                                         |                            |
|  Is it unambiguous about what is being |                            |
| measured?                               |                            |
|                                         |                            |
|  Is there general agreement over the   |                            |
| interpretation of the results?          |                            |
|                                         |                            |
|  Is it unidimensional (i.e., does it   |                            |
| measure only one phenomenon at a time)? |                            |
|                                         |                            |
|  Is it operationally precise (i.e., is |                            |
| there no ambiguity over what kind of    |                            |
| data should be collected)?              |                            |
+-----------------------------------------+----------------------------+
| ***Is the indicator USEFUL for          |                            |
| management?***                          |                            |
|                                         |                            |
| -   Useful at what level? (SO? Project? |                            |
|     Agency?)                            |                            |
|                                         |                            |
| -   How will it be used?                |                            |
+-----------------------------------------+----------------------------+
| ***Is the indicator PRACTICAL?***       |                            |
|                                         |                            |
|  Are timely data available (i.e., is   |                            |
| data current and available on regular   |                            |
| basis)?                                 |                            |
|                                         |                            |
|  Can the data be collected frequently  |                            |
| enough to inform management decisions?  |                            |
|                                         |                            |
|  Are data valid and reliable?          |                            |
|                                         |                            |
|  Are the costs of data collection      |                            |
| reasonable?                             |                            |
+-----------------------------------------+----------------------------+
| Is the indicator ATTRIBUTABLE to USAID  |                            |
| effort?                                 |                            |
|                                         |                            |
| -   Are the links between USAID         |                            |
|     supported activities and the result |                            |
|     being measured clear and            |                            |
|     significant?                        |                            |
|                                         |                            |
| -   Can the result be attributed, at    |                            |
|     least in part, to USAID efforts?    |                            |
+-----------------------------------------+----------------------------+
| ***Is the indicator TIMELY?***          |                            |
|                                         |                            |
| -   Are data available when needed for  |                            |
|     decision making?                    |                            |
|                                         |                            |
| -   Are data available frequently       |                            |
|     enough for decision making          |                            |
+-----------------------------------------+----------------------------+
| ***Is the indicator ADEQUATE?***        |                            |
|                                         |                            |
|  Does it merely indicate progress      |                            |
| rather than attempt to fully describe   |                            |
| everything an activity accomplishes?    |                            |
|                                         |                            |
| -   Taken as a group, are the indicator |                            |
|     and its companion indicators the    |                            |
|     minimum necessary to ensure that    |                            |
|     progress toward the given result is |                            |
|     sufficiently captured?              |                            |
+-----------------------------------------+----------------------------+
| ***Should the indicator be              |                            |
| DISAGGREGATED?***                       |                            |
|                                         |                            |
| -   Is dissaggregation                  |                            |
|     necessary/appropriate?              |                            |
+-----------------------------------------+----------------------------+
| ***Does the indicator reflect GENDER    |                            |
| CONSIDERATIONS? (if technical analysis  |                            |
| demonstrates the need for this)***      |                            |
+-----------------------------------------+----------------------------+

**OTHER COMMENTS:**

**RECOMMENDATION:**

## Worksheet 6: Performance Indicator Reference Sheet 

*Use this comprehensive reference sheet to record and update all
relevant specifications and details for a particular indicator. Imagine
that you are providing a new staff member with a quick but complete
overview of this performance indicator, including where the raw data
comes from and how it can be analyzed. Edit the headings to make them
more relevant to your situation, or modify the sheet to meet Operating
Unit requirements as needed. For suggestions on how to complete this
form, see following page. Also, an example of a completed performance
indicator reference sheet can be found in Part 2, Task 2.4 of the
Toolkit.*

  ---------------------- -------------- -------------- ---------------------------------
  **Performance                                        
  Indicator Reference                                  
  Sheet**                                              

  **Name of Strategic                                  
  Objective:**                                         

  **Name of Intermediate                               
  Result:**                                            

  **Name of Indicator:**                               

  **Is this an Annual                                  
  Report indicator?** No                               
  \_\_\_ Yes \_\_\_\_,                                 
  for Reporting Year(s)                                
  \_\_\_\_\_\_\_\_\_                                   

  **DESCRIPTION *(Refer                                
  to Toolkit Part 2,                                   
  Task 2)***                                           

  **Precise                                            
  Definition(s):**                                     

  **Unit of Measure:**                                 

  **Disaggregated by:**                                

  **Justification &                                    
  Management Utility:**                                

  **PLAN FOR DATA                                      
  ACQUISITION BY USAID                                 
  *(Refer to Toolkit                                   
  Part 2, Task 3)***                                   

  **Data collection                                    
  method:**                                            

  **Data Source:**                                     

  **Method of data                                     
  acquisition by                                       
  USAID:**                                             

  **Frequency and timing                               
  of data acquisition by                               
  USAID:**                                             

  **Estimated Cost of                                  
  Data Acquisition:**                                  

  **Individual                                         
  responsible at                                       
  USAID:**                                             

  **Individual                                         
  responsible for                                      
  providing data to                                    
  USAID:**                                             

  **Location of Data                                   
  Storage:**                                           

  **DATA QUALITY ISSUES                                
  *(Refer to Toolkit                                   
  Part 2, Task 4)***                                   

  **Date of Initial Data                               
  Quality Assessment:**                                

  **Known Data                                         
  Limitations and                                      
  Significance (if                                     
  any):**                                              

  **Actions Taken or                                   
  Planned to Address                                   
  Data Limitations:**                                  

  **Date of Future Data                                
  Quality Assessments:**                               

  **Procedures for                                     
  Future Data Quality                                  
  Assessments:**                                       

  **PLAN FOR DATA                                      
  ANALYSIS, REVIEW, &                                  
  REPORTING *(Refer to                                 
  Toolkit Part 2, Task                                 
  6)***                                                

  **Data Analysis:**                                   

  **Presentation of                                    
  Data:**                                              

  **Review of Data:**                                  

  **Reporting of Data:**                               

  **OTHER NOTES *(Refer                                
  to Toolkit Part 2,                                   
  Tasks 4 & 5)***                                      

  **Notes on                                           
  Baselines/Targets:**                                 

  **Other Notes:**                                     

  **PERFORMANCE                                        
  INDICATOR VALUES**                                   

  **Year**               **Target**     **Actual**     **Notes**

                                                       

                                                       

                                                       

                                                       

  **THIS SHEET LAST                                    
  UPDATED ON:**                                        
  ---------------------- -------------- -------------- ---------------------------------

+------+--------------+--------------+---------------------------------+
| **   |              |              |                                 |
| Inst |              |              |                                 |
| ruct |              |              |                                 |
| ions |              |              |                                 |
| for  |              |              |                                 |
| Co   |              |              |                                 |
| mple |              |              |                                 |
| ting |              |              |                                 |
| the  |              |              |                                 |
| Per  |              |              |                                 |
| form |              |              |                                 |
| ance |              |              |                                 |
| I    |              |              |                                 |
| ndic |              |              |                                 |
| ator |              |              |                                 |
| R    |              |              |                                 |
| efer |              |              |                                 |
| ence |              |              |                                 |
| She  |              |              |                                 |
| et** |              |              |                                 |
+------+--------------+--------------+---------------------------------+
| **   |              |              |                                 |
| Name |              |              |                                 |
| of   |              |              |                                 |
| S    |              |              |                                 |
| trat |              |              |                                 |
| egic |              |              |                                 |
| Obje |              |              |                                 |
| ctiv |              |              |                                 |
| e:** |              |              |                                 |
| E    |              |              |                                 |
| nter |              |              |                                 |
| the  |              |              |                                 |
| nu   |              |              |                                 |
| mber |              |              |                                 |
| and  |              |              |                                 |
| full |              |              |                                 |
| name |              |              |                                 |
| of   |              |              |                                 |
| the  |              |              |                                 |
| SO.  |              |              |                                 |
+------+--------------+--------------+---------------------------------+
| **   |              |              |                                 |
| Name |              |              |                                 |
| of   |              |              |                                 |
| Inte |              |              |                                 |
| rmed |              |              |                                 |
| iate |              |              |                                 |
| R    |              |              |                                 |
| esul |              |              |                                 |
| t:** |              |              |                                 |
| E    |              |              |                                 |
| nter |              |              |                                 |
| the  |              |              |                                 |
| nu   |              |              |                                 |
| mber |              |              |                                 |
| and  |              |              |                                 |
| full |              |              |                                 |
| name |              |              |                                 |
| of   |              |              |                                 |
| the  |              |              |                                 |
| IR,  |              |              |                                 |
| if   |              |              |                                 |
| ap   |              |              |                                 |
| plic |              |              |                                 |
| able |              |              |                                 |
+------+--------------+--------------+---------------------------------+
| **   |              |              |                                 |
| Name |              |              |                                 |
| of   |              |              |                                 |
| Indi |              |              |                                 |
| cato |              |              |                                 |
| r:** |              |              |                                 |
| E    |              |              |                                 |
| nter |              |              |                                 |
| the  |              |              |                                 |
| full |              |              |                                 |
| t    |              |              |                                 |
| itle |              |              |                                 |
| of   |              |              |                                 |
| the  |              |              |                                 |
| in   |              |              |                                 |
| dica |              |              |                                 |
| tor. |              |              |                                 |
+------+--------------+--------------+---------------------------------+
| **Is |              |              |                                 |
| this |              |              |                                 |
| an   |              |              |                                 |
| An   |              |              |                                 |
| nual |              |              |                                 |
| Re   |              |              |                                 |
| port |              |              |                                 |
| indi |              |              |                                 |
| cato |              |              |                                 |
| r?** |              |              |                                 |
| E    |              |              |                                 |
| nter |              |              |                                 |
| yes  |              |              |                                 |
| or   |              |              |                                 |
| no,  |              |              |                                 |
| and  |              |              |                                 |
| cla  |              |              |                                 |
| rify |              |              |                                 |
| w    |              |              |                                 |
| hich |              |              |                                 |
| r    |              |              |                                 |
| epor |              |              |                                 |
| ting |              |              |                                 |
| y    |              |              |                                 |
| ears |              |              |                                 |
| (s). |              |              |                                 |
+------+--------------+--------------+---------------------------------+
| *    |              |              |                                 |
| *DES |              |              |                                 |
| CRIP |              |              |                                 |
| TION |              |              |                                 |
| *(R  |              |              |                                 |
| efer |              |              |                                 |
| to   |              |              |                                 |
| Too  |              |              |                                 |
| lkit |              |              |                                 |
| Part |              |              |                                 |
| 2,   |              |              |                                 |
| Task |              |              |                                 |
| 2    |              |              |                                 |
| )*** |              |              |                                 |
+------+--------------+--------------+---------------------------------+
| *    |              |              |                                 |
| *Pre |              |              |                                 |
| cise |              |              |                                 |
| Defi |              |              |                                 |
| niti |              |              |                                 |
| on(s |              |              |                                 |
| ):** |              |              |                                 |
| De   |              |              |                                 |
| fine |              |              |                                 |
| the  |              |              |                                 |
| spec |              |              |                                 |
| ific |              |              |                                 |
| w    |              |              |                                 |
| ords |              |              |                                 |
| or   |              |              |                                 |
| elem |              |              |                                 |
| ents |              |              |                                 |
| used |              |              |                                 |
| in   |              |              |                                 |
| the  |              |              |                                 |
| in   |              |              |                                 |
| dica |              |              |                                 |
| tor. |              |              |                                 |
+------+--------------+--------------+---------------------------------+
| **   |              |              |                                 |
| Unit |              |              |                                 |
| of   |              |              |                                 |
| Me   |              |              |                                 |
| asur |              |              |                                 |
| e:** |              |              |                                 |
| E    |              |              |                                 |
| nter |              |              |                                 |
| the  |              |              |                                 |
| unit |              |              |                                 |
| of   |              |              |                                 |
| mea  |              |              |                                 |
| sure |              |              |                                 |
| (*nu |              |              |                                 |
| mber |              |              |                                 |
| of.  |              |              |                                 |
| ..*, |              |              |                                 |
| *per |              |              |                                 |
| cent |              |              |                                 |
| of.  |              |              |                                 |
| ..*, |              |              |                                 |
| or   |              |              |                                 |
| *US  |              |              |                                 |
| do   |              |              |                                 |
| llar |              |              |                                 |
| s*). |              |              |                                 |
| Cla  |              |              |                                 |
| rify |              |              |                                 |
| the  |              |              |                                 |
| min  |              |              |                                 |
| imum |              |              |                                 |
| or   |              |              |                                 |
| max  |              |              |                                 |
| imum |              |              |                                 |
| va   |              |              |                                 |
| lues |              |              |                                 |
| if   |              |              |                                 |
| ne   |              |              |                                 |
| eded |              |              |                                 |
| (    |              |              |                                 |
| *min |              |              |                                 |
| imum |              |              |                                 |
| s    |              |              |                                 |
| core |              |              |                                 |
| is   |              |              |                                 |
| 1.0  |              |              |                                 |
| and  |              |              |                                 |
| max  |              |              |                                 |
| imum |              |              |                                 |
| s    |              |              |                                 |
| core |              |              |                                 |
| is   |              |              |                                 |
| 5.   |              |              |                                 |
| 0*). |              |              |                                 |
| Cla  |              |              |                                 |
| rify |              |              |                                 |
| if   |              |              |                                 |
| the  |              |              |                                 |
| nu   |              |              |                                 |
| mber |              |              |                                 |
| is   |              |              |                                 |
| cu   |              |              |                                 |
| mula |              |              |                                 |
| tive |              |              |                                 |
| or   |              |              |                                 |
| spec |              |              |                                 |
| ific |              |              |                                 |
| to   |              |              |                                 |
| the  |              |              |                                 |
| y    |              |              |                                 |
| ear. |              |              |                                 |
| Cla  |              |              |                                 |
| rify |              |              |                                 |
| n    |              |              |                                 |
| umer |              |              |                                 |
| ator |              |              |                                 |
| and  |              |              |                                 |
| den  |              |              |                                 |
| omin |              |              |                                 |
| ator |              |              |                                 |
| if   |              |              |                                 |
| app  |              |              |                                 |
| lica |              |              |                                 |
| ble. |              |              |                                 |
+------+--------------+--------------+---------------------------------+
| **D  |              |              |                                 |
| isag |              |              |                                 |
| greg |              |              |                                 |
| ated |              |              |                                 |
| b    |              |              |                                 |
| y:** |              |              |                                 |
| List |              |              |                                 |
| any  |              |              |                                 |
| pla  |              |              |                                 |
| nned |              |              |                                 |
| ways |              |              |                                 |
| of   |              |              |                                 |
| di   |              |              |                                 |
| sagg |              |              |                                 |
| rega |              |              |                                 |
| ting |              |              |                                 |
| the  |              |              |                                 |
| data |              |              |                                 |
| (*   |              |              |                                 |
| male |              |              |                                 |
| /fem |              |              |                                 |
| ale, |              |              |                                 |
| yout |              |              |                                 |
| h/ad |              |              |                                 |
| ult, |              |              |                                 |
| urba |              |              |                                 |
| n/ru |              |              |                                 |
| ral, |              |              |                                 |
| reg  |              |              |                                 |
| ion, |              |              |                                 |
| et   |              |              |                                 |
| c*.) |              |              |                                 |
| and  |              |              |                                 |
| jus  |              |              |                                 |
| tify |              |              |                                 |
| why  |              |              |                                 |
| use  |              |              |                                 |
| ful. |              |              |                                 |
+------+--------------+--------------+---------------------------------+
| **J  |              |              |                                 |
| usti |              |              |                                 |
| fica |              |              |                                 |
| tion |              |              |                                 |
| &    |              |              |                                 |
| Ma   |              |              |                                 |
| nage |              |              |                                 |
| ment |              |              |                                 |
| Ut   |              |              |                                 |
| ilit |              |              |                                 |
| y:** |              |              |                                 |
| Bri  |              |              |                                 |
| efly |              |              |                                 |
| desc |              |              |                                 |
| ribe |              |              |                                 |
| *    |              |              |                                 |
| why* |              |              |                                 |
| this |              |              |                                 |
| pa   |              |              |                                 |
| rtic |              |              |                                 |
| ular |              |              |                                 |
| i    |              |              |                                 |
| ndic |              |              |                                 |
| ator |              |              |                                 |
| was  |              |              |                                 |
| sele |              |              |                                 |
| cted |              |              |                                 |
| and  |              |              |                                 |
| *    |              |              |                                 |
| how* |              |              |                                 |
| it   |              |              |                                 |
| will |              |              |                                 |
| be   |              |              |                                 |
| us   |              |              |                                 |
| eful |              |              |                                 |
| for  |              |              |                                 |
| mana |              |              |                                 |
| ging |              |              |                                 |
| per  |              |              |                                 |
| form |              |              |                                 |
| ance |              |              |                                 |
| of   |              |              |                                 |
| the  |              |              |                                 |
| SO   |              |              |                                 |
| te   |              |              |                                 |
| am's |              |              |                                 |
| po   |              |              |                                 |
| rtfo |              |              |                                 |
| lio? |              |              |                                 |
| If   |              |              |                                 |
| the  |              |              |                                 |
| v    |              |              |                                 |
| alue |              |              |                                 |
| of   |              |              |                                 |
| this |              |              |                                 |
| i    |              |              |                                 |
| ndic |              |              |                                 |
| ator |              |              |                                 |
| chan |              |              |                                 |
| ges, |              |              |                                 |
| what |              |              |                                 |
| does |              |              |                                 |
| this |              |              |                                 |
| indi |              |              |                                 |
| cate |              |              |                                 |
| a    |              |              |                                 |
| bout |              |              |                                 |
| the  |              |              |                                 |
| prog |              |              |                                 |
| ram? |              |              |                                 |
+------+--------------+--------------+---------------------------------+
| **   |              |              |                                 |
| PLAN |              |              |                                 |
| FOR  |              |              |                                 |
| DATA |              |              |                                 |
| ACQ  |              |              |                                 |
| UISI |              |              |                                 |
| TION |              |              |                                 |
| BY   |              |              |                                 |
| U    |              |              |                                 |
| SAID |              |              |                                 |
| *(R  |              |              |                                 |
| efer |              |              |                                 |
| to   |              |              |                                 |
| Too  |              |              |                                 |
| lkit |              |              |                                 |
| Part |              |              |                                 |
| 2,   |              |              |                                 |
| Task |              |              |                                 |
| 3    |              |              |                                 |
| )*** |              |              |                                 |
+------+--------------+--------------+---------------------------------+
| **   |              |              |                                 |
| Data |              |              |                                 |
| co   |              |              |                                 |
| llec |              |              |                                 |
| tion |              |              |                                 |
| m    |              |              |                                 |
| etho |              |              |                                 |
| d:** |              |              |                                 |
| Desc |              |              |                                 |
| ribe |              |              |                                 |
| the  |              |              |                                 |
| *to  |              |              |                                 |
| ols* |              |              |                                 |
| and  |              |              |                                 |
| *    |              |              |                                 |
| meth |              |              |                                 |
| ods* |              |              |                                 |
| for  |              |              |                                 |
| co   |              |              |                                 |
| llec |              |              |                                 |
| ting |              |              |                                 |
| the  |              |              |                                 |
| raw  |              |              |                                 |
| d    |              |              |                                 |
| ata. |              |              |                                 |
| Exam |              |              |                                 |
| ples |              |              |                                 |
| incl |              |              |                                 |
| ude: |              |              |                                 |
| le   |              |              |                                 |
| dger |              |              |                                 |
| of   |              |              |                                 |
| pat  |              |              |                                 |
| ient |              |              |                                 |
| na   |              |              |                                 |
| mes, |              |              |                                 |
| docu |              |              |                                 |
| ment |              |              |                                 |
| rev  |              |              |                                 |
| iew, |              |              |                                 |
| st   |              |              |                                 |
| ruct |              |              |                                 |
| ured |              |              |                                 |
| int  |              |              |                                 |
| ervi |              |              |                                 |
| ews, |              |              |                                 |
| f    |              |              |                                 |
| ocus |              |              |                                 |
| g    |              |              |                                 |
| roup |              |              |                                 |
| int  |              |              |                                 |
| ervi |              |              |                                 |
| ews, |              |              |                                 |
| wri  |              |              |                                 |
| tten |              |              |                                 |
| sur  |              |              |                                 |
| vey, |              |              |                                 |
| di   |              |              |                                 |
| rect |              |              |                                 |
| obse |              |              |                                 |
| rvat |              |              |                                 |
| ion, |              |              |                                 |
| s    |              |              |                                 |
| elf- |              |              |                                 |
| repo |              |              |                                 |
| rted |              |              |                                 |
| info |              |              |                                 |
| rmat |              |              |                                 |
| ion, |              |              |                                 |
| and  |              |              |                                 |
| so   |              |              |                                 |
| on.  |              |              |                                 |
| Who  |              |              |                                 |
| coll |              |              |                                 |
| ects |              |              |                                 |
| the  |              |              |                                 |
| raw  |              |              |                                 |
| data |              |              |                                 |
| and  |              |              |                                 |
| w    |              |              |                                 |
| here |              |              |                                 |
| is   |              |              |                                 |
| it   |              |              |                                 |
| st   |              |              |                                 |
| ored |              |              |                                 |
| be   |              |              |                                 |
| fore |              |              |                                 |
| it   |              |              |                                 |
| gets |              |              |                                 |
| to   |              |              |                                 |
| US   |              |              |                                 |
| AID? |              |              |                                 |
+------+--------------+--------------+---------------------------------+
| **   |              |              |                                 |
| Data |              |              |                                 |
| S    |              |              |                                 |
| ourc |              |              |                                 |
| e:** |              |              |                                 |
| Iden |              |              |                                 |
| tify |              |              |                                 |
| the  |              |              |                                 |
| so   |              |              |                                 |
| urce |              |              |                                 |
| of   |              |              |                                 |
| data |              |              |                                 |
| (e   |              |              |                                 |
| .g., |              |              |                                 |
| DHS  |              |              |                                 |
| sur  |              |              |                                 |
| vey; |              |              |                                 |
| mini |              |              |                                 |
| stry |              |              |                                 |
| d    |              |              |                                 |
| ata; |              |              |                                 |
| par  |              |              |                                 |
| tner |              |              |                                 |
| reco |              |              |                                 |
| rds) |              |              |                                 |
+------+--------------+--------------+---------------------------------+
| **Me |              |              |                                 |
| thod |              |              |                                 |
| of   |              |              |                                 |
| data |              |              |                                 |
| acq  |              |              |                                 |
| uisi |              |              |                                 |
| tion |              |              |                                 |
| by   |              |              |                                 |
| USAI |              |              |                                 |
| D:** |              |              |                                 |
| How  |              |              |                                 |
| does |              |              |                                 |
| U    |              |              |                                 |
| SAID |              |              |                                 |
| acq  |              |              |                                 |
| uire |              |              |                                 |
| the  |              |              |                                 |
| data |              |              |                                 |
| or   |              |              |                                 |
| rep  |              |              |                                 |
| ort? |              |              |                                 |
| Desc |              |              |                                 |
| ribe |              |              |                                 |
| the  |              |              |                                 |
| form |              |              |                                 |
| in   |              |              |                                 |
| w    |              |              |                                 |
| hich |              |              |                                 |
| the  |              |              |                                 |
| SO   |              |              |                                 |
| team |              |              |                                 |
| will |              |              |                                 |
| rec  |              |              |                                 |
| eive |              |              |                                 |
| the  |              |              |                                 |
| data |              |              |                                 |
| (    |              |              |                                 |
| such |              |              |                                 |
| as   |              |              |                                 |
| peri |              |              |                                 |
| odic |              |              |                                 |
| mo   |              |              |                                 |
| nito |              |              |                                 |
| ring |              |              |                                 |
| rep  |              |              |                                 |
| ort, |              |              |                                 |
| or   |              |              |                                 |
| comp |              |              |                                 |
| iled |              |              |                                 |
| su   |              |              |                                 |
| rvey |              |              |                                 |
| anal |              |              |                                 |
| ysis |              |              |                                 |
| repo |              |              |                                 |
| rt). |              |              |                                 |
+------+--------------+--------------+---------------------------------+
| **F  |              |              |                                 |
| requ |              |              |                                 |
| ency |              |              |                                 |
| and  |              |              |                                 |
| ti   |              |              |                                 |
| ming |              |              |                                 |
| of   |              |              |                                 |
| data |              |              |                                 |
| ac   |              |              |                                 |
| quis |              |              |                                 |
| itio |              |              |                                 |
| n:** |              |              |                                 |
| Desc |              |              |                                 |
| ribe |              |              |                                 |
| *how |              |              |                                 |
| of   |              |              |                                 |
| ten* |              |              |                                 |
| data |              |              |                                 |
| will |              |              |                                 |
| be   |              |              |                                 |
| rece |              |              |                                 |
| ived |              |              |                                 |
| by   |              |              |                                 |
| the  |              |              |                                 |
| SO   |              |              |                                 |
| Team |              |              |                                 |
| or   |              |              |                                 |
| O    |              |              |                                 |
| pera |              |              |                                 |
| ting |              |              |                                 |
| U    |              |              |                                 |
| nit, |              |              |                                 |
| and  |              |              |                                 |
| *wh  |              |              |                                 |
| en*. |              |              |                                 |
+------+--------------+--------------+---------------------------------+
| **E  |              |              |                                 |
| stim |              |              |                                 |
| ated |              |              |                                 |
| cost |              |              |                                 |
| of   |              |              |                                 |
| data |              |              |                                 |
| ac   |              |              |                                 |
| quis |              |              |                                 |
| itio |              |              |                                 |
| n:** |              |              |                                 |
| Esti |              |              |                                 |
| mate |              |              |                                 |
| the  |              |              |                                 |
| cost |              |              |                                 |
| (in  |              |              |                                 |
| dol  |              |              |                                 |
| lars |              |              |                                 |
| an   |              |              |                                 |
| d/or |              |              |                                 |
| l    |              |              |                                 |
| evel |              |              |                                 |
| of   |              |              |                                 |
| eff  |              |              |                                 |
| ort) |              |              |                                 |
| of   |              |              |                                 |
| col  |              |              |                                 |
| lect |              |              |                                 |
| ing, |              |              |                                 |
| a    |              |              |                                 |
| naly |              |              |                                 |
| zing |              |              |                                 |
| and  |              |              |                                 |
| p    |              |              |                                 |
| rovi |              |              |                                 |
| ding |              |              |                                 |
| the  |              |              |                                 |
| data |              |              |                                 |
| to   |              |              |                                 |
| US   |              |              |                                 |
| AID. |              |              |                                 |
| Cla  |              |              |                                 |
| rify |              |              |                                 |
| if   |              |              |                                 |
| t    |              |              |                                 |
| here |              |              |                                 |
| are  |              |              |                                 |
| any  |              |              |                                 |
| di   |              |              |                                 |
| rect |              |              |                                 |
| c    |              |              |                                 |
| osts |              |              |                                 |
| to   |              |              |                                 |
| U    |              |              |                                 |
| SAID |              |              |                                 |
| for  |              |              |                                 |
| co   |              |              |                                 |
| llec |              |              |                                 |
| ting |              |              |                                 |
| this |              |              |                                 |
| d    |              |              |                                 |
| ata, |              |              |                                 |
| or   |              |              |                                 |
| if   |              |              |                                 |
| the  |              |              |                                 |
| c    |              |              |                                 |
| osts |              |              |                                 |
| are  |              |              |                                 |
| incl |              |              |                                 |
| uded |              |              |                                 |
| in   |              |              |                                 |
| an   |              |              |                                 |
| exis |              |              |                                 |
| ting |              |              |                                 |
| acti |              |              |                                 |
| vity |              |              |                                 |
| or   |              |              |                                 |
| c    |              |              |                                 |
| ontr |              |              |                                 |
| act. |              |              |                                 |
+------+--------------+--------------+---------------------------------+
| **I  |              |              |                                 |
| ndiv |              |              |                                 |
| idua |              |              |                                 |
| l(s) |              |              |                                 |
| res  |              |              |                                 |
| pons |              |              |                                 |
| ible |              |              |                                 |
| at   |              |              |                                 |
| USAI |              |              |                                 |
| D:** |              |              |                                 |
| Iden |              |              |                                 |
| tify |              |              |                                 |
| the  |              |              |                                 |
| spec |              |              |                                 |
| ific |              |              |                                 |
| SO   |              |              |                                 |
| team |              |              |                                 |
| me   |              |              |                                 |
| mber |              |              |                                 |
| who  |              |              |                                 |
| will |              |              |                                 |
| be   |              |              |                                 |
| *    |              |              |                                 |
| dire |              |              |                                 |
| ctly |              |              |                                 |
| resp |              |              |                                 |
| onsi |              |              |                                 |
| ble* |              |              |                                 |
| for  |              |              |                                 |
| a    |              |              |                                 |
| cqui |              |              |                                 |
| ring |              |              |                                 |
| the  |              |              |                                 |
| d    |              |              |                                 |
| ata. |              |              |                                 |
+------+--------------+--------------+---------------------------------+
| **I  |              |              |                                 |
| ndiv |              |              |                                 |
| idua |              |              |                                 |
| l(s) |              |              |                                 |
| res  |              |              |                                 |
| pons |              |              |                                 |
| ible |              |              |                                 |
| for  |              |              |                                 |
| p    |              |              |                                 |
| rovi |              |              |                                 |
| ding |              |              |                                 |
| data |              |              |                                 |
| to   |              |              |                                 |
| USAI |              |              |                                 |
| D:** |              |              |                                 |
| Iden |              |              |                                 |
| tify |              |              |                                 |
| who  |              |              |                                 |
| is   |              |              |                                 |
| res  |              |              |                                 |
| pons |              |              |                                 |
| ible |              |              |                                 |
| for  |              |              |                                 |
| p    |              |              |                                 |
| rovi |              |              |                                 |
| ding |              |              |                                 |
| the  |              |              |                                 |
| data |              |              |                                 |
| to   |              |              |                                 |
| US   |              |              |                                 |
| AID. |              |              |                                 |
+------+--------------+--------------+---------------------------------+
| **   |              |              |                                 |
| Loca |              |              |                                 |
| tion |              |              |                                 |
| of   |              |              |                                 |
| Data |              |              |                                 |
| St   |              |              |                                 |
| orag |              |              |                                 |
| e:** |              |              |                                 |
| Iden |              |              |                                 |
| tify |              |              |                                 |
| w    |              |              |                                 |
| here |              |              |                                 |
| the  |              |              |                                 |
| data |              |              |                                 |
| will |              |              |                                 |
| be   |              |              |                                 |
| ma   |              |              |                                 |
| inta |              |              |                                 |
| ined |              |              |                                 |
| in   |              |              |                                 |
| the  |              |              |                                 |
| O    |              |              |                                 |
| pera |              |              |                                 |
| ting |              |              |                                 |
| Unit |              |              |                                 |
| (e   |              |              |                                 |
| .g., |              |              |                                 |
| spec |              |              |                                 |
| ific |              |              |                                 |
| file |              |              |                                 |
| cabi |              |              |                                 |
| net, |              |              |                                 |
| or   |              |              |                                 |
| spec |              |              |                                 |
| ific |              |              |                                 |
| fo   |              |              |                                 |
| lder |              |              |                                 |
| on   |              |              |                                 |
| sh   |              |              |                                 |
| ared |              |              |                                 |
| co   |              |              |                                 |
| mput |              |              |                                 |
| er). |              |              |                                 |
+------+--------------+--------------+---------------------------------+
| **   |              |              |                                 |
| DATA |              |              |                                 |
| QUA  |              |              |                                 |
| LITY |              |              |                                 |
| IS   |              |              |                                 |
| SUES |              |              |                                 |
| *(R  |              |              |                                 |
| efer |              |              |                                 |
| to   |              |              |                                 |
| Too  |              |              |                                 |
| lkit |              |              |                                 |
| Part |              |              |                                 |
| 2,   |              |              |                                 |
| Task |              |              |                                 |
| 4    |              |              |                                 |
| )*** |              |              |                                 |
+------+--------------+--------------+---------------------------------+
| **   |              |              |                                 |
| Date |              |              |                                 |
| of   |              |              |                                 |
| Ini  |              |              |                                 |
| tial |              |              |                                 |
| Data |              |              |                                 |
| Qua  |              |              |                                 |
| lity |              |              |                                 |
| A    |              |              |                                 |
| sses |              |              |                                 |
| smen |              |              |                                 |
| t:** |              |              |                                 |
| E    |              |              |                                 |
| nter |              |              |                                 |
| the  |              |              |                                 |
| date |              |              |                                 |
| of   |              |              |                                 |
| ini  |              |              |                                 |
| tial |              |              |                                 |
| data |              |              |                                 |
| qua  |              |              |                                 |
| lity |              |              |                                 |
| as   |              |              |                                 |
| sess |              |              |                                 |
| ment |              |              |                                 |
| and  |              |              |                                 |
| the  |              |              |                                 |
| res  |              |              |                                 |
| pons |              |              |                                 |
| ible |              |              |                                 |
| pa   |              |              |                                 |
| rty. |              |              |                                 |
+------+--------------+--------------+---------------------------------+
| **K  |              |              |                                 |
| nown |              |              |                                 |
| Data |              |              |                                 |
| Lim  |              |              |                                 |
| itat |              |              |                                 |
| ions |              |              |                                 |
| and  |              |              |                                 |
| Sign |              |              |                                 |
| ific |              |              |                                 |
| ance |              |              |                                 |
| (if  |              |              |                                 |
| any  |              |              |                                 |
| ):** |              |              |                                 |
| Desc |              |              |                                 |
| ribe |              |              |                                 |
| any  |              |              |                                 |
| data |              |              |                                 |
| lim  |              |              |                                 |
| itat |              |              |                                 |
| ions |              |              |                                 |
| di   |              |              |                                 |
| scov |              |              |                                 |
| ered |              |              |                                 |
| du   |              |              |                                 |
| ring |              |              |                                 |
| the  |              |              |                                 |
| ini  |              |              |                                 |
| tial |              |              |                                 |
| data |              |              |                                 |
| qua  |              |              |                                 |
| lity |              |              |                                 |
| ass  |              |              |                                 |
| essm |              |              |                                 |
| ent. |              |              |                                 |
| Dis  |              |              |                                 |
| cuss |              |              |                                 |
| the  |              |              |                                 |
| sign |              |              |                                 |
| ific |              |              |                                 |
| ance |              |              |                                 |
| of   |              |              |                                 |
| any  |              |              |                                 |
| data |              |              |                                 |
| weak |              |              |                                 |
| ness |              |              |                                 |
| that |              |              |                                 |
| may  |              |              |                                 |
| af   |              |              |                                 |
| fect |              |              |                                 |
| con  |              |              |                                 |
| clus |              |              |                                 |
| ions |              |              |                                 |
| a    |              |              |                                 |
| bout |              |              |                                 |
| the  |              |              |                                 |
| ex   |              |              |                                 |
| tent |              |              |                                 |
| to   |              |              |                                 |
| w    |              |              |                                 |
| hich |              |              |                                 |
| per  |              |              |                                 |
| form |              |              |                                 |
| ance |              |              |                                 |
| g    |              |              |                                 |
| oals |              |              |                                 |
| have |              |              |                                 |
| been |              |              |                                 |
| a    |              |              |                                 |
| chie |              |              |                                 |
| ved. |              |              |                                 |
+------+--------------+--------------+---------------------------------+
| *    |              |              |                                 |
| *Act |              |              |                                 |
| ions |              |              |                                 |
| T    |              |              |                                 |
| aken |              |              |                                 |
| or   |              |              |                                 |
| Pla  |              |              |                                 |
| nned |              |              |                                 |
| to   |              |              |                                 |
| Add  |              |              |                                 |
| ress |              |              |                                 |
| Data |              |              |                                 |
| Li   |              |              |                                 |
| mita |              |              |                                 |
| tion |              |              |                                 |
| s:** |              |              |                                 |
| Desc |              |              |                                 |
| ribe |              |              |                                 |
| how  |              |              |                                 |
| you  |              |              |                                 |
| have |              |              |                                 |
| or   |              |              |                                 |
| will |              |              |                                 |
| take |              |              |                                 |
| co   |              |              |                                 |
| rrec |              |              |                                 |
| tive |              |              |                                 |
| act  |              |              |                                 |
| ion, |              |              |                                 |
| if   |              |              |                                 |
| p    |              |              |                                 |
| ossi |              |              |                                 |
| ble, |              |              |                                 |
| to   |              |              |                                 |
| add  |              |              |                                 |
| ress |              |              |                                 |
| data |              |              |                                 |
| qua  |              |              |                                 |
| lity |              |              |                                 |
| iss  |              |              |                                 |
| ues. |              |              |                                 |
+------+--------------+--------------+---------------------------------+
| **   |              |              |                                 |
| Date |              |              |                                 |
| of   |              |              |                                 |
| Fu   |              |              |                                 |
| ture |              |              |                                 |
| Data |              |              |                                 |
| Qua  |              |              |                                 |
| lity |              |              |                                 |
| As   |              |              |                                 |
| sess |              |              |                                 |
| ment |              |              |                                 |
| s:** |              |              |                                 |
| E    |              |              |                                 |
| nter |              |              |                                 |
| the  |              |              |                                 |
| pla  |              |              |                                 |
| nned |              |              |                                 |
| date |              |              |                                 |
| for  |              |              |                                 |
| su   |              |              |                                 |
| bseq |              |              |                                 |
| uent |              |              |                                 |
| data |              |              |                                 |
| qua  |              |              |                                 |
| lity |              |              |                                 |
| asse |              |              |                                 |
| ssme |              |              |                                 |
| nts. |              |              |                                 |
+------+--------------+--------------+---------------------------------+
| **Pr |              |              |                                 |
| oced |              |              |                                 |
| ures |              |              |                                 |
| for  |              |              |                                 |
| Fu   |              |              |                                 |
| ture |              |              |                                 |
| Data |              |              |                                 |
| Qua  |              |              |                                 |
| lity |              |              |                                 |
| As   |              |              |                                 |
| sess |              |              |                                 |
| ment |              |              |                                 |
| s:** |              |              |                                 |
| Desc |              |              |                                 |
| ribe |              |              |                                 |
| *    |              |              |                                 |
| how* |              |              |                                 |
| the  |              |              |                                 |
| data |              |              |                                 |
| will |              |              |                                 |
| be   |              |              |                                 |
| asse |              |              |                                 |
| ssed |              |              |                                 |
| in   |              |              |                                 |
| the  |              |              |                                 |
| fu   |              |              |                                 |
| ture |              |              |                                 |
| (e   |              |              |                                 |
| .g., |              |              |                                 |
| spot |              |              |                                 |
| ch   |              |              |                                 |
| ecks |              |              |                                 |
| of   |              |              |                                 |
| par  |              |              |                                 |
| tner |              |              |                                 |
| d    |              |              |                                 |
| ata, |              |              |                                 |
| f    |              |              |                                 |
| inan |              |              |                                 |
| cial |              |              |                                 |
| au   |              |              |                                 |
| dit, |              |              |                                 |
| site |              |              |                                 |
| vis  |              |              |                                 |
| its, |              |              |                                 |
| or   |              |              |                                 |
| soft |              |              |                                 |
| ware |              |              |                                 |
| edit |              |              |                                 |
| che  |              |              |                                 |
| ck). |              |              |                                 |
+------+--------------+--------------+---------------------------------+
| **   |              |              |                                 |
| PLAN |              |              |                                 |
| FOR  |              |              |                                 |
| DATA |              |              |                                 |
| A    |              |              |                                 |
| NALY |              |              |                                 |
| SIS, |              |              |                                 |
| REV  |              |              |                                 |
| IEW, |              |              |                                 |
| &    |              |              |                                 |
| R    |              |              |                                 |
| EPOR |              |              |                                 |
| TING |              |              |                                 |
| *(R  |              |              |                                 |
| efer |              |              |                                 |
| to   |              |              |                                 |
| Too  |              |              |                                 |
| lkit |              |              |                                 |
| Part |              |              |                                 |
| 2,   |              |              |                                 |
| Task |              |              |                                 |
| 6    |              |              |                                 |
| )*** |              |              |                                 |
+------+--------------+--------------+---------------------------------+
| **   |              |              |                                 |
| Data |              |              |                                 |
| Ana  |              |              |                                 |
| lysi |              |              |                                 |
| s:** |              |              |                                 |
| Desc |              |              |                                 |
| ribe |              |              |                                 |
| *    |              |              |                                 |
| how* |              |              |                                 |
| the  |              |              |                                 |
| raw  |              |              |                                 |
| data |              |              |                                 |
| will |              |              |                                 |
| be   |              |              |                                 |
| a    |              |              |                                 |
| naly |              |              |                                 |
| zed, |              |              |                                 |
| *    |              |              |                                 |
| who* |              |              |                                 |
| will |              |              |                                 |
| do   |              |              |                                 |
| it,  |              |              |                                 |
| and  |              |              |                                 |
| *wh  |              |              |                                 |
| en*. |              |              |                                 |
+------+--------------+--------------+---------------------------------+
| **   |              |              |                                 |
| Pres |              |              |                                 |
| enta |              |              |                                 |
| tion |              |              |                                 |
| of   |              |              |                                 |
| Dat  |              |              |                                 |
| a:** |              |              |                                 |
| Desc |              |              |                                 |
| ribe |              |              |                                 |
| how  |              |              |                                 |
| tab  |              |              |                                 |
| les, |              |              |                                 |
| cha  |              |              |                                 |
| rts, |              |              |                                 |
| gra  |              |              |                                 |
| phs, |              |              |                                 |
| or   |              |              |                                 |
| o    |              |              |                                 |
| ther |              |              |                                 |
| dev  |              |              |                                 |
| ices |              |              |                                 |
| will |              |              |                                 |
| be   |              |              |                                 |
| used |              |              |                                 |
| to   |              |              |                                 |
| pre  |              |              |                                 |
| sent |              |              |                                 |
| d    |              |              |                                 |
| ata, |              |              |                                 |
| ei   |              |              |                                 |
| ther |              |              |                                 |
| in   |              |              |                                 |
| tern |              |              |                                 |
| ally |              |              |                                 |
| wi   |              |              |                                 |
| thin |              |              |                                 |
| the  |              |              |                                 |
| SO   |              |              |                                 |
| team |              |              |                                 |
| or   |              |              |                                 |
| O    |              |              |                                 |
| pera |              |              |                                 |
| ting |              |              |                                 |
| U    |              |              |                                 |
| nit, |              |              |                                 |
| or   |              |              |                                 |
| ex   |              |              |                                 |
| tern |              |              |                                 |
| ally |              |              |                                 |
| to   |              |              |                                 |
| Wa   |              |              |                                 |
| shin |              |              |                                 |
| gton |              |              |                                 |
| or   |              |              |                                 |
| o    |              |              |                                 |
| ther |              |              |                                 |
| au   |              |              |                                 |
| dien |              |              |                                 |
| ces. |              |              |                                 |
+------+--------------+--------------+---------------------------------+
| **Re |              |              |                                 |
| view |              |              |                                 |
| of   |              |              |                                 |
| Dat  |              |              |                                 |
| a:** |              |              |                                 |
| Desc |              |              |                                 |
| ribe |              |              |                                 |
| *w   |              |              |                                 |
| hen* |              |              |                                 |
| and  |              |              |                                 |
| *    |              |              |                                 |
| how* |              |              |                                 |
| the  |              |              |                                 |
| SO   |              |              |                                 |
| team |              |              |                                 |
| or   |              |              |                                 |
| O    |              |              |                                 |
| pera |              |              |                                 |
| ting |              |              |                                 |
| Unit |              |              |                                 |
| will |              |              |                                 |
| re   |              |              |                                 |
| view |              |              |                                 |
| the  |              |              |                                 |
| data |              |              |                                 |
| and  |              |              |                                 |
| anal |              |              |                                 |
| ysis |              |              |                                 |
| (e   |              |              |                                 |
| .g., |              |              |                                 |
| du   |              |              |                                 |
| ring |              |              |                                 |
| p    |              |              |                                 |
| ortf |              |              |                                 |
| olio |              |              |                                 |
| rev  |              |              |                                 |
| iew, |              |              |                                 |
| mis  |              |              |                                 |
| sion |              |              |                                 |
| inte |              |              |                                 |
| rnal |              |              |                                 |
| rev  |              |              |                                 |
| iew, |              |              |                                 |
| or   |              |              |                                 |
| ac   |              |              |                                 |
| tivi |              |              |                                 |
| ty-l |              |              |                                 |
| evel |              |              |                                 |
| rev  |              |              |                                 |
| iews |              |              |                                 |
| with |              |              |                                 |
| impl |              |              |                                 |
| emen |              |              |                                 |
| ting |              |              |                                 |
| pa   |              |              |                                 |
| rtne |              |              |                                 |
| rs). |              |              |                                 |
+------+--------------+--------------+---------------------------------+
| **R  |              |              |                                 |
| epor |              |              |                                 |
| ting |              |              |                                 |
| of   |              |              |                                 |
| Dat  |              |              |                                 |
| a:** |              |              |                                 |
| List |              |              |                                 |
| any  |              |              |                                 |
| inte |              |              |                                 |
| rnal |              |              |                                 |
| or   |              |              |                                 |
| exte |              |              |                                 |
| rnal |              |              |                                 |
| rep  |              |              |                                 |
| orts |              |              |                                 |
| that |              |              |                                 |
| will |              |              |                                 |
| fea  |              |              |                                 |
| ture |              |              |                                 |
| the  |              |              |                                 |
| data |              |              |                                 |
| an   |              |              |                                 |
| d/or |              |              |                                 |
| anal |              |              |                                 |
| ysis |              |              |                                 |
| of   |              |              |                                 |
| this |              |              |                                 |
| i    |              |              |                                 |
| ndic |              |              |                                 |
| ator |              |              |                                 |
| (e   |              |              |                                 |
| .g., |              |              |                                 |
| An   |              |              |                                 |
| nual |              |              |                                 |
| Re   |              |              |                                 |
| port |              |              |                                 |
| data |              |              |                                 |
| tab  |              |              |                                 |
| les, |              |              |                                 |
| An   |              |              |                                 |
| nual |              |              |                                 |
| Re   |              |              |                                 |
| port |              |              |                                 |
| na   |              |              |                                 |
| rrat |              |              |                                 |
| ive, |              |              |                                 |
| Bu   |              |              |                                 |
| dget |              |              |                                 |
| Ju   |              |              |                                 |
| stif |              |              |                                 |
| icat |              |              |                                 |
| ion, |              |              |                                 |
| re   |              |              |                                 |
| port |              |              |                                 |
| to   |              |              |                                 |
| Amb  |              |              |                                 |
| assa |              |              |                                 |
| dor, |              |              |                                 |
| or   |              |              |                                 |
| acti |              |              |                                 |
| vity |              |              |                                 |
| m    |              |              |                                 |
| anag |              |              |                                 |
| er's |              |              |                                 |
| repo |              |              |                                 |
| rt). |              |              |                                 |
+------+--------------+--------------+---------------------------------+
| **O  |              |              |                                 |
| THER |              |              |                                 |
| N    |              |              |                                 |
| OTES |              |              |                                 |
| *(R  |              |              |                                 |
| efer |              |              |                                 |
| to   |              |              |                                 |
| Too  |              |              |                                 |
| lkit |              |              |                                 |
| Part |              |              |                                 |
| 2,   |              |              |                                 |
| Task |              |              |                                 |
| 5    |              |              |                                 |
| )*** |              |              |                                 |
+------+--------------+--------------+---------------------------------+
| **N  |              |              |                                 |
| otes |              |              |                                 |
| on   |              |              |                                 |
| Base |              |              |                                 |
| line |              |              |                                 |
| s/Ta |              |              |                                 |
| rget |              |              |                                 |
| s:** |              |              |                                 |
| Exp  |              |              |                                 |
| lain |              |              |                                 |
| how  |              |              |                                 |
| the  |              |              |                                 |
| b    |              |              |                                 |
| asel |              |              |                                 |
| ines |              |              |                                 |
| and  |              |              |                                 |
| tar  |              |              |                                 |
| gets |              |              |                                 |
| were |              |              |                                 |
| set  |              |              |                                 |
| and  |              |              |                                 |
| iden |              |              |                                 |
| tify |              |              |                                 |
| any  |              |              |                                 |
| ass  |              |              |                                 |
| umpt |              |              |                                 |
| ions |              |              |                                 |
| m    |              |              |                                 |
| ade. |              |              |                                 |
| If   |              |              |                                 |
| b    |              |              |                                 |
| asel |              |              |                                 |
| ines |              |              |                                 |
| and  |              |              |                                 |
| tar  |              |              |                                 |
| gets |              |              |                                 |
| have |              |              |                                 |
| not  |              |              |                                 |
| been |              |              |                                 |
| set, |              |              |                                 |
| iden |              |              |                                 |
| tify |              |              |                                 |
| *w   |              |              |                                 |
| hen* |              |              |                                 |
| and  |              |              |                                 |
| *    |              |              |                                 |
| how* |              |              |                                 |
| this |              |              |                                 |
| will |              |              |                                 |
| be   |              |              |                                 |
| d    |              |              |                                 |
| one. |              |              |                                 |
+------+--------------+--------------+---------------------------------+
| **O  |              |              |                                 |
| ther |              |              |                                 |
| Note |              |              |                                 |
| s:** |              |              |                                 |
| Use  |              |              |                                 |
| this |              |              |                                 |
| s    |              |              |                                 |
| pace |              |              |                                 |
| as   |              |              |                                 |
| nee  |              |              |                                 |
| ded. |              |              |                                 |
+------+--------------+--------------+---------------------------------+
| *    |              |              |                                 |
| *PER |              |              |                                 |
| FORM |              |              |                                 |
| ANCE |              |              |                                 |
| I    |              |              |                                 |
| NDIC |              |              |                                 |
| ATOR |              |              |                                 |
| VALU |              |              |                                 |
| ES** |              |              |                                 |
+------+--------------+--------------+---------------------------------+
| **Ye | **Target**   | **Actual**   | **Notes**                       |
| ar** |              |              |                                 |
+------+--------------+--------------+---------------------------------+
|      |              |              |                                 |
+------+--------------+--------------+---------------------------------+
|      |              |              |                                 |
+------+--------------+--------------+---------------------------------+
|      |              |              |                                 |
+------+--------------+--------------+---------------------------------+
|      |              |              |                                 |
+------+--------------+--------------+---------------------------------+
| **   |              |              |                                 |
| THIS |              |              |                                 |
| S    |              |              |                                 |
| HEET |              |              |                                 |
| LAST |              |              |                                 |
| UPD  |              |              |                                 |
| ATED |              |              |                                 |
| O    |              |              |                                 |
| N:** |              |              |                                 |
| mm/d |              |              |                                 |
| d/yy |              |              |                                 |
|      |              |              |                                 |
| To   |              |              |                                 |
| a    |              |              |                                 |
| void |              |              |                                 |
| ver  |              |              |                                 |
| sion |              |              |                                 |
| con  |              |              |                                 |
| trol |              |              |                                 |
| p    |              |              |                                 |
| robl |              |              |                                 |
| ems, |              |              |                                 |
| type |              |              |                                 |
| the  |              |              |                                 |
| date |              |              |                                 |
| of   |              |              |                                 |
| most |              |              |                                 |
| re   |              |              |                                 |
| cent |              |              |                                 |
| revi |              |              |                                 |
| sion |              |              |                                 |
| or   |              |              |                                 |
| up   |              |              |                                 |
| date |              |              |                                 |
| to   |              |              |                                 |
| this |              |              |                                 |
| r    |              |              |                                 |
| efer |              |              |                                 |
| ence |              |              |                                 |
| sh   |              |              |                                 |
| eet. |              |              |                                 |
+------+--------------+--------------+---------------------------------+

## Worksheet 7: Data Quality Assessment Checklist

*Refer to this checklist when the SO team conducts both initial and
periodic data quality assessments. The full list does not have to be
completed---the SO team may wish to identify the most critical data
quality issues for formal or informal assessment.*

  -----------------------------------------------------------------------
  **Name of Strategic Objective:**

  **Name of Intermediate Result** (if applicable):

  **Name of Performance indicator**:

  **Data source(s):**

  **Partner or contractor who provided the data** (if applicable):

  **Year or period for which the data are being reported:**

  **Is this indicator reported in the Annual Report?** *(circle one)* YES
  NO

  **Date(s) of assessment:**

  **Location(s) of assessment:**

  **Assessment team members:**
  -----------------------------------------------------------------------

  -------------------------------------------------------------------------------------------------------------------
  *For Office Use Only*

  SO team leader approval:
  X\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_Date\_\_\_\_\_\_\_\_\_\_\_\_\_\_

  Mission director or delegate approval:
  X\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_Date\_\_\_\_\_\_\_\_\_\_\_\_\_\_

  Copies to:

  Comments
  -------------------------------------------------------------------------------------------------------------------

+-------------------------+----+----+---------------------------------+
| **1. VALIDITY---Do the  |    |    |                                 |
| data adequately         |    |    |                                 |
| represent               |    |    |                                 |
| performance?**          |    |    |                                 |
+=========================+====+====+=================================+
|                         | *  | ** | **Comments**                    |
|                         | *Y | No |                                 |
|                         | es | ** |                                 |
|                         | ** |    |                                 |
+-------------------------+----+----+---------------------------------+
| **Face Validity**       |    |    |                                 |
+-------------------------+----+----+---------------------------------+
| -   Is there a solid,   |   |   |                                 |
|     logical relation    |    |    |                                 |
|     between the         |    |    |                                 |
|     activity or program |    |    |                                 |
|     and what is being   |    |    |                                 |
|     measured, or are    |    |    |                                 |
|     there significant   |    |    |                                 |
|     uncontrollable      |    |    |                                 |
|     factors?            |    |    |                                 |
+-------------------------+----+----+---------------------------------+
|                         |    |    |                                 |
+-------------------------+----+----+---------------------------------+
| **Measurement Error**   |    |    |                                 |
+-------------------------+----+----+---------------------------------+
| *Sampling Error* (only  |    |    |                                 |
| applies when the data   |    |    |                                 |
| source is a survey)     |    |    |                                 |
+-------------------------+----+----+---------------------------------+
| -   Were samples        |   |   |                                 |
|     representative?     |    |    |                                 |
+-------------------------+----+----+---------------------------------+
| -   Were the questions  |   |   |                                 |
|     in the              |    |    |                                 |
|                         |    |    |                                 |
|    survey/questionnaire |    |    |                                 |
|     clear, direct, easy |    |    |                                 |
|     to understand?      |    |    |                                 |
+-------------------------+----+----+---------------------------------+
| -   If the instrument   |   |   |                                 |
|     was self-reporting  |    |    |                                 |
|     were adequate       |    |    |                                 |
|     instructions        |    |    |                                 |
|     provided?           |    |    |                                 |
+-------------------------+----+----+---------------------------------+
| -   Were response rates |   |   |                                 |
|     sufficiently large? |    |    |                                 |
+-------------------------+----+----+---------------------------------+
| -   Has non-response    |   |   |                                 |
|     rate been followed  |    |    |                                 |
|     up?                 |    |    |                                 |
+-------------------------+----+----+---------------------------------+
| *Non Sampling Error*    |    |    |                                 |
+-------------------------+----+----+---------------------------------+
| -   Is the data         |   |   |                                 |
|     collection          |    |    |                                 |
|     instrument well     |    |    |                                 |
|     designed?           |    |    |                                 |
+-------------------------+----+----+---------------------------------+
| -   Were there          |   |   |                                 |
|     incentives for      |    |    |                                 |
|     respondents to give |    |    |                                 |
|     incomplete or       |    |    |                                 |
|     untruthful          |    |    |                                 |
|     information?        |    |    |                                 |
+-------------------------+----+----+---------------------------------+
| -   Are definitions for |   |   |                                 |
|     data to be          |    |    |                                 |
|     collected           |    |    |                                 |
|     operationally       |    |    |                                 |
|     precise?            |    |    |                                 |
+-------------------------+----+----+---------------------------------+
| -   Are enumerators     |   |   |                                 |
|     well trained? How   |    |    |                                 |
|     were they trained?  |    |    |                                 |
|     Were they insiders  |    |    |                                 |
|     or outsiders? Was   |    |    |                                 |
|     there any quality   |    |    |                                 |
|     control in the      |    |    |                                 |
|     selection process?  |    |    |                                 |
+-------------------------+----+----+---------------------------------+
| -   Were there efforts  |   |   |                                 |
|     to reduce the       |    |    |                                 |
|     potential for       |    |    |                                 |
|     personal bias by    |    |    |                                 |
|     enumerators?        |    |    |                                 |
+-------------------------+----+----+---------------------------------+
|                         |    |    |                                 |
+-------------------------+----+----+---------------------------------+
| **Transcription Error** |    |    |                                 |
+-------------------------+----+----+---------------------------------+
| -   What is the data    |   |   |                                 |
|     transcription       |    |    |                                 |
|     process? Is there   |    |    |                                 |
|     potential for       |    |    |                                 |
|     error?              |    |    |                                 |
+-------------------------+----+----+---------------------------------+
| -   Are steps being     |   |   |                                 |
|     taken to limit      |    |    |                                 |
|     transcription       |    |    |                                 |
|     error? (e.g.,       |    |    |                                 |
|     double keying of    |    |    |                                 |
|     data for large      |    |    |                                 |
|     surveys, electronic |    |    |                                 |
|     edit checking       |    |    |                                 |
|     program to clean    |    |    |                                 |
|     data, random checks |    |    |                                 |
|     of partner data     |    |    |                                 |
|     entered by          |    |    |                                 |
|     supervisors)        |    |    |                                 |
+-------------------------+----+----+---------------------------------+
| -   Have data errors    |   |   |                                 |
|     been tracked to     |    |    |                                 |
|     their original      |    |    |                                 |
|     source and mistakes |    |    |                                 |
|     corrected?          |    |    |                                 |
+-------------------------+----+----+---------------------------------+
| -   If raw data need to |   |   |                                 |
|     be manipulated to   |    |    |                                 |
|     produce the data    |    |    |                                 |
|     required for the    |    |    |                                 |
|     indicator:          |    |    |                                 |
+-------------------------+----+----+---------------------------------+
| -   Are the correct     |   |   |                                 |
|     formulae being      |    |    |                                 |
|     applied?            |    |    |                                 |
+-------------------------+----+----+---------------------------------+
| -   Are the same        |   |   |                                 |
|     formulae applied    |    |    |                                 |
|     consistently from   |    |    |                                 |
|     year to year, site  |    |    |                                 |
|     to site, data       |    |    |                                 |
|     source to data      |    |    |                                 |
|     source (if data     |    |    |                                 |
|     from multiple       |    |    |                                 |
|     sources need to be  |    |    |                                 |
|     aggregated)?        |    |    |                                 |
+-------------------------+----+----+---------------------------------+
| -   Have procedures for |   |   |                                 |
|     dealing with        |    |    |                                 |
|     missing data been   |    |    |                                 |
|     correctly applied?  |    |    |                                 |
+-------------------------+----+----+---------------------------------+
| -   Are final numbers   |   |   |                                 |
|     reported accurate?  |    |    |                                 |
|     (E.g., does a       |    |    |                                 |
|     number reported as  |    |    |                                 |
|     a "total" actually  |    |    |                                 |
|     add up?)            |    |    |                                 |
+-------------------------+----+----+---------------------------------+
|                         |    |    |                                 |
+-------------------------+----+----+---------------------------------+
| **Representativeness of |    |    |                                 |
| Data**                  |    |    |                                 |
+-------------------------+----+----+---------------------------------+
| -   Is the sample from  |   |   |                                 |
|     which the data are  |    |    |                                 |
|     drawn               |    |    |                                 |
|     representative of   |    |    |                                 |
|     the population      |    |    |                                 |
|     served by the       |    |    |                                 |
|     activity?           |    |    |                                 |
+-------------------------+----+----+---------------------------------+
| -   Did all units of    |   |   |                                 |
|     the population have |    |    |                                 |
|     an equal chance of  |    |    |                                 |
|     being selected for  |    |    |                                 |
|     the sample?         |    |    |                                 |
+-------------------------+----+----+---------------------------------+
| -   Is the sampling     |   |   |                                 |
|     frame (i.e., the    |    |    |                                 |
|     list of units in    |    |    |                                 |
|     the target          |    |    |                                 |
|     population) up to   |    |    |                                 |
|     date?               |    |    |                                 |
|     Comprehensive?      |    |    |                                 |
|     Mutually exclusive  |    |    |                                 |
|     (for geographic     |    |    |                                 |
|     frames)             |    |    |                                 |
+-------------------------+----+----+---------------------------------+
| -   Is the sample of    |   |   |                                 |
|     adequate size?      |    |    |                                 |
+-------------------------+----+----+---------------------------------+
| -   Are the data        |   |   |                                 |
|     complete? (i.e.,    |    |    |                                 |
|     have all data       |    |    |                                 |
|     points been         |    |    |                                 |
|     recorded?)          |    |    |                                 |
+-------------------------+----+----+---------------------------------+
| **Recommendations for   |    |    |                                 |
| improvement**:          |    |    |                                 |
+-------------------------+----+----+---------------------------------+

+-------------------------+----+----+---------------------------------+
| **2. RELIABILITY---Are  |    |    |                                 |
| data collection         |    |    |                                 |
| processes stable and    |    |    |                                 |
| consistent over time?** |    |    |                                 |
+=========================+====+====+=================================+
|                         | *  | ** | **Comments**                    |
|                         | *Y | No |                                 |
|                         | es | ** |                                 |
|                         | ** |    |                                 |
+-------------------------+----+----+---------------------------------+
| **Consistency**         |    |    |                                 |
+-------------------------+----+----+---------------------------------+
| -   Is a consistent     |   |   |                                 |
|     data collection     |    |    |                                 |
|     process used from   |    |    |                                 |
|     year to year,       |    |    |                                 |
|     location to         |    |    |                                 |
|     location, data      |    |    |                                 |
|     source to data      |    |    |                                 |
|     source (if data     |    |    |                                 |
|     come from different |    |    |                                 |
|     sources)?           |    |    |                                 |
+-------------------------+----+----+---------------------------------+
| -   Is the same         |   |   |                                 |
|     instrument used to  |    |    |                                 |
|     collect data from   |    |    |                                 |
|     year to year,       |    |    |                                 |
|     location to         |    |    |                                 |
|     location? If data   |    |    |                                 |
|     come from different |    |    |                                 |
|     sources are the     |    |    |                                 |
|     instruments similar |    |    |                                 |
|     enough that the     |    |    |                                 |
|     reliability of the  |    |    |                                 |
|     data are not        |    |    |                                 |
|     compromised?        |    |    |                                 |
+-------------------------+----+----+---------------------------------+
| -   Is the same         |   |   |                                 |
|     sampling method     |    |    |                                 |
|     used from year to   |    |    |                                 |
|     year, location to   |    |    |                                 |
|     location, data      |    |    |                                 |
|     source to data      |    |    |                                 |
|     source?             |    |    |                                 |
+-------------------------+----+----+---------------------------------+
|                         |    |    |                                 |
+-------------------------+----+----+---------------------------------+
| **Internal quality      |    |    |                                 |
| control**               |    |    |                                 |
+-------------------------+----+----+---------------------------------+
| -   Are there           |   |   |                                 |
|     procedures to       |    |    |                                 |
|     ensure that data    |    |    |                                 |
|     are free of         |    |    |                                 |
|     significant error   |    |    |                                 |
|     and that bias is    |    |    |                                 |
|     not introduced?     |    |    |                                 |
+-------------------------+----+----+---------------------------------+
| -   Are there           |   |   |                                 |
|     procedures in place |    |    |                                 |
|     for periodic review |    |    |                                 |
|     of data collection, |    |    |                                 |
|     maintenance, and    |    |    |                                 |
|     processing?         |    |    |                                 |
+-------------------------+----+----+---------------------------------+
| -   Do these procedures |   |   |                                 |
|     provide for         |    |    |                                 |
|     periodic sampling   |    |    |                                 |
|     and quality         |    |    |                                 |
|     assessment of data? |    |    |                                 |
+-------------------------+----+----+---------------------------------+
|                         |    |    |                                 |
+-------------------------+----+----+---------------------------------+
| **Transparency**        |    |    |                                 |
+-------------------------+----+----+---------------------------------+
| -   Are data            |   |   |                                 |
|     collection,         |    |    |                                 |
|     cleaning, analysis, |    |    |                                 |
|     reporting, and      |    |    |                                 |
|     quality assessment  |    |    |                                 |
|     procedures          |    |    |                                 |
|     documented in       |    |    |                                 |
|     writing?            |    |    |                                 |
+-------------------------+----+----+---------------------------------+
| -   Are data problems   |   |   |                                 |
|     at each level       |    |    |                                 |
|     reported to the     |    |    |                                 |
|     next level?         |    |    |                                 |
+-------------------------+----+----+---------------------------------+
| -   Are data quality    |   |   |                                 |
|     problems clearly    |    |    |                                 |
|     described in final  |    |    |                                 |
|     reports?            |    |    |                                 |
+-------------------------+----+----+---------------------------------+
| **Recommendations for   |    |    |                                 |
| improvement:**          |    |    |                                 |
+-------------------------+----+----+---------------------------------+

+-------------------------+----+----+---------------------------------+
| **3. TIMELINESS---Are   |    |    |                                 |
| data collected          |    |    |                                 |
| frequently and are they |    |    |                                 |
| current?**              |    |    |                                 |
+=========================+====+====+=================================+
|                         | *  | ** | **Comments**                    |
|                         | *Y | No |                                 |
|                         | es | ** |                                 |
|                         | ** |    |                                 |
+-------------------------+----+----+---------------------------------+
| **Frequency**           |    |    |                                 |
+-------------------------+----+----+---------------------------------+
| -   Are data available  |   |   |                                 |
|     on a frequent       |    |    |                                 |
|     enough basis to     |    |    |                                 |
|     inform program      |    |    |                                 |
|     management          |    |    |                                 |
|     decisions?          |    |    |                                 |
+-------------------------+----+----+---------------------------------+
| -   Is a regularized    |   |   |                                 |
|     schedule of data    |    |    |                                 |
|     collection in place |    |    |                                 |
|     to meet program     |    |    |                                 |
|     management needs?   |    |    |                                 |
+-------------------------+----+----+---------------------------------+
|                         |    |    |                                 |
+-------------------------+----+----+---------------------------------+
| **Currency**            |    |    |                                 |
+-------------------------+----+----+---------------------------------+
| -   Are the data        |   |   |                                 |
|     reported in a given |    |    |                                 |
|     timeframe the most  |    |    |                                 |
|     current practically |    |    |                                 |
|     available?          |    |    |                                 |
+-------------------------+----+----+---------------------------------+
| -   Are data from       |   |   |                                 |
|     within the policy   |    |    |                                 |
|     period of interest? |    |    |                                 |
|     (i.e., are data     |    |    |                                 |
|     from a point in     |    |    |                                 |
|     time after          |    |    |                                 |
|     intervention has    |    |    |                                 |
|     begun?)             |    |    |                                 |
+-------------------------+----+----+---------------------------------+
| -   Are the data        |   |   |                                 |
|     reported as soon as |    |    |                                 |
|     possible after      |    |    |                                 |
|     collection?         |    |    |                                 |
+-------------------------+----+----+---------------------------------+
| -   Is the date of      |   |   |                                 |
|     collection clearly  |    |    |                                 |
|     identified in the   |    |    |                                 |
|     report?             |    |    |                                 |
+-------------------------+----+----+---------------------------------+
| **Recommendations for   |    |    |                                 |
| improvement:**          |    |    |                                 |
+-------------------------+----+----+---------------------------------+

+-------------------------+----+----+---------------------------------+
| **4. PRECISION---Do the |    |    |                                 |
| data have an acceptable |    |    |                                 |
| margin of error?**      |    |    |                                 |
+=========================+====+====+=================================+
|                         | *  | ** | **Comments**                    |
|                         | *Y | No |                                 |
|                         | es | ** |                                 |
|                         | ** |    |                                 |
+-------------------------+----+----+---------------------------------+
| -   Is the margin of    |   |   |                                 |
|     error less than the |    |    |                                 |
|     expected change     |    |    |                                 |
|     being measured?     |    |    |                                 |
+-------------------------+----+----+---------------------------------+
| -   Is the margin of    |   |   |                                 |
|     error is acceptable |    |    |                                 |
|     given the likely    |    |    |                                 |
|     management          |    |    |                                 |
|     decisions to be     |    |    |                                 |
|     affected? (consider |    |    |                                 |
|     the consequences of |    |    |                                 |
|     the program or      |    |    |                                 |
|     policy decisions    |    |    |                                 |
|     based on the data)  |    |    |                                 |
+-------------------------+----+----+---------------------------------+
| -   Have targets been   |   |   |                                 |
|     set for the         |    |    |                                 |
|     acceptable margin   |    |    |                                 |
|     of error?           |    |    |                                 |
+-------------------------+----+----+---------------------------------+
| -   Has the margin of   |   |   |                                 |
|     error been reported |    |    |                                 |
|     along with the      |    |    |                                 |
|     data?               |    |    |                                 |
+-------------------------+----+----+---------------------------------+
| -   Would an increase   |   |   |                                 |
|     in the degree of    |    |    |                                 |
|     accuracy be more    |    |    |                                 |
|     costly than the     |    |    |                                 |
|     increased value of  |    |    |                                 |
|     the information?    |    |    |                                 |
+-------------------------+----+----+---------------------------------+
| **Recommendations for   |    |    |                                 |
| improvement:**          |    |    |                                 |
+-------------------------+----+----+---------------------------------+

+-------------------------+----+----+---------------------------------+
| **5. INTEGRITY---Are    |    |    |                                 |
| data are free of        |    |    |                                 |
| manipulation?**         |    |    |                                 |
+=========================+====+====+=================================+
|                         | *  | ** | **Comments**                    |
|                         | *Y | No |                                 |
|                         | es | ** |                                 |
|                         | ** |    |                                 |
+-------------------------+----+----+---------------------------------+
| -   Are mechanisms in   |   |   |                                 |
|     place to reduce the |    |    |                                 |
|     possibility that    |    |    |                                 |
|     data are            |    |    |                                 |
|     manipulated for     |    |    |                                 |
|     political or        |    |    |                                 |
|     personal reasons?   |    |    |                                 |
+-------------------------+----+----+---------------------------------+
| -   Is there            |   |   |                                 |
|     objectivity and     |    |    |                                 |
|     independence in key |    |    |                                 |
|     data collection,    |    |    |                                 |
|     management, and     |    |    |                                 |
|     assessment          |    |    |                                 |
|     procedures?         |    |    |                                 |
+-------------------------+----+----+---------------------------------+
| -   Has there been      |   |   |                                 |
|     independent review? |    |    |                                 |
+-------------------------+----+----+---------------------------------+
| -   If data is from a   |   |   |                                 |
|     secondary source,   |    |    |                                 |
|     is USAID management |    |    |                                 |
|     confident in the    |    |    |                                 |
|     credibility of the  |    |    |                                 |
|     data?               |    |    |                                 |
+-------------------------+----+----+---------------------------------+
| **Recommendations for   |    |    |                                 |
| improvement:**          |    |    |                                 |
+-------------------------+----+----+---------------------------------+

  -----------------------------------------------------------------------
  **For indicators for which no recent relevant data are available**

  **If no recent relevant data are available for this indicator, why
  not?**

  **What concrete actions are now being undertaken to collect and report
  this data as soon as possible?**

  **On what date will data be reported?**
  -----------------------------------------------------------------------

## Worksheet 8: Summary Performance Data Table 

*Use this worksheet to keep track of baseline values and target values
for the life of the SO for each SO and IR indicator. Some Operating
Units prefer to consolidate baseline and target data values for all
indicators into one table (and other prefer to keep the data on the
lower section of Worksheet 6). Modify the table to include additional
indicators and years as needed. An excerpt from a completed table can be
found in the Toolkit in Part 2, Task 5.3.*

  -------------------------------------------------------------------------------------------------------------------------------------------------------------------
  **SO or  **Results     **Indicator**   **Unit of   **Disaggregation**   **Baseline   **Baseline   **2005     **2005     **2006     **2006     **2007     **2007
  IR**     Statement**                   Measure**                        Year**       Value**      Target**   Actual**   Target**   Actual**   Target**   Actual**
  -------- ------------- --------------- ----------- -------------------- ------------ ------------ ---------- ---------- ---------- ---------- ---------- ----------
  SO                                                                                                                                                       

                                                                                                                                                           

  IR                                                                                                                                                       

                                                                                                                                                           

  Sub-IR                                                                                                                                                   

                                                                                                                                                           
  -------------------------------------------------------------------------------------------------------------------------------------------------------------------

## Worksheet 9: Performance Management Task Schedule

Use this worksheet to schedule all of the SO team\'s monitoring and
reporting activities over the life of the SO. Modify the table to
include additional indicators and years as needed. An excerpt from a
completed schedule can be found in Part 2, Task 6.1 of the Toolkit.

  -----------------------------------------------------------------------------------------------------------------------------------
  **PERFORMANCE       **FY                    **FY                    **FY                    **FY2007**                  **Notes**
  MANAGEMENT TASKS**  2004**                  2005**                  2006**                                              
  ------------------- -------- ---- ---- ---- -------- ---- ---- ---- -------- ---- ---- ---- ------------ ---- ---- ---- -----------
                      Q1       Q2   Q1   Q2   Q3       Q4   Q3   Q4   Q1       Q2   Q3   Q4   Q1           Q2   Q3   Q4   

  **COLLECT                                                                                                               
  CONTEXT-LEVEL                                                                                                           
  DATA**                                                                                                                  

                                                                                                                          

                                                                                                                          

  **COLLECT SO-LEVEL                                                                                                      
  PERFORMANCE DATA**                                                                                                      

                                                                                                                          

                                                                                                                          

  **COLLECT IR-LEVEL                                                                                                      
  PERFORMANCE DATA**                                                                                                      

                                                                                                                          

                                                                                                                          

                                                                                                                          

                                                                                                                          

  **COLLECT                                                                                                               
  ACTIVITY-LEVEL                                                                                                          
  DATA**                                                                                                                  

                                                                                                                          

                                                                                                                          

  **CONDUCT                                                                                                               
  EVALUATIONS &                                                                                                           
  SPECIAL STUDIES**                                                                                                       

                                                                                                                          

                                                                                                                          

  **REVIEW                                                                                                                
  PERFORMANCE                                                                                                             
  INFORMATION**                                                                                                           

                                                                                                                          

                                                                                                                          

                                                                                                                          

  **REPORT                                                                                                                
  PERFORMANCE                                                                                                             
  RESULTS**                                                                                                               

                                                                                                                          

  **ASSESS DATA                                                                                                           
  QUALITY**                                                                                                               

                                                                                                                          

                                                                                                                          

                                                                                                                          

  **REVIEW & UPDATE                                                                                                       
  PMP**                                                                                                                   

                                                                                                                          

                                                                                                                          
  -----------------------------------------------------------------------------------------------------------------------------------

## Worksheet 10: Evaluations and Special Studies Planning 

*Use this worksheet during a facilitated discussion with the PMP
development team to determine whether and when evaluations and special
studies might be conducted during the life of the SO. A completed
version of this worksheet can be found in Part 2, Task 6.3 of the
Toolkit.*

  ----------------------- ----------------------- -----------------------
  **Evaluation/Study      **When**                **Key Research
  Subject**                                       Question(s)**

                                                  

                                                  

                                                  

                                                  
  ----------------------- ----------------------- -----------------------

## Worksheet 11: Evaluation Scope of Work (SOW) Planning 

*Use this worksheet as soon as the SO team has determined that an
evaluation should take place in the near future. Reviewing this list of
questions will help formulate a well-developed SOW.*

  --------------------------------- -------------------------------------
  **PLANNING ELEMENTS**             **DESCRIPTION**

  **What is the activity, or        
  strategy being evaluated?**       

  **Provide a brief background on   
  the implementation.**             

  **What are existing performance   
  information sources?**            

  **What is the purpose of the      
  evaluation?**                     

  **Who is the audience for the     
  evaluation?**                     

  **How will the evaluation be      
  used?**                           

  **What are the key evaluation     
  questions**                       

  **What evaluation methods will be 
  used to answer the evaluation     
  questions?**                      

  **What is the proposed            
  composition of the evaluation     
  team?**                           

  **What customers, partners, or    
  stakeholders will participate in  
  the evaluation?**                 

  **What is the schedule for the    
  evaluation?**                     

  **What logistics are necessary    
  for the evaluation?**             

  **What are requirements for       
  reporting and dissemination of    
  the evaluation?**                 

  **What is the budget for the      
  evaluation?**                     
  --------------------------------- -------------------------------------

#  Appendix B: Helpful Hints

  --------------------------------------------------------------- -------
  Helpful Hint 1: Facilitating Group Discussions and              
  Decision-Making                                                 

  Helpful Hint 2: Indicators for Hard-to-Measure Results          

  Helpful Hint 3: Performance Indicator Brainstorming Session     

  Helpful Hint 4: Rapid Low-Cost Data Collection Methods          

  Helpful Hint 5: Information Systems for Data Collection and     
  Performance Monitoring                                          

  Helpful Hint 6: Key Features of Quality Data Sources            

  Helpful Hint 7: Tips to Minimize Bias                           

  Helpful Hint 8: Tips for Communicating Performance Information  
  in Reports                                                      

  Helpful Hint 9: Questions to Guide Portfolio Reviews            

  Helpful Hint 10: Maintaining Official SO Team Files             
  --------------------------------------------------------------- -------

## Helpful Hint 1: Facilitating Group Discussions and Decision-Making

There are two critical dimensions of an effective group discussion:
"what" is discussed, and "how" the discussion takes place. The latter
focuses on the process of the meeting -- how decisions are made, how
problems are resolved, how group members interact and is often where
problems arise. Facilitators are especially helpful with managing the
"how" aspects a meeting. However, they also sometimes assist with the
"what" aspects as well.

***Hints for Facilitators***

-   Encourage a creative environment

-   Warm up

-   Break the ice

-   Review ground rules

-   Introduce problem statement/discussion objective

-   Manage flow of ideas/discussion

-   Listen actively

-   Solicit clarification by asking questions

-   Provide feedback

-   Record ideas/discussion on flip charts

-   Facilitate decision-making. Consider consensus-building,
    multi-voting, pair-wise ranking, polling, or other techniques.

***Ground Rules for Brainstorming***

When used skillfully, brainstorming is a great technique to help people
get involved in the process of generating creative ideas. A facilitator
writes the topic or question to be brainstormed at the top of a large
sheet of paper, then asks the group to call out their ideas in short
phrases that can be written down quickly. In order to set a creative,
high energy atmosphere, the following guidelines should be stated to the
group at the outset:

-   *Everyone participates*---no one dominates. Have each person share
    at least one idea before opening the floor for free flow.

-   *Record every idea in full view* to prevents misunderstandings and
    reminds others of ideas.

-   *Go for quantity*. The more ideas the better.

-   *Don't criticize or evaluate ideas*. This is a key rule, and is
    often forgotten! Premature judgment will curb the essential creative
    flow. The time to evaluate is after you have created a large list of
    possibilities.

-   *Encourage creative, free-wheeling ideas*. Wild ideas can lead to
    innovative approaches.

-   *Ideas may be "piggy-backed."* One person may build upon another's
    idea.

-   *Seek to combine ideas in creative ways*.

Do it quickly---5-15 minutes works well. After brainstorming, focus the
group's thinking by jointly identifying the most promising ideas or
combinations of ideas. Consider waiting a day before making a final
decision in order to allow your sub-conscious mind to continue to work
on it. Select the best idea based on agreed-upon criteria.

## Helpful Hint 2: Indicators for Hard-to-Measure Results 

Here are some key points to consider when using scales, indexes, and
scorecards for hard-to-

measure results.

+----------+-----------------------+----------------+----------------+
| **       | **Definition**        | **Strengths**  | **Weaknesses** |
| Method** |                       |                |                |
+----------+-----------------------+----------------+----------------+
| **Rating | A rating device that  | -              | -   Derived    |
| Scales** | presents a range of   |    Facilitates |     numbers    |
|          | responses of a single |     data       |     can be     |
|          | issue or a single     |     collection |     misleading |
|          | dimension of an       |     on "soft"  |     if         |
|          | issue. The rating is  |     dimensions |     underlying |
|          | done by trained       |     of         |     data are   |
|          | observers or experts. |                |     invalid or |
|          | There are three major |    development |     unreliable |
|          | types of rating       |                |                |
|          | systems: written      | -   Enables    |                |
|          | descriptions,         |                |                |
|          | photographs, and      | transformation |                |
|          | other visual scales   |     of         |                |
|          |                       |                |                |
|          |                       |   "subjective" |                |
|          |                       |                |                |
|          |                       |    information |                |
|          |                       |     into       |                |
|          |                       |     numbers    |                |
+----------+-----------------------+----------------+----------------+
| **I      | A combination of      | -   A          | -   Incorrect  |
| ndexes** | multiple ratings to   |     weighting  |     weighting  |
|          | assess an overall     |     system can |     can lead   |
|          | concept or issue      |     be applied |     to         |
|          |                       |     to assign  |     erroneous  |
|          |                       |     greater or |                |
|          |                       |     lesser     |    conclusions |
|          |                       |     value to   |                |
|          |                       |     each item  | -   Combining  |
|          |                       |     in the     |     too many   |
|          |                       |     index      |     elements   |
|          |                       |                |     into a     |
|          |                       | -   Useful for |     single     |
|          |                       |     measuring  |     number has |
|          |                       |     progress   |     limited    |
|          |                       |     in areas   |     management |
|          |                       |     where      |     utility    |
|          |                       |     complex,   |                |
|          |                       |                |                |
|          |                       |    qualitative |                |
|          |                       |     judgments  |                |
|          |                       |     are        |                |
|          |                       |     required   |                |
+----------+-----------------------+----------------+----------------+
| **Scor   | A simple index that   | -   Useful in  | -   May        |
| ecards** | is based on yes/no    |                |                |
|          | responses             |    determining |   oversimplify |
|          |                       |     whether    |     complex    |
|          |                       |     certain    |                |
|          |                       |     c          |    information |
|          |                       | haracteristics |     to the     |
|          |                       |     are        |     extent     |
|          |                       |     present    |     that       |
|          |                       |                |     scores     |
|          |                       | -   Most       |     lack       |
|          |                       |     useful for |     meaning    |
|          |                       |     s          |                |
|          |                       | traightforward |                |
|          |                       |     judgments  |                |
+----------+-----------------------+----------------+----------------+

Some points to note regarding rating scales and indices:

-   Different people use scales differently

-   You need to determine how many values the scale should have

-   Raters should be trained

-   Consistency in ratings is key

-   Using the same team of raters helps standardize ratings

-   Weights should be assigned with care

Refer to the *Democracy and Governance Handbook's* annex on using
scales, indexes, and scorecards for performance measurement for
additional examples.

## Helpful Hint 3: Performance Indicator Brainstorming Session

Your SO team can use the following interactive technique to brainstorm
indicators for your results framework. Adapt the process to your needs
and the resources available.

***1. Set Up***

On a large piece of newsprint, draw out the results framework for the
SO. Under each results statement box, arrange the following:

-   Blue Post-Its for each of the performance indicators initially
    proposed by the mission

-   Mini Post-Its labeled with the name of each program under the
    result. Include the names of implementing partners and collaborating
    NGOs on each program Post-It

-   Large Post-Its for general notes and comments

***2. Discuss Initial or Existing Indicators***

Discuss each of the initial indicators based upon your detailed quality
assessment and notes from the review by the entire SO team. On each
indicator Post-It, use Post-It "flags" or colored markers to record
initial thoughts as follows:

-   "Note!" flags for each indicator that the mission had previously
    proposed to report in the Annual Report

-   Green flags for each indicator that was outstanding or needed
    minimal improvement

-   Yellow flags for each indicator that could be considered if it were
    significantly improved

-   Red flags for each indicator that was completely unacceptable

-   Blue flags for each red-flagged indicator that was not an acceptable
    performance indicator, but was a good indicator of context and that
    the mission should track

***3. Brainstorm for New Indicators***

Based upon discussions with partners, stakeholders, and others,
brainstorm for additional indicators, with an eye for "adequacy"
(including "balance"), using:

-   Yellow Post-Its for each potential indicator

-   For the new indicators, follow a rating process using flags as
    above.

-   Engage in an iterative process of improvement, rating, and selection
    among all indicators

## Helpful Hint 4: Rapid Low-Cost Data Collection Methods

**Key Informant Interviews:** In-depth discussions on a specific topic
with a knowledgeable person.

+----------------------+-------------------+-------------+-------------+
| **When Appropriate** | **Advantages**    | **Li        | **Skills    |
|                      |                   | mitations** | Required**  |
+----------------------+-------------------+-------------+-------------+
| -   General          | -   Provides      | -   Does    | -           |
|     information is   |     in-depth,     |     not     | Interviewer |
|     sufficient for   |     inside        |     provide |     must    |
|     decision-making  |     information   |     q       |     have:   |
|                      |     because it    | uantitative |             |
| -   An understanding |     comes from    |     data    | -           |
|     is required of   |     knowledgeable |             | Substantial |
|     the motivations  |     persons.      | -           |             |
|     and attitudes    |                   |    Findings |   knowledge |
|     that direct      | -   Provides      |     are     |     of the  |
|     behavior         |     flexibility   |             |     subject |
|                      |     to explore    | susceptible |     and     |
| -   Available        |     new idea &    |     to      |             |
|     quantitative     |     issues that   |             |   practical |
|     data needs to be |     had not been  | interviewer |             |
|     interpreted      |     anticipated   |     biases  |  experience |
|                      |     during        |             |             |
| -   Primary purpose  |     planning      |             | -           |
|     of the study is  |                   |             |    Exposure |
|     to generate      | -   Easy to find  |             |     to      |
|     suggestions and  |     people with   |             |             |
|     recommendations  |     right skills  |             |  techniques |
|                      |     to conduct    |             |     of      |
| -   The need is to   |     these         |             |             |
|     develop          |     interviews    |             |  conducting |
|     questions,       |                   |             |             |
|     hypotheses, and  | -   Can be        |             | qualitative |
|     propositions for |     completed     |             |             |
|     further testing  |     quickly.      |             |  interviews |
|     and refinement.  |                   |             |             |
|                      |                   |             | -           |
|                      |                   |             |   Knowledge |
|                      |                   |             |     of      |
|                      |                   |             |     local   |
|                      |                   |             |             |
|                      |                   |             |    language |
+----------------------+-------------------+-------------+-------------+

**Focus Group Interviews:** Participants discuss ideas, issues, and
information among themselves under general supervision of a moderator

+----------------------+-------------------+-------------+-------------+
| **When Appropriate** | **Advantages**    | **Li        | **Skills    |
|                      |                   | mitations** | Required**  |
+----------------------+-------------------+-------------+-------------+
| -   Ideas and        | -   Enable        | -           | Interviewer |
|     hypotheses for   |     information   | Susceptible | must have   |
|     designing a      |     to be         |     to same |             |
|     development      |     gathered      |             | -   Un      |
|     intervention are |     rapidly       |   moderator | derstanding |
|     needed           |                   |     biases  |     of the  |
|                      | -   Are           |     as are  |     subject |
| -   Reactions to the |     economical-do |     key     |             |
|     recommended      |     not require   |             | -   Local   |
|     innovations need |     large sample  |   informant |             |
|     to be determined |     surveys, etc. |             |    language |
|                      |                   |  interviews |             |
| -   Responses of the | -   Individual    |             | proficiency |
|     local            |     inhibitions   | -           |             |
|     populations need |     are often     | Discussions | -           |
|     to be determined |     reduced       |     can be  |   Training/ |
|                      |                   |             |             |
| -   Major            | -   Generate      |   dominated |  experience |
|     implementation   |     fresh ideas   |     by a    |     in      |
|     problems, whose  |                   |     few     |             |
|     nature and       |                   |             |  conducting |
|     implications are |                   |  articulate |     group   |
|     not clear, are   |                   |     people  |             |
|     to be examined   |                   |             | discussions |
|     and analyzed     |                   | -   Cannot  |             |
|                      |                   |     provide |             |
| -   Recommendations  |                   |     q       |             |
|     and suggestions  |                   | uantifiable |             |
|     are needed       |                   |             |             |
|                      |                   | information |             |
+----------------------+-------------------+-------------+-------------+

**Community Interviews:** Community/village meetings open to all members

+----------------------+-------------------+-------------+-------------+
| **When Appropriate** | **Advantages**    | **Li        | **Skills    |
|                      |                   | mitations** | Required**  |
+----------------------+-------------------+-------------+-------------+
| -   Vil              | -   Permit direct | -   Can be  | -           |
| lage/Community-level |     interactions  |     easily  | Interviewer |
|     data are         |     between the   |             |     must    |
|     required         |     investigator  | manipulated |     have:   |
|                      |     and a large   |     by      |             |
| -   Support for a    |     number of     |             | -   Un      |
|     specific         |     local         |   community | derstanding |
|     initiative needs |     population    |     elite   |     of the  |
|     to be assessed   |                   |             |     subject |
|                      | -   Generate      | -   Can be  |             |
| -   An assessment of |     quantitative  |             | -   Local   |
|     the needs of     |     data          | monopolized |             |
|     communities is   |                   |     by      |    language |
|     to be made in    | -   Built-in      |             |             |
|     order to develop |     mechanism for |  articulate | proficiency |
|     suitable         |     correcting    |     p       |             |
|     programs         |     inaccurate    | articipants | -           |
|                      |     information   |             |   Training/ |
| -   An evaluation is |                   | -   Many    |             |
|     to be conducted  | -   Data can be   |     issues  |  experience |
|     of the           |     collected     |     cannot  |     in      |
|     development      |     quickly and   |     be      |             |
|     initiative       |     can be cost   |             |  conducting |
|     affecting a      |     effective     |   discussed |     group   |
|     majority of the  |                   |     in      |             |
|     community        |                   |     group   | discussions |
|     members.         |                   |             |             |
|                      |                   |  encounters |             |
+----------------------+-------------------+-------------+-------------+

**Direct Observation:** Intensive and systematic observation of a
phenomenon or process in is natural setting. May involve interviews of
key informants.

+----------------------+-------------------+-------------+-------------+
| **When Appropriate** | **Advantages**    | **Li        | **Skills    |
|                      |                   | mitations** | Required**  |
+----------------------+-------------------+-------------+-------------+
| -   When trying to   | -   Enables       | -           | -           |
|     understand       |     investigator  | Susceptible |   Observers |
|     on-going         |     to study a    |     to      |     must    |
|     behavior or an   |     phenomenon in |             |     have:   |
|     unfolding event  |     its natural   |    observer |             |
|                      |     setting       |     bias    | -           |
| -   Information      |                   |             | Specialized |
|     about physical   | -   May reveal    | -   Units   |     subject |
|     infrastructure   |     social and    |     under   |             |
|     is required      |     economic      |             |   knowledge |
|                      |     conditions    | observation |             |
| -   Delivery systems |     and problems  |     must    | -           |
|     or the services  |     of which the  |             |  Experience |
|     offered by       |     informants    |   represent |     in      |
|     public and       |     are unaware   |     the     |     field   |
|     private agencies |                   |             |             |
|     are to be        | -   Rapid and     |  population | observation |
|     examined         |     economical    |             |             |
|                      |                   | -           | -           |
| -   Preliminary      |                   | Observation |   Knowledge |
|     information is   |                   |     affects |     of      |
|     required         |                   |             |     local   |
|                      |                   |   people\'s |             |
|                      |                   |             |    language |
|                      |                   |    behavior |             |
+----------------------+-------------------+-------------+-------------+

**Informal Surveys:** Differ from samples surveys in that they: focus on
few variables, use a small sample size, use non-probability sampling,
and permit more flexibility to interviewers in the field.

+----------------------+-------------------+-------------+-------------+
| **When Appropriate** | **Advantages**    | **Li        | **Skills    |
|                      |                   | mitations** | Required**  |
+----------------------+-------------------+-------------+-------------+
| -   When             | -   Can generate  | -   Not     | -   I       |
|     quantitative     |     quantitative  |     good    | nvestigator |
|     information is   |     data when     |     for     |     must    |
|     needed about a   |     sample        |             |     have:   |
|     homogeneous      |     surveys are   |  collecting |             |
|     population       |     difficult     |             | -   Strong  |
|                      |                   |    in-depth |             |
| -   Difficult to     | -   Non-random    |             |   knowledge |
|     construct a      |     sampling      | information |     of      |
|     probability      |     errors are    |             |     survey  |
|     sample without   |     low           | -           |     topic   |
|     major investment |                   | Susceptible |             |
|                      | -   Provide       |     to      | -   Formal  |
| -   Some qualitative |     relevant      |             |             |
|     information is   |     quantitative  |    sampling |   training/ |
|     already          |     data in short |     biases  |             |
|     available        |     time with     |             |  experience |
|                      |     limited       | -   Complex |     in      |
| -   Quantitative     |     personnel.    |             |             |
|     data about       |                   | statistical |  conducting |
|     attitudes,       |                   |             |             |
|     beliefs and      |                   |    analyses |    informal |
|     responses of     |                   |     are not |     surveys |
|     target           |                   |     always  |             |
|     population are   |                   |             | -           |
|     required         |                   |    feasible | Familiarity |
|     immediately      |                   |             |     with    |
|                      |                   |             |     local   |
|                      |                   |             |     so      |
|                      |                   |             | cioeconomic |
|                      |                   |             |             |
|                      |                   |             |  conditions |
+----------------------+-------------------+-------------+-------------+

## Helpful Hint 5: Information Systems for Data Collection and Performance Monitoring

Operating Units have a variety of options to choose from when it comes
to selecting an information system to manage the performance monitoring
and reporting process. In order to decide what system is right for you,
conduct a simple needs analysis by asking:

-   How much will data collection and processing cost?

-   What is the value of the information to decision-making?

-   What level of data quality to decision-makers need?

-   Who needs access to the data and when?

-   What are the needs for security/control of data?

-   What is the appropriate frequency of data collection and reporting?

-   Are quality vendors/contractors available locally?

**Microsoft Excel**

Using a well-thought out spreadsheet, such as Microsoft Excel, may
accomplish the mission's goals and be a more effective system than more
complicated computer programs. In Microsoft Excel, Workbooks (files) can
be used to work and store your data. Because each workbook can contain
many worksheets, you can organize various kinds of related information
into a single spreadsheet file. You can enter and edit data on several
worksheets simultaneously and perform calculations based on data from
multiple worksheets. When you create a chart, you can place the chart on
the worksheet with its related data or on a separate chart sheet.

If using an Excel spreadsheet, take time to determine:

-   How many fields of data (pieces of information) do you want to
    store?

-   How many columns (in a database, a "field") and rows (in a database,
    a "record") would you need?

*"Strengths" depend on a lot of things but could include:*

-   Relatively low cost to buy program (already included on many
    computers as part of the Microsoft Office suite)

-   Relatively user-friendly

-   Ability to generate graphs and charts based on data entered into the
    worksheet

-   Help for users and developers is available within the program or
    from the Microsoft help centers in various countries and on the
    internet

*"Weaknesses" include:*

-   Not robust/large enough for integration with enterprise-wide
    computer systems like Oracle

-   Cannot provide meaningful analysis if data set is incomplete

**Microsoft Access**

Microsoft Access is a database program. You may need a programmer,
although many non-IT specialists can learn Access fairly easily (the
fact that the screens look similar to other Microsoft products helps).
An average computer, or even a laptop, can run Access. Access is
appropriate for a "mid-sized" database ("Large" would be Oracle). Access
is often used by missions and other offices as their first database,
when they are unlikely to need a large or complex database.

*"Strengths" depend on a lot of things but could include:*

-   Relatively low cost to buy program (already included on many
    computers as part of the Microsoft Office suite)

-   Relatively low cost of programmers (it is not a difficult
    programming language)

-   Relatively user-friendly

-   Help for users and developers is available within the program or
    from the Microsoft help centers in various countries and on the
    internet

*"Weaknesses" include:*

-   Not robust/large enough for integration with enterprise-wide
    computer systems like Oracle

-   Cannot provide meaningful analysis if data set is incomplete

-   Fails when the data is too large (too many records) or there are too
    many relationships in the database

**Geographic Information Systems (GIS)**

GIS is the general term used for applications that allow you to
demonstrate the analysis of data in a map form; all GIS systems have a
database inside. Although many GIS applications are available (do a
search on the web to see some brand names) they require more expense and
more development time than simple databases. GIS systems need a
programmer, and the programmers are harder to find. It may be necessary
to have a larger and/or dedicated computer to run a GIS system.

But ask questions first to determine if GIS is even needed. Why/how do
you want the data to be linked to a map? Would it be enough just to have
a chart showing the number of kids in each province or is it really
important to show that information on a map? Although GIS systems are
very powerful, especially for presenting uneven distributions of social
indicators across geography, the cost of developing a system may not be
justifiable.

*"Strengths" depend on a lot of things but could:*

-   Can present data analysis linked to geography

*"Weaknesses":*

-   Expense

-   Less powerful analysis and results if you have an incomplete or
    small data set

Here are some examples of companies who sell GIS applications. On each
of these web pages, there are descriptions of how GIS analysis was used
in various sectors.

ESRI
[(http://www.esri.com/industries/index.html)](http://www.esri.com/industries/index.html)

ArcInfo (<http://www.esri.com/software/arcinfo/index.html>)

ArcView (<http://www.esri.com/software/arcview/index.html>)

Atlas GIS (<http://www.esri.com/software/atlas/index.html)>.

Clarklabs (<http://www.clarklabs.org/03prod/gallery/imgalfrm.htm>)

MapInfo
(<http://www.mapinfo.com/solutions_and_services/interest_areas/government/government_solutions.html>[)]{.underline}

GRASS GIS (<http://www.baylor.edu/~grass/links.html>[)]{.underline}

## Helpful Hint 6: Key Features of Quality Data Sources

**Primary Data Sources**

You may occasionally use primary data collected on your own or through
independent entities that you engage for this purpose. For example, you
may require primary data to be collected scientifically to serve as a
baseline, interim, or final evaluation of SO achievement.

In these settings, you should ensure that the contractor doing the data
collection and analysis follows good statistical and operational
methodology, as outlined in TIPS No. 12 and ADS 203.3.5. These
requirements should be written into the contracts so they become
enforceable if and when necessary. As part of data quality assessments,
look for and document evidence of the following key features:

-   Are there written descriptions of the data verification and
    validation procedures to be used to minimize the chance that
    significant error, including bias, are not added during collection,
    maintenance or processing of data?

-   Are general procedures in place to control data quality, such as
    supervision of data collection?

-   Are sufficient descriptions available of the instruments and
    procedures to be used to ensure that data are collected consistently
    over the course of the activity?

```{=html}
<!-- -->
```
-   Do mechanisms exist to ensure timely availability of data for
    management purposes?

-   Is source documentation readily available?

**Partner Data**

Much of the data that you will collect will come from implementing
partners, including contractors, cooperating agencies, and grantees.
This data is typically derived from partners' ongoing performance
monitoring systems. To assess the quality of partner data, you may:

-   Periodically sample and review the partner data to ensure
    completeness, accuracy and consistency

-   Use independent audits or other procedures for ensuring quality
    financial information when financial data is used for performance
    measurement

-   Determine whether the partner appropriately addressed known data
    quality problems

As you seek to monitor whether implementation is on track towards
expected results, you may want to use field visits, data from other
sources, and independent surveys or evaluations to ensure quality data.
All assessments should be documented and available. In particular, look
for and document evidence of the following:

-   Does the partner have procedures for data collection, maintenance
    and processing that are consistently applied and continue to be
    adequate?

-   Does the partner continue to utilize these procedures to ensure a
    consistent flow of quality data?

**Secondary Data Sources**

Occasionally, you may rely on secondary data sources over which the
Agency has no control. These may include government ministries or other
bilateral donors or development agencies.

When such data are the only, or best, source of performance information,
obtain as much information as possible as to how the data are collected
and analyzed, and what quality control mechanisms exist. To perform data
assessments, you may want to:

-   Interview managers responsible for data collection within the source
    agency, and report findings in a memorandum of conversation.

-   Discuss the quality of data---often off the record---with
    counterparts in other agencies. These discussions will be documented
    and summarized in memoranda that assess everything that is known
    about the reliability of such data.

-   Share data collection experiences with secondary sources, and
    discuss how reliable they feel their information is. This
    information should be documented in trip reports.

When performing data quality assessments of secondary sources, you might
choose to seek evidence of the following:

-   Does the secondary source have procedures that periodically review
    data collection, maintenance and processing to ensure that these
    procedures are consistently applied and continue to be adequate?

-   Were independent audits or other independent evaluations of the data
    available to ensure that quality data are available? This applies
    particularly to financial data, but may apply to other data if
    another donor or independent body assesses the adequacy of the data
    collection, analysis and reporting system.

-   Does the secondary source address known data quality problems?

-   Are there independent sources that confirm that secondary source
    data are accurate?

If the data from secondary sources turn out to be reliable, they can
serve as an inexpensive, easily accessible source of performance
information. If not, develop other data sources that can be used to
monitor performance.

##  

## Helpful Hint 7: Tips to Minimize Bias

+--------------+-------------------------------------------------------+
| **Bias       | **Tips to Minimize Bias**                             |
| Type**       |                                                       |
+--------------+-------------------------------------------------------+
| *            | -   Train interviewers thoroughly                     |
| *Interviewer |                                                       |
| bias**       | -   Standardize the interview protocol                |
|              |                                                       |
|              | -   Use highly objective, closed ended questions      |
|              |                                                       |
|              | -   Have each collector or team gather information    |
|              |     from different areas, both in baseline and        |
|              |     subsequent evaluation surveys                     |
|              |                                                       |
|              | -   For longitudinal surveys, the same data collector |
|              |     (or team) should collect information for the same |
|              |     individuals throughout the duration of the        |
|              |     evaluation                                        |
+--------------+-------------------------------------------------------+
| **Instrument | -   Pilot test the instrument and revise accordingly  |
| or           |                                                       |
| measurement  | -   Standardize measurement instruments and           |
| bias**       |     procedures                                        |
|              |                                                       |
|              | -   Calibrate instruments frequently                  |
+--------------+-------------------------------------------------------+
| **Response   | -   Train interviewers thoroughly on how to probe for |
| bias**       |     information                                       |
|              |                                                       |
|              | -   Use highly objective, closed ended questions      |
+--------------+-------------------------------------------------------+
| **Recall     | -   Train interviewers thoroughly on how to probe for |
| bias**       |     information and how to help respondents remember  |
|              |     past events                                       |
|              |                                                       |
|              | -   Use specific and meaningful reference/recall      |
|              |     period                                            |
+--------------+-------------------------------------------------------+
| **Time or    | -   Standardize the time of day or season of data     |
| seasonal     |     collection so information on treatment groups and |
| bias**       |     controls is collected during the same period      |
+--------------+-------------------------------------------------------+

## Helpful Hint 8: Tips for Communicating Performance Information in Reports

-   Begin reports with a brief executive summary

-   Include a table of contents and list of acronyms, if applicable

-   Keep reports concise

-   Use simple, clear language

-   Use tables, charts, and graphs to summarize results

-   Use simple, eye catching graphics to support text

-   Synthesize available findings

-   Use real examples

-   Make concrete recommendations

-   List "lessons learned"

-   Provide references for additional sources of information

##  

## Helpful Hint 9: Questions to Guide Portfolio Reviews 

*The following tables (excerpted from ADS 203.3.7) provide a list of
questions that can be used to plan and carry out portfolio reviews.
Other questions may be relevant and useful to address. For those
questions answered in the negative, the SO team should seek to
understand the reason behind the answer and what corrective actions, if
any, might be necessary.*

**Strategy and Activity Issues to Address during Portfolio Reviews**

+-------------+--------------------------------------------------------+
| **Area of   | **Suggested Questions**                                |
| Concern**   |                                                        |
+-------------+--------------------------------------------------------+
| **Results** | -   Are the desired results being achieved?            |
|             |                                                        |
|             | -   Are the results within USAID's manageable          |
|             |     interest?                                          |
|             |                                                        |
|             | -   Will planned targets set in the previous Annual    |
|             |     Report be met?                                     |
|             |                                                        |
|             | -   Is the performance management system in place      |
|             |     adequate to capture data on the achievement of     |
|             |     results?                                           |
+-------------+--------------------------------------------------------+
| **Outputs** | -   Are planned outputs being completed on schedule?   |
|             |                                                        |
|             | -   Are the outputs leading to the achievement of the  |
|             |     desired results as anticipated?                    |
+-------------+--------------------------------------------------------+
| **Inputs**  | -   Are the necessary inputs being provided on         |
|             |     schedule by USAID and/or its customers/partners?   |
|             |                                                        |
|             | -   Are inputs effective in producing the desired      |
|             |     outputs?                                           |
|             |                                                        |
|             | -   Are funding pipelines adequate to finance          |
|             |     activities until new funds become available for    |
|             |     obligation?                                        |
|             |                                                        |
|             | -   If there are significant differences between       |
|             |     planned and actual expenditures, do these point to |
|             |     potentially problematic delays or cost overruns ?  |
+-------------+--------------------------------------------------------+
| **          | -   Has the logic identified in the development        |
| Development |     hypothesis in the Results Framework been found to  |
| h           |     hold true?                                         |
| ypothesis** |                                                        |
|             | -   If not, what adjustments, if any, are needed to    |
|             |     the strategy?                                      |
+-------------+--------------------------------------------------------+
| **Critical  | -   Do the assumptions stated in the Results Framework |
| assumptions |     still hold true?                                   |
| inherent in |                                                        |
| results     | -   If not, what effect does this have on the SO       |
| framework** |     activities and expected results?                   |
+-------------+--------------------------------------------------------+
| **Non-USAID | -   Are situations or circumstances beyond USAID       |
| circ        |     control and influence, other than the identified   |
| umstances** |     critical assumptions, affecting USAID activities?  |
|             |                                                        |
|             | -   If so, what are they, and what are the effects on  |
|             |     USAID activities?                                  |
+-------------+--------------------------------------------------------+
| **Interface | -   At the current rate of progress, is USAID on track |
| between     |     to achieve the results that have been targeted in  |
| tactics and |     the future?                                        |
| strategy**  |                                                        |
|             | -   Have significant problems or issues been           |
|             |     identified in their early stages in order to take  |
|             |     corrective action, or are they dealt with after    |
|             |     major problems have occurred?                      |
+-------------+--------------------------------------------------------+

**Process Issues to Address during Portfolio Reviews**

+-------------+--------------------------------------------------------+
| **Area of   | **Suggested Questions**                                |
| Concern**   |                                                        |
+-------------+--------------------------------------------------------+
| *           | -   Are the performance indicators appropriate for     |
| *Indicators |     management needs?                                  |
| and         |                                                        |
| targets**   | -   Are the established indicators being monitored     |
|             |     regularly?                                         |
|             |                                                        |
|             | -   Will USAID be able to attribute progress in the    |
|             |     indicator to USAID?                                |
|             |                                                        |
|             | -   Were the set targets realistic?                    |
|             |                                                        |
|             | -   If not, what targets are more appropriate?         |
|             |                                                        |
|             | -   Do performance data meet quality standards for     |
|             |     reporting?                                         |
+-------------+--------------------------------------------------------+
| **Ev        | -   Have any evaluations been completed to fill        |
| aluations** |     performance information gaps?                      |
|             |                                                        |
|             | -   Is the information from prior evaluations          |
|             |     informing decisions and action on relevant         |
|             |     activities?                                        |
|             |                                                        |
|             | -   Are new evaluations needed to inform future        |
|             |     decisions?                                         |
+-------------+--------------------------------------------------------+
| *           | -   Do team members have clear roles and               |
| *Teamwork** |     responsibilities and adequate authority for        |
|             |     implementing activities?                           |
|             |                                                        |
|             | -   Is the team receiving adequate support from other  |
|             |     units in the Mission, Operating Unit or Bureau?    |
|             |                                                        |
|             | -   Is the team regularly involving non-USAID members  |
|             |     in information sharing and decision-making?        |
|             |                                                        |
|             | -   Is staffing of the team adequate?                  |
|             |                                                        |
|             | -   Are any changes to roles or new team members       |
|             |     needed?                                            |
|             |                                                        |
|             | -   Are sub-teams (if any) functioning adequately?     |
+-------------+--------------------------------------------------------+
| **Custo     | -   Are customer/partner expectations and needs being  |
| mer/partner |     regularly assessed?                                |
| pe          |                                                        |
| rceptions** | -   Are customers/partners involved in performance     |
|             |     management and assessing effort?                   |
|             |                                                        |
|             | -   Are gender concerns being addressed, and are there |
|             |     new gender issues that the SO team needs to take   |
|             |     into account?                                      |
|             |                                                        |
|             | -   What opportunities do customers have to obtain     |
|             |     information and to provide ongoing feedback to     |
|             |     USAID on priorities and activity implementation?   |
+-------------+--------------------------------------------------------+

**Vulnerability Issues to Address during Portfolio Reviews**

+-------------+--------------------------------------------------------+
| **Area of   | **Suggested Questions**                                |
| Concern**   |                                                        |
+-------------+--------------------------------------------------------+
| **Financial | -   Do recipient institutions meet financial           |
| vuln        |     management and accountability standards?           |
| erability** |                                                        |
|             | -   Are the funds received from USAID being handled    |
|             |     properly?                                          |
|             |                                                        |
|             | -   Are previously identified problem areas being      |
|             |     corrected?                                         |
+-------------+--------------------------------------------------------+
| **Other     | -   Are activities in compliance with any applicable   |
| vuln        |     legal or legislative restrictions?                 |
| erability** |                                                        |
|             | -   Are potential conflict of interest or procurement  |
|             |     integrity issues being adequately managed?         |
|             |                                                        |
|             | -   Are activities in compliance with the              |
|             |     environmental impact mitigation provisions of the  |
|             |     22 CFR environmental determination? (see ADS 204   |
|             |     and ADS 201.3.4.11b)                               |
+-------------+--------------------------------------------------------+
| **Audit     | -   Are filing systems and documentation adequate to   |
| readiness** |     establish an audit trail?                          |
|             |                                                        |
|             | -   Are approval authorities and procedures clear and  |
|             |     being followed?                                    |
|             |                                                        |
|             | -   Has the necessary post-obligation documentation    |
|             |     been developed (e.g., financial and substantive    |
|             |     tracking)?                                         |
|             |                                                        |
|             | -   Are the performance indicators supported by        |
|             |     documentation that show reported data accurately   |
|             |     represent real progress?                           |
+-------------+--------------------------------------------------------+

##  Helpful Hint 10: Maintaining Official SO Team Files 

Each SO Team is responsible for ensuring there is official documentation
to support the SO team\'s performance management and reporting
activities. ADS 203.3.3 provides the guidance about what is mandatory
and what is suggested in a performance management plan (this information
is also summarized in Table 2.14 of the Toolkit).

As you begin implementing the PMP you should organize and maintain a set
of performance management files that document your performance
management activities. Here are some suggestions that you should
consider maintaining to support the PMP:

**Recommended Documentation for Performance Management Files**

-   Data tables with targets and actual data for all indicators in PMP

-   Source documents/supporting documentation for all data recorded in
    data tables

-   Calculations to support data recorded in performance data tables

-   Documentation of data quality assessments

-   Documentation of indicator assessments

-   Justification for any changes to Annual Report indicators

-   Copies of all special studies and/or evaluations

-   Copies of all surveys and other instruments used to collect
    performance data

-   Reports of site visits by USAID staff to monitor activities

-   Activity progress reports from partners

-   Summary of outcomes of annual portfolio review process and activity
    implementation review if any

Also see ADS 202.3.4.6 and ADS 502 about maintaining official SO Team
files.

# Appendix C: Helpful Resources

### 

## Resources available on the Internet

+-----------------------------------+-----------------------------------+
| **Description**                   | **Location**                      |
+===================================+===================================+
| General USAID Resources           |                                   |
+-----------------------------------+-----------------------------------+
| 1.  ADS 200                       | [www.usaid.go                     |
|                                   | v/pubs/ads/200/200.pdf](http://ww |
|                                   | w.usaid.gov/pubs/ads/200/200.pdf) |
+-----------------------------------+-----------------------------------+
| 2.  ADS 201                       | [www.usaid.go                     |
|                                   | v/pubs/ads/200/201.pdf](http://ww |
|                                   | w.usaid.gov/pubs/ads/200/201.pdf) |
+-----------------------------------+-----------------------------------+
| 3.  ADS 202                       | [www.usaid.go                     |
|                                   | v/pubs/ads/200/202.pdf](http://ww |
|                                   | w.usaid.gov/pubs/ads/200/202.pdf) |
+-----------------------------------+-----------------------------------+
| 4.  ADS 203                       | [www.usaid.go                     |
|                                   | v/pubs/ads/200/203.pdf](http://ww |
|                                   | w.usaid.gov/pubs/ads/200/203.pdf) |
+-----------------------------------+-----------------------------------+
| 5.  Dialogue about ADS            | [www.USAIDResults.o               |
|     Programming Policies          | rg](http://www.USAIDResults.org/) |
|                                   | (click on Town Hall)              |
+-----------------------------------+-----------------------------------+
| 6.  Economic and Social Data      | [http://cdie.u                    |
|     Services                      | said.gov](http://cdie.usaid.gov/) |
|                                   | (internal USAID only)             |
+-----------------------------------+-----------------------------------+
| 7.  Research and Reference        | [http://cdie.u                    |
|     Services                      | said.gov](http://cdie.usaid.gov/) |
|                                   | (internal USAID only)             |
+-----------------------------------+-----------------------------------+
| 8.  Performance Management        | [www.USAIDResults.o               |
|     Toolkit (including            | rg](http://www.USAIDResults.org/) |
|     worksheets)                   | (click on Tools)                  |
+-----------------------------------+-----------------------------------+
| 9.  "Legal and Policy             | [www.usaid.gov/pubs               |
|     Considerations when involving | /ads/200/2016s1.doc](http://www.u |
|     partners and customers on     | said.gov/pubs/ads/200/2016s1.doc) |
|     Strategic Objective Teams and |                                   |
|     other consultations"          |                                   |
+-----------------------------------+-----------------------------------+
| 10. Database of R4s and Annual    | [www.dec.org/partners/pmdb/](h    |
|     Reports                       | ttp://www.dec.org/partners/pmdb/) |
+-----------------------------------+-----------------------------------+
| 11. Database of "Development      | [                                 |
|     Experience"                   | www.dec.org](http://www.dec.org/) |
+-----------------------------------+-----------------------------------+
| TIPS Series                       |                                   |
+-----------------------------------+-----------------------------------+
| 12. TIPS No. 1: Conducting a      | [www.dec.or                       |
|     Participatory Evaluation      | g/pdf_docs/PNABS539.pdf](http://w |
|     (1996)                        | ww.dec.org/pdf_docs/PNABS539.pdf) |
+-----------------------------------+-----------------------------------+
| 13. TIPS No. 2: Conducting Key    | [www.dec.or                       |
|     Informant Interviews          | g/pdf_docs/PNABS541.pdf](http://w |
|                                   | ww.dec.org/pdf_docs/PNABS541.pdf) |
+-----------------------------------+-----------------------------------+
| 14. TIPS No. 3: Preparing an      | [www.dec.or                       |
|     Evaluation Scope of Work      | g/pdf_docs/PNABY207.pdf](http://w |
|     (1996)                        | ww.dec.org/pdf_docs/PNABY207.pdf) |
+-----------------------------------+-----------------------------------+
| 15. TIPS No. 4: Using Direct      | [www.dec.or                       |
|     Observation Techniques        | g/pdf_docs/PNABY208.pdf](http://w |
|                                   | ww.dec.org/pdf_docs/PNABY208.pdf) |
+-----------------------------------+-----------------------------------+
| 16. TIPS No. 5: Using Rapid       | [www.dec.or                       |
|     Appraisal Methods (1996)      | g/pdf_docs/PNABY209.pdf](http://w |
|                                   | ww.dec.org/pdf_docs/PNABY209.pdf) |
+-----------------------------------+-----------------------------------+
| 17. TIPS No. 6: Selecting         | [www.dec.or                       |
|     Performance Indicators (1996) | g/pdf_docs/PNABY214.pdf](http://w |
|                                   | ww.dec.org/pdf_docs/PNABY214.pdf) |
+-----------------------------------+-----------------------------------+
| 18. TIPS No. 7: Preparing a       | [www.dec.or                       |
|     Performance Monitoring Plan   | g/pdf_docs/PNABY215.pdf](http://w |
|     (1996)                        | ww.dec.org/pdf_docs/PNABY215.pdf) |
+-----------------------------------+-----------------------------------+
| 19. TIPS No. 8: Establishing      | [www.dec.or                       |
|     Performance Targets (1996)    | g/pdf_docs/PNABY226.pdf](http://w |
|                                   | ww.dec.org/pdf_docs/PNABY226.pdf) |
+-----------------------------------+-----------------------------------+
| 20. TIPS No. 9: Conducting        | [www.dec.or                       |
|     Customer Service Assessments  | g/pdf_docs/PNABY227.pdf](http://w |
|     (1996)                        | ww.dec.org/pdf_docs/PNABY227.pdf) |
+-----------------------------------+-----------------------------------+
| 21. TIPS No. 10: Conducting Focus | [www.dec.or                       |
|     Group Interviews (1996)       | g/pdf_docs/PNABY233.pdf](http://w |
|                                   | ww.dec.org/pdf_docs/PNABY233.pdf) |
+-----------------------------------+-----------------------------------+
| 22. TIPS No. 11: Role of          | [www.dec.or                       |
|     Evaluation in USAID (1997)    | g/pdf_docs/PNABY239.pdf](http://w |
|                                   | ww.dec.org/pdf_docs/PNABY239.pdf) |
+-----------------------------------+-----------------------------------+
| 23. TIPS No. 12: Guidelines for   | [www.dec.or                       |
|     Indicator and Data Quality    | g/pdf_docs/PNACA927.pdf](http://w |
|     (1998)                        | ww.dec.org/pdf_docs/PNACA927.pdf) |
+-----------------------------------+-----------------------------------+
| 24. TIPS No. 13: Building a       | [www.dec.or                       |
|     Results Framework (2000)      | g/pdf_docs/PNACA947.pdf](http://w |
|                                   | ww.dec.org/pdf_docs/PNACA947.pdf) |
+-----------------------------------+-----------------------------------+
| 25. TIPS No. 14: Monitoring the   | [www.dec.or                       |
|     Policy Reform Process (2000)  | g/pdf_docs/PNACA949.pdf](http://w |
|                                   | ww.dec.org/pdf_docs/PNACA949.pdf) |
+-----------------------------------+-----------------------------------+
| 26. TIPS No. 15: Measuring        | [www.dec.org                      |
|     Institutional Capacity (2000) | /pdf_docs/PNACG612.pdf](http://ww |
|                                   | w.dec.org/pdf_docs/PNACG612.pdf); |
|                                   | Annex available at                |
|                                   | [www.dec.or                       |
|                                   | g/pdf_docs/PNACG624.pdf](http://w |
|                                   | ww.dec.org/pdf_docs/PNACG624.pdf) |
+-----------------------------------+-----------------------------------+
| Selected Sector-Specific          |                                   |
| Resources                         |                                   |
+-----------------------------------+-----------------------------------+
| 27. Food Security Indicators and  | [www.dec.or                       |
|     Framework for Use in the      | g/pdf_docs/PNACG170.pdf](http://w |
|     Monitoring and Evaluation of  | ww.dec.org/pdf_docs/PNACG170.pdf) |
|     Food Aid Programs             |                                   |
+-----------------------------------+-----------------------------------+
| 28. Handbook of Democracy and     | [www.dec.or                       |
|     Governance Program Indicators | g/pdf_docs/PNACC390.pdf](http://w |
|                                   | ww.dec.org/pdf_docs/PNACC390.pdf) |
+-----------------------------------+-----------------------------------+
| 29. "Synthesis of Democracy and   | [www.dec.or                       |
|     Governance Cross Sectoral     | g/pdf_docs/PNACJ950.pdf](http://w |
|     Case Studies"                 | ww.dec.org/pdf_docs/PNACJ950.pdf) |
|                                   | or                                |
|                                   | [w                                |
|                                   | ww.usaid.gov/regions/afr/pubs/dem |
|                                   | ocracy.html](http://www.usaid.gov |
|                                   | /regions/afr/pubs/democracy.html) |
+-----------------------------------+-----------------------------------+
| 30. Kumar, Krishna, "Rapid Low    | [www.dec.or                       |
|     Cost Data Collection Methods  | g/pdf_docs/PNAAL100.pdf](http://w |
|     for AID."                     | ww.dec.org/pdf_docs/PNAAL100.pdf) |
+-----------------------------------+-----------------------------------+
| 31. "Pocketbook of Family         | [www.dec.or                       |
|     Planning and Reproductive     | g/pdf_docs/PNACG519.pdf](http://w |
|     Health Indicators for Program | ww.dec.org/pdf_docs/PNACG519.pdf) |
|     Design and Evaluation"        |                                   |
+-----------------------------------+-----------------------------------+
| 32. "Health and family planning   | [www.dec.or                       |
|     indicators: a tools for       | g/pdf_docs/PNACM806.pdf](http://w |
|     results frameworks, Volume I" | ww.dec.org/pdf_docs/PNACM806.pdf) |
+-----------------------------------+-----------------------------------+
| 33. "Health and family planning   | [www.dec.or                       |
|     indicators: measuring         | g/pdf_docs/PNACE795.pdf](http://w |
|     sustainability, Volume II"    | ww.dec.org/pdf_docs/PNACE795.pdf) |
+-----------------------------------+-----------------------------------+
| 34. "Measuring results of health  | [www.dec.or                       |
|     sector reform for system      | g/pdf_docs/PNACH329.pdf](http://w |
|     performance: a handbook of    | ww.dec.org/pdf_docs/PNACH329.pdf) |
|     indicators"                   |                                   |
+-----------------------------------+-----------------------------------+
| 35. "Indicators for monitoring    | [www.dec.or                       |
|     and evaluation of AIDS        | g/pdf_docs/PNACN335.pdf](http://w |
|     programs"                     | ww.dec.org/pdf_docs/PNACN335.pdf) |
+-----------------------------------+-----------------------------------+
| 36. "Handbook of indicators for   | [www.dec.or                       |
|     HIV/AIDS/STI programs"        | g/pdf_docs/PNACK416.pdf](http://w |
|                                   | ww.dec.org/pdf_docs/PNACK416.pdf) |
+-----------------------------------+-----------------------------------+
| 37. "Expanded response guide to   | [www.dec.org/p                    |
|     core indicators for           | df_docs/PNACS452.pdf]{.underline} |
|     monitoring and reporting      |                                   |
|     HIV/AIDS programs"            |                                   |
+-----------------------------------+-----------------------------------+
| 38. "Developing survey-based      | [www.dec.org/p                    |
|     indicators for national AIDS  | df_docs/PNACM547.pdf]{.underline} |
|     programs"                     |                                   |
+-----------------------------------+-----------------------------------+
| 39. "A framework to identify      | [www.dec.org/p                    |
|     gender indicators for         | df_docs/PNACR626.pdf]{.underline} |
|     reproductive health and       |                                   |
|     nutrition programming"        |                                   |
+-----------------------------------+-----------------------------------+
| Other Donors                      |                                   |
+-----------------------------------+-----------------------------------+
| 40. CIDA: Results Based           | [                                 |
|     Management Handbook           | www.acdi-cida.gc.ca/cida_ind.nsf/ |
|                                   | vLUallDocByIDEn/98D3AA0A746EECF08 |
|                                   | 52569840046FA4D?OpenDocument](htt |
|                                   | p://www.acdi-cida.gc.ca/cida_ind. |
|                                   | nsf/vLUallDocByIDEn/98D3AA0A746EE |
|                                   | CF0852569840046FA4D?OpenDocument) |
+-----------------------------------+-----------------------------------+
| 41. ADB: Handbook for the         | [www.adb.or                       |
|     Economic Analysis of Health   | g/Documents/Handbooks/Health_Sect |
|     Sector Projects               | or_Projects/default.asp](http://w |
|                                   | ww.adb.org/Documents/Handbooks/He |
|                                   | alth_Sector_Projects/default.asp) |
+-----------------------------------+-----------------------------------+
| 42. World Bank: Performance       | [www.worldbank.org/ht             |
|     Monitoring Indicators (1996,  | ml/opr/pmi/pmi.pdf](http://www.wo |
|     based on logical frameworks)  | rldbank.org/html/opr/pmi/pmi.pdf) |
+-----------------------------------+-----------------------------------+
| 43. OECD Public Management and    | [www.oecd.org/                    |
|     Governance                    | puma/](http://www.oecd.org/puma/) |
+-----------------------------------+-----------------------------------+
| General Accounting Office         | **www.gao.gov/**                  |
+-----------------------------------+-----------------------------------+
| 44. GAO Guide to Implementing     | [www.gao.go                       |
|     GPRA                          | v/special.pubs/gpra.htm](http://w |
|                                   | ww.gao.gov/special.pubs/gpra.htm) |
+-----------------------------------+-----------------------------------+
| 45. Government Performance and    | [w                                |
|     Results Act                   | ww.whitehouse.gov/OMB/mgmt-gpra/g |
|                                   | plaw2m.html](http://www.whitehous |
|                                   | e.gov/OMB/mgmt-gpra/gplaw2m.html) |
+-----------------------------------+-----------------------------------+
| 46. GAO Results Act Evaluation    | [w                                |
|     Guide                         | ww.gao.gov/](http://www.gao.gov/) |
+-----------------------------------+-----------------------------------+
| 47. GAO, \"Performance Plans:     | [w                                |
|     Selected Approaches for       | ww.gao.gov/](http://www.gao.gov/) |
|     Verification and Validation   |                                   |
|     of Agency Performance         |                                   |
|     Information\"                 |                                   |
+-----------------------------------+-----------------------------------+
| 48. GAO Quantitative Data         | [w                                |
|     Analysis                      | ww.gao.gov/](http://www.gao.gov/) |
+-----------------------------------+-----------------------------------+
| 49. GAO, Standards for Internal   | [w                                |
|     Controls                      | ww.gao.gov/](http://www.gao.gov/) |
+-----------------------------------+-----------------------------------+
| Department of Energy              | [**www.                           |
|                                   | orau.gov**](http://www.orau.gov/) |
+-----------------------------------+-----------------------------------+
| 50. Department of Energy,         | http://                           |
|     \"Guidelines for Performance  | [www.orau.gov/pbm/doc             |
|     Measurement\"                 | uments/g1201-5.pdf](http://www.or |
|                                   | au.gov/pbm/documents/g1201-5.pdf) |
+-----------------------------------+-----------------------------------+
| 51. Department of Energy, \"The   | [w                                |
|     Performance-Based Management  | ww.orau.gov/pbm/pbmhandbook/pbmha |
|     Handbook, Volumes 1-6         | ndbook.html](http://www.orau.gov/ |
|                                   | pbm/pbmhandbook/pbmhandbook.html) |
+-----------------------------------+-----------------------------------+
| 52. DOE Performance-Based         | [www.orau.go                      |
|     Management Special Interest   | v/pbm/](http://www.orau.gov/pbm/) |
|     Group                         |                                   |
+-----------------------------------+-----------------------------------+
| Other                             |                                   |
+-----------------------------------+-----------------------------------+
| 53. Hatry, Harry P. and Joseph S. | None available                    |
|     Wholey. *Performance          |                                   |
|     Measurement: Getting          |                                   |
|     Results,* 1999                |                                   |
+-----------------------------------+-----------------------------------+

## USAID Economic and Social Data Service (ESDS)

The Economic and Social Data Service (ESDS) is managed by DevTech
Systems, Inc., under contract with the U.S. Agency for International
Development. ESDS staff provide economic analysts and policy makers with
access to social and economic statistics on the developing countries of
Asia, Africa, Latin America and the emerging market economies of Eastern
Europe and the former Soviet republics. The ESDS project serves as a
focal point for collecting, analyzing and disseminating a wide range of
data in support of the analysis, planning, management, and monitoring of
programs and projects conducted by USAID and its development partners.
You can access ESDS via CDIE Online at
[http://cdie.usaid.gov](http://cdie.usaid.gov/) (The link is
\'Statistics\' at the top of the homepage.).

## Research and Reference Services (R&RS)

R&RS is a research and reference service project designed to support
USAID\'s development assistance programs, by providing relevant and
timely information and analysis to USAID field and Washington staff, as
well as USAID contractors and development partners. The project serves
to link those who need information with the essential literature and
resources.

R&RS staff assist development practitioners in clarifying their
information needs, and respond by identifying, analyzing and
disseminating appropriate information in a useful form. Products and
services include analytical memoranda and papers, lessons learned
analyses, reference, computerized database searches, bibliographies,
interlibrary loans, tailored information packages, referrals, a current
awareness update service, and several regular publications. Technical
assistance and training is available to USAID mission libraries and
other USAID units concerned with the Management of development
information resources.

## USAID Library and Learning Resources Center

The R&RS project also staffs and manages the USAID Library and Learning
Resources Center, the heart of the R&RS reference service, specializing
in providing ready reference responses, ready access to information
resources, and USAID database search service for USAID patrons,
development partners and the public. The collection comprises USAID
reports and serials, World Bank publications, commercially published
books and journals, reference materials, newspapers, and CD-ROMs. The
library\'s online catalogue is located at
<http://library.info.usaid.gov/> (available only within the USAID
firewall). The telephone number is 202-712-0579.

You can access R&RS via CDIE Online at
[http://cdie.usaid.gov](http://cdie.usaid.gov/) (available only within
the USAID firewall). Follow the links for \'Research\' and \'Library\'
at the top of the homepage.

## Development Experience Clearinghouse

As part of PPC's development information services team, the USAID
Development Experience Clearinghouse (DEC) collects information that
describes USAID\'s own development assistance activities. These
materials include: evaluations, research studies, contracts and grant
agreements, technical assessments, and deliverables such as annual and
final reports, training materials, conference proceedings, videotapes,
and computer software.

The Development Experience Clearinghouse has created an electronic
library catalogue, the Development Experience System (DEXS), which
contains citations for over 100,000 documents. The DEXS also contains a
growing collection of complete, full-text publications saved in Adobe
Acrobat that can be downloaded to your computer. To search the DEXS,
visit the Clearinghouse\'s website at <http://www.dec.org/partners/>
(public website). Agency personnel may access the DEXS through CDIE
Online at [http://cdie.usaid.gov](http://cdie.usaid.gov/) (availably
only within the USAID firewall). The DEX is also available on CD-ROM
(CD-DEXS).

PPC also manages the R4 and Annual Report database, which is available
through the internal USAID intranet at <http://cdie.usaid.gov/r4/>
(available only within the USAID firewall) and the public USAID internet
at <http://www.dec.org/partners/ardb/>.

# Appendix D: ADS 201.3.7.6 and ADS Chapter 203