Special Inspector General for 

Afghanistan Reconstruction

SIGAR 17-10 Audit Report

USAID’s Measuring Impacts of Stabilization 

Initiatives: Program Generally Achieved Its 

Objectives, but USAID’s Lack of a 

Geospatial Data Policy and Standards 

Affected Its Implementation

SIGAR 17-10-AR/USAID’s MISTI Program

SIGAR

O C T O B E R

2016
For more information, contact SIGAR Public Affairs at (703) 545-5974 or sigar.pentagon.ccr.mbx.public-affairs@mail.mil. 

WHAT SIGAR REVIEWED

From September 2003 through 

December 2015, the U.S. Agency for 

International Development (USAID) spent 

more than $2.3 billion on more than two 

dozen stabilization activities and 

programs in Afghanistan. The stabilization 

programs were intended to support at-risk 

populations, extend the reach of the 

Afghan government to unstable areas, 

provide job opportunities, build trust 

between citizens and their government, 

and encourage local populations to take 

an active role in their development. 

Beginning in 2011, with the drawdown of 

coalition troops throughout Afghanistan, 

USAID faced increasing challenges in 

overseeing its stabilization programs. To 

address these challenges, in March

2012, USAID awarded Management 

Systems International Inc. (MSI) a 

contract to implement the Measuring 

Impacts of Stabilization Initiatives (MISTI) 

program to monitor and evaluate eight 

ongoing stabilization programs costing 

approximately $762 million. The agency 

estimated that MISTI would last 3 years 

and cost approximately $15 million. The 

contract ended in October 2015 and 

ultimately cost $19.3 million.

The objectives of this audit were to (1) 

assess the extent to which MSI met its 

contract requirements and USAID 

provided oversight; (2) assess the extent 

to which the MISTI program met its 

objectives; and (3) identify challenges MSI 

faced in conducting third-party monitoring 

under MISTI and the extent to which 

USAID has addressed those challenges. 

WHAT SIGAR FOUND

SIGAR found that although early monitoring and evaluation (M&E) challenges 

caused USAID to make multiple modifications to the MISTI contract, MSI met 

its contract requirements, and USAID generally performed contract oversight in 

accordance with agency regulations. USAID modified the MISTI contract 

multiple times to address the agency’s inability to verify directly whether the 

contractors implementing the stabilization programs were meeting their 

contract requirements, and to assess the quality of the data being collected 

from them. The modifications to and expansions in MISTI’s scope added to the 

cost of the contract with MSI. Within the first year, the contract’s maximum 

value increased from approximately $15 million to $21 million, though USAID 

ultimately spent $19.3 million on the program. Despite this increased cost, 

SIGAR determined that USAID’s decisions appropriately followed contract 

requirements, helped ensure oversight of its stabilization activities, and 

potentially benefitted the agency’s overall understanding of the impact of its 

stabilization efforts.

Although MSI completed all 10 contract deliverables, it was late, through no 

apparent fault of its own, in establishing the program baselines required to 

complete two of the deliverables. Specifically, the contract required MSI to 

establish baselines for its evaluation of the eight USAID stabilization programs 

within the contract’s first year, or by March 2013. However, MSI did not do this 

until 4 months later. USAID officials said that during MISTI’s first year, the four 

Stability in Key Areas (SIKA) programs, the Community Cohesion Initiative, and 

the Community Development Program were not yet performing activities 

necessary for MSI to generate baseline data, largely because of security 

concerns in areas where the programs operated and because implementers 

were not completing required activities. As a result, MSI did not have sufficient 

performance data to use to establish the baselines.

USAID generally followed Federal Acquisition Regulation (FAR) and USAID 

requirements for contract oversight by performing quality assurance and 

technical performance monitoring functions on the MISTI contract. For 

example, the contracting officer’s representatives (COR) generally maintained 

adequate files to document correspondence with the contractor and actions or 

decisions taken related to the contract, as required. However, USAID could not 

provide documentation supporting MSI’s request and the agency’s approval for 

MSI to compare the SIKA programs to the World Bank’s National Solidarity 

Program when the two operated in the same areas.

SIGAR found that MSI met MISTI program objectives to complete independent 

evaluations and impact assessments of USAID’s eight stabilization programs 

and develop recommendations for improving them. For example, by March 

2015, MSI had conducted 6 mid-term evaluations and made 82 

recommendations to USAID, most of which USAID accepted. USAID also 

received information from MSI that could have influenced its decisions about 

the direction and design of the stabilization programs.

SIGAR

Special Inspector General for 

Afghanistan Reconstruction

SIGAR 17-10 AUDIT REPORT

October 2016

USAID’s Measuring Impacts of Stabilization Initiatives: Program 

Generally Achieved Its Objectives, but USAID’s Lack of a Geospatial Data 

Policy and Standards Affected Its Implementation 
For more information, contact SIGAR Public Affairs at (703) 545-5974 or sigar.pentagon.ccr.mbx.public-affairs@mail.mil. 

However, many of the recommendations USAID officials told SIGAR that they considered through November 2015 were of 

limited value in Afghanistan because seven of the eight stabilization programs ended in 2015. Moreover, before MISTI began 

in 2012, the U.S. government had already decided to discontinue the stabilization mission and stop implementing new 

programs, because of a U.S. policy shift to support the transition to building the capacity of the Afghan government to lead its 

own development efforts. MSI’s findings and recommendations did not influence this decision to discontinue the stabilization 

mission.

MSI also met program objectives to track stabilization trends and provide best practices for stabilization programs. For 

example, MSI reached several conclusions and identified trends from its tracking of the programs’ performance and impacts. 

MSI noted that: (1) USAID’s stabilization programs, though having some positive impacts throughout implementation, were 

unsuccessful overall as implemented in Afghanistan; (2) insurgents targeted programs in areas where the Afghan government 

was in control; and (3) the programs should include more programming aimed at empowering women and at literate Afghans 

because these groups were least supportive of the Taliban.

SIGAR identified two significant issues that MSI faced in conducting third-party monitoring under the MISTI contract: (1) the 

lack of accurate geospatial data and (2) lack of sharing existing geospatial data from, for example, USAID. USAID and MSI 

officials told SIGAR that the geospatial data they received from the stabilization programs’ implementers were of poor quality. 

MSI also reported that the implementers did not have standardized collection platforms or methods for using geospatial data, 

and that they used different formats and Global Positioning System (GPS) devices for recording coordinates and storing 

geospatial data. Because of the inaccurate data it received, MSI told us it could not properly locate where USAID conducted 

stabilization activities, and as a result, it could not begin conducting verification work in February 2014, as the contract 

required. In order to meet the contract requirement, MSI officials had to first create its own geospatial database and work 

closely with the stabilization program implementers to address errors. MSI officials said they initially spent up to 60 percent of 

their time addressing these errors instead of performing verification work.

In addition to these problems, MSI did not have access to existing government geospatial data, as it was supposed to 

according to the contract. MSI officials stated that they understood this would include geospatial information from the 

Department of Defense’s (DOD) databases and USAID’s Afghan Info database, which had information on village locations and 

naming conventions. MSI did not gain access to existing DOD geographic data, and USAID officials did not provide an 

explanation for not getting this data from DOD. Moreover, although MSI had access to Afghan Info, its employees told us they 

had concerns about its accuracy. 

Since 1990, the Office of Management and Budget’s (OMB) Circular A-16 has required agencies—including USAID—to issue 

agency standards for the collection and reporting of geospatial data. Officials from USAID’s Bureau for Policy, Planning and 

Learning and the GeoCenter confirmed that OMB Circular A-16 applies to USAID. However, USAID officials told SIGAR that they 

do not have any agency or mission-level—for example, USAID Mission for Afghanistan (USAID/Afghanistan)—policies to govern 

or guide the collection, maintenance, use, or sharing of geospatial data. When SIGAR asked why the agency has not followed 

OMB Circular A-16, the officials could not provide an explanation.

These challenges with geospatial data are not new to USAID or limited to Afghanistan. Over the past decade, USAID and SIGAR 

have repeatedly identified problems with the agency’s ability to use accurate geospatial data and an inability to standardize a 

practice and process for collecting, managing, and reporting that data. Without an established policy and standards for how 

USAID and its implementers should collect, maintain, use, and share geospatial data, the agency will continue to face 

problems with inaccuracies and inconsistencies in the data, and, therefore, maintain a limited understanding of the locations, 

and visual verification, of its activities being conducted in Afghanistan.

WHAT SIGAR RECOMMENDS

To enhance USAID/Afghanistan’s ability to monitor, evaluate, and oversee its development efforts, SIGAR recommends that 

the USAID Administrator, as soon as possible, establish an agency policy implementing OMB Circular A-16 requirements that 

institute agency-wide standards for collecting, using, and sharing geospatial data both within USAID and with other U.S. 

government agencies and nongovernmental entities, including those conducting reconstruction efforts in Afghanistan.

SIGAR received written comments on a draft of this report from USAID’s Office of Afghanistan and Pakistan Affairs. USAID 

concurred in principle with our recommendation and noted several actions either taken, underway, or planned that will 

address the weaknesses SIGAR identified.
October 26, 2016 

The Honorable John F. Kerry 

Secretary of State 

The Honorable P. Michael McKinley 

U.S. Ambassador to Afghanistan 

The Honorable Gayle E. Smith 

Administrator, U.S. Agency for International Development 

Mr. Herbert B. Smith 

USAID Mission Director for Afghanistan 

This report discusses the results of SIGAR’s audit of the U.S. Agency for International Development’s (USAID) 

Measuring Impacts of Stabilization Initiatives (MISTI) program. In March 2012, USAID awarded a contract, 

which ultimately cost $19.3 million, to Management Systems International Inc. (MSI) to implement MISTI. The 

program was intended to monitor and evaluate eight ongoing USAID stabilization programs in Afghanistan. This

report focuses on MSI’s implementation of MISTI and USAID’s oversight of the program from March 2012 to 

October 2015. 

We recommend that the USAID Administrator, as soon as possible, establish an agency policy implementing 

Office of Management and Budget Circular A-16 requirements. This policy should set agency-wide standards 

for collecting, using, and sharing geospatial data both within USAID and with other U.S. government agencies 

and nongovernmental entities, including those conducting reconstruction efforts in Afghanistan. 

We received written comments on a draft of this report from USAID’s Office of Afghanistan and Pakistan Affairs. 

USAID concurred in principle with our recommendation and noted several actions either taken, underway, or 

planned that will address the weaknesses we identified. 

SIGAR conducted this work under the authority of Public Law No. 110‐181, as amended, and the Inspector 

General Act of 1978, as amended; and in accordance with generally accepted government auditing standards. 

John F. Sopko 

Special Inspector General 

 for Afghanistan Reconstruction 
SIGAR 17-10-AR/USAID’s MISTI Program Page iv

TABLE OF CONTENTS

Background .................................................................................................................................................................. 2

USAID Modified the MISTI Contract Due to M&E Challenges; Despite Those Challenges, MSI Met Its Contract 

Requirements, and USAID Generally Oversaw the Contract in Accordance with Agency Regulations ................... 6

MISTI Met Its Objectives and Provided USAID with Information Relevant to the Management of Stabilization 

Programs in Afghanistan ............................................................................................................................................. 9

The MISTI Program Faced Implementation Problems Because of a Lack of Geospatial Data and USAID 

Standards for Collecting and Reporting this Data ...................................................................................................11

Conclusion ..................................................................................................................................................................14

Recommendation ......................................................................................................................................................14

Agency Comments .....................................................................................................................................................15

Appendix I - Scope and Methodology .......................................................................................................................16

Appendix II - Comments from the U.S. Agency for International Development ......................................................18

Appendix III - Acknowledgments ...............................................................................................................................24

TABLES 

Table 1 - USAID/Afghanistan Stabilization Programs under MISTI Oversight .......................................................... 4

Table 2 - MISTI Contract Deliverables ........................................................................................................................ 7
SIGAR 17-10-AR/USAID’s MISTI Program Page v 

ABBREVIATIONS 

ADS Automated Directives System

COR contracting officer’s representative

DOD Department of Defense

FAR Federal Acquisition Regulation

FGDC Federal Geographic Data Committee

GPS Global Positioning System

M&E monitoring and evaluation

MISTI Measuring Impacts of Stabilization Initiatives

MSI Management Systems International Inc.

NGO nongovernmental organization

OMB Office of Management and Budget

SIKA Stability in Key Areas

USAID U.S. Agency for International Development

USAID/Afghanistan USAID Mission for Afghanistan

USAID/OAPA USAID Office of Afghanistan and Pakistan Affairs

USAID OIG USAID Office of Inspector General
SIGAR 17-10-AR/USAID’s MISTI Program Page 1 

From September 2003 through December 2015, the U.S. Agency for International Development (USAID) spent 

more than $2.3 billion on more than two dozen stabilization programs in Afghanistan. The programs were 

intended to work with and support at-risk populations, extend the Afghan government’s reach to unstable 

areas, provide job opportunities, build trust between citizens and their government, and encourage local 

populations to take an active role in their development.1 With the drawdown of coalition troops throughout 

Afghanistan beginning in 2011, USAID faced increasing challenges in overseeing its stabilization programs. 

These challenges included high levels of violence and reduced security and protection, which impeded USAID 

personnel from traveling to or accessing locations where the agency was implementing its programs. USAID’s

stabilization programs ended in August 2016. 

To address those challenges, in March 2012, USAID awarded a contract to Management Systems International 

Inc. (MSI) to implement the Measuring Impacts of Stabilization Initiatives (MISTI) program to monitor and 

evaluate the agency’s eight ongoing stabilization programs. As of March 2016, those eight programs had cost 

a total of approximately $762 million.2 MISTI was a third-party monitoring and evaluation (M&E) program 

designed “to measure and map stabilization trends and impacts in areas such as security, rule of law, and 

economic activity; build a community of practice for rigorous monitoring and evaluation of Afghan 

reconstruction programs; and communicate lessons learned for the transition to Afghan-led sustainable 

development.”3 MISTI’s three main objectives were to: 

1. Provide independent evaluation and impact assessment of USAID stabilization programs. 

2. Collect, synthesize, and analyze data at the district, provincial, and regional levels to track higherorder stabilization trends and inform U.S. and Afghan government policy and practice related to 

the transition from short-term, donor-driven, counterinsurgency-focused programming to Afghan 

government-led, long-term traditional development activities. 

3. Contribute to the larger body of knowledge on best practices and lessons learned related to the 

design, implementation, and assessment of stabilization activities within a counterinsurgency 

context. 

Initially valued at a maximum of $15 million, the MISTI contract ultimately cost $19.3 million. 

The objectives of this audit were to (1) assess the extent to which MSI met its contract requirements and 

USAID provided oversight; (2) assess the extent to which the MISTI program met its objectives; and (3) identify 

challenges MSI faced in conducting third-party monitoring under MISTI and the extent to which USAID has 

addressed those challenges.4 

 

1 SIGAR, Quarterly Report to the United States Congress, January 30, 2016. 

2 According to USAID, performance monitoring “is the ongoing and routine collection of performance indicator data to 

reveal whether desired results are being achieved and whether implementation is on track. Performance monitoring 

continues throughout the life of an activity [and] project.” A performance evaluation “focuses on what a particular project or 

program has achieved (either at an intermediate point in execution or at the conclusion of an implementation period), how 

it is implemented, how it is perceived and valued, whether expected results occur, and other questions that are pertinent to 

project design, management and operational decision making.” See USAID, Automated Directives System (ADS) Chapter 

200, Introduction to Programming, partially revised July 18, 2014; and USAID, ADS Chapter 203, Assessing and Learning, 

partially revised November 2, 2012. 

3 USAID, Stabilization Unit Afghanistan: Performance Management Plan, Fiscal Years 2012–15, June 18, 2013. 

4 For this audit, we did not assess MSI’s management of the MISTI contract’s costs, which was the subject of a separate 

SIGAR financial audit of costs incurred under the contract (see SIGAR, USAID’s Measuring Impact of Stabilization Initiative: 

Audit of Costs Incurred by Management Systems International, SIGAR 15-53-FA, April 21, 2015). In that financial audit, we 

identified two significant deficiencies in MSI’s internal control and two instances of noncompliance with the terms and 

conditions of the contract, all of which related to cash management procedures and financial reporting. We did not identify 

any questioned costs. In addition, we did not review or assess the implementation or results of the eight stabilization 

programs MSI was required to monitor and evaluate under the MISTI program. 
SIGAR 17-10-AR/USAID’s MISTI Program Page 2 

To accomplish these objectives, we reviewed USAID’s MISTI contract, task orders, modifications, and MSI’s 

contract requirements and deliverables. We interviewed current and former USAID officials responsible for 

overseeing MISTI, including staff from USAID’s Office of Afghanistan and Pakistan Affairs (OAPA) and the USAID 

Mission for Afghanistan’s (USAID/Afghanistan) Office of Program and Project Development; MSI officials 

responsible for implementing the MISTI contract; and a nongovernmental organization (NGO), the Global Public 

Policy Institute. We reviewed USAID’s Learning Lab best practices for conducting M&E. We also reviewed U.S. 

government standards and requirements for geospatial data collection, processing, maintenance, and 

information sharing, including Office of Management and Budget (OMB) circulars and USAID policies. We 

conducted our work from November 2014 through October 2016 in accordance with generally accepted 

government auditing standards. A more detailed discussion of our scope and methodology is in appendix I. 

BACKGROUND 

Due to ongoing security concerns and the drawdown of coalition military and civilian personnel in Afghanistan, 

in December 2013, USAID took steps to develop a more robust means of conducting oversight and M&E of its 

programs. The agency ultimately developed a five-tiered strategy to provide alternative means of M&E and 

oversight through (1) U.S. government representatives, (2) implementing partners, (3) Afghan government 

representatives and other international donors, (4) Afghan civil society groups and beneficiaries, and (5) 

independent monitoring contractors, also known as third-party or “remote” monitors.

5 USAID M&E practices in 

Afghanistan rely significantly on geographic location information, also referred to as geospatial data, among 

other activities, to identify project sites, visually verify whether program activities are being conducted, and 

report and present information visually and spatially in maps, charts, and reports.6

As early as 2003, USAID authorized the use of alternative M&E methods to, among other things, keep its 

personnel safe while satisfying its need to visit project sites and meet project beneficiaries to assess project 

implementation.7 USAID’s Automated Directives System (ADS) Chapter 202, Achieving, supports, and even 

encourages, the use of third-party monitors as a way of doing this.8 USAID hires third-party monitors as 

contractors to conduct M&E activities, such as verifying project data through site visits, collection of geographic 

location data, and interviews with, among others, program implementers and beneficiaries.9 The monitors 

report to USAID as part of the agency’s contract oversight, but do not replace USAID program staff and 

managers, or completely relieve them of their responsibility for overseeing how agency programs are being 

implemented. 

In December 2015, the USAID Office of Inspector General (OIG) issued an audit report on USAID/Afghanistan’s

five-tiered M&E strategy. The report identified several issues with USAID/Afghanistan’s M&E policy and 

processes.10 For example, USAID OIG reported that USAID/Afghanistan did not have standards for what 

 

5 USAID introduced this strategy over a year after MISTI began. 

6 OMB defines “geospatial” as relating to or denoting data that is associated with a particular location. OMB Circular A-16, 

“Coordination of Geographic Information and Related Spatial Data Activities,” revised August 19, 2002.

7 SIGAR, USAID Spent Almost $400 Million on an Afghan Stabilization Project Despite Uncertain Results, but Has Taken 

Steps to Better Assess Similar Efforts, SIGAR Audit-12-8, April 25, 2012; and SIGAR, Statement of John F. Sopko Before the 

Subcommittee on National Security, Committee on Oversight and Government Reform, Lessons Learned from Oversight of 

the U.S. Agency for International Development’s Efforts in Afghanistan, April 3, 2014. 

8 USAID, ADS Chapter 202, Achieving, partially revised January 25, 2012; and USAID/Afghanistan Mission Order 203.2, 

Monitoring and Evaluation, October 24, 2012. 

9 After the Withdrawal: the Way Forward in Afghanistan and Pakistan (Part III), Joint Hearing before the Subcommittee on 

the Middle East and North Africa and the Subcommittee on Asia and the Pacific of the U.S. House of Representatives 

Committee of Foreign Affairs, H.R. Rep No. 113-233; Statement for the Record, Donald L. “Larry” Sampler, Assistant to the 

Administrator and Director of the Office of Afghanistan and Pakistan Affairs, USAID, December 10, 2014. 

10 USAID OIG, Audit of USAID/Afghanistan's Strategy for Monitoring and Evaluating Programs Throughout Afghanistan, F306-16-001-P, December 10, 2015. 
SIGAR 17-10-AR/USAID’s MISTI Program Page 3 

constitutes effective M&E and that Afghan Info—the database USAID program managers used to track foreign 

assistance activities, spending, and monitoring data—was not updated or used as directed. 

The use of third-party monitors is not new or unique to Afghanistan or USAID. The United Nations, World Bank, 

the foreign aid agencies of other nations, and numerous NGOs use them for oversight in conflict or high-threat 

areas, such as Afghanistan. USAID has been using third-party monitors for at least two decades in countries 

such as Afghanistan, Yemen, South Sudan, Iraq, and Pakistan. According to MSI officials, as of August 2015, 

the company had 12 third-party M&E contracts covering USAID programs around the world.11 

USAID Designed the MISTI Program to Improve M&E of Its Stabilization Programs 

In February 2010, USAID/Afghanistan formed the Stabilization Unit to unite all U.S. government stabilization 

planning and programs under one office.12 The unit’s responsibilities were managing, coordinating, and 

monitoring and evaluating USAID’s stabilization programs. Prior to awarding the MISTI contract in March 2012, 

USAID/Afghanistan’s M&E of these programs consisted of the contractor for each program performing M&E 

requirements on its own program and the Stabilization Unit using three additional separate contracts to 

perform additional M&E on three of the programs.13 In September 2010, USAID reported that this approach 

was inefficient and raised the possibility of using only one independent M&E contractor for all stabilization 

programs. 

The agency was concerned then about its inability to measure the impacts of its stabilization programs and 

access the programs to provide direct oversight. As a result, USAID/Afghanistan replaced the three separate 

M&E contracts with one contract—MISTI.14 USAID reported that using one monitor would help ensure 

contractor compliance with USAID requirements and increase the level of transparency on its stabilization 

efforts. 

On March 14, 2012, using a competitive process, USAID awarded the MISTI contract to MSI. The agency 

estimated that the program would last 3 years—with an original 18-month base year period of performance of 

March 2012 to September 2013 and one 18-month option year—and cost approximately $15 million. 

Ultimately, USAID modified the MISTI contract 12 times and issued 2 extensions, increasing the contract’s 

period of performance to approximately 3 and a half years—through October 7, 2015—and its cost to 

approximately $19.3 million. Under MISTI, MSI was to provide M&E of eight stabilization programs that 

collectively cost approximately $762 million as of March 2016 (see table 1). 

 

11 USAID also contracted with MSI to implement its Monitoring Support Project in Afghanistan, which aims to assist 

USAID/Afghanistan in monitoring its portfolio of development assistance projects. 

12 USAID/Afghanistan closed the Stabilization Unit in 2014 and made the Office of Democracy and Governance responsible 

for the mission’s stabilization programs.

13 The three stabilization programs were Local Governance and Community Development implemented by Checchi and 

Company Consulting Inc. (Checchi), the Afghan Stability Initiative implemented by Altai Consulting, and Community 

Development implemented by A.E. Ferguson. The first two programs ended before MISTI began. 

14 According to USAID Acquisition Regulation requirements, each contractor was still required to conduct its own M&E. 
SIGAR 17-10-AR/USAID’s MISTI Program Page 4 

Table 1 - USAID/Afghanistan Stabilization Programs under MISTI Oversight

Stabilization 

Program Program Objective(s) Period of Performance

Total 

Estimated 

Cost 

($ millions)

Amount 

Spent1

($ millions)

Community

Development 

Program2

Promote links between the Afghan government 

and communities, primarily those in insecure rural 

areas, through, for example, labor-intensive 

infrastructure projects.

March 2009–

August 2013 $266.0 $264.0

Community 

Cohesion 

Initiative

Build community cohesion and resilience in areas 

vulnerable to insurgent exploitation by increasing 

cohesion among and between communities; 

supporting peaceful and legitimate governance 

processes and outcomes; and countering violent 

extremism. 

March 2012–

December 2015 $133.6 $117.0

Stability in Key 

Areas (SIKA)-

North

Promote stabilization in key areas by supporting 

Afghan government efforts to implement 

community-led development and governance 

initiatives that respond to the population’s needs 

and concerns, with an aim to build confidence in 

local government and increase the provision of 

basic services.

March 2012–

July 2015 $38.0 $37.0

SIKA-West 

Expand and improve the legitimacy of the Afghan 

government in unstable districts and communities 

by assisting district entities to better understand 

their operating environment and the challenges to 

stability they face.

January 2012–

August 2015 $54.0 $51.2

SIKA-South

Support the Afghan Independent Directorate of 

Local Governance and the Ministry of Rural 

Rehabilitation and Development to build 

governance, build development projects, facilitate 

governance and democracy initiatives, and deliver 

training and mentoring to district and provincial 

entities.

April 2012–

July 2015 $82.1 $79.3

SIKA-East

Expand and improve the legitimacy of the Afghan 

government in unstable districts and communities 

by assisting district entities to better understand 

their operating environment and the challenges to 

stability they face, and then enable district 

governments to implement activities aimed at 

addressing those identified sources of instability.

December 2011–

September 2015 $145.0 $138.7

Afghan Civilian 

Assistance 

Program II

Partner with the Afghan Ministry of Labor, Social 

Affairs, Martyrs, and Disabled to provide 

education and vocational training to affected 

families.

September 2011–

February 2015 $64.0 $52.4
SIGAR 17-10-AR/USAID’s MISTI Program Page 5 

Kandahar 

Food Zone

Identify and attempt to address drivers of poppy 

cultivation in seven districts of Kandahar province, 

and work with the Afghan government to reduce 

poppy cultivation and increase the effectiveness 

and legitimacy of national and subnational 

administrations 

July 2013–

August 2016 $27.7 $22.5

Totals $810.4 $762.1

Source: SIGAR analysis of USAID data

1 As of March 31, 2016.

2 From April 2012 to August 2013, MISTI provided oversight for the last two of the five phases of the program.

Ongoing, Systemic Challenges Affect Program Implementation and M&E in 

Afghanistan 

USAID faces a host of systemic challenges in implementing and conducting oversight of its stabilization 

programs in Afghanistan. These challenges are not unique to MISTI and cut across all U.S. government and 

stakeholder development efforts in Afghanistan. These challenges include the high levels of violence endemic 

to a war zone and reduced security and protection assets available to USAID/Afghanistan personnel, which 

impede them from traveling to and accessing places where programs are under way. In addition, reductions in 

USAID/Afghanistan staff at the U.S. Embassy in Kabul have left fewer employees to oversee program 

implementation. Furthermore, USAID and MSI reported challenges with the timely vetting of contractors and 

their personnel, managing frequent staff turnover, and the need to include Afghan women in programming and 

M&E activities, while facing cultural and gender issues that continue to affect the agency’s ability to implement 

programs and conduct M&E. 

USAID Has Requirements and Guidance for Managing Contracts, Monitoring and 

Evaluating Programs, and Developing Policies and Standards for Using Geospatial 

Data 

The Federal Acquisition Regulation (FAR) requires U.S. government agencies, such as USAID, to clearly define 

their requirements when contracting for services;15 safeguard the interests of the United States in its 

contractual relationships;16 and perform contract quality assurance “as may be necessary to determine that 

the supplies or services conform to contract requirements.”

17 The FAR also requires contracting officers to 

designate contracting officer’s representatives (COR)18 to assist “in the technical monitoring or administration 

of a contract,” including the maintaining of contract files documenting actions taken in “accordance with the 

delegation of authority” from the contracting officer.

19

Additionally, USAID has established agency guidance for conducting M&E of its programs and contractors. For 

example, ADS Chapter 202, Achieving, (1) requires USAID officials to assess the performance of contractors to 

determine whether the outputs, or deliverables, produced by the contractors are timely and of acceptable 

quality, and (2) supports the use of third-party monitoring as a way of recognizing the need to keep USAID 

mission personnel safe while ensuring that project monitoring is conducted in a “high threat environment,” 

15 FAR 37.503.

16 FAR 1.602-2. 

17 FAR 46.401. 

18 Unless the contracting officer retains and executes the COR duties. 

19 FAR Subpart 1.6. 
SIGAR 17-10-AR/USAID’s MISTI Program Page 6 

such as Afghanistan. In addition, ADS Chapter 203, Assessing and Learning, requires agency programs to 

establish performance baselines from which to assess program effectiveness and assess the quality of data 

and processes used to establish and report program results. 

OMB Circular A-16 provides direction and requirements to federal agencies related to the collecting and 

reporting of geospatial data.20 This circular and related supplemental guidance also established the 

interagency Federal Geographic Data Committee (FGDC), which develops government standards and promotes 

the development, use, sharing, and dissemination of geospatial data throughout the federal government. 

USAID MODIFIED THE MISTI CONTRACT DUE TO M&E CHALLENGES; DESPITE 

THOSE CHALLENGES, MSI MET ITS CONTRACT REQUIREMENTS, AND USAID 

GENERALLY OVERSAW THE CONTRACT IN ACCORDANCE WITH AGENCY 

REGULATIONS 

USAID Modified the MISTI Contract’s Scope to Address Early Implementation 

Challenges 

USAID revised the scope of the MISTI contract multiple times in order to address the agency’s inability to (1) 

verify directly whether the contractors for its eight stabilization programs were meeting their contract 

requirements, and (2) assess the quality of the data being collected from those contractors. For example, in its 

Federal Managers’ Financial Integrity certification reports, USAID/Afghanistan stated that it was increasingly 

unable to perform its required M&E responsibilities for the programs to verify whether the contractors were 

completing their contractual obligations and the programs were meeting their objectives, due to security and 

travel restrictions placed on USAID/Afghanistan staff.21 As a result, the agency modified the MISTI contract in 

February 2014 to require that MSI conduct this verification, in addition to its existing requirement to perform 

M&E of the programs. 

In addition, MSI and USAID reported that they were concerned with the accuracy and quality of the data that 

program implementers were collecting, and had questions about the experimental methodologies MSI was 

using to measure the programs’ effectiveness. As a result, in May 2014—2 years into the 3.5 year MISTI 

contract—USAID modified the contract to allow a third party, the RAND Corporation, to conduct an independent 

peer review of MSI’s evaluation process and the data collected under USAID’s stabilization programs. 

The modifications and expansions in MISTI’s scope added to the contract’s cost. Within the first year, the 

contract’s maximum value increased from approximately $15 million to $21 million. However, we determined 

that USAID’s decisions to increase the scope appropriately followed contract requirements, helped ensure 

oversight of its stabilization programs, and potentially benefitted the agency’s overall understanding of the 

impact of its stabilization efforts. 

MSI Completed All Required Contract Deliverables 

ADS Chapter 202 requires contracting officers to ensure that contractors perform in accordance with the terms 

in their contracts or other agreements by, for example, reviewing and approving deliverables and performance 

 

20 OMB Circular A-16, “Coordination of Geographic Information and Related Spatial Data Activities,” revised August 19, 

2002; and Executive Office of the President, OMB, Memorandum to Heads of Executive Departments and Agencies, 

Subject: Issuance of OMB Circular A-16 Supplemental Guidance, M-11-03, November 10, 2010. 

21 To comply with the Federal Managers’ Financial Integrity Act of 1982, Pub. L. No. 97-255, USAID requires its missions to 

report annually on significant internal control deficiencies and action plans to correct them. 
SIGAR 17-10-AR/USAID’s MISTI Program Page 7 

reports; reporting variations and problems; recommending modifications; and analyzing financial reports.22 For 

the purposes of this report, we analyzed MISTI contract records that MSI and USAID provided to identify the 

contract deliverables MSI was required to produce and determined the extent to which MSI completed them.23

We found that MSI completed all 10 deliverables. Table 2 lists the deliverables and their completion dates. 

Table 2 - MISTI Contract Deliverables

Deliverables Description of Requirements or Objectives Date Completed

1. Review of stabilization 

resources

Identify challenges, best practices, and lessons learned for thirdparty monitoring in Afghanistan.February 2013

2. Establish a knowledge 

management platform

Deliver a U.S. government web-accessible database and fully 

operational knowledge management platform for USAID and M&E 

stakeholders to use in program performance management.

March 2015

3. Review and revise the 

USAID/Afghanistan 

Stabilization Unit’s 

Performance Management 

Plan

Review and provide suggested edits to USAID’s Stabilization Unit 

Performance Management Plan, including data quality 

assessments.

June 2014

4. Performance evaluations 

of stabilization programs

Spatial analysis of baseline stabilization trends at sub-district, 

district, provincial, and regional scales.

September 2015

5. Impact evaluations of 

stabilization programs

Conduct midterm and final performance evaluations for the 

Community Development Program, Community Cohesion 

Initiative, all four SIKAs, Kandahar Food Zone, and Afghan Civilian 

Assistance Program II stabilization programs.

April 2015

6. Evaluation and 

assessment of best 

practices for the district

stability 

framework/stability 

analysis methodologies

Review and assess the effectiveness of the concepts and tools for 

stabilization analysis, resiliency assessment, relationship building, 

and community engagement currently employed by stabilization 

programs in Afghanistan.

May 2014

7. Operationalize 

stabilization trends and 

impact evaluation survey

Conduct a baseline survey as a basis for measuring program 

impacts for the Wave 1 impact evaluation survey. Conduct four 

additional surveys as the basis of midterm and final impact 

evaluations.

April 2015

 

22 ADS Chapter 202 requires USAID to assess the performance of contractors, including whether the outputs, or 

deliverables, produced by the contractor are timely and of acceptable quality. See USAID, ADS Chapter 202, section 

202.3.6.1, Assessing Performance of Contractors and Recipients, partially revised January 25, 2012. 

23 These deliverables are identified in “Section C: Description. Specifications/Statement of Work” of the MISTI contract. The 

contract lists a number of other documents that MSI was required to produce for USAID, such as annual work plans and 

quarterly performance monitoring reports. These documents, largely directed at contract management and oversight, were 

not in included in the list of deliverables that USAID tracked. However, we obtained these documents and included them as 

part of our analysis of USAID’s contract oversight. 
SIGAR 17-10-AR/USAID’s MISTI Program Page 8 

8. Lessons learned and best 

practices

Deliver lessons learned/best practices briefings; annual reports 

and one final report documenting project performance, key 

findings, lessons learned, best practices, and implementation 

challenges; conduct quarterly stabilization M&E summit meetings 

and report on the results; and three research reports, suitable for 

publication in scholarly journals, that document lessons learned 

and best practices from the program.

November 2015

9. Stabilization program 

project and activity 

verification

Conduct site visits, verify, and report on the status or completion 

of stabilization projects and activities conducted by USAID’s 

implementers.

September 2015

10. RAND peer review of MISTI 

data

Provide an assessment of the MISTI program survey design 

methodology, tools used to measure and map stabilization trends, 

and approach used to evaluate the impacts of stabilization 

projects to help validate the quality of MISTI data independently.

October 2014

Source: SIGAR analysis of USAID data

Although MSI completed all 10 contract deliverables, it was, through no apparent fault of its own, late in

establishing the program baseline required to complete the impact evaluations of stabilization programs (item 

#5 in table 2) and to operationalize stabilization trends and impact evaluation survey (item #7 in table 2).24

USAID’s ADS Chapter 203, Assessing and Learning, requires the establishment of performance baselines from 

which to assess the effectiveness of the agency’s programs and activities.25 In addition, ADS 203 states that 

the program baseline should be established at the beginning of the program. The MISTI contract specifically 

required MSI to establish a baseline for its evaluation of USAID’s stabilization programs within the contract’s 

first year, or by March 2013. However, MSI did not do this until 4 months later. USAID reported that during 

MISTI’s first year the four SIKAs, the Community Cohesion Initiative, and the Community Development Program 

were not yet performing activities necessary to generate baseline data. This was largely because of security 

concerns in areas where the programs operated and implementers not completing required activities. As a 

result, MSI did not have sufficient performance data to use to establish the baseline.26 Because MSI could not 

establish baselines early in MISTI implementation, USAID lacked a starting point from which to monitor and 

evaluate the progress of the stabilization programs over the first year of the MISTI contract. 

USAID Generally Met Its Contract Oversight Requirements 

Through our analysis of MISTI contract documentation and discussions with USAID contracting officers and 

CORs, we determined that USAID generally met its contract oversight requirements and provided sufficient 

oversight of the contract. The FAR requires contracting officers to designate CORs to assist in the contract 

activities, unless the contracting officer retains and executes the COR duties.27 The FAR also requires agencies 

to directly manage and oversee the administration of their contracts,28 and to follow an effective quality 

 

24 A baseline is the value of a performance indicator before the implementation of projects or activities. See USAID, ADS 

Chapter 203, Assessing and Learning, partially revised November 2, 2012. 

25 USAID, ADS Chapter 203, Assessing and Learning, partially revised November 2, 2012.

26 As we reported in July 2013, the SIKA programs had not met essential program objectives because USAID did not get a 

formal working agreement with the Afghan government and did not award grants on a timely basis (See SIGAR, Stability in 

Key Areas (SIKA) Program: After 16 Months and $47 Million Spent, USAID Had Not Met Essential Program Objectives, Audit 

13-16, July 29, 2013.)

27 FAR 1.602-2.

28 FAR 37.504.
SIGAR 17-10-AR/USAID’s MISTI Program Page 9 

assurance and monitoring process.29 Additionally, the FAR requires that CORs maintain complete records in 

their files that document their activities and any significant decisions made during contract execution.30 For 

example, in accordance with the FAR, contracting officers appropriately designated CORs in writing with 

appointment letters and documented they were trained to oversee the MISTI contract. 

USAID followed the FAR and its own internal requirements while performing contract quality assurance and 

technical performance monitoring functions related to its contract with MSI. The agency established and 

tracked requirements for MSI to produce annual work plans describing the activities it initiated and planned to 

execute in the next year. The contract also required that MSI produce quarterly performance reports describing 

items such as the activities and accomplishments in the preceding quarter, highlights of any problems 

affecting the delivery of services, and program progress against specific benchmarks. MSI fulfilled both of 

these reporting requirements. 

USAID’s CORs generally established and maintained adequate files to document correspondence with MSI and 

actions taken, as USAID guidance requires. However, the agency could not provide documentation approving 

MSI’s request and the agency’s approval for MSI to perform additional work comparing the performance of the 

SIKA programs with the World Bank’s National Solidarity Program when the two programs operated in the 

same areas.31 USAID and MSI personnel stated that the CORs asked for the additional analysis to help the 

agency compare the results from the World Bank’s methodology in its program to the results from MSI’s 

methodology used in MISTI. 

MISTI MET ITS OBJECTIVES AND PROVIDED USAID WITH INFORMATION 

RELEVANT TO THE MANAGEMENT OF STABILIZATION PROGRAMS IN 

AFGHANISTAN 

MSI Met MISTI Program Objectives to Evaluate USAID’s Stabilization Programs, but 

Its Recommendations to Improve Stabilization Efforts Are of Limited Use 

According to ADS Chapter 203, USAID’s overall success in managing programs depends on its ability to use 

program evaluation findings to strengthen programming efforts and decision making. By measuring program 

effectiveness, relevance, and efficiency, evaluation results can ensure accountability to stakeholders and 

inform resource allocation and other decisions.32 USAID/Afghanistan guidance states that the mission should 

determine whether it accepts and supports recommendations that come either from external evaluations or 

from a program’s final reporting in the mission’s design of future or follow-up work, and incorporate the 

recommendations as appropriate.33 

Based on our analysis, MSI met USAID guidance and MISTI program objectives to complete independent 

evaluations and impact assessments of USAID’s stabilization programs, and to develop recommendations for 

improving them. For example, by March 2015, MSI had conducted 6 MISTI midterm evaluations that resulted 

 

29 FAR 46.401.

30 FAR 1.604.

31 The Afghan Ministry for Rural Rehabilitation and Development began implementing the National Solidarity Program in 

2003 to extend the government’s administrative reach. It is a community-driven development program that aimed to create 

gender-balanced democratically elected community development councils and fund small-scale development projects. See 

World Bank Group, The National Solidarity Program: Assessing the Effects of Community-Driven Development in 

Afghanistan, Policy Research Working Paper 7415, September 2015. 

32 USAID, ADS Chapter 203, Assessing and Learning, partially revised November 2, 2012; and USAID Evaluation Policy, 

January 2011. 

33 USAID/Afghanistan Mission Order 203.2, Monitoring and Evaluation, October 24, 2012. 
SIGAR 17-10-AR/USAID’s MISTI Program Page 10

in 82 recommendations to USAID.34 We assessed each recommendation and the agency’s response. We 

determined that USAID accepted 72 of the recommendations, rejected 8 of them, and neither accepted or 

rejected remaining 2. 

The 82 recommendations made by MSI provided USAID with information that influenced its decisions 

regarding the direction and design of its Afghanistan stabilization programs. We found several examples in 

which USAID reviewed, changed, or updated the planning and implementation of programs that were still 

ongoing, or had sufficient remaining implementation time, based on MSI’s recommendations. For instance, in 

response to a recommendation on SIKA-East, USAID adjusted the program to focus during its last year on 

improving the responsiveness of local government institutions to Afghan citizens, instead of developing the 

capabilities of Afghan ministries.35 In another case, USAID reevaluated SIKA-South’s capacity-building 

assessments, curriculum, and workshop designs based on feedback from district and provincial 

stakeholders.36

However, many of the remaining midterm and final recommendations USAID reported as having considered 

through November 2015 were of limited value or benefit in Afghanistan because seven of the eight 

stabilization programs ended by 2015, leaving insufficient time to make meaningful program changes.37

Moreover, USAID officials told us that MSI’s findings and recommendations did not influence USAID’s decision 

to conclude its stabilization efforts in Afghanistan. Before MISTI started in 2012, the U.S. government had 

already decided to discontinue the stabilization mission and stop implementing new programs because of a 

policy shift to support the transition to building the capacity of the Afghan government to lead its own 

development efforts. 

MSI Met MISTI Program Objectives to Track Stabilization Trends and Provide Best 

Practices for Stabilization Activities 

Our review and analysis of USAID and MSI documentation showed that MSI addressed MISTI’s objective to 

collect, synthesize, and analyze district, provincial, and regional data to track stabilization trends and inform 

U.S. policy regarding the transition of responsibility for development efforts to the Afghan government. For 

example, USAID and MSI documents mention five semiannual MSI surveys of between 36,000 and 41,000 

households that provided data on how Afghan citizens perceived the impacts of USAID’s stabilization programs

in their communities. The surveys used “treated” groups (i.e., people in villages that received stability 

programming) and control groups (i.e., those in villages that did not receive such programming) to develop a 

comparative baseline against which to assess impacts. In addition, MSI reached several conclusions and 

identified trends from its tracking of the programs’ performance and impacts. For example: 

 USAID’s stabilization programs, though having some positive impacts throughout implementation, 

were unsuccessful overall as implemented in Afghanistan. 

 The programs generally were more successful when used in concert with other, similar development or 

stabilization programs, such as the World Bank’s National Solidarity Program. 

 Insurgents targeted USAID’s stabilization programs in areas where the Afghan government was in 

control. 

 

34 In November 2015, MSI provided us with documentation intended to show that it produced 52 additional 

recommendations in its final evaluations, and, 4 months later, USAID provided information that officials said showed how 

the agency considered them. However, we found that this information lacked enough evidence to include it in our overall 

analysis. 

35 MSI, Stability in Key Areas–East, Mid-Term Performance Evaluation, November 2014. 

36 MSI, Stability in Key Areas–South, Mid-Term Performance Evaluation, October 2014.

37 As noted in a previous footnote, we did not have sufficient information on the 52 final recommendations to determine a 

specific amount of all the recommendations that we consider to be of limited value or benefit in Afghanistan.
SIGAR 17-10-AR/USAID’s MISTI Program Page 11

 USAID’s stabilization programs should include more “soft” programming, such as capacity 

development, vocational training, and communications, aimed more at empowering women and at 

literate Afghans because these groups were least supportive of the Taliban. Such programming would 

likely have a greater impact on support for the Afghan government than the most common, “hard” 

infrastructure-focused stabilization activities. 

Officials in USAID/Afghanistan’s Office of Program and Project Development told us the agency has not 

responded officially to or confirmed the findings by MSI, nor is it required to do so. However, MSI’s data and 

analyses provide USAID’s technical and programming officers, as well as the broader development community, 

information on the impacts of the agency’s stabilization efforts in Afghanistan. 

Finally, MSI addressed MISTI’s objective “to contribute to the larger body of knowledge on best practices and 

lessons learned related to the design, implementation, and assessment of stabilization activities within a 

counterinsurgency context.”

38 USAID is in the process of publishing MISTI data and evaluation and 

performance reports on the Internet to make the information publically available. In addition, MSI produced

lessons learned briefings and reports, and shared this information with Afghan national and local government 

officials, other U.S. government employees, and NGOs. For example, MSI noted that stabilization programs 

should clearly outline which specific Afghan ministries and departments are responsible for overseeing 

programming, and identify their particular responsibilities, deliverables, communication channels, and 

necessary coordination activities.

According to USAID officials, some of MSI’s lessons learned are being considered in USAID’s implementation of other 

programs, such as the Services Under Program and Project Offices for Results Tracking II program, and in the design 

of its new Monitoring Support Program, and will be used to improve the way the agency reviews evaluation reports.

39

Those officials added that as part of its implementation of the Monitoring Support Program, MSI is maintaining MISTI 

documents to use for historical and lessons learned purposes. 

THE MISTI PROGRAM FACED IMPLEMENTATION PROBLEMS BECAUSE OF A 

LACK OF GEOSPATIAL DATA AND USAID STANDARDS FOR COLLECTING AND 

REPORTING THIS DATA 

USAID Lacked Accurate Geospatial Data Early in MISTI’s Implementation, and 

Sharing and Access to Geospatial Data Did Not Occur as Expected

We identified two significant challenges that MSI faced in conducting third-party monitoring under the MISTI 

program: (1) the lack of accurate geospatial data and (2) lack of sharing existing geospatial data. As part of the 

MISTI contract, MSI staff conducted third-party verification of USAID’s implementation of its stabilization 

programs, in order to help USAID better track performance and monitor progress on the programs. USAID and 

MSI officials told us the data they received from the stabilization programs’ implementers were poor quality. 

MSI also reported that the implementers did not have standardized collection platforms or methods for using 

geospatial data, and that they used different formats and Global Positioning System (GPS) devices for 

recording coordinates and storing geospatial data. USAID/OAPA officials and MSI contractors added that 

accurate GPS coordinates are important to their verification efforts for several reasons, particularly to enable 

them to discern whether programs have been executed and whether those programs are operating at the 

proper locations. 

 

38 USAID-306-TO-12-00004, MISTI Task Order with Management Systems International, March 14, 2012. 

39 The Services Under Program and Project Offices for Results Tracking II program is a follow-on to a contract that Checchi 

has executed since 2006 to improve the efficiency and effectiveness of USAID M&E, strategic communications and public 

information products, and performance management. 
SIGAR 17-10-AR/USAID’s MISTI Program Page 12

MSI told us that, because of the inaccurate data it received, it could not properly locate where USAID 

conducted stabilization activities, and as a result, it could not begin conducting verification work in February 

2014, as the contract required. In order to meet the contract requirement, MSI had to first create its own 

geospatial database and work closely with the stabilization program implementers to address errors. MSI 

officials said they initially spent up to 60 percent of their time addressing these errors instead of performing 

verification work.40 

In addition to these problems, MSI officials stated they did not have full access to existing government 

geospatial data, even though the MISTI contract stated that “USAID shall assist the Contractor [MSI] in 

obtaining access to data from the military and USAID-supported stabilization programs.”41 MSI officials said 

they had understood that this would include geospatial information from the Department of Defense’s (DOD) 

databases and from USAID’s Afghan Info database, which had information on village locations and naming 

conventions.42 However, MSI did not gain access to DOD’s geographic data. A USAID COR made at least one 

request to DOD for the data. We did not find any evidence that DOD responded to that request or that USAID 

followed up. USAID officials did not provide an explanation for not obtaining the data from DOD. Moreover, 

although MSI had access to Afghan Info, its officials told us they had concerns about its accuracy. 

While MSI and USAID MISTI personnel identified data-sharing issues, they did not bring these issues to the 

attention of USAID offices responsible for geospatial data. USAID officials could not provide us an explanation 

for this. Specifically, USAID officials responsible for MISTI stated they did not consult with USAID/OAPA’s 

Mapping Unit or the agency’s GeoCenter regarding the use of existing geospatial data or the lack of information 

sharing, and therefore, did not make use of all of its own geospatial resources.43 

USAID Has Not Developed Policies or Standards for Collecting and Reporting 

Geospatial Data 

Since 1990, the U.S. government has required agencies—including USAID—to adopt agency standards for 

collecting and reporting geospatial data.44 According to the FGDC, “Standards might include requirements 

and/or recommendations for products, systems, processes or services. They might describe a measurement or 

 

40 We could not quantify a dollar amount for the time MSI lost implementing MISTI. 

41 USAID-306-TO-12-00004, MISTI Task Order with Management Systems International, March 14, 2012. 

42 USAID officials stated it is common for locations in Afghanistan to have several names. For example, a single town may 

have different names in Dari or Pashto; it could be annotated according to boundary levels like village, district, region, or 

province. A GPS coordinate or corresponding latitudinal and longitudinal data would denote the name and location. 

43 According to USAID officials, the Mapping Unit has one employee who receives on-site direction from USAID/OAPA’s 

Strategy, Monitoring, and Evaluation Division. The employee’s duties include providing geospatial services to the client in 

support of program management, analysis, planning, and advocacy as a member of the Geographic Information Unit; 

communicating the geographic operating picture of the client’s international development work through the acquisition, 

production, and provision of geospatial data and cartography; and collaborating closely with the client and with peers in U.S 

agencies, academia, NGOs, and the private sector to achieve the client’s mandate. Additionally, a USAID summary of the 

GeoCenter states that the center’s goal “is to improve the effectiveness and efficiency of USAID’s development programs 

by geographically prioritizing resources where they will maximize development impact. The GeoCenter team works directly 

with field missions and Washington-based bureaus to integrate a geographic approach into the strategic planning, design, 

monitoring, and evaluation of USAID’s development programs.” The center, now part of the Learning Lab, launched on 

November 10, 2011, as part of USAID’s Office of Science and Technology. It provides services such as mapping and 

analysis, geospatial training, coordination among USAID geographic information system specialists, and external 

engagements. See USAID/U.S. Global Development Lab GeoCenter, The Geographic Approach to Development, undated. 

44 OMB Circular A-16 was originally issued in 1953 and revised in 1967, 1990, and 2002. The 1990 revision expanded the 

circular to include related spatial activities, in addition to surveying and mapping; specifically, it included computerreadable (digital) data. The 2002 update includes continued improvements in spatial coordination, the use of geographic 

data, and clearly defines agency and Federal Geographic Data Committee responsibilities. Supplemental guidance, 

published on November 10, 2010, clarifies and further defines the guidance for more coordinated and effective federal 

geospatial asset management. 
SIGAR 17-10-AR/USAID’s MISTI Program Page 13

test method or establish common [geospatial-data-related] terminology so that there is no misunderstanding 

among users.”45

OMB Circular A-16 and the related supplemental guidance also state that federal agencies that produce, 

maintain, or use geospatial data, either directly or indirectly in the fulfillment of their mission will: 

 Prepare, maintain, publish, and implement a strategy for advancing geographic information and 

related spatial data activities appropriate to their mission. 

 Collect, maintain, disseminate, and preserve spatial information such that the resulting data, 

information, or products can be readily shared with other federal agencies and non-federal uses. 

 Allocate agency resources to fulfill the responsibilities of effective spatial data collection, production, 

and stewardship. 

 Appoint a contact to coordinate with lead agencies for collection, acquisition, maintenance, or 

dissemination of the spatial data themes used by their organization.46

However, USAID has not created an agency-level policy addressing OMB Circular A-16 requirements or how the 

agency intends to implement them. USAID officials confirmed that it is normal practice for federal agencies to 

create and implement agency policy related to OMB Circulars. USAID officials also told us they do not have any 

agency or mission-level—for example, USAID/Afghanistan—policies to govern or guide the collection, 

maintenance, use, or sharing of geospatial data. Officials from USAID’s Bureau for Policy, Planning and 

Learning, and the GeoCenter confirmed that OMB Circular A-16 applies to USAID and that the agency is a 

member of the FGDC. They added that although USAID does have an agency policy for sharing data, it does not 

specifically relate to geospatial data.47 

When we asked USAID officials why the agency has not followed OMB’s requirements or FGDC guidance, they 

could not provide an explanation. One senior official stated that the agency’s missions and contractors—which 

implement most of its programs—are not required to follow any specific standards and use their own methods. 

To address this problem, in January 2016, officials in USAID’s GeoCenter provided draft language to update 

ADS 579, USAID Development Data, to include a requirement that contracts include specific standards for the 

use of geospatial data. However, the agency has yet to make a decision to incorporate the revision into its 

regulations. 

The challenges identified with using geospatial data as part of the MISTI program are not new to USAID or

limited to Afghanistan. Over the past decade of USAID reconstruction efforts in Afghanistan, the agency has 

repeatedly identified a problem in the agency’s ability to use accurate geospatial data and standardize a 

practice and process for collecting, managing, and reporting that data. USAID and its contractors have 

highlighted the need for geospatial precision and the lack of geospatial standards, tools, and baseline data. 

For example, in a 2006 USAID report, MSI recommended that “monitoring information and databases must 

code activities according to [geo]spatial coordinates and be able to report and present information visually and 

spatially.”48 

We also have reported on problems with USAID’s geospatial data and the agency’s inability to adequately use 

geographic data to show where its projects and activities are located.49 For example, in January 2016, we 

 

45 Federal Geographic Data Committee, Presentation on Standards Benefits, https://www.fgdc.gov/standards, accessed 

February 26, 2016. 

46 OMB Circular A-16, “Coordination of Geographic Information and Related Spatial Data Activities,” revised August 19, 

2002; and Executive Office of the President, OMB, Memorandum to Heads of Executive Departments and Agencies, 

Subject: Issuance of OMB Circular A-16 Supplemental Guidance, M-11-03, November 10, 2010. 

47 USAID, ADS Chapter 579, USAID Development Data, partially revised March 13, 2015. 

48 USAID, Monitoring and Evaluation in Post-conflict Settings, PN-ADG-193, March 15, 2006. 

49 SIGAR, USAID-Supported Health Facilities in Kabul, SIGAR-16-09-SP, January 5, 2016; SIGAR, USAID-Supported Health 

Facilities in Herat, SIGAR-16-01-SP, October 20, 2015; and SIGAR, Geospatial Coordinates for PCH Health Facilities, SIGAR-
SIGAR 17-10-AR/USAID’s MISTI Program Page 14

reported on substantial inaccuracies in the geospatial coordinates USAID provided for many of the health 

facilities it funded in Afghanistan.50 Moreover, a senior USAID official responsible for studying USAID’s 

geospatial data practices explained that the agency’s challenges with collecting and maintaining accurate 

geospatial data affect missions around the world. 

Without an established policy and standards for how USAID and its implementers should collect, maintain, use, 

and share geospatial data, USAID will continue to face problems with inaccuracies and inconsistencies in its 

geospatial data. As a result, the agency will continue to have only a limited understanding of the locations and 

limited visual verification of its activities being conducted in Afghanistan. 

CONCLUSION 

Overall, USAID’s $19.3 million MISTI program met its overall objective to monitor and evaluate approximately 

$762 million in USAID stabilization programs. MSI produced all of the required contract deliverables, and 

USAID generally provided the necessary contract oversight. As a result of the program, USAID received a wealth 

of information on the performance and progress of eight of its programs intended to bring stability to 

Afghanistan. However, USAID decided around the time MISTI began that it would discontinue its stabilization 

mission in Afghanistan—on which USAID has spent more than $2.3 billion—a decision that was made before 

the results of MISTI were known. Therefore, the utility of the information provided by MSI was limited by 

USAID’s decision not to continue its stabilization programs in Afghanistan. Nevertheless, we would expect that 

USAID, as it has mentioned, would use the program’s results to inform its decisions and other programming in 

Afghanistan and other countries. 

USAID has not addressed a longstanding need for accurate, standardized geospatial data. MSI’s reporting that 

the agency’s stabilization programs in Afghanistan had poor, inaccurate data underscores a significant 

problem that SIGAR continues to highlight. Geospatial data plays a crucial role in USAID’s M&E practices by

helping identify USAID program sites and visually verifying whether activities are being conducted. Although 

OMB guidance requires USAID to develop standards for collecting, sharing, and reporting geospatial data, the 

agency has not adhered to it. As a result, USAID and its implementers continue to reinvent standard operating 

procedures for handling geospatial data on a case-by-case basis and, in turn, add unnecessary burdens to the 

implementation of programs and contracts. Without an established policy and standards that explain how to 

handle and use geospatial data, USAID will continue to operate with inaccurate, problematic geospatial data, 

not knowing where its program activities are being conducted. This, in turn, will continue to limit the agency’s 

ability to provide effective oversight and to mitigate potential fraud, waste, and abuse in connection with its 

programs in Afghanistan. 

RECOMMENDATION 

To enhance USAID/Afghanistan’s ability to monitor, evaluate, and oversee its development efforts, SIGAR 

recommends that the USAID Administrator, as soon as possible: 

1. Establish an agency policy implementing OMB Circular A-16 requirements that institute agency-wide 

standards for collecting, using, and sharing geospatial data both within USAID and with other U.S. 

government agencies and nongovernmental entities, including those conducting reconstruction efforts 

in Afghanistan. 

 

15-67-SP, June 25, 2015. These letters highlighted our concerns about the inaccurate geospatial coordinates for 

numerous health-care facilities funded by USAID and requested that the agency to correct the information. 

50 SIGAR, USAID-Supported Health Facilities in Kabul, SIGAR-16-09-SP, January 5, 2016. 
SIGAR 17-10-AR/USAID’s MISTI Program Page 15

AGENCY COMMENTS 

We provided a draft of this report to USAID for review and comment. USAID/OAPA provided written comments, 

which are reproduced in appendix II. Additionally, USAID/OAPA provided technical comments, which we 

incorporated into this report, as appropriate. 

USAID concurred in principle with our recommendation and noted several actions either taken, underway, or 

planned that will address the weaknesses we identified in its approaches to geospatial data and policy in 

accordance with OMB Circular A-16. For example, USAID/Afghanistan updated its guidance for mission project 

managers and implementing partners on collecting geospatial data. According to USAID, this guidance is 

intended to standardize collection and use of the data, and is being circulated among USAID staff and 

implementing partners. In addition, USAID plans to strengthen its two policies that cover the use of geospatial 

data by including mandatory references that further define how the agency collects, uses, and shares 

geospatial data. USAID’s target date for completing these actions is March 31, 2017. We commend USAID’s 

immediate attention to this issue. The recommendation will remain open until we receive evidence that the 

actions are complete. 
SIGAR 17-10-AR/USAID’s MISTI Program Page 16

APPENDIX I - SCOPE AND METHODOLOGY 

This audit examines the implementation of the U.S. Agency for International Development’s (USAID) Measuring 

Impacts of Stabilization Initiatives (MISTI) contract from March 2012 to October 2015. The objectives of this 

audit were to: (1) assess the extent to which Management Systems International, Inc. (MSI) met its contract 

requirements and USAID provided oversight; (2) assess the extent to which the MISTI program met its 

objectives; and (3) identify challenges MSI faced in conducting third-party monitoring under MISTI and the 

extent to which USAID has addressed those challenges. 

To assess the extent to which MSI met its contract requirements, and USAID provided oversight, we analyzed 

MISTI’s contract requirements and deliverables, and contract modifications, specifically those impacting the 

contract’s cost and schedule. We reviewed relevant sections and clauses of the Federal Acquisition Regulation 

and USAID’s Acquisition Regulation. We interviewed the USAID contracting officer responsible for the MISTI 

program during our audit, as well as prior and current contracting officer’s representatives (COR) responsible 

for monitoring MSI’s performance. We obtained records from the MISTI COR files to help us understand the 

extent to which USAID and MSI documented and reported on specific oversight and management activities, 

such as contract deliverables, annual work plans, and periodic reporting and briefings of contract and program 

status.51 

To assess the extent to which MISTI met its first program objective—to provide independent evaluations and 

impact assessments of USAID stabilization programs and to develop recommendations for improving them—we 

reviewed MSI reports and evaluations that focused primarily on the performance and implementation of 

USAID’s stabilization programs. We analyzed recommendations MSI made in six MISTI mid-term evaluations on 

how to improve USAID’s stabilization programs. Next, we sent a table to USAID listing the recommendations we 

identified and asked officials to verify our information, and provide information on the extent to which USAID 

considered each recommendation. We then performed qualitative and quantitative analysis on the data and 

USAID’s responses to determine if and how the agency considered and applied the recommendations to 

ongoing stabilization programs. 

To assess the extent to which MISTI met its second objective—to collect, synthesize, and analyze data at the 

district, provincial, and regional levels to track higher-order stabilization trends and inform U.S. and Afghan 

government policy and practice related to transition—we reviewed MSI’s evaluation reports and semi-annual 

surveys to assess the contractor’s efforts to collect data, quantify trends, and draw conclusions about USAID 

stabilization programming. 

To assess the extent to which MISTI met its third and final objective—to contribute to the larger body of 

knowledge on best practices and lessons learned related to the design, implementation, and assessment of 

stabilization activities within a counterinsurgency context—we reviewed MISTI contract records and USAID 

Learning Lab best practices for conducting monitoring and evaluation (M&E). We also interviewed USAID 

officials and MSI staff. 

To identify any challenges MSI faced in conducting third-party monitoring under MISTI and the extent to which 

USAID has addressed those challenges, we first defined “challenges” as they relate to the MISTI program and 

third-party monitoring in Afghanistan. For the purpose of this audit, we defined “challenges” as impediments to

the successful, timely, and accurate fulfillment of MISTI objectives. We reviewed MISTI contract records and 

interviewed USAID and MSI officials to identify potential challenges. We also interviewed a nongovernmental 

organization, the Global Public Policy Institute, to learn about its independent research on third-party 

monitoring in high threat environments, including Afghanistan. To narrow the list of potential challenges, we 

established three general criteria the challenges had to meet for inclusion in our analysis: 

 

51 In a separate financial audit of the MISTI contract, we assessed MSI’s cost management. (See SIGAR, USAID’s 

Measuring Impact of Stabilization Initiative: Audit of Costs Incurred by Management Systems International, SIGAR 15-53-

FA, April 21, 2015.) 
SIGAR 17-10-AR/USAID’s MISTI Program Page 17

1. The challenge must be directly related to and within the scope of the MISTI program: Given our audit 

objective, we decided to consider only challenges identified within the scope of and directly relevant to 

the MISTI program. We excluded challenges that we determined to be outside the scope of the MISTI 

program. This means those challenges focused primarily on USAID’s stabilization program 

implementing partners, and not on MISTI implementation, or on broad systemic issues within USAID 

were not included. 

2. The challenge must be directly related to third-party monitoring in Afghanistan: We only considered 

challenges that related to performing third-party monitoring in Afghanistan and excluded those related 

to M&E practices as a whole. 

3. The challenge must be one for which there can be realistic, actionable remedies: We included 

challenges for which USAID can implement corrective action to address root causes or fully address 

the issue, but excluded challenges that neither the contractor, subcontractors, nor USAID could readily 

remedy. For example, security and the hostile, violent environment in Afghanistan are common 

challenges all U.S. reconstruction efforts face. While USAID programs may contribute to stabilization or 

even indirectly reduce violence, it is not reasonable to expect USAID to resolve this challenge itself. 

Using this criteria, we identified two significant challenges that MSI faced conducting third-party monitoring 

under MISTI: (1) lack of accurate geospatial data and (2) lack of sharing existing geospatial data. We then 

identified and reviewed U.S. government requirements for geospatial data collection, processing, standards, 

maintenance, and information sharing, including the Office of Management and Budget’s (OMB) Circular A-16 

and supplemental guidance; Executive Orders 12906 and 12333; the Federal Geographic Data Committee 

website; USAID Automated Directives System (ADS) Chapters 200, 201, 202, 203, and 579; and 

USAID/Afghanistan Mission Orders 201.03, 203.01, and 203.2.52

For all of our objectives, we interviewed current and former USAID officials responsible for overseeing the MISTI 

contract, including staff from USAID’s Office of Afghanistan and Pakistan Affairs and the USAID Mission for 

Afghanistan’s (USAID/Afghanistan) the Office of Program and Project Development, the former 

USAID/Afghanistan Stabilization Unit, and the Office of Acquisition and Assistance. We also interviewed MSI 

officials responsible for implementing the MISTI contract. 

We did not rely on computer-processed data for the purpose of the audit objectives. We assessed internal 

controls to determine the extent to which the USAID had systems in place to track and report on its efforts 

supporting the implementation and oversight of the MISTI contract. The results of our assessment are included 

in the body of the report. 

We conducted our audit work in Washington, D.C., from November 2014 to October 2016, in accordance with 

generally accepted government auditing standards. Those standards require that we plan and perform the 

audit to obtain sufficient, appropriate evidence to provide a reasonable basis for our findings and conclusions 

based on our audit objectives. We believe that the evidence obtained provides a reasonable basis for our 

findings and conclusions based on our audit objectives. This audit was performed by SIGAR under the authority 

of Public Law No. 110-181, as amended, and the Inspector General Act of 1978, as amended. 

 

52 OMB Circular A-16, “Coordination of Geographic Information and related Spatial Data Activities,” revised August 19, 

2002; Executive Office of the President, OMB, Memorandum to Heads of Executive Departments and Agencies, Subject: 

Issuance of OMB Circular A-16 Supplemental Guidance, M-11-03, November 10, 2010; Executive Order 12906, 

Coordinating Geographic Data Acquisition and Access: The National Spatial Data Infrastructure, April 11, 1994; Executive 

Order 12333, United States Intelligence Activities, December 4, 1981; USAID, ADS Chapter 200, Introduction to 

Programming, partially revised July 18, 2014; USAID, ADS Chapter 201, Planning, partially revised July 22, 2013; USAID, 

ADS Chapter 202, Achieving, January, 25, 2012; USAID, ADS Chapter 203, Assessing and Learning, partial revised 

November 2, 2012; USAID, ADS Chapter 579, USAID Development Data, partially revised March 13, 2015; USAID Mission 

for Afghanistan Mission Order 201.3, Terrorist Financing Risk Assessment, May 27, 2009; USAID Mission for Afghanistan 

Mission Order 203.1, Review of Project Performance Management Plans (PMP), July 30, 2009; and USAID Mission for 

Afghanistan Mission Order 203.2, Mission Order on Monitoring and Evaluation, October 24, 2012. 
SIGAR 17-10-AR/USAID’s MISTI Program Page 18

APPENDIX II - COMMENTS FROM THE U.S. AGENCY FOR INTERNATIONAL 

DEVELOPMENT 
SIGAR 17-10-AR/USAID’s MISTI Program Page 19
SIGAR 17-10-AR/USAID’s MISTI Program Page 20
SIGAR 17-10-AR/USAID’s MISTI Program Page 21
SIGAR 17-10-AR/USAID’s MISTI Program Page 22
SIGAR 17-10-AR/USAID’s MISTI Program Page 23

SIGAR’s Response to Comments from USAID

1. The summary statement in our report is based on MSI’s findings presented in MISTI’s fifth impact 

evaluation survey of USAID stabilization programs, which stated that throughout all five surveys, “the 

overall stability trend across the 55 districts surveyed in all five waves was largely flat.”

53 MSI did 

report some positive impacts of USAID’s stabilization programs, and we updated the report to note 

that. In addition, we removed the statement suggesting that there was a correlation between the 

effectiveness of stabilization programs and gender or literacy. 

 

 

53 MSI, MISTI Stabilization Trends and Impact Evaluation Survey, Analytical Report, Wave 5 (with Addendum), November 

25, 2015. 
SIGAR 17-10-AR/USAID’s MISTI Program Page 24

APPENDIX III - ACKNOWLEDGMENTS 

Eugene Gray, Senior Program Manager 

Clifton Spruill, Analyst-in-Charge 

Carlos Torres, Senior Program Analyst 

Daniel Vergamini, Senior Program Analyst 
SIGAR 17-10-AR/USAID’s MISTI Program Page 25

This performance audit was conducted 

under project code SIGAR-103A.
Obtaining Copies of SIGAR 

Reports and Testimonies

To Report Fraud, Waste, and 

Abuse in Afghanistan 

Reconstruction Programs

Public Affairs

SIGAR’s Mission

The mission of the Special Inspector General for Afghanistan 

Reconstruction (SIGAR) is to enhance oversight of programs for the 

reconstruction of Afghanistan by conducting independent and 

objective audits, inspections, and investigations on the use of 

taxpayer dollars and related funds. SIGAR works to provide accurate 

and balanced information, evaluations, analysis, and 

recommendations to help the U.S. Congress, U.S. agencies, and 

other decision-makers to make informed oversight, policy, and 

funding decisions to: 

 improve effectiveness of the overall reconstruction 

strategy and its component programs; 

 improve management and accountability over funds 

administered by U.S. and Afghan agencies and their 

contractors; 

 improve contracting and contract management 

processes; 

 prevent fraud, waste, and abuse; and 

 advance U.S. interests in reconstructing Afghanistan. 

To obtain copies of SIGAR documents at no cost, go to SIGAR’s Web 

site (www.sigar.mil). SIGAR posts all publicly released reports, 

testimonies, and correspondence on its Web site. 

To help prevent fraud, waste, and abuse by reporting allegations of 

fraud, waste, abuse, mismanagement, and reprisal, contact SIGAR’s 

hotline: 

 Web: www.sigar.mil/fraud 

 Email: sigar.pentagon.inv.mbx.hotline@mail.mil 

 Phone Afghanistan: +93 (0) 700-10-7300 

 Phone DSN Afghanistan: 318-237-3912 ext. 7303 

 Phone International: +1-866-329-8893 

 Phone DSN International: 312-664-0378 

 U.S. fax: +1-703-601-4065 

Public Affairs Officer 

 Phone: 703-545-5974

 Email: sigar.pentagon.ccr.mbx.public-affairs@mail.mil 

 Mail: SIGAR Public Affairs 

2530 Crystal Drive 

Arlington, VA 22202