Foreign Assistance: Agencies Can Improve the Quality and Dissemination of Program Evaluations
Published: 2017-03-03T00:00:00

Fast Facts:
The U.S. government plans to spend about $35 billion on foreign assistance programs in 2017, and program evaluations can help assess and improve the results of this spending.
However, our review of the six agencies providing the most on foreign aid (shown below) found that about a quarter of their program evaluations in 2015 lacked adequate information on results to inform future programs.
We recommended that each agency develop a plan to improve the quality of its evaluations. We also recommended that some agencies improve their procedures to disseminate their evaluation reports—which may help future programs benefit from lessons learned.

Highlights:
section_title: Why GAO Did This Study
paragraphs: The U.S. government plans to spend approximately $35 billion on foreign assistance in 2017. Evaluation is an essential tool for U.S. agencies to assess and improve the results of their programs. Government-wide guidance emphasizes the importance of evaluation, and the Foreign Aid Transparency and Accountability Act of 2016 requires the President to establish guidelines for conducting evaluations. However, evaluations can be challenging to conduct. GAO has previously reported on challenges in the design, implementation, and dissemination of the evaluations of individual foreign assistance programs.
GAO was asked to review foreign aid evaluations across multiple agencies. This report examines the (1) quality, (2) cost, and (3) dissemination of foreign aid program evaluations. GAO assessed a representative sample of 173 fiscal year 2015 evaluations for programs at the six agencies providing the largest amounts of U.S. foreign aid —USAID, State, MCC, HHS's Centers for Disease Control and Prevention under the President's Emergency Plan for AIDS Relief, USDA's Foreign Agricultural Service, and DOD's Global Train and Equip program—against leading evaluation quality criteria; analyzed cost and contract documents; and reviewed agency websites and dissemination procedures.
section_title: What GAO Found
paragraphs: An estimated 73 percent of evaluations completed in fiscal year 2015 by the six U.S. agencies GAO reviewed generally or partially addressed all of the quality criteria GAO identified for evaluation design, implementation, and conclusions (see fig.). Agencies met some elements of the criteria more often than others. For example, approximately 90 percent of all evaluations addressed questions that are generally aligned with program goals and were thus able to provide useful information about program results. About 40 percent of evaluations did not use generally appropriate sampling, data collection, or analysis methods. Although implementing evaluations overseas poses significant methodological challenges, GAO identified opportunities for each agency to improve evaluation quality and thereby strengthen its ability to manage aid funds more effectively based on results.
Evaluation costs ranged widely and were sometimes difficult to determine, but the majority of evaluations GAO examined cost less than $200,000. Millennium Challenge Corporation (MCC) evaluations had a median cost of about $269,000, while median costs for the U.S. Agency for International Development (USAID), the U.S. Department of Agriculture (USDA), and the Department of State (State) ranged from about $88,000 to about $178,000. GAO was unable to identify the specific costs for the Department of Defense (DOD) and Department of Health and Human Services (HHS) evaluations. High-quality evaluations tend to be more costly, but some well-designed lower-cost evaluations also met all quality criteria. Other factors related to evaluation costs include the evaluation's choice of methodology, its duration, and its location.
Agencies generally posted and distributed evaluations for the use of internal and external stakeholders. However, shortfalls in some agency efforts may limit the evaluations' usefulness.
Public posting . USDA has not developed procedures for reviewing and preparing its evaluations for public posting, but the other agencies posted nonsensitive reports on a public website.
Timeliness . Some HHS reports and more than half of MCC reports were posted a year or more after completion.
Dissemination planning. State does not currently have a policy requiring a plan that identifies potential users and the means of dissemination.
section_title: What GAO Recommends
paragraphs: GAO recommends that each of the six agencies develop a plan to improve the quality of its evaluations and that HHS, MCC, State, and USDA improve their procedures and planning for disseminating evaluation reports.
The agencies concurred with our recommendations.

section_title: Letter
paragraphs: The U.S. government plans to spend approximately $35 billion on foreign  assistance in 2017 to improve the lives and health of millions living in  poverty, support democracy, enhance global security, and achieve other  U.S. foreign policy goals. For U.S. agencies that provide foreign  assistance, evaluations are essential to assess and help improve  program results. Preparing and disseminating high-quality evaluations  helps agencies and their implementing partners assess their program  results, adjust program designs, and make evidence-based decisions  about the use of their resources. Both the 2010 GPRA Modernization Act  and the 2010 Presidential Policy Directive on Global Development Policy  called for an increased focus on evaluations of agency programs. In  addition, in July 2016, the Foreign Aid Transparency and Accountability  Act of 2016 required the President to set forth guidelines for the  establishment of measurable goals, performance metrics, and monitoring  and evaluation (M&E) plans for U.S. foreign assistance within 18 months  of its enactment. In recent years, foreign assistance agencies have  adopted or updated their guidance on evaluations. However, prior GAO  work has identified challenges in the design, implementation, and  dissemination of evaluations of individual foreign assistance programs.
We were asked to review U.S. agencies’ evaluation of foreign assistance  programs. Focusing on evaluations completed in fiscal year 2015 by the  six agencies that administer the largest amounts of U.S. foreign  assistance—the Department of Defense (DOD), the Department of Health  and Human Services (HHS), the Millennium Challenge Corporation  (MCC), the Department of State (State), the U.S. Department of  Agriculture (USDA), and the U.S. Agency for International  Development  (USAID)—this report examines (1) the extent to which foreign assistance  program evaluations met key evaluation quality criteria, (2) the costs of  the agencies’ evaluations and factors that affect these costs, and (3) the  extent to which the agencies ensure the dissemination of evaluation  reports within the agency and to the public.
To identify the six agencies that administer the largest amounts of foreign  assistance, we reviewed obligations data that the agencies reported to  USAID’s U.S. Overseas Loans and Grants database for fiscal years 2008  through 2012. To identify evaluations completed in fiscal year 2015, we  requested that each agency provide a list of all foreign aid evaluation  reports completed in that year. We did not separately review agency files  to identify if agencies had additional evaluations beyond those listed in  the registries. We performed an initial review of the evaluation lists and  documents provided by agencies and excluded some documents from  our review because they were incomplete, were not evaluation reports, or  were not completed in fiscal year 2015.
To address our first objective, we reviewed all State, DOD, and MCC  evaluation reports completed in fiscal year 2015. We reviewed  representative samples of USAID, USDA, and HHS evaluation reports to  create estimates about the population of all evaluation reports at the  sampled agencies. We reviewed the selected evaluations against eight  criteria for high-quality evaluations related to the appropriateness of  design, data collection methods, and analysis and the extent of support  for conclusions and any recommendations. We developed these criteria  on the basis of a review of federal, international, and evaluation organization guidance and our prior reports (See app. II  for the criteria  and data by agency for our review). We assessed each agency’s  evaluations as “generally,” “partially,” or “not at all” meeting each criterion;  we also rated some evaluations as providing insufficient information to  make an assessment. In addition to assessing evaluation quality, we also  collected information about the characteristics of each evaluation, such as  the location of the study and its methodology.
To address our second objective, we reviewed contract documents,  invoices, and related documents to determine the cumulative cost of final  evaluations conducted by an outside evaluator. We defined the  cumulative costs as the cost of conducting the final evaluation and any  related activities that informed the final evaluation,  such as a midterm or  baseline evaluation. In some cases, we were unable to determine an  evaluation’s precise cost if it was procured under a contract that covered  additional activities. In these cases, we approximated the cost on the  basis of estimates provided by the agency or contractor. We did not  determine the cost of evaluations prepared by agency staff because  agencies did not separately track these costs. To identify factors that  affect the costs of foreign aid evaluations, we analyzed the cost of MCC,  State, USDA, and USAID evaluations in relation to the data we collected  on these evaluations’ quality and other characteristics. We report only  limited data on the cost of DOD’s GT&E and HHS’s PEPFAR evaluations  because the evaluation contracts or implementing partner agreements did  not separately track evaluation costs, and we concluded that the available  estimates were too limited to include in our statistical analysis.
To address our third objective, we identified leading practices for the  dissemination of evaluation findings. We identified these leading practices  using federal guidance that encourages the timely public posting of  agency information on a searchable website, as well as plans and  additional efforts to actively disseminate agency information. In addition to  the federal guidance, we also used the American Evaluation Association’s  (AEA) An Evaluation Roadmap for a More Effective Government (AEA  Roadmap), as well as some other nonfederal sources that also cite  timely public posting, dissemination planning, and additional active efforts  to disseminate results as important communication tools for evaluations.  We then reviewed each agency’s evaluation policies to identify their requirements for dissemination of evaluation reports and interviewed  cognizant officials. We compared agency policies and practices with the  leading practices we identified. We reviewed agency websites to  determine whether evaluation reports were posted online and examined  each agency website to determine whether it provided a search engine  that could be used to locate evaluations.
See appendix I for a more detailed discussion of our scope and  methodology.
We conducted this performance audit from October 2015 to March 2017  in accordance with generally accepted government auditing standards.  Those standards require that we plan and perform the audit to obtain  sufficient, appropriate evidence to provide a reasonable basis for our  findings and conclusions based on our audit objectives. We believe that  the evidence obtained provides a reasonable basis for our findings and  conclusions based on our audit objectives.
subsections: section_title: Background
paragraphs: The six agencies whose evaluations we reviewed focus on foreign  assistance to varying degrees. DOD, HHS, and USDA provide foreign  assistance as part of their larger portfolios of programs, while MCC,  State, and USAID focus exclusively on foreign affairs or foreign  assistance.
DOD’s GT&E program provides training, equipment, and small-scale  military construction activities to partner nations to build their capacity  and enable them to conduct counterterrorism operations or to support  ongoing allied or coalition military or stability operations that benefit  the national security interests of the United States.
HHS’s CDC implements a portion of the President’s Emergency Plan  for AIDS Relief (PEPFAR) programs under the direction of State’s  Office of the U.S. Global AIDS Coordinator and Health Diplomacy.
MCC, a U.S. government corporation, provides aid to developing  countries that have demonstrated a commitment to ruling justly,  encouraging economic freedom, and investing in people. MCC  supplies this assistance to eligible countries primarily through 5-year  compacts with the goal of reducing poverty by stimulating economic  growth.
State, the lead U.S. foreign affairs agency, implements programs that  provide, for example, counternarcotics assistance; refugee  assistance; and support for democracy, governance, and human  rights.
USAID, the lead U.S. foreign assistance agency, implements  programs intended to both further America’s interests and improve  lives in the developing world. USAID’s broad portfolio includes  programs that address democracy and human rights, water and  sanitation, food security, education, poverty, the environment, global  health, and other areas.
USDA’s Foreign Agricultural Service (FAS) administers two  nonemergency food aid programs: (1) The Food for Progress program  supports agricultural value chain development, expanding revenue  and production capacity, and increasing incomes in food-insecure  countries; (2) The McGovern-Dole International  Food for Education  and Child Nutrition program supports education and nutrition for  schoolchildren, particularly girls, expectant mothers, and infants.
subsections: section_title: Agency Evaluation Guidance
paragraphs: Each of the six agencies has adopted evaluation guidance for the  programs included in our review.
DOD’s November 2012 Section 1206 Assessment Handbook serves  as a guide to evaluation planners and implementers for conducting  evaluations of DOD’s GT&E programs as required by federal law. The  fiscal year 2012 National Defense Authorization Act (NDAA) required  DOD, no later than 90 days after the end of each fiscal year, to submit  to Congress a report including an assessment of the effectiveness of  GT&E programs conducted that fiscal year in building the capacity of  the recipient foreign country. The fiscal year 2015 NDAA maintained  this requirement through 2020. DOD did not have agency-wide  evaluation guidance for security cooperation at the time we performed  our review but issued such guidance in January 2017.
For PEPFAR programs, including those implemented by HHS, State’s  Office of the U.S. Global AIDS Coordinator and Health Diplomacy  issued the PEPFAR Evaluation Standards of Practice in January 2014  and an updated version (version 2) in September 2015.
MCC’s May 2012 Policy for Monitoring and Evaluation of Compacts  and Threshold Programs requires that compact M&E plans identify  and describe the evaluations that will be conducted, key evaluation  questions and methodologies, and data collection strategies.
State issued its current evaluation policy and an additional guidance  document for evaluations in January 2015 and issued a revised and  updated version of the guidance in January 2016.
USAID lays out its evaluation policies in its Automated Directives  System (ADS). USAID issued a fully revised ADS 201, addressing  evaluation guidance, planning, and implementation, in September  2016.
USDA’s FAS evaluations are guided by its May 2013 Monitoring and  Evaluation Policy, which requires both interim and final program  evaluations.
section_title: Agency Evaluation Procurement and Cost Tracking
paragraphs: With the exception of HHS, the agencies we selected for our review  generally rely on outside contractors to conduct evaluations. DOD, MCC,  State, and USAID directly contract for third-party evaluation services. The  HHS PEPFAR evaluations we reviewed were prepared (1) by CDC staff  using existing program data; (2) by an implementing partner as part of the  partner’s cooperative agreement; or (3) in one instance, under a separate  agreement. USDA implementing partners procured the USDA evaluations  whose costs we reviewed.
The six agencies track evaluation costs to varying extents. DOD, MCC,  and State procured the evaluations we reviewed through centrally  managed contracts, and cost information for these evaluations was  available through the program or agency evaluation office. HHS’s  PEPFAR, USDA, and USAID evaluations were often procured and  managed at the country, mission, or implementing partner level. Cost  information was not centrally available and could be obtained only from  each mission or implementing partner.
section_title: Evaluation Types, Timing, and Methods
paragraphs: Foreign assistance evaluations may vary in type, timing, and method.  Two common types of evaluation are the following:
Performance evaluations assess the extent to which a program is operating as was intended or the extent to which it achieves its outcome-oriented objectives. Performance evaluations often judge program effectiveness against criteria, such as progress against baselines, whether program goals were met, or whether expected targets were met.
Net impact evaluations assess the net effect of a program by  comparing program outcomes with an estimate of what would have happened in the program’s absence. Net impact evaluations use a  variety of experimental and quasi-experimental designs, including randomized methods in which participants are assigned to separate  control or treatment groups to isolate the program’s effect. Net impact evaluations have more complex methodologies than the other evaluation types.
Agencies may conduct evaluations during or after the completion of a  program. Interim or midterm evaluations are conducted while a program  is in progress, and final evaluations are conducted after the program  ends. Baseline evaluations are also sometimes conducted before a  program begins as a basis for determining any effects of the program.
Evaluations may use one or more methods to produce their results. For  example, evaluations may use random or nonrandom sampling from the  target population to select cases for inclusion in the study. Evaluations  may also use one or more methods to collect data on the chosen  indicators and measures—for example, structured or unstructured  interviews, focus groups, surveys, direct observations, or collection and  analysis of existing data. Each of these methods has potential benefits  and limitations that an evaluator must consider in assessing the  evaluation’s evidence as a basis for its conclusions and  recommendations.
section_title: Most Foreign Aid Evaluations Were of High or Acceptable Quality Overall, though Quality Varied by Criterion and Agency
paragraphs: Overall, about three quarters of all 2015 foreign aid evaluations from the  six agencies we reviewed generally or partially met the quality criteria we  identified. The remaining evaluations did not meet one or more of these  criteria or provided insufficient information. While we generally found that  evaluations met quality criteria related to design, implementation, and  conclusions, we more often found limitations in implementation—including  sampling methods, data collection, and analysis. In addition, we found  that the independence of evaluators was not always clearly evident. While  the quality of evaluations varied by agency, we identified shortcomings at  all six of the selected agencies that could limit evaluation reliability and  usefulness.
subsections: section_title: About Three-Quarters of Agencies’ Evaluations Showed High or Acceptable Quality Overall
paragraphs: By reviewing policies of federal agencies, international organizations,  and evaluation organizations, and our prior reporting, we identified  common characteristics of high-quality evaluations, from which we  developed eight criteria for assessing evaluation quality. These quality  criteria are associated with the (1) design, (2) implementation, and (3)  conclusions of an evaluation,  as follows. (See app. I for a full description  of how we developed our evaluation criteria.)
Evaluation questions are aligned with program goals.
Performance indicators are appropriate for measuring progress  against program goals.
Design is appropriate for answering the evaluation questions.
Target population and sampling method are appropriate, given the  scope and nature of the evaluation questions.
Data collection is appropriate for answering the evaluation questions.
Data analysis is appropriate to answer the evaluation questions.
Conclusions are supported by the available evidence.
Recommendations and lessons learned are justified by the available  evidence.
Based on an assessment of agency evaluations against these criteria, we  rated 73 percent of all evaluations as high quality (26 percent) or  acceptable quality (47 percent), because they generally or partially met all  applicable quality criteria. We rated the remaining 27 percent as lower  quality because they either did not meet or did not provide sufficient  information related to at least one applicable criterion. These evaluations  may not provide sufficiently reliable evidence to inform agency program  and budget decisions. Overall, we encountered more instances when  evaluations did not provide sufficient information about a certain criterion  than instances when evaluations did not meet a quality criterion at all.  Table 1 summarizes our observations about the quality of evaluations at  the six selected agencies in our review.
section_title: Criteria Related to Evaluation Implementation Posed Greatest Challenge for Agencies’ Evaluations
paragraphs: The quality of evaluations varied by the type of quality criterion we  applied. As figure 1 shows, many evaluations generally met the criteria  related to the appropriateness of evaluation design, implementation, and  conclusions. However, overall, evaluations generally met fewer criteria  related to implementation, reflecting limitations in the way evidence was  collected or analyzed.
A relatively high percentage of evaluations generally met each of the  criteria we used to assess the alignment of the study questions with the  program goals, the appropriateness of the evaluation design for the study  questions, and the use of indicators for measuring progress.
Alignment of questions with program goals. Evaluation questions  generally aligned with one or more of the evaluated program’s goals  in more than 90 percent of the evaluations. Thus, evaluations were  designed to provide useful information about program results.
Appropriate evaluation design for the study questions. About 80  percent of the evaluations used a design that was generally  appropriate for the study questions, and the remainder of the designs  was at least partially appropriate.
Appropriate use of indicators. Indicators for measuring progress  were generally appropriate in about 80 percent of the evaluations. As  a result, successes or failures identified by these evaluations are likely  to be directly relevant to assessing achievement of the evaluated  programs’ goals.
subsections: section_title: Implementation of Sampling, Data Collection, and Analysis More Often Had Limitations
paragraphs: We found more limitations in the implementation of evaluations than in  their design. On average, about 60 percent of evaluations generally met  each of the criteria related to this aspect of quality—sampling, data  collection, and analysis. Limitations we identified revealed that conducting  evaluations overseas can pose challenges for evaluators. For example,  travel to remote areas with safety and security concerns may limit an  evaluator’s ability to conduct appropriate sampling and collect primary  data for the study. Also, insufficient local resources to implement certain  methodologies, such as implementing survey instruments, or a lack of  local administrative data on the study population may constitute additional  obstacles to sampling and data collection.
About 40 percent of evaluations had limitations in, or provided insufficient  information about, their sampling methodology. If an evaluation does not  clearly describe how sampling was conducted, it may raise questions  about the quality of the evidence, including concerns regarding selection  bias of respondents, sufficiency of the sample size for the findings, and  the relevance of the evaluation’s findings and conclusions for the entire  study population. Sampling methods were particularly problematic or  unclear in evaluations that used nonrandom sampling. For evaluations  that relied primarily or exclusively on testimonial evidence, the method for  selecting participants for interviews or focus group discussions was  sometimes inappropriate or unclear. For example, one evaluation we  reviewed relied largely on interviews but did not describe the process  used for selecting participants, and it indicated that the available list of  potential participants was incomplete and inaccurate. Several evaluations  provided insufficient information about the target population, other than  identifying them as program beneficiaries, and included no discussion of  how participants were selected for interviews, focus groups, or surveys.
Limitations in Data Collection Methods  About 40 percent of the evaluations had limitations in, or provided  insufficient information about, their data collection methods. We  identified a number of deficiencies in the data collection process,  including a lack of documentation of data collection instruments (DCI),  such as questionnaires or structured interview protocols. In cases where  evidence was gathered through a DCI, some evaluations were unclear  about how the instrument was designed and administered. For example,  an evaluation of a program intended to increase access to mobile  technologies and improve mothers’ health used a survey to gather data.  However, the evaluation did not provide sufficient details about the  survey, such as the questionnaire itself or the sampling strategy, for the  reader to be able to determine the validity and reliability of the data  collected.
In addition, about half of the evaluations did not collect baseline data from  which to calculate change after the program was implemented, and about  half generally did not set targets, which makes an assessment of  progress toward meeting the goals of the program difficult. For example,  an evaluation of two community training programs in an Asian country  identified the study question but did not collect baseline data and did not  establish targets. Although some baseline data may have been gathered  by the implementing partner, such information was not used for  comparative purposes, making it impossible to assess the net effects of  the program.
Further, an estimated 60 percent of the evaluations used data collection  procedures that only partially ensured the reliability of the data, or there  was not sufficient information to assess data reliability. For example, an  evaluation of a small business program in Latin America acknowledged  numerous data quality problems, including serious attrition among the  group used as a comparison group to program participants. Such  limitations raise questions about the strength of the conclusions.
About 40 percent of evaluations did not demonstrate that they had  conducted appropriate data analysis. These evaluations often did not  specify the analysis methods for each question, such as how interview  responses were analyzed. For example, an evaluation of a program  serving women living with HIV  analyzed data through a content analysis  but did not clearly explain the categories created for the analysis or the  numbers of individual  responses that fell into each category. The lack of  clarity in the analysis makes it difficult for the reader to determine whether  the findings from this program have broader applicability. Several  evaluations relied on focus group discussions but analyzed and reported  the percentages of informants expressing the stated views in ways that  did not appropriately account for the potential influence of other focus  group members on informants’ responses. Evaluations that used some  quantitative data analysis also had certain shortcomings. For example, an  evaluation reported a statistically significant change from baseline but did  not include a discussion of the type of statistical test that supported this  result.
Finally, while about 90 percent of the evaluations assessed processes  such as program implementation, about half of those evaluations did not  establish any criteria, such as evaluation plans, budgets, timeframes, and  targets. Without such benchmarks, it is difficult to define what constituted  success for the evaluated program.
section_title: Evaluation Conclusions and Recommendations Were Generally Supported
paragraphs: The majority of evaluations generally met each of our criteria related to  conclusions. These evaluations considered the strengths and limitations  of the available evidence from the evaluation’s design and  implementation and included conclusions that were generally supported  and recommendations that were generally justified.
Conclusions supported by the available evidence. About 70  percent of the evaluations had conclusions that were generally  supported by the evidence, and nearly all of the evaluations had  conclusions that were partially supported. This indicates that these  evaluations did not reach beyond what was supported by the  evidence and justified given the limitations.
Recommendations and lessons learned justified by the available  evidence. About 75 percent of evaluations with recommendations  included evidence that generally supported the recommendations,  and all evaluations with recommendations included evidence that at  least partially supported them. This indicates that the collected  evidence justified the follow-up steps the evaluations recommended.
section_title: Independence of Evaluators Was Not Always Clearly Documented
paragraphs: Our analysis found that, in addition to meeting the eight criteria to varying  extents, the evaluations did not always provide documentation of the  evaluator’s independence and whether there were any potential conflicts  of interest. In instances where an evaluation was not conducted by a third  party, a statement about conflicts of interest may be especially important  to forestall any potential concerns about the evaluator’s impartiality. In all,  about 80 percent of the agency evaluations documented that they were  conducted by third-party evaluators, while about 13 percent were not  conducted by a third-party evaluator, and another 6 percent did not  indicate whether they were performed by a third-party evaluator. About 70  percent of HHS evaluations, about 30 percent of State evaluations, and  about 40 percent of USAID evaluations included a conflict-of-interest  statement, while no DOD, USDA, or MCC evaluations included such a  statement. If an evaluation does not address the independence of the  evaluation organization and of individual evaluators, questions could arise  about the objectivity and reliability of the evaluation’s findings.
section_title: Extent to Which Evaluations Met Each Applicable Quality Criterion Varied among Agencies
paragraphs: As table 2 shows, the extent to which the evaluations met each quality  criterion varied among the six agencies we reviewed. While our  assessments revealed strengths in each agency’s evaluations, all six  agencies’ evaluations also showed shortcomings in quality that could limit  the agencies’ ability to ensure the effectiveness of foreign assistance  based on evaluation results.
Each applicable quality criterion was generally met by a majority of HHS,  MCC, and USAID evaluations. However, evaluations for all three  agencies scored generally lower on the criteria related to evaluation  implementation—that is, the appropriateness of the target population and  sampling, data collection, and data analysis. While most HHS, MCC, and  USAID evaluations used generally appropriate sampling methods, our  analysis showed that overall about half of the evaluations did not use  appropriate nonrandom sampling techniques. In addition, we estimate  that overall only about half of the three agencies’ evaluations generally  used data collection methods that ensured data reliability,  and only about  10 to 20 percent of USAID and HHS evaluations generally specified the  key assumptions of the data analysis methods used.
DOD’s GT&E program evaluations’ study questions met the first quality  criterion—aligning with the program’s goals—but overall did not generally  meet the other criteria. For example, we identified weaknesses in the  implementation of the evaluations’ designs in terms of target population  and sampling, data collection, and analysis. In particular, some  evaluations did not describe the target population and did not discuss the  methods the evaluators used for their selection of the equipment items  they observed or the persons they interviewed. In addition, we found  limited discussion about how the data were summarized and analyzed,  incomplete baseline metrics, and a lack of targets. Without systematic  selection of equipment to observe or respondents to interview, it is difficult  to know whether the selections were justifiable and selected in a way that  supports the intervention’s objectives and the conclusions drawn.  Because of these implementation weaknesses as well as a lack of  discussion of study limitations, we determined that the DOD GT&E  program evaluations provided only partial support for their conclusions.
State and USDA evaluations each met more than one quality criterion  about half the time or less. About half or fewer of State evaluations  generally met four criteria: appropriate indicators, appropriate target  population and sampling, appropriate data collection, and appropriate  data analysis. Regarding the appropriateness of chosen indicators, less  than a third of State evaluations had indicators with baselines or  established criteria such as plans or budgets, and almost none of the  evaluations had indicators with targets against which progress could be  assessed. In addition, about 80 percent of State evaluations used a data  collection process that did not generally ensure the reliability  of the data,  and about half of the State evaluations generally did not specify data  analysis methods for each question and the key assumptions used in the  analysis. State officials noted that their programs are often implemented  rapidly in response to specific events, making it difficult to design an  evaluation for the program and to gather baseline data. We estimate that  overall about half of USDA evaluations had generally appropriate target  population and sampling, generally appropriate data analysis, or support  for conclusions.
section_title: Foreign Aid Evaluation Costs Range Widely and Are Influenced by Methodology, Location, and Evaluation Quality
paragraphs: Most foreign aid evaluations we reviewed cost less than $200,000, but  costs ranged widely and varied by agency and type. We identified costs  for MCC, State, USAID, and USDA final evaluations but could not obtain  specific cost information for DOD’s GT&E and HHS’s PEPFAR  evaluations because these programs used procurement methods for their  evaluations that did not separately track evaluation costs. Evaluation  costs were related to the evaluation’s methodology and location, and  higher-cost evaluations tended to meet more evaluation quality criteria,  though we also identified lower-cost evaluations that met all quality  criteria.
subsections: section_title: Evaluation Costs Ranged Widely and Varied by Type and Agency, but Most Cost Less Than $200,000
paragraphs: Costs for the majority of the foreign aid evaluations whose costs we  reviewed were less than $200,000, but the costs ranged widely and  varied by type of evaluation and agency. Of the 76 MCC, State, USAID,  and USDA evaluations, 48 cost less than $200,000 while 6 cost more  than $900,000. The costs of net impact evaluations ranged from  approximately $36,100 to $2.2 million, with a median of $117,500. The costs of performance evaluations ranged from $9,600 to $902,100, with a  median cost of $169,600.
While the median cost was higher for the performance evaluations than  the net impact evaluations, the net impact evaluations had a higher  average cost than the performance evaluations; five of the six evaluations  that cost more than $900,000 were net impact evaluations. Net impact  evaluations that used randomized controlled trials were the most  expensive evaluations in our sample, with a median cost of $926,600  compared with $154,700 for all other evaluations. Figure 2 shows the  range of costs for the net impact and performance evaluations whose  costs we reviewed.
Of the four agencies’ evaluations whose costs we reviewed, MCC’s  evaluations had the highest median cost, at $268,900, and USDA’s  evaluations had the lowest median cost, at $87,900 (see table 3). Seven  of 12 MCC evaluations cost over $200,000, including 3 net impact  evaluations that cost over $900,000. In contrast, 8 of the 10 USDA  evaluations were performance evaluations that cost less than $200,000.  Most State evaluations were performance evaluations, which were  generally more expensive than performance evaluations at the other  agencies. USAID costs for impact and performance evaluations both  ranged widely, and USAID’s net impact evaluations had a lower median  cost than its performance evaluations.
Costs for DOD’s GT&E evaluations and HHS’s PEPFAR evaluations were  not specifically identifiable  because they were not separately tracked by  the agencies, contractors, or implementing partners.
The contract for the DOD GT&E evaluations included many activities  in addition to the evaluations and was not structured to show the cost  of each activity. Additionally,  according to DOD officials, neither DOD  nor the contractor separately tracked the evaluation costs in their  financial records. However, on the basis of the contractor’s estimate  of contract time spent on 20 GT&E evaluations in fiscal years 2012  through 2015 (including the four evaluations in our sample), we  estimated the total cost of these evaluations at approximately $1.1  million—an average of approximately $56,300 per evaluation.  According to DOD officials, actual costs likely varied across  evaluations due to differences in the size of the evaluation teams, the  foreign country in which the evaluation took place, and the amount of  time each team spent abroad.
HHS’s PEPFAR programs typically conduct evaluations as part of  larger cooperative agreements that are not structured to specify  evaluation costs. We reviewed cost information for 10 HHS  evaluations. We identified a specific cost—$15,400—for only one  evaluation, which was conducted under a cooperative agreement  specifically for the evaluation; the remaining nine evaluations were  conducted as part of cooperative agreements that did not specify  evaluation costs. Using budget documents and informed estimates  from CDC staff and implementing partners, we estimated that the  costs of these nine evaluations ranged from $25,100 to $356,200.  Additionally,  11 of the 34 HHS evaluations in our sample had no  external costs because they were conducted solely by HHS staff  using existing datasets. CDC officials stated that CDC intends to track  evaluation costs in the future. For example, according to HHS, upon  continuation of a cooperative agreement, an implementing partner will  be required to report on progress on its Evaluation and Performance  Monitoring  Plan, as well as on expenditures to date and plans and  budgets for the following year.
section_title: Evaluation Methods, Period of Performance, and Location Influence Evaluation Costs
paragraphs: Our analysis found that data collection methods, frequency of data  collection, evaluation duration, and evaluation location all affect  evaluations’ cost. For example, evaluations that collected data by  surveying program beneficiaries had a median cost of $202,500— approximately $74,000 higher than the median cost of those that did  not—and evaluations that collected data repeatedly over time had a  median cost of $194,500—approximately $44,500 higher than those that  did not. Evaluations that took longer to perform also tended to be more  expensive. Other factors that might influence costs include unstable  locations and evaluations conducted at multiple sites. For example, a  performance evaluation conducted in an unstable country cost $365,700  for 78 days of work, and a performance evaluation that conducted data  collection in 12 countries cost $902,100. In addition, conducting an  evaluation in multiple sites within the same country might increase  evaluation costs. For example, a performance evaluation conducted in  eight cities and seven states in India cost $407,500, including the cost of  a midterm evaluation that was also conducted in multiple cities. These  costs greatly exceeded the median costs for all evaluations.
section_title: High-Quality Evaluations Tend to Cost More, but Some Lower-Cost Evaluations also Met All Quality Criteria
paragraphs: Our analysis found that high-quality evaluations tend to be more  expensive, but well-designed lower-cost evaluations also met the criteria  we identified for a high-quality evaluation. Overall, as table 4 shows, the  median cost of high-quality evaluations (i.e., evaluations that met all  quality criteria) was $137,800 more than the median cost of acceptable- quality evaluations (i.e., evaluations that partially or generally met all  quality criteria) and $208,600 more than the median cost of lower-quality  evaluations (i.e., evaluations that did not meet, or provided insufficient  information for, one or more quality criteria).
High-quality  evaluations also tended to include factors associated with  higher evaluation costs. For example, the most expensive evaluation in  our sample cost $2.2 million and generally met all quality criteria. This net  impact evaluation assessed multiple civil society and governance  programs in an African country using different methodologies, including a  randomized controlled trial, over 4 years and conducted two rounds of  surveys of program beneficiaries. Another high-quality evaluation cost  $1.4 million and took almost 4 years to complete; this evaluation  conducted three surveys of program beneficiaries and used a quasi- experimental methodology to assess the net impacts of energy-efficient  stoves in Asia. However, some lower-cost evaluations also met all of the  quality criteria. Of the 15 high-quality evaluations for which we identified  costs, 4 cost less than $150,000.
section_title: Selected Agencies’ Evaluations Are Generally Available Online, but Some Agencies Can Improve Dissemination
paragraphs: We assessed DOD’s, HHS’s, MCC’s, State’s, USAID’s, and USDA’s use  of six dissemination practices that federal, AEA, and other guidance  indicate agencies should generally use to ensure effective dissemination  of evaluations. We found that the agencies varied in their performance  of these practices for the fiscal year 2015 evaluations we reviewed (see  table 5). All except USDA generally made nonsensitive evaluations  publicly available online. These nonsensitive evaluations could  generally be located with the agencies’ website search engines. However,  some agencies’ evaluations were not posted in a timely manner. Each of  the agencies posted its sensitive evaluations internally for access by  internal users. Only USAID included dissemination plans in most  nonsensitive evaluations to help ensure their dissemination to potential  users of the evaluation, but most of the other agencies now require such  plans to be prepared for future evaluations. In addition to publicly posting  the report, all of the agencies used other means to actively disseminate  evaluation findings. Following these practices can help agencies ensure  that their evaluation reports are accessible, timely, and useful to decision  makers and other stakeholders.
subsections: section_title: All Agencies except USDA Make Nonsensitive Evaluations Publicly Available Online
paragraphs: Every agency with nonsensitive evaluations requires public, online  posting of nonsensitive evaluation documents, and all except USDA  publicly posted all of the nonsensitive evaluations we reviewed on publicly  accessible websites. Making evaluation reports publically available on  their websites helps agencies share evaluation findings with partners,  program beneficiaries, and the wider public and facilitates the  incorporation of evaluation findings into program management decisions.
We examined the agencies’ dissemination of 193 evaluations. The  agencies did not require 22 evaluations to be publicly posted due to  their sensitivity—all 4 DOD evaluations as well as 17 evaluations from  State and 1 from USAID. Of the remaining 171 nonsensitive  evaluations, we found that more than three-quarters (133) were publicly  posted. USDA did not publicly post any of its 38 nonsensitive  evaluations. According to USDA officials, the department is in the  process of developing procedures for making these nonsensitive  evaluations public, which would include reviewing the documents to  ensure that they did not contain, for example, personally identifiable or  proprietary information. Without posting all nonsensitive evaluations  online, agencies cannot ensure that the evaluations’ findings reach  intended audiences and are available to inform future program design  or budget decisions.
section_title: Publicly Posted Evaluations Can Generally Be Found with Agency Search Engines
paragraphs: Most of the nonsensitive, publicly posted evaluations we reviewed could  be located with a search engine on the agencies’ websites. Providing a  search engine that potential evaluation users can employ to find the  evaluation reports ensures that users can locate the information they  seek, in a format that matches their expectations. Websites at three of  the four agencies with publicly posted evaluations—MCC, State, and  USAID—have search engines that enable users to find each specific  evaluation. The PEPFAR website, which hosts evaluations of PEPFAR  programs implemented by CDC and other agencies, has these  evaluations listed in a spreadsheet locatable on the site. Many  evaluations of PEPFAR programs implemented by CDC can also be  found using a search engine at a separate website, called “CDC Stacks.”  According to CDC, almost all of the evaluations that we reviewed were  posted on the CDC Stacks website. The agency reported that it is in the  process of adding the remaining CDC evaluations from fiscal year 2015 to  this website.
section_title: Some Evaluations Were Not Posted within Required Time Frames
paragraphs: Some of the nonsensitive evaluations we reviewed were not posted on  the agencies’ websites within required timeframes. Making evaluation  reports accessible in a timely manner ensures that interested parties can  access the findings of these evaluations in time to incorporate them into  program management decisions. MCC and HHS did not post some  evaluations within the timeframes they require, limiting stakeholders’  ability to make optimal use of the evaluation findings.
We found that MCC did not post 10 of its 16 evaluations, as MCC  requires, within 6 months after MCC received them, and it did not post 8  of these 10 evaluations until a year or more after MCC received them.  According to MCC officials, the agency’s internal evaluation quality review  process for evaluations, in which the agency reviews the document  before releasing it to the public, has been a major factor in these delays.  MCC officials reported that for some of the evaluations—for instance,  those written in a language other than English—this process took  significantly longer than usual.
HHS did not post 11 HHS evaluations in the timeframe required by the  agency. It did not post 6 of these 11 evaluations online within 90 days as  required by PEPFAR. PEPFAR guidance requires that evaluations be  posted within 90 days of completion, while HHS requires that evaluations  be publicly posted within a year of their completion. One HHS official  stated that the delay in the posting of these six evaluations was due to the  conflicting policies. However, the remaining five evaluations were also not posted within the year as required by HHS/CDC. These five evaluations  have since been posted online. Since evaluated conditions may change  over time, not posting evaluations online within the required timeframe  limits internal and external stakeholders’ access to current, actionable  information. In comments on a draft of this report, CDC noted that, as of  December 2016, CDC is providing guidance that all evaluations be  posted online within 90 days, as required by PEPFAR. CDC published  this guidance in January 2017.
section_title: All Agencies Have Websites to Make Sensitive Evaluations Available Internally
paragraphs: Of the three agencies with sensitive evaluations—DOD, State, and  USAID—all have websites to make these evaluations available to internal  stakeholders. While sensitive evaluations are not required to be made  available to the public on an agency’s website, disseminating sensitive  evaluation findings to the appropriate audience will facilitate their use.  DOD, State, and USAID all reported that they have internal websites that  can be used to post sensitive evaluations. In addition, State updated its  policy for 2015 to require that State officials post a nonsensitive summary  of sensitive evaluations on State’s public website. While USDA does not  currently publicly post its evaluations, USDA reported that it makes these  evaluations internally available through its grant management system.  HHS and MCC did not have sensitive evaluations in fiscal year 2015.
section_title: USAID Included Dissemination Plans in Most Evaluations, and Other Agencies Will Require Such Plans in the Future
paragraphs: USAID requires the development of dissemination plans and included  evidence of such planning in the majority of the evaluations we reviewed,  and all of the other agencies except State now require such plans for  nonsensitive evaluations. Dissemination planning identifies potential  users of an evaluation and describes an approach to providing users with  the evaluation results. Such planning can help agencies ensure that  evaluation reports are disseminated effectively  Among the six agencies, only USAID required the development of  dissemination plans for fiscal year 2015 evaluations and included  evidence of such planning in the majority of the evaluations whose  dissemination we reviewed. Of the 62 USAID evaluations, 44 included  evidence that dissemination planning had been completed. HHS, MCC,  State, and USDA did not require dissemination plans for their evaluations completed in fiscal year 2015. Agency officials at HHS and MCC provided  evidence that dissemination planning took place for at least one of their  respective evaluations we reviewed, but this dissemination planning was  not required by agency policy, and therefore this planning was ad hoc.  DOD plans for evaluation dissemination by identifying  potential users of  the evaluation and sending e-mails to these internal and congressional  stakeholders after the evaluations are completed.
HHS, MCC, and USDA guidance now require dissemination plans for  future evaluations. State officials reported that, as of November 2016,  State was planning to revise its policy to require the use of dissemination  plans for evaluations but had not instituted this requirement. Without  dissemination planning, State cannot ensure that its evaluations are  disseminated as effectively as possible to potential users.
section_title: Agencies Used Additional Means to Actively Disseminate Evaluation Findings
paragraphs: In addition to posting evaluations online, each of the six agencies  reported disseminating evaluation findings through other means. Our prior  work has shown that taking such additional steps to actively disseminate  evaluation reports—for example, briefing stakeholders on evaluation  findings and distributing the evaluations to interested stakeholders via e- mail—facilitates dissemination of evaluation report findings and  encourages their use.
Agency officials reported using various means besides web posting to  disseminate evaluation findings. For example, officials at all six agencies  reported using briefings to share evaluation findings with various  stakeholders within and outside of the agency. Additionally,  HHS, State,  MCC, and USAID officials reported that they shared evaluation results  with interested parties at various professional conferences. USAID  officials stated that the agency also disseminates evaluation findings by  posting its evaluations on partner websites, creating video companions to  evaluation reports to provide to stakeholders, and posting syntheses of  evaluation findings on the agency’s website.
section_title: Conclusions
paragraphs: Foreign assistance evaluations can be challenging to implement, but they  are an essential tool for guiding agency decision making and allocation of  resources. Agencies’ foreign assistance evaluations assess a wide  variety of programs around the world, using many different designs and  methodologies, and the wide range of evaluation costs reflects this  diverse context. However, regardless of the location, design, or cost, an  evaluation should provide sufficient and reliable evidence to support its  findings. A high-quality evaluation helps agencies and stakeholders  identify successful programs to expand or pitfalls to avoid. Evaluations  that do not meet all quality criteria that we identified may not provide  sufficiently reliable evidence to inform these decisions. In addition, for  evaluations to inform decision making, stakeholders must be able to find  them. While foreign assistance agencies have generally made their  evaluations available online in a timely manner, several agencies can  take additional steps to ensure that stakeholders have improved access  to these evaluations to make better-informed decisions about future  program design and implementation. A growing body of high-quality,  broadly disseminated evaluations can help the United States continuously  improve its foreign assistance programs and thereby support democracy,  enhance security, reduce poverty and suffering, and achieve other U.S.  foreign policy goals.
section_title: Recommendations for Executive Action
paragraphs: To improve the reliability and usefulness of program evaluations for  agency program and budget decisions, we recommend that the Chief  Executive Officer of MCC, the Administrator of USAID, the Secretary of  Agriculture, the Secretary of Defense, the Secretary of State, and the  Secretary of Health and Human Services (in cooperation with State’s  Office of the U.S. Global AIDS Coordinator and Health Diplomacy) each  develop a plan for improving the quality of evaluations for the programs  included in our review, focusing on areas where our analysis has shown  the largest areas for potential improvement.
To better ensure that the evaluation findings reach their intended  audiences and are available to facilitate incorporating lessons learned  into future program design or budget decisions, we recommend that  the Secretary of Health and Human Services direct the Centers for  Disease Control and Prevention to update its guidance and practices  on the posting of evaluations to require PEPFAR evaluations to be  posted within the timeframe required by PEPFAR guidance;  the Chief Executive Officer of MCC adjust MCC evaluation practices  to make evaluation reports available within the timeframe required by  MCC guidance;  the Secretary of State amend State’s evaluation policy to require the  completion of dissemination plans for all agency evaluations; and  the Secretary of Agriculture implement guidance and procedures for  making FAS evaluations available online and searchable on a single  website that can be accessed by the general public.
section_title: Agency Comments and Our Evaluation
paragraphs: We provided a draft of this report to DOD, State, HHS, MCC, USAID and  USDA for review and comment. DOD, State, HHS, MCC, USAID, and  USDA provided official comments, which are reproduced in appendixes III  through VIII  with, where relevant, our responses. DOD, HHS, and USAID  also provided technical comments, which we incorporated as appropriate.
The following summarizes DOD, State’s, HHS’s, MCC’s, USAID’s, and  USDA’s official comments and our responses.
DOD stated that it partially concurred with our recommendation,  noting that in many cases, certain methodologies are not well suited  for security assistance evaluation. DOD observed that, for example, it  would be unethical to establish randomized control groups for security  assistance evaluations and that foreign military organizations may be  unwilling to provide DOD significant access to some military units  solely for the purpose of the evaluation. We recognize that certain  methodologies are not appropriate in every context, and we did not  advocate the use of randomized control groups in the DOD  evaluations we reviewed. Our main concerns about the DOD  evaluations focused on implementation of the methods used. In  particular, we found limitations in sampling methods including  descriptions of the target population, data collection methods, and  data analysis. We adjusted pertinent wording in our report to clarify  these points.
State concurred with our recommendations and noted that its  forthcoming Program Design and Performance Management Policy  for Programs, Projects, and Processes, recently published Program  Design and Performance Management toolkit, and updated policy  guidance will constitute a plan for improvement. We will monitor the  implementation of this plan to verify that State takes appropriate steps  to address our recommendation.
HHS concurred with our recommendation that it update guidance and  practices on the posting of PEPFAR evaluations and stated that CDC  guidance now requires evaluation reports to be posted on a publically  accessible website within 90 days of the evaluation’s completion. HHS  did not comment on our recommendation that it develop a plan for  improving the quality of evaluations.
MCC stated that it welcomed our findings and recommendations for  improvement but noted that it could not agree or disagree with our  quality assessments because we did not provide data on our  determinations for individual evaluations. In response to our  observation that MCC evaluations did not contain conflict-of-interest  statements, MCC noted that it has required independent third-party  evaluation of all its projects since 2009 and that, in 2013, it  standardized the language in its evaluation contracts to explicitly  establish the independent role of evaluators. While these are positive  steps, we believe that including in MCC’s published evaluations  explicit statements about the evaluators’ independence and any  potential conflicts of interest would bolster the evaluations’ credibility  and usefulness. With regard to the timeliness of public access to its  evaluations, MCC indicated that when it established its internal review  process for evaluations in 2013, it did not anticipate the length of time  that would be required to finalize evaluation reports. MCC noted that  its forthcoming revised policy on monitoring and evaluation states that  “MCC expects to make each interim and final evaluation report  publicly available as soon as practical after receiving the draft report.”  However, the revised policy does not establish a target time frame for  completing internal reviews of the reports. Establishing such a time  frame could help MCC ensure that evaluation reports are published in  a timely fashion that maximizes their usefulness.
USAID stated that it has established a plan to improve the quality of  evaluations, including an update and clarification of the requirements  and quality standards for evaluations. USAID also stated that it plans  to provide additional training and other capacity-building efforts to help  ensure that staff have the necessary skills to manage evaluations. We  will monitor implementation of this plan to verify that USAID takes  appropriate steps to address our recommendation.
USDA agreed with our recommendations. To address the  recommendations, USDA stated that it would update its guidance on  reviewing evaluation terms of reference to include a section on quality  that specifically focuses on the areas where the GAO analysis has   shown the largest areas for potential improvement. USDA further  stated that FAS will continue its current efforts to make nonsensitive  evaluations publicly available online and will make them searchable  as well.
We are sending copies of this report to the appropriate congressional  committees and to the Secretaries of Agriculture, Defense, State, and  Health and Human Services; the Chief Executive Officer of the Millennium  Challenge Corporation; and the Administrator of the U.S. Agency for  International  Development. In addition, the report will be available at no  charge on GAO’s website at http://www.gao.gov.
If you or your staff have questions about this report, please contact me at  (202) 512-6991, or farbj@gao.gov. Contact points for our Offices of  Congressional Relations and Public Affairs may be found on the last page  of this report. GAO staff who made major contributions to this report are  listed in appendix IX.
section_title: Appendix I: Objectives, Scope, and Methodology
paragraphs: In response to congressional requests, we examined (1) the extent to  which foreign assistance program evaluations met key evaluation quality  criteria; (2) the costs of the evaluations, as well as factors that affect  these costs; and (3) the extent to which the agencies ensure the  dissemination of evaluation reports within the agency and to the public.
To address our objectives, we identified the six major agencies  administering the most foreign assistance on the basis of obligations  reported to the U.S. Agency for International  Development’s (USAID) U.S.  Overseas Loans and Grants database for fiscal years 2008 through 2012.  The six agencies we identified are USAID, the Department of State  (State), the Millennium Challenge Corporation (MCC), the Department of  Health and Human Services (HHS), the U.S. Department of Agriculture  (USDA) and the Department of Defense (DOD). For the three agencies  that are not focused exclusively on foreign aid or foreign affairs (HHS,  USDA, and DOD), we limited our scope to selected programs. For HHS  and USDA, we selected programs that account for the vast majority of  foreign assistance program dollars that the agency implemented. At HHS  we examined evaluations of the President’s Emergency Plan for AIDS  Relief (PEPFAR) programs implemented by HHS’s Centers for Disease  Control and Prevention (CDC). At USDA we examined evaluations for the  Food for Progress and McGovern-Dole food assistance programs,  implemented by the Foreign Agricultural Service (FAS). At DOD we  examined evaluations prepared for the Global Train and Equip (GT&E)  program. While our previous review of agency evaluation policies did not  identify DOD-wide evaluation policies, we did identify GT&E as having  relevant policies to guide its evaluations.
To identify evaluations completed in fiscal year 2015, the most recently  completed fiscal year as we undertook our review, we requested that  each agency provide a list of all foreign aid evaluation reports completed  in that year. We did not separately review agency files to identify if  agencies had additional evaluations beyond those listed in the registries.  To assess the reliability  of the agency evaluation lists, we reviewed the  documents provided to ensure that each was a completed evaluation and  to confirm that the date of the document fell within our specified  timeframe. If necessary, we followed up with agency officials to clarify the  date or status of the document. Based on their responses, we removed  documents that were not evaluations or fell outside of our timeframe. We  also did not review evaluation reports that were not written in English. We  determined that the data in the evaluation lists were sufficiently reliable  for the purposes of this engagement. In all, we identified a study  population of 361 evaluations: 4 DOD evaluations, 51 HHS evaluations,  17 MCC evaluations, 28 State evaluations, 221 USAID evaluations, and  40 USDA evaluations. We examined the evaluations themselves and any  appendices that the agency provided which were directly referred to in the  evaluations. We did not consider evaluation plans and protocols,  underlying documents and other work papers as evidence that the  planned design was implemented. Similarly, we did not consider contracts  with third-party evaluators or evaluation organizations as evidence that  the evaluator had maintained independence. Instead we required  statements in the reports or methodological appendices that steps and  procedures were actually taken and that no threats to independence have  been identified.
From the study population of fiscal year 2015 evaluations, we reviewed all  DOD, MCC, and State evaluations; all USAID net impact evaluations; and  a sample of HHS, USDA, and USAID performance evaluations. We  randomly selected a probability sample from the study population of HHS,  USDA, and USAID performance evaluations. With this probability sample,  each member of the study population had a nonzero probability of being  included, and that probability could be computed for any member. For  USAID, we included all net impact evaluations in the sample because net  impact evaluations constituted less than 20 percent of all the evaluations  provided, and if we had not included them all, we would not have been  able to comment on this type of evaluation.
Based on the review of the evaluation documents after the initial  screening, an additional 16 evaluations were found not to be within our  scope, and we substituted for these evaluations when possible. For  example, we excluded documents that did not evaluate a specific  program, were monitoring or grant reports, or were plans for an  evaluation rather than an evaluation report. We included two substitute  HHS and two substitute USDA evaluations to replace those that were found to not be in scope and also reviewed additional USDA evaluations.  However, because we had initially included all MCC, State, and USAID  net impact evaluations, there were no additional evaluations available to  substitute if those were excluded. The original sample and the final  respondents across the six agencies can be found in table 6. Each  sample selection was subsequently weighted in the analysis to represent  the evaluations in the population that were not selected.
We reviewed the full population of DOD, MCC, and State evaluations;  therefore, our results from the quality review of these evaluations do not  have an associated margin of error. The results from our review of the  HHS, USDA, and USAID evaluations are reported with an associated  margin of error. Because we followed a probability procedure based on  random selections, our sample is only one of a large number of samples  that we might have drawn. Since each sample could have provided  different estimates, we express our confidence in the precision of our  particular sample’s results as a 95-percent confidence interval. This is the  interval that would contain the actual population value for 95 percent of  the samples we could have drawn. All percentage estimates for  aggregated results from our review have margins of error at the 95  percent confidence level of plus or minus 8 percentage points or less,  unless otherwise noted, and all percentage estimates for individual  agencies from our review have margins of error at the 95 percent  confidence level of plus or minus 11 percentage points or less, unless  otherwise noted.
To assess the extent to which the results of foreign assistance program  evaluations are supported by their evidence and whether they assess if  programs have met their goals, we assessed the sample of agency fiscal  year 2015 evaluation reports against quality criteria we identified. We  identified these criteria based on our review and analysis of evaluation  guidance from agencies included in our review (including any agency  internal evaluation review checklists), international organizations,  evaluation organizations, and prior GAO reporting. These criteria  include necessary high-level elements in designing, implementing and  reporting on evaluations that could serve as standards across different  agencies and evaluation types. Prior to undertaking our quality review,  these criteria were discussed and reviewed within the engagement team,  as well as by other GAO staff with experience in program evaluation and  methodologies.
We incorporated the identified criteria into a standardized data collection  instrument (DCI) in order to consistently review the sampled evaluation  reports. The DCI contained evaluative questions against which to assess  evaluation quality as well as descriptive questions to gather information  about the evaluations, such as its location and methodology. The high- level criteria each included subquestions about elements the reviewer  should consider in making his or her overall decision. The evaluation  quality criteria were judged on a four-part scale for most of the judgmental  questions: generally addressed: the evaluation mostly addressed the key  element(s) of the criterion but did not have to completely address all  elements in the subquestions; partially addressed: the evaluation had one or more clear area(s) for  improvement on the criterion; not at all addressed: the evaluation did not show that steps were  taken to address the criterion; and  insufficient information: reviewers could not make a determination due  to a lack of information in the evaluation and any other associated  materials.
If a criterion was partially or not at all addressed, or if there was  insufficient information in the evaluation to assess the criterion, we  considered it a deficiency. We did not consider study protocols or design  documents that indicated plans for a particular evaluation step as  sufficient evidence that such a step was performed unless the evaluation  report also provided evidence that it had.
The descriptive questions in the DCI about evaluation types and  methodology were based on prior GAO work and asked about designs  that examined net impacts of interventions, outcomes of interventions,  and processes. The questions on net impact evaluations asked about the  type of design using four categories: (1) randomized controlled trials or  groups, (2) comparison groups, (3) time series that would allow for trends  to be determined pre- and post- intervention, and (4) quasi-experimental  statistical modelling techniques. The questions on outcome and process  evaluations asked whether baselines and targets had been established  for the outcomes and whether criteria had been established to assess  processes. From these types of evaluation, we created two broad  categories to use in our analysis of evaluation cost and quality: net impact  evaluations, and performance evaluations. The net impact category  included all four impact design types, while the performance evaluation  category included outcome and process evaluations as well as a few  evaluations that did not fall within the outcome and process categories.  The performance evaluations included some that had established targets  or baselines and others that had not, while the process evaluations  included some that had established criteria for assessment and others  that had not. We noted some overlap between the net impact,  performance, and process categories. For example, net impact  evaluations often considered outcomes or processes, and performance  evaluations often considered both outcomes and processes. This overlap  was a key reason we decided to develop and analyze two broad types of  evaluation categories rather than attempt to develop more refined types.
A key assumption underlying our analysis was that different types of  evaluations were appropriate for different types of study questions. The  evaluative questions in the DCI were relative, rather than absolute, with  respect to the study questions and were intended to be applicable across  evaluation types. We did not assess the study questions in terms of their  scope or rigor. We instead took the study questions as given, thereby  giving every evaluation an equal chance to receive a high score if its  design and implementation were appropriate for the study questions.  Evaluation reviewers were instructed to consider design, implementation,  and reporting in terms of the study questions the evaluations set out to  answer rather than against an absolute standard. In this way, we  determined that it would be as possible for a qualitative midterm  evaluation that considered program implementation to achieve high  scores as it would for a final net impact evaluation that considered effects  attributable to the program.
The main criteria questions in the DCI asked about the appropriateness of  the design, implementation, and conclusions in light of the study  objectives. We did not determine a single definition of appropriateness  because we recognized that it is dependent on the study objectives and  data collection conditions. For example, the standards for  appropriateness of a final net impact evaluation of a pilot health care  program that seeks to establish whether the intervention is achieving  positive outcomes are different from the standards for a mid-term  performance evaluation of a well-established water and sanitation  program supported by a solid evidence base and focused on whether the  program was implemented as planned. Given the variation in agency  goals and programs, evaluation types, and evaluation timing, we  determined that we would rely on expert professional judgment rather  than attempt to use a single definition of appropriateness for every  situation.
While we designed our DCI to apply broadly across agencies and  evaluation types, differences in agency evaluation practices and areas of  responsibility may limit comparisons between the agencies. For example,  the target audience of an evaluation may determine whether it includes  certain reporting elements. HHS’s PEPFAR evaluations are generally  produced for dissemination in research publications and journals, while  USAID evaluations evaluate a wide range of programs and are generally  directed to an audience of program officials and managers. In addition,  the evaluations we reviewed assessed a wide range of foreign assistance  programs with varying characteristics. These characteristics include the  nature of the foreign assistance intervention, the type of program  responsible for the intervention, whether the program was designed to be  evaluated, and the timing of the evaluation. For example, some  evaluations consider ongoing development assistance in areas such as  education or health, while others consider emergency responses to  humanitarian crises. Agency officials noted that it could be harder to  ensure quality in an evaluation of a program that had to respond quickly  to a crisis and therefore did not have the opportunity to plan for an  evaluation. They also noted that if an evaluation is not started until after  the program has begun, there may not be any baseline data available.
The evaluation review consisted of multiple reviews by a team of GAO  staff with experience and familiarity with research methods as well as with  reviewing studies and evaluations across a wide range of subject areas  and disciplines. After completing his or her initial review, the first reviewer  notified the second reviewer that the evaluation report was available for  his or her review. The second reviews were not independent; as the  second reviewer saw the decisions made by the first reviewer and could  review the first reviewer’s notes on sources and justifications for his or her  decisions. The second reviewers read the evaluation and indicated  whether he or she agreed with the first reviewers’ decisions or whether he  or she proposed another decision. The first and second reviewers  subsequently met to reconcile any differences. After the reconciliations  were completed, a supervisor then reviewed the work of the two  reviewers for internal consistency and completeness according to a  standard protocol but did not re-review the evaluation documents. The  supervisor related the identified issues as needed to the first and second  reviewers, who addressed them before the supervisor recorded the  review as final.
We took several steps to ensure consistency among the reviewers. We  conducted two pretests of the DCI on sample evaluations. The first  pretest included members of the engagement team as well as GAO staff  with experience in the design of survey instruments or in the review of  foreign assistance evaluations. The second pretest included members of  the evaluation review team. After each round of pretests, we made  appropriate revisions to the DCI. To help ensure consistency of  interpretation, we created a guidance document where reviewers  recorded questions about certain decision rules to follow in specific  instances. Answers to the questions were then posted after discussion  among the review and engagement team members. Additionally, the  engagement and review team held regular weekly meetings to discuss  any methodological issues that arose and preliminary tabulations of the  review data.
To analyze the responses to the DCI, we examined how evaluations’  quality varied by the eight quality criteria, by agency, by timing of the  evaluation relative to the stage of program implementation (midterm or  interim vs. final), and type of evaluation (net impact vs. performance).  While there were some differences between the final and midterm  evaluations, these were not statistically significant. However, a higher  percentage of evaluations that attempted to assess net impacts were of  high quality than those that did not attempt to assess net impacts. We  determined that these differences were due primarily to specific  weaknesses in the implementation of the design of the evaluations. For  example, performance evaluations used nonrandom sampling more often  than net impact evaluations, which used at least some random sampling.  While our DCI and subsequent analysis treated both methods of sampling  equally, we focused our assessment on the extent to which each method  had been appropriate for the study questions. We found that the  performance evaluations’ nonrandom sampling was carried out  appropriately less often than the random sampling typically used in net  impact evaluations. In the body of our report, therefore, we focus on the  specific weaknesses that we found, such as the one regarding  nonrandom sampling, rather than differences at the level of the two broad  evaluation types.
Our analysis found a reasonably high degree of overlap between several  approaches that we considered for categorizing the evaluations. For  example, we categorized the evaluations into three groups: high quality,  acceptable quality, and lower quality based on the number of quality  criteria that were generally or partially met for each evaluation, as well as  instances when quality areas were either not met or there was insufficient  information to determine if a certain criterion was met by a particular  evaluation. Those evaluations that fell into the lowest and middle  categories based on these three categories also generally fell into the  lowest and middle categories using another approach that we examined,  as table 7 shows. This comparison also shows that some evaluations in  the highest category had a relatively higher number of criteria generally  met, while some in the middle category had a relatively lower number of  criteria generally met.
To determine the cost of foreign aid evaluations, we reviewed contracts,  invoices, and related documents to determine the cumulative cost of final  evaluations that were conducted by an outside evaluator. We defined the  cumulative cost as the cost of the final evaluation and any related  activities that informed the evaluation’s findings, such as a separate data  collection effort or a midterm or baseline evaluation. We did not determine  the cost of evaluations that had a midterm evaluation but not a final  evaluation because midterm evaluation costs do not reflect the total  cumulative cost of evaluating a program. State and USAID officials noted  that some of their midterm evaluations may ultimately be a program’s only  evaluation. We did not review the State and USAID fiscal year 2015  midterm evaluations to determine if the agency intends to conduct an  additional, final evaluation or include these midterms in our cost analysis.  Table 8 shows the total number of evaluations in GAO’s quality sample,  the number of those evaluations whose costs we reviewed, and the  number of evaluations whose costs we reviewed that we included in the  statistical analysis by agency.
We used contracts and invoices to determine the cost of 42 of the 76  MCC, State, USAID, and USDA final evaluations. To determine an  evaluation’s cost, we used either the final invoiced amount or the  contract’s total obligations. For each evaluation, we also read the  statement of work to determine if the contract covered only the evaluation  in our sample or if it covered additional activities as well. In some cases,  while the contract covered additional activities, the evaluation’s cost was  clearly identifiable  in a separate line item or invoice. We identified the  evaluation start and end dates using the period of performance in the  contract or statement of work. In some cases we determined the period of  performance using dates in the evaluation report if the contract’s period of  performance covered a broader time period than the evaluation in our  sample.
For six USAID evaluations and nine State Department evaluations, we  determined the evaluation cost using data from the Federal Procurement  Data System – Next Generation (FPDS-NG). FPDS-NG provides a  contract’s obligations, start and end dates, and other descriptive data. In  each case, we confirmed that the contract covered only the evaluation in  our sample by reviewing the statement of work or by confirming with  agency officials. We used total obligations to determine the evaluation’s  cost and also used the date signed and end date listed in FPDS-NG to  determine the period of performance. To assess the reliability of the  FPDS-NG data, we (1) reviewed related documentation, (2) traced to or  from source documents, and (3) confirmed FPDS-NG data with  knowledgeable agency officials. We determined that the FPDS-NG data  were sufficiently reliable for the purposes of this engagement.
For MCC, State, and USAID evaluations without clearly identifiable cost  information from contract documents or FPDS-NG, we estimated the cost  based on budget documents or cost estimates provided by the agency or  contractor, where available.  We relied on budgets or cost estimates to  determine the cost of 5 MCC evaluations, 4 State Department  evaluations, and 10 USAID evaluations. We excluded one MCC  evaluation from the cost sample because MCC provided a wide range for  the estimated cost, and we concluded that this range was not sufficiently  reliable to report. We could not determine the cost of one State  Department evaluation and one USAID evaluation that were each  procured under large agreements that did not separately track evaluation  costs. Additionally,  USAID officials did not provide cost information for  one evaluation.
We report only limited data on the cost of DOD’s GT&E and HHS’s  PEPFAR evaluations because the evaluation contracts or implementing  partner agreements did not separately track evaluation costs, and we  concluded that the available estimates were too limited to include in our  statistical analysis. To estimate the cost of DOD’s GT&E evaluations, we  reviewed the associated contract and invoices, which included the  evaluations as well as additional services. Since the contract and related  documents did not contain a separate line item for the evaluations, we  requested a cost estimate from agency officials and the contractor. The  contractor was able to provide only the broad estimate that we include in  our report with appropriate caveats but which we concluded was not  sufficiently reliable to include in our statistical analysis. The costs of the  HHS PEPFAR evaluations were also not separately tracked by the  agency and implementing partners. Evaluation costs were instead  estimated by HHS country teams or implementing partners based on their  review of previous years’ financial records, budgets, or cooperative  agreements. Because of the volume of records involved, we judgmentally  selected a subsample of 10 HHS evaluations to review the cost estimates  provided by HHS officials. To review these estimates, we traced the  estimates to source documentation and spoke with knowledgeable  agency officials to understand the methodology used to prepare the  source estimates. Because of the uncertainty of these cost estimates, we  include them in our report with appropriate caveats but concluded that  they were not sufficiently reliable to include in our statistical analysis.
To determine the factors that are associated with the costs of foreign aid  evaluations, we analyzed the costs of MCC, State, USDA, and USAID  evaluations in relation to the data that we collected on these evaluations’  quality scores, duration, and other characteristics. We then produced  summary statistics showing the cost differences of various characteristics.  For example, we compared the average cost of evaluations with a survey  to those without surveys. We conducted difference-in-means tests to  determine if any of the characteristics were statistically significant at the  95-percent confidence level and reported characteristics that were  significantly related to costs. We also reviewed the evaluations to obtain  insights into other likely cost factors, such as unstable locations and the  number of sites, for which systematic data were not available for  difference-in-means tests. We included location among the characteristics  we considered after observing that evaluations that were more costly than  others that were of the same type, or required the same performance  period to complete, tended to be conducted in unstable or multiple  locations.
To assess our third objective, we identified leading practices for the  dissemination of evaluation findings. We identified these leading practices  using federal guidance, including the President’s Open Government  Directive and Office of Management and Budget (OMB) guidance,  which encourages or requires the timely public posting of agency  information on a searchable website, as well as plans and additional  efforts to actively disseminate agency information. In addition to the  federal guidance, we also used the American Evaluation Association’s  (AEA) An Evaluation Roadmap for a More Effective Government (AEA  Roadmap); the Organization for Economic Co-operation and  Development, Development Assistance Committee’s (OECD DAC)  Quality Standards for Development Evaluation and Evaluating  Development Activities: 12 Lessons from the OECD DAC; and HHS and  the General Services Administration’s (GSA) Research-Based Web  Design and Usability Guidelines  that cite timely public posting,  dissemination planning, and additional active efforts to disseminate  results as important communication tools for evaluations. We used these  sources to identify six practices that agencies should use in order to  successfully disseminate the results of foreign aid evaluations. We  reviewed the dissemination of all evaluations from fiscal year 2015 for five  of the agencies and a sample of USAID evaluations. In total, we  examined the dissemination of 193 evaluations: 4 at DOD, 49 at HHS, 16  at MCC, 23 at State, 63 at USAID, and 38 at USDA.
To assess the availability  and timeliness of the dissemination of  evaluation reports, we reviewed agency policies and websites and  interviewed agency officials. We reviewed agency evaluation websites to  determine if the evaluation reports in agency evaluation lists had been  publicly posted. If an evaluation report had not been posted, we followed  up with agency officials regarding the reasons it had not been. We also  reviewed the evaluation reports to ensure the documents contained the  information necessary for a user to determine if the findings were valid.  For example, we reviewed evaluations to ensure that any related annexes  had been included when the document had been posted. We examined  each agency website to determine whether it provided a search engine  that could be used to locate evaluations. We also checked whether the  search engine included additional search filters such as the year the  evaluation was completed or its location. To assess timeliness, we  reviewed agency policies and guidance to determine how soon it required  evaluation reports to be posted after the completion of the report. We  compared the date an evaluation was considered complete by the agency  to the date that it was posted online to determine whether it had been  posted within the timeframe required by the agency. To determine  whether sensitive evaluations were made available to identified  stakeholders via an internal digital system, we reviewed agency lists of  sensitive evaluations and interviewed agency officials about agency  processes for making sensitive evaluations available internally. We also  received an in-person demonstration of the internal posting of USAID’s  sensitive evaluations and documentation of the internal systems that  DOD and USDA use to post evaluations.
To assess agency dissemination planning and its use of additional means  for dissemination, we interviewed agency officials and reviewed agency  policies, practices, and evaluation documents. To determine if the agency  required dissemination planning, we reviewed the dissemination  requirements in its evaluation guidance. If the agency required  dissemination plans, we reviewed its evaluation reports, contracts, and  related documents to determine if they included an identification of the  potential users of an evaluation and a description of the approach that will  provide users with the evaluation results. If the agency guidance did not  require dissemination plans, we asked the agencies if dissemination  planning had occurred without the policy in place. If such ad hoc planning  had occurred, we asked that the agencies provide examples. We also  provided written questions to agency officials regarding additional agency  practices for disseminating evaluations other than posting the evaluation  online. If agency officials identified additional means of dissemination, we  reviewed additional documentary evidence that evaluation findings had  been disseminated using these means.
We conducted this performance audit from October 2015 to March 2017  in accordance with generally accepted government auditing standards.  Those standards require that we plan and perform the audit to obtain  sufficient, appropriate evidence to provide a reasonable basis for our  findings and conclusions based on our audit objectives. We believe that  the evidence obtained provides a reasonable basis for our findings and  conclusions based on our audit objectives.
section_title: Appendix II: Evaluation Review Data, by Agency
paragraphs: Agencies varied in the extent to which they met the applicable quality  criteria for evaluations that we identified. Tables 9 through 26 below  provide further detail on the characteristics and quality of the design,  implementation, and conclusions of fiscal year 2015 evaluations we  reviewed summarized for all six agencies and then individually  for (1) the  President’s Emergency Plan for AIDS Relief (PEPFAR) programs  implemented by the Centers for Disease Control and Prevention (CDC) of  the Department of Health and Human Services (HHS), (2) the Millennium  Challenge Corporation (MCC), (3) the Department of State (State), (4) the  U.S. Agency for International  Development (USAID) , and (5) the U.S.  Department of Agriculture’s (USDA) Foreign Agricultural Service’s food  aid programs.
section_title: Appendix III: Comments from the Department of Defense
subsections: section_title: GAO Comment
paragraphs: DOD partially concurs with our recommendation and notes that in many  cases certain methodologies are not well suited for security assistance  evaluation. DOD observed that, for example, it would be unethical for  DOD to establish a randomized control group for security assistance  evaluation and that some foreign military organizations may be unwilling  to provide significant access to military units solely for the purpose of an  evaluation. We recognize that certain methodologies are not appropriate  in every context, and we do not advocate the use of randomized control  groups in the evaluations we reviewed for DOD. Our main concerns about  the DOD evaluations focus on implementation of the methods used. In  particular, we found limitations in sampling methods, including  descriptions of the target population; data collection methods; and data  analysis. We adjusted pertinent wording in our report to clarify these  points.
section_title: Appendix IV: Comments from the Department of Health and Human Services
section_title: Appendix V: Comments from the Millennium Challenge Corporation
subsections: section_title: GAO Comments
paragraphs: 1.  MCC notes that it has required independent third-party evaluation of  all its projects since 2009 and that, in 2013, it standardized the  language in its independent evaluation contracts to explicitly define an  independent role for evaluators. While these are positive steps, we  believe that including in MCC’s published evaluations explicit  statements about the evaluators’ independence and any potential  conflicts of interest would bolster the evaluations’ credibility and the  usefulness.  2.  MCC states that it had not anticipated the length of time required by  the review process for all evaluations that it implemented beginning in  2013. MCC notes that its forthcoming revised policy on monitoring  and evaluation will state that “MCC expects to make each interim and  final evaluation report publicly available as soon as practical after  receiving the draft report.” This revised guidance does not set a  specific time frame for the reviews. While agency review efforts may  help ensure quality, a specific target for the length of time for the  reviews would provide a metric for assessing whether reports are  being published in a timely fashion that maximizes their usefulness.
section_title: Appendix VI: Comments from the Department of State
section_title: Appendix VII: Comments from the U.S. Agency for International Development
section_title: Appendix VIII: Comments from the U.S. Department of Agriculture
section_title: Appendix IX: GAO Contact and Staff Acknowledgments
subsections: section_title: GAO Contact
section_title: Staff Acknowledgments
paragraphs: In addition to the contact named above, James Michels, Assistant  Director; Thomas Beall, Miranda Berry, Anthony Costulas, Gergana  Danailova-Trainor, Martin De Alteriis, Neil Doherty, Mark Dowling, Laurie  Ekstrand, Justin Fisher, Georgette Hagans, Kay Halpern, Reid Lowe,  Luann Moy, Barry Seltser, Stephanie Shipman, Michael Simon, Douglas  Sloane, and Gregory Wilmoth made key contributions to this report.