Foreign Assistance: Agencies Can Improve the Quality and Dissemination of Program Evaluations
Released: 2017-03-03T00:00:00

Letter

Background

The six agencies whose evaluations we reviewed focus on foreign assistance to varying degrees. DOD, HHS, and USDA provide foreign assistance as part of their larger portfolios of programs, while MCC, State, and USAID focus exclusively on foreign affairs or foreign assistance.

DOD’s GT&E program provides training, equipment, and small-scale military construction activities to partner nations to build their capacity and enable them to conduct counterterrorism operations or to support ongoing allied or coalition military or stability operations that benefit the national security interests of the United States.

HHS’s CDC implements a portion of the President’s Emergency Plan for AIDS Relief (PEPFAR) programs under the direction of State’s Office of the U.S. Global AIDS Coordinator and Health Diplomacy.

MCC, a U.S. government corporation, provides aid to developing countries that have demonstrated a commitment to ruling justly, encouraging economic freedom, and investing in people. MCC supplies this assistance to eligible countries primarily through 5-year compacts with the goal of reducing poverty by stimulating economic growth.

State, the lead U.S. foreign affairs agency, implements programs that provide, for example, counternarcotics assistance; refugee assistance; and support for democracy, governance, and human rights.

USAID, the lead U.S. foreign assistance agency, implements programs intended to both further America’s interests and improve lives in the developing world. USAID’s broad portfolio includes programs that address democracy and human rights, water and sanitation, food security, education, poverty, the environment, global health, and other areas.

USDA’s Foreign Agricultural Service (FAS) administers two nonemergency food aid programs: (1) The Food for Progress program supports agricultural value chain development, expanding revenue and production capacity, and increasing incomes in food-insecure countries; (2) The McGovern-Dole International Food for Education and Child Nutrition program supports education and nutrition for schoolchildren, particularly girls, expectant mothers, and infants.

Agency Evaluation Guidance

Each of the six agencies has adopted evaluation guidance for the programs included in our review.

DOD’s November 2012 Section 1206 Assessment Handbook serves as a guide to evaluation planners and implementers for conducting evaluations of DOD’s GT&E programs as required by federal law. The fiscal year 2012 National Defense Authorization Act (NDAA) required DOD, no later than 90 days after the end of each fiscal year, to submit to Congress a report including an assessment of the effectiveness of GT&E programs conducted that fiscal year in building the capacity of the recipient foreign country. The fiscal year 2015 NDAA maintained this requirement through 2020. DOD did not have agency-wide evaluation guidance for security cooperation at the time we performed our review but issued such guidance in January 2017.

For PEPFAR programs, including those implemented by HHS, State’s Office of the U.S. Global AIDS Coordinator and Health Diplomacy issued the PEPFAR Evaluation Standards of Practice in January 2014 and an updated version (version 2) in September 2015.

MCC’s May 2012 Policy for Monitoring and Evaluation of Compacts and Threshold Programs requires that compact M&E plans identify and describe the evaluations that will be conducted, key evaluation questions and methodologies, and data collection strategies.

State issued its current evaluation policy and an additional guidance document for evaluations in January 2015 and issued a revised and updated version of the guidance in January 2016.

USAID lays out its evaluation policies in its Automated Directives System (ADS). USAID issued a fully revised ADS 201, addressing evaluation guidance, planning, and implementation, in September 2016.

USDA’s FAS evaluations are guided by its May 2013 Monitoring and Evaluation Policy, which requires both interim and final program evaluations.

Agency Evaluation Procurement and Cost Tracking

With the exception of HHS, the agencies we selected for our review generally rely on outside contractors to conduct evaluations. DOD, MCC, State, and USAID directly contract for third-party evaluation services. The HHS PEPFAR evaluations we reviewed were prepared (1) by CDC staff using existing program data; (2) by an implementing partner as part of the partner’s cooperative agreement; or (3) in one instance, under a separate agreement. USDA implementing partners procured the USDA evaluations whose costs we reviewed.

The six agencies track evaluation costs to varying extents. DOD, MCC, and State procured the evaluations we reviewed through centrally managed contracts, and cost information for these evaluations was available through the program or agency evaluation office. HHS’s PEPFAR, USDA, and USAID evaluations were often procured and managed at the country, mission, or implementing partner level. Cost information was not centrally available and could be obtained only from each mission or implementing partner.

Evaluation Types, Timing, and Methods

Foreign assistance evaluations may vary in type, timing, and method. Two common types of evaluation are the following:

Performance evaluations assess the extent to which a program is operating as was intended or the extent to which it achieves its outcome-oriented objectives. Performance evaluations often judge program effectiveness against criteria, such as progress against baselines, whether program goals were met, or whether expected targets were met.

Net impact evaluations assess the net effect of a program by comparing program outcomes with an estimate of what would have happened in the program’s absence. Net impact evaluations use a variety of experimental and quasi-experimental designs, including randomized methods in which participants are assigned to separate control or treatment groups to isolate the program’s effect. Net impact evaluations have more complex methodologies than the other evaluation types.

Agencies may conduct evaluations during or after the completion of a program. Interim or midterm evaluations are conducted while a program is in progress, and final evaluations are conducted after the program ends. Baseline evaluations are also sometimes conducted before a program begins as a basis for determining any effects of the program.

Evaluations may use one or more methods to produce their results. For example, evaluations may use random or nonrandom sampling from the target population to select cases for inclusion in the study. Evaluations may also use one or more methods to collect data on the chosen indicators and measures—for example, structured or unstructured interviews, focus groups, surveys, direct observations, or collection and analysis of existing data. Each of these methods has potential benefits and limitations that an evaluator must consider in assessing the evaluation’s evidence as a basis for its conclusions and recommendations.

Most Foreign Aid Evaluations Were of High or Acceptable Quality Overall, though Quality Varied by Criterion and Agency

Overall, about three quarters of all 2015 foreign aid evaluations from the six agencies we reviewed generally or partially met the quality criteria we identified. The remaining evaluations did not meet one or more of these criteria or provided insufficient information. While we generally found that evaluations met quality criteria related to design, implementation, and conclusions, we more often found limitations in implementation—including sampling methods, data collection, and analysis. In addition, we found that the independence of evaluators was not always clearly evident. While the quality of evaluations varied by agency, we identified shortcomings at all six of the selected agencies that could limit evaluation reliability and usefulness.

About Three-Quarters of Agencies’ Evaluations Showed High or Acceptable Quality Overall

By reviewing policies of federal agencies, international organizations, and evaluation organizations, and our prior reporting, we identified common characteristics of high-quality evaluations, from which we developed eight criteria for assessing evaluation quality. These quality criteria are associated with the (1) design, (2) implementation, and (3) conclusions of an evaluation, as follows. (See app. I for a full description of how we developed our evaluation criteria.)

Evaluation questions are aligned with program goals.

Performance indicators are appropriate for measuring progress against program goals.

Design is appropriate for answering the evaluation questions.

Target population and sampling method are appropriate, given the scope and nature of the evaluation questions.

Data collection is appropriate for answering the evaluation questions.

Data analysis is appropriate to answer the evaluation questions.

Conclusions are supported by the available evidence.

Recommendations and lessons learned are justified by the available evidence.

Based on an assessment of agency evaluations against these criteria, we rated 73 percent of all evaluations as high quality (26 percent) or acceptable quality (47 percent), because they generally or partially met all applicable quality criteria. We rated the remaining 27 percent as lower quality because they either did not meet or did not provide sufficient information related to at least one applicable criterion. These evaluations may not provide sufficiently reliable evidence to inform agency program and budget decisions. Overall, we encountered more instances when evaluations did not provide sufficient information about a certain criterion than instances when evaluations did not meet a quality criterion at all. Table 1 summarizes our observations about the quality of evaluations at the six selected agencies in our review.

Criteria Related to Evaluation Implementation Posed Greatest Challenge for Agencies’ Evaluations

The quality of evaluations varied by the type of quality criterion we applied. As figure 1 shows, many evaluations generally met the criteria related to the appropriateness of evaluation design, implementation, and conclusions. However, overall, evaluations generally met fewer criteria related to implementation, reflecting limitations in the way evidence was collected or analyzed.

A relatively high percentage of evaluations generally met each of the criteria we used to assess the alignment of the study questions with the program goals, the appropriateness of the evaluation design for the study questions, and the use of indicators for measuring progress.

Alignment of questions with program goals. Evaluation questions generally aligned with one or more of the evaluated program’s goals in more than 90 percent of the evaluations. Thus, evaluations were designed to provide useful information about program results.

Appropriate evaluation design for the study questions. About 80 percent of the evaluations used a design that was generally appropriate for the study questions, and the remainder of the designs was at least partially appropriate.

Appropriate use of indicators. Indicators for measuring progress were generally appropriate in about 80 percent of the evaluations. As a result, successes or failures identified by these evaluations are likely to be directly relevant to assessing achievement of the evaluated programs’ goals.

Implementation of Sampling, Data Collection, and Analysis More Often Had Limitations

We found more limitations in the implementation of evaluations than in their design. On average, about 60 percent of evaluations generally met each of the criteria related to this aspect of quality—sampling, data collection, and analysis. Limitations we identified revealed that conducting evaluations overseas can pose challenges for evaluators. For example, travel to remote areas with safety and security concerns may limit an evaluator’s ability to conduct appropriate sampling and collect primary data for the study. Also, insufficient local resources to implement certain methodologies, such as implementing survey instruments, or a lack of local administrative data on the study population may constitute additional obstacles to sampling and data collection.

About 40 percent of evaluations had limitations in, or provided insufficient information about, their sampling methodology. If an evaluation does not clearly describe how sampling was conducted, it may raise questions about the quality of the evidence, including concerns regarding selection bias of respondents, sufficiency of the sample size for the findings, and the relevance of the evaluation’s findings and conclusions for the entire study population. Sampling methods were particularly problematic or unclear in evaluations that used nonrandom sampling. For evaluations that relied primarily or exclusively on testimonial evidence, the method for selecting participants for interviews or focus group discussions was sometimes inappropriate or unclear. For example, one evaluation we reviewed relied largely on interviews but did not describe the process used for selecting participants, and it indicated that the available list of potential participants was incomplete and inaccurate. Several evaluations provided insufficient information about the target population, other than identifying them as program beneficiaries, and included no discussion of how participants were selected for interviews, focus groups, or surveys.

Limitations in Data Collection Methods About 40 percent of the evaluations had limitations in, or provided insufficient information about, their data collection methods. We identified a number of deficiencies in the data collection process, including a lack of documentation of data collection instruments (DCI), such as questionnaires or structured interview protocols. In cases where evidence was gathered through a DCI, some evaluations were unclear about how the instrument was designed and administered. For example, an evaluation of a program intended to increase access to mobile technologies and improve mothers’ health used a survey to gather data. However, the evaluation did not provide sufficient details about the survey, such as the questionnaire itself or the sampling strategy, for the reader to be able to determine the validity and reliability of the data collected.

In addition, about half of the evaluations did not collect baseline data from which to calculate change after the program was implemented, and about half generally did not set targets, which makes an assessment of progress toward meeting the goals of the program difficult. For example, an evaluation of two community training programs in an Asian country identified the study question but did not collect baseline data and did not establish targets. Although some baseline data may have been gathered by the implementing partner, such information was not used for comparative purposes, making it impossible to assess the net effects of the program.

Further, an estimated 60 percent of the evaluations used data collection procedures that only partially ensured the reliability of the data, or there was not sufficient information to assess data reliability. For example, an evaluation of a small business program in Latin America acknowledged numerous data quality problems, including serious attrition among the group used as a comparison group to program participants. Such limitations raise questions about the strength of the conclusions.

About 40 percent of evaluations did not demonstrate that they had conducted appropriate data analysis. These evaluations often did not specify the analysis methods for each question, such as how interview responses were analyzed. For example, an evaluation of a program serving women living with HIV analyzed data through a content analysis but did not clearly explain the categories created for the analysis or the numbers of individual responses that fell into each category. The lack of clarity in the analysis makes it difficult for the reader to determine whether the findings from this program have broader applicability. Several evaluations relied on focus group discussions but analyzed and reported the percentages of informants expressing the stated views in ways that did not appropriately account for the potential influence of other focus group members on informants’ responses. Evaluations that used some quantitative data analysis also had certain shortcomings. For example, an evaluation reported a statistically significant change from baseline but did not include a discussion of the type of statistical test that supported this result.

Finally, while about 90 percent of the evaluations assessed processes such as program implementation, about half of those evaluations did not establish any criteria, such as evaluation plans, budgets, timeframes, and targets. Without such benchmarks, it is difficult to define what constituted success for the evaluated program.

Evaluation Conclusions and Recommendations Were Generally Supported

The majority of evaluations generally met each of our criteria related to conclusions. These evaluations considered the strengths and limitations of the available evidence from the evaluation’s design and implementation and included conclusions that were generally supported and recommendations that were generally justified.

Conclusions supported by the available evidence. About 70 percent of the evaluations had conclusions that were generally supported by the evidence, and nearly all of the evaluations had conclusions that were partially supported. This indicates that these evaluations did not reach beyond what was supported by the evidence and justified given the limitations.

Recommendations and lessons learned justified by the available evidence. About 75 percent of evaluations with recommendations included evidence that generally supported the recommendations, and all evaluations with recommendations included evidence that at least partially supported them. This indicates that the collected evidence justified the follow-up steps the evaluations recommended.

Independence of Evaluators Was Not Always Clearly Documented

Our analysis found that, in addition to meeting the eight criteria to varying extents, the evaluations did not always provide documentation of the evaluator’s independence and whether there were any potential conflicts of interest. In instances where an evaluation was not conducted by a third party, a statement about conflicts of interest may be especially important to forestall any potential concerns about the evaluator’s impartiality. In all, about 80 percent of the agency evaluations documented that they were conducted by third-party evaluators, while about 13 percent were not conducted by a third-party evaluator, and another 6 percent did not indicate whether they were performed by a third-party evaluator. About 70 percent of HHS evaluations, about 30 percent of State evaluations, and about 40 percent of USAID evaluations included a conflict-of-interest statement, while no DOD, USDA, or MCC evaluations included such a statement. If an evaluation does not address the independence of the evaluation organization and of individual evaluators, questions could arise about the objectivity and reliability of the evaluation’s findings.

Extent to Which Evaluations Met Each Applicable Quality Criterion Varied among Agencies

As table 2 shows, the extent to which the evaluations met each quality criterion varied among the six agencies we reviewed. While our assessments revealed strengths in each agency’s evaluations, all six agencies’ evaluations also showed shortcomings in quality that could limit the agencies’ ability to ensure the effectiveness of foreign assistance based on evaluation results.

Each applicable quality criterion was generally met by a majority of HHS, MCC, and USAID evaluations. However, evaluations for all three agencies scored generally lower on the criteria related to evaluation implementation—that is, the appropriateness of the target population and sampling, data collection, and data analysis. While most HHS, MCC, and USAID evaluations used generally appropriate sampling methods, our analysis showed that overall about half of the evaluations did not use appropriate nonrandom sampling techniques. In addition, we estimate that overall only about half of the three agencies’ evaluations generally used data collection methods that ensured data reliability, and only about 10 to 20 percent of USAID and HHS evaluations generally specified the key assumptions of the data analysis methods used.

DOD’s GT&E program evaluations’ study questions met the first quality criterion—aligning with the program’s goals—but overall did not generally meet the other criteria. For example, we identified weaknesses in the implementation of the evaluations’ designs in terms of target population and sampling, data collection, and analysis. In particular, some evaluations did not describe the target population and did not discuss the methods the evaluators used for their selection of the equipment items they observed or the persons they interviewed. In addition, we found limited discussion about how the data were summarized and analyzed, incomplete baseline metrics, and a lack of targets. Without systematic selection of equipment to observe or respondents to interview, it is difficult to know whether the selections were justifiable and selected in a way that supports the intervention’s objectives and the conclusions drawn. Because of these implementation weaknesses as well as a lack of discussion of study limitations, we determined that the DOD GT&E program evaluations provided only partial support for their conclusions.

State and USDA evaluations each met more than one quality criterion about half the time or less. About half or fewer of State evaluations generally met four criteria: appropriate indicators, appropriate target population and sampling, appropriate data collection, and appropriate data analysis. Regarding the appropriateness of chosen indicators, less than a third of State evaluations had indicators with baselines or established criteria such as plans or budgets, and almost none of the evaluations had indicators with targets against which progress could be assessed. In addition, about 80 percent of State evaluations used a data collection process that did not generally ensure the reliability of the data, and about half of the State evaluations generally did not specify data analysis methods for each question and the key assumptions used in the analysis. State officials noted that their programs are often implemented rapidly in response to specific events, making it difficult to design an evaluation for the program and to gather baseline data. We estimate that overall about half of USDA evaluations had generally appropriate target population and sampling, generally appropriate data analysis, or support for conclusions.

Foreign Aid Evaluation Costs Range Widely and Are Influenced by Methodology, Location, and Evaluation Quality

Most foreign aid evaluations we reviewed cost less than $200,000, but costs ranged widely and varied by agency and type. We identified costs for MCC, State, USAID, and USDA final evaluations but could not obtain specific cost information for DOD’s GT&E and HHS’s PEPFAR evaluations because these programs used procurement methods for their evaluations that did not separately track evaluation costs. Evaluation costs were related to the evaluation’s methodology and location, and higher-cost evaluations tended to meet more evaluation quality criteria, though we also identified lower-cost evaluations that met all quality criteria.

Evaluation Costs Ranged Widely and Varied by Type and Agency, but Most Cost Less Than $200,000

Costs for the majority of the foreign aid evaluations whose costs we reviewed were less than $200,000, but the costs ranged widely and varied by type of evaluation and agency. Of the 76 MCC, State, USAID, and USDA evaluations, 48 cost less than $200,000 while 6 cost more than $900,000. The costs of net impact evaluations ranged from approximately $36,100 to $2.2 million, with a median of $117,500. The costs of performance evaluations ranged from $9,600 to $902,100, with a median cost of $169,600.

While the median cost was higher for the performance evaluations than the net impact evaluations, the net impact evaluations had a higher average cost than the performance evaluations; five of the six evaluations that cost more than $900,000 were net impact evaluations. Net impact evaluations that used randomized controlled trials were the most expensive evaluations in our sample, with a median cost of $926,600 compared with $154,700 for all other evaluations. Figure 2 shows the range of costs for the net impact and performance evaluations whose costs we reviewed.

Of the four agencies’ evaluations whose costs we reviewed, MCC’s evaluations had the highest median cost, at $268,900, and USDA’s evaluations had the lowest median cost, at $87,900 (see table 3). Seven of 12 MCC evaluations cost over $200,000, including 3 net impact evaluations that cost over $900,000. In contrast, 8 of the 10 USDA evaluations were performance evaluations that cost less than $200,000. Most State evaluations were performance evaluations, which were generally more expensive than performance evaluations at the other agencies. USAID costs for impact and performance evaluations both ranged widely, and USAID’s net impact evaluations had a lower median cost than its performance evaluations.

Costs for DOD’s GT&E evaluations and HHS’s PEPFAR evaluations were not specifically identifiable because they were not separately tracked by the agencies, contractors, or implementing partners.

The contract for the DOD GT&E evaluations included many activities in addition to the evaluations and was not structured to show the cost of each activity. Additionally, according to DOD officials, neither DOD nor the contractor separately tracked the evaluation costs in their financial records. However, on the basis of the contractor’s estimate of contract time spent on 20 GT&E evaluations in fiscal years 2012 through 2015 (including the four evaluations in our sample), we estimated the total cost of these evaluations at approximately $1.1 million—an average of approximately $56,300 per evaluation. According to DOD officials, actual costs likely varied across evaluations due to differences in the size of the evaluation teams, the foreign country in which the evaluation took place, and the amount of time each team spent abroad.

HHS’s PEPFAR programs typically conduct evaluations as part of larger cooperative agreements that are not structured to specify evaluation costs. We reviewed cost information for 10 HHS evaluations. We identified a specific cost—$15,400—for only one evaluation, which was conducted under a cooperative agreement specifically for the evaluation; the remaining nine evaluations were conducted as part of cooperative agreements that did not specify evaluation costs. Using budget documents and informed estimates from CDC staff and implementing partners, we estimated that the costs of these nine evaluations ranged from $25,100 to $356,200. Additionally, 11 of the 34 HHS evaluations in our sample had no external costs because they were conducted solely by HHS staff using existing datasets. CDC officials stated that CDC intends to track evaluation costs in the future. For example, according to HHS, upon continuation of a cooperative agreement, an implementing partner will be required to report on progress on its Evaluation and Performance Monitoring Plan, as well as on expenditures to date and plans and budgets for the following year.

Evaluation Methods, Period of Performance, and Location Influence Evaluation Costs

Our analysis found that data collection methods, frequency of data collection, evaluation duration, and evaluation location all affect evaluations’ cost. For example, evaluations that collected data by surveying program beneficiaries had a median cost of $202,500— approximately $74,000 higher than the median cost of those that did not—and evaluations that collected data repeatedly over time had a median cost of $194,500—approximately $44,500 higher than those that did not. Evaluations that took longer to perform also tended to be more expensive. Other factors that might influence costs include unstable locations and evaluations conducted at multiple sites. For example, a performance evaluation conducted in an unstable country cost $365,700 for 78 days of work, and a performance evaluation that conducted data collection in 12 countries cost $902,100. In addition, conducting an evaluation in multiple sites within the same country might increase evaluation costs. For example, a performance evaluation conducted in eight cities and seven states in India cost $407,500, including the cost of a midterm evaluation that was also conducted in multiple cities. These costs greatly exceeded the median costs for all evaluations.

High-Quality Evaluations Tend to Cost More, but Some Lower-Cost Evaluations also Met All Quality Criteria

Our analysis found that high-quality evaluations tend to be more expensive, but well-designed lower-cost evaluations also met the criteria we identified for a high-quality evaluation. Overall, as table 4 shows, the median cost of high-quality evaluations (i.e., evaluations that met all quality criteria) was $137,800 more than the median cost of acceptable- quality evaluations (i.e., evaluations that partially or generally met all quality criteria) and $208,600 more than the median cost of lower-quality evaluations (i.e., evaluations that did not meet, or provided insufficient information for, one or more quality criteria).

High-quality evaluations also tended to include factors associated with higher evaluation costs. For example, the most expensive evaluation in our sample cost $2.2 million and generally met all quality criteria. This net impact evaluation assessed multiple civil society and governance programs in an African country using different methodologies, including a randomized controlled trial, over 4 years and conducted two rounds of surveys of program beneficiaries. Another high-quality evaluation cost $1.4 million and took almost 4 years to complete; this evaluation conducted three surveys of program beneficiaries and used a quasi- experimental methodology to assess the net impacts of energy-efficient stoves in Asia. However, some lower-cost evaluations also met all of the quality criteria. Of the 15 high-quality evaluations for which we identified costs, 4 cost less than $150,000.

Selected Agencies’ Evaluations Are Generally Available Online, but Some Agencies Can Improve Dissemination

We assessed DOD’s, HHS’s, MCC’s, State’s, USAID’s, and USDA’s use of six dissemination practices that federal, AEA, and other guidance indicate agencies should generally use to ensure effective dissemination of evaluations. We found that the agencies varied in their performance of these practices for the fiscal year 2015 evaluations we reviewed (see table 5). All except USDA generally made nonsensitive evaluations publicly available online. These nonsensitive evaluations could generally be located with the agencies’ website search engines. However, some agencies’ evaluations were not posted in a timely manner. Each of the agencies posted its sensitive evaluations internally for access by internal users. Only USAID included dissemination plans in most nonsensitive evaluations to help ensure their dissemination to potential users of the evaluation, but most of the other agencies now require such plans to be prepared for future evaluations. In addition to publicly posting the report, all of the agencies used other means to actively disseminate evaluation findings. Following these practices can help agencies ensure that their evaluation reports are accessible, timely, and useful to decision makers and other stakeholders.

All Agencies except USDA Make Nonsensitive Evaluations Publicly Available Online

Every agency with nonsensitive evaluations requires public, online posting of nonsensitive evaluation documents, and all except USDA publicly posted all of the nonsensitive evaluations we reviewed on publicly accessible websites. Making evaluation reports publically available on their websites helps agencies share evaluation findings with partners, program beneficiaries, and the wider public and facilitates the incorporation of evaluation findings into program management decisions.

We examined the agencies’ dissemination of 193 evaluations. The agencies did not require 22 evaluations to be publicly posted due to their sensitivity—all 4 DOD evaluations as well as 17 evaluations from State and 1 from USAID. Of the remaining 171 nonsensitive evaluations, we found that more than three-quarters (133) were publicly posted. USDA did not publicly post any of its 38 nonsensitive evaluations. According to USDA officials, the department is in the process of developing procedures for making these nonsensitive evaluations public, which would include reviewing the documents to ensure that they did not contain, for example, personally identifiable or proprietary information. Without posting all nonsensitive evaluations online, agencies cannot ensure that the evaluations’ findings reach intended audiences and are available to inform future program design or budget decisions.

Publicly Posted Evaluations Can Generally Be Found with Agency Search Engines

Most of the nonsensitive, publicly posted evaluations we reviewed could be located with a search engine on the agencies’ websites. Providing a search engine that potential evaluation users can employ to find the evaluation reports ensures that users can locate the information they seek, in a format that matches their expectations. Websites at three of the four agencies with publicly posted evaluations—MCC, State, and USAID—have search engines that enable users to find each specific evaluation. The PEPFAR website, which hosts evaluations of PEPFAR programs implemented by CDC and other agencies, has these evaluations listed in a spreadsheet locatable on the site. Many evaluations of PEPFAR programs implemented by CDC can also be found using a search engine at a separate website, called “CDC Stacks.” According to CDC, almost all of the evaluations that we reviewed were posted on the CDC Stacks website. The agency reported that it is in the process of adding the remaining CDC evaluations from fiscal year 2015 to this website.

Some Evaluations Were Not Posted within Required Time Frames

Some of the nonsensitive evaluations we reviewed were not posted on the agencies’ websites within required timeframes. Making evaluation reports accessible in a timely manner ensures that interested parties can access the findings of these evaluations in time to incorporate them into program management decisions. MCC and HHS did not post some evaluations within the timeframes they require, limiting stakeholders’ ability to make optimal use of the evaluation findings.

We found that MCC did not post 10 of its 16 evaluations, as MCC requires, within 6 months after MCC received them, and it did not post 8 of these 10 evaluations until a year or more after MCC received them. According to MCC officials, the agency’s internal evaluation quality review process for evaluations, in which the agency reviews the document before releasing it to the public, has been a major factor in these delays. MCC officials reported that for some of the evaluations—for instance, those written in a language other than English—this process took significantly longer than usual.

HHS did not post 11 HHS evaluations in the timeframe required by the agency. It did not post 6 of these 11 evaluations online within 90 days as required by PEPFAR. PEPFAR guidance requires that evaluations be posted within 90 days of completion, while HHS requires that evaluations be publicly posted within a year of their completion. One HHS official stated that the delay in the posting of these six evaluations was due to the conflicting policies. However, the remaining five evaluations were also not posted within the year as required by HHS/CDC. These five evaluations have since been posted online. Since evaluated conditions may change over time, not posting evaluations online within the required timeframe limits internal and external stakeholders’ access to current, actionable information. In comments on a draft of this report, CDC noted that, as of December 2016, CDC is providing guidance that all evaluations be posted online within 90 days, as required by PEPFAR. CDC published this guidance in January 2017.

All Agencies Have Websites to Make Sensitive Evaluations Available Internally

Of the three agencies with sensitive evaluations—DOD, State, and USAID—all have websites to make these evaluations available to internal stakeholders. While sensitive evaluations are not required to be made available to the public on an agency’s website, disseminating sensitive evaluation findings to the appropriate audience will facilitate their use. DOD, State, and USAID all reported that they have internal websites that can be used to post sensitive evaluations. In addition, State updated its policy for 2015 to require that State officials post a nonsensitive summary of sensitive evaluations on State’s public website. While USDA does not currently publicly post its evaluations, USDA reported that it makes these evaluations internally available through its grant management system. HHS and MCC did not have sensitive evaluations in fiscal year 2015.

USAID Included Dissemination Plans in Most Evaluations, and Other Agencies Will Require Such Plans in the Future

USAID requires the development of dissemination plans and included evidence of such planning in the majority of the evaluations we reviewed, and all of the other agencies except State now require such plans for nonsensitive evaluations. Dissemination planning identifies potential users of an evaluation and describes an approach to providing users with the evaluation results. Such planning can help agencies ensure that evaluation reports are disseminated effectively Among the six agencies, only USAID required the development of dissemination plans for fiscal year 2015 evaluations and included evidence of such planning in the majority of the evaluations whose dissemination we reviewed. Of the 62 USAID evaluations, 44 included evidence that dissemination planning had been completed. HHS, MCC, State, and USDA did not require dissemination plans for their evaluations completed in fiscal year 2015. Agency officials at HHS and MCC provided evidence that dissemination planning took place for at least one of their respective evaluations we reviewed, but this dissemination planning was not required by agency policy, and therefore this planning was ad hoc. DOD plans for evaluation dissemination by identifying potential users of the evaluation and sending e-mails to these internal and congressional stakeholders after the evaluations are completed.

HHS, MCC, and USDA guidance now require dissemination plans for future evaluations. State officials reported that, as of November 2016, State was planning to revise its policy to require the use of dissemination plans for evaluations but had not instituted this requirement. Without dissemination planning, State cannot ensure that its evaluations are disseminated as effectively as possible to potential users.

Agencies Used Additional Means to Actively Disseminate Evaluation Findings

In addition to posting evaluations online, each of the six agencies reported disseminating evaluation findings through other means. Our prior work has shown that taking such additional steps to actively disseminate evaluation reports—for example, briefing stakeholders on evaluation findings and distributing the evaluations to interested stakeholders via e- mail—facilitates dissemination of evaluation report findings and encourages their use.

Agency officials reported using various means besides web posting to disseminate evaluation findings. For example, officials at all six agencies reported using briefings to share evaluation findings with various stakeholders within and outside of the agency. Additionally, HHS, State, MCC, and USAID officials reported that they shared evaluation results with interested parties at various professional conferences. USAID officials stated that the agency also disseminates evaluation findings by posting its evaluations on partner websites, creating video companions to evaluation reports to provide to stakeholders, and posting syntheses of evaluation findings on the agency’s website.

Conclusions

Foreign assistance evaluations can be challenging to implement, but they are an essential tool for guiding agency decision making and allocation of resources. Agencies’ foreign assistance evaluations assess a wide variety of programs around the world, using many different designs and methodologies, and the wide range of evaluation costs reflects this diverse context. However, regardless of the location, design, or cost, an evaluation should provide sufficient and reliable evidence to support its findings. A high-quality evaluation helps agencies and stakeholders identify successful programs to expand or pitfalls to avoid. Evaluations that do not meet all quality criteria that we identified may not provide sufficiently reliable evidence to inform these decisions. In addition, for evaluations to inform decision making, stakeholders must be able to find them. While foreign assistance agencies have generally made their evaluations available online in a timely manner, several agencies can take additional steps to ensure that stakeholders have improved access to these evaluations to make better-informed decisions about future program design and implementation. A growing body of high-quality, broadly disseminated evaluations can help the United States continuously improve its foreign assistance programs and thereby support democracy, enhance security, reduce poverty and suffering, and achieve other U.S. foreign policy goals.

Recommendations for Executive Action

To improve the reliability and usefulness of program evaluations for agency program and budget decisions, we recommend that the Chief Executive Officer of MCC, the Administrator of USAID, the Secretary of Agriculture, the Secretary of Defense, the Secretary of State, and the Secretary of Health and Human Services (in cooperation with State’s Office of the U.S. Global AIDS Coordinator and Health Diplomacy) each develop a plan for improving the quality of evaluations for the programs included in our review, focusing on areas where our analysis has shown the largest areas for potential improvement.

To better ensure that the evaluation findings reach their intended audiences and are available to facilitate incorporating lessons learned into future program design or budget decisions, we recommend that the Secretary of Health and Human Services direct the Centers for Disease Control and Prevention to update its guidance and practices on the posting of evaluations to require PEPFAR evaluations to be posted within the timeframe required by PEPFAR guidance; the Chief Executive Officer of MCC adjust MCC evaluation practices to make evaluation reports available within the timeframe required by MCC guidance; the Secretary of State amend State’s evaluation policy to require the completion of dissemination plans for all agency evaluations; and the Secretary of Agriculture implement guidance and procedures for making FAS evaluations available online and searchable on a single website that can be accessed by the general public.

Agency Comments and Our Evaluation

We provided a draft of this report to DOD, State, HHS, MCC, USAID and USDA for review and comment. DOD, State, HHS, MCC, USAID, and USDA provided official comments, which are reproduced in appendixes III through VIII with, where relevant, our responses. DOD, HHS, and USAID also provided technical comments, which we incorporated as appropriate.

The following summarizes DOD, State’s, HHS’s, MCC’s, USAID’s, and USDA’s official comments and our responses.

DOD stated that it partially concurred with our recommendation, noting that in many cases, certain methodologies are not well suited for security assistance evaluation. DOD observed that, for example, it would be unethical to establish randomized control groups for security assistance evaluations and that foreign military organizations may be unwilling to provide DOD significant access to some military units solely for the purpose of the evaluation. We recognize that certain methodologies are not appropriate in every context, and we did not advocate the use of randomized control groups in the DOD evaluations we reviewed. Our main concerns about the DOD evaluations focused on implementation of the methods used. In particular, we found limitations in sampling methods including descriptions of the target population, data collection methods, and data analysis. We adjusted pertinent wording in our report to clarify these points.

State concurred with our recommendations and noted that its forthcoming Program Design and Performance Management Policy for Programs, Projects, and Processes, recently published Program Design and Performance Management toolkit, and updated policy guidance will constitute a plan for improvement. We will monitor the implementation of this plan to verify that State takes appropriate steps to address our recommendation.

HHS concurred with our recommendation that it update guidance and practices on the posting of PEPFAR evaluations and stated that CDC guidance now requires evaluation reports to be posted on a publically accessible website within 90 days of the evaluation’s completion. HHS did not comment on our recommendation that it develop a plan for improving the quality of evaluations.

MCC stated that it welcomed our findings and recommendations for improvement but noted that it could not agree or disagree with our quality assessments because we did not provide data on our determinations for individual evaluations. In response to our observation that MCC evaluations did not contain conflict-of-interest statements, MCC noted that it has required independent third-party evaluation of all its projects since 2009 and that, in 2013, it standardized the language in its evaluation contracts to explicitly establish the independent role of evaluators. While these are positive steps, we believe that including in MCC’s published evaluations explicit statements about the evaluators’ independence and any potential conflicts of interest would bolster the evaluations’ credibility and usefulness. With regard to the timeliness of public access to its evaluations, MCC indicated that when it established its internal review process for evaluations in 2013, it did not anticipate the length of time that would be required to finalize evaluation reports. MCC noted that its forthcoming revised policy on monitoring and evaluation states that “MCC expects to make each interim and final evaluation report publicly available as soon as practical after receiving the draft report.” However, the revised policy does not establish a target time frame for completing internal reviews of the reports. Establishing such a time frame could help MCC ensure that evaluation reports are published in a timely fashion that maximizes their usefulness.

USAID stated that it has established a plan to improve the quality of evaluations, including an update and clarification of the requirements and quality standards for evaluations. USAID also stated that it plans to provide additional training and other capacity-building efforts to help ensure that staff have the necessary skills to manage evaluations. We will monitor implementation of this plan to verify that USAID takes appropriate steps to address our recommendation.

USDA agreed with our recommendations. To address the recommendations, USDA stated that it would update its guidance on reviewing evaluation terms of reference to include a section on quality that specifically focuses on the areas where the GAO analysis has shown the largest areas for potential improvement. USDA further stated that FAS will continue its current efforts to make nonsensitive evaluations publicly available online and will make them searchable as well.

We are sending copies of this report to the appropriate congressional committees and to the Secretaries of Agriculture, Defense, State, and Health and Human Services; the Chief Executive Officer of the Millennium Challenge Corporation; and the Administrator of the U.S. Agency for International Development. In addition, the report will be available at no charge on GAO’s website at http://www.gao.gov.

If you or your staff have questions about this report, please contact me at (202) 512-6991, or farbj@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. GAO staff who made major contributions to this report are listed in appendix IX.

Appendix I: Objectives, Scope, and Methodology

In response to congressional requests, we examined (1) the extent to which foreign assistance program evaluations met key evaluation quality criteria; (2) the costs of the evaluations, as well as factors that affect these costs; and (3) the extent to which the agencies ensure the dissemination of evaluation reports within the agency and to the public.

To address our objectives, we identified the six major agencies administering the most foreign assistance on the basis of obligations reported to the U.S. Agency for International Development’s (USAID) U.S. Overseas Loans and Grants database for fiscal years 2008 through 2012. The six agencies we identified are USAID, the Department of State (State), the Millennium Challenge Corporation (MCC), the Department of Health and Human Services (HHS), the U.S. Department of Agriculture (USDA) and the Department of Defense (DOD). For the three agencies that are not focused exclusively on foreign aid or foreign affairs (HHS, USDA, and DOD), we limited our scope to selected programs. For HHS and USDA, we selected programs that account for the vast majority of foreign assistance program dollars that the agency implemented. At HHS we examined evaluations of the President’s Emergency Plan for AIDS Relief (PEPFAR) programs implemented by HHS’s Centers for Disease Control and Prevention (CDC). At USDA we examined evaluations for the Food for Progress and McGovern-Dole food assistance programs, implemented by the Foreign Agricultural Service (FAS). At DOD we examined evaluations prepared for the Global Train and Equip (GT&E) program. While our previous review of agency evaluation policies did not identify DOD-wide evaluation policies, we did identify GT&E as having relevant policies to guide its evaluations.

To identify evaluations completed in fiscal year 2015, the most recently completed fiscal year as we undertook our review, we requested that each agency provide a list of all foreign aid evaluation reports completed in that year. We did not separately review agency files to identify if agencies had additional evaluations beyond those listed in the registries. To assess the reliability of the agency evaluation lists, we reviewed the documents provided to ensure that each was a completed evaluation and to confirm that the date of the document fell within our specified timeframe. If necessary, we followed up with agency officials to clarify the date or status of the document. Based on their responses, we removed documents that were not evaluations or fell outside of our timeframe. We also did not review evaluation reports that were not written in English. We determined that the data in the evaluation lists were sufficiently reliable for the purposes of this engagement. In all, we identified a study population of 361 evaluations: 4 DOD evaluations, 51 HHS evaluations, 17 MCC evaluations, 28 State evaluations, 221 USAID evaluations, and 40 USDA evaluations. We examined the evaluations themselves and any appendices that the agency provided which were directly referred to in the evaluations. We did not consider evaluation plans and protocols, underlying documents and other work papers as evidence that the planned design was implemented. Similarly, we did not consider contracts with third-party evaluators or evaluation organizations as evidence that the evaluator had maintained independence. Instead we required statements in the reports or methodological appendices that steps and procedures were actually taken and that no threats to independence have been identified.

From the study population of fiscal year 2015 evaluations, we reviewed all DOD, MCC, and State evaluations; all USAID net impact evaluations; and a sample of HHS, USDA, and USAID performance evaluations. We randomly selected a probability sample from the study population of HHS, USDA, and USAID performance evaluations. With this probability sample, each member of the study population had a nonzero probability of being included, and that probability could be computed for any member. For USAID, we included all net impact evaluations in the sample because net impact evaluations constituted less than 20 percent of all the evaluations provided, and if we had not included them all, we would not have been able to comment on this type of evaluation.

Based on the review of the evaluation documents after the initial screening, an additional 16 evaluations were found not to be within our scope, and we substituted for these evaluations when possible. For example, we excluded documents that did not evaluate a specific program, were monitoring or grant reports, or were plans for an evaluation rather than an evaluation report. We included two substitute HHS and two substitute USDA evaluations to replace those that were found to not be in scope and also reviewed additional USDA evaluations. However, because we had initially included all MCC, State, and USAID net impact evaluations, there were no additional evaluations available to substitute if those were excluded. The original sample and the final respondents across the six agencies can be found in table 6. Each sample selection was subsequently weighted in the analysis to represent the evaluations in the population that were not selected.

We reviewed the full population of DOD, MCC, and State evaluations; therefore, our results from the quality review of these evaluations do not have an associated margin of error. The results from our review of the HHS, USDA, and USAID evaluations are reported with an associated margin of error. Because we followed a probability procedure based on random selections, our sample is only one of a large number of samples that we might have drawn. Since each sample could have provided different estimates, we express our confidence in the precision of our particular sample’s results as a 95-percent confidence interval. This is the interval that would contain the actual population value for 95 percent of the samples we could have drawn. All percentage estimates for aggregated results from our review have margins of error at the 95 percent confidence level of plus or minus 8 percentage points or less, unless otherwise noted, and all percentage estimates for individual agencies from our review have margins of error at the 95 percent confidence level of plus or minus 11 percentage points or less, unless otherwise noted.

To assess the extent to which the results of foreign assistance program evaluations are supported by their evidence and whether they assess if programs have met their goals, we assessed the sample of agency fiscal year 2015 evaluation reports against quality criteria we identified. We identified these criteria based on our review and analysis of evaluation guidance from agencies included in our review (including any agency internal evaluation review checklists), international organizations, evaluation organizations, and prior GAO reporting. These criteria include necessary high-level elements in designing, implementing and reporting on evaluations that could serve as standards across different agencies and evaluation types. Prior to undertaking our quality review, these criteria were discussed and reviewed within the engagement team, as well as by other GAO staff with experience in program evaluation and methodologies.

We incorporated the identified criteria into a standardized data collection instrument (DCI) in order to consistently review the sampled evaluation reports. The DCI contained evaluative questions against which to assess evaluation quality as well as descriptive questions to gather information about the evaluations, such as its location and methodology. The high- level criteria each included subquestions about elements the reviewer should consider in making his or her overall decision. The evaluation quality criteria were judged on a four-part scale for most of the judgmental questions: generally addressed: the evaluation mostly addressed the key element(s) of the criterion but did not have to completely address all elements in the subquestions; partially addressed: the evaluation had one or more clear area(s) for improvement on the criterion; not at all addressed: the evaluation did not show that steps were taken to address the criterion; and insufficient information: reviewers could not make a determination due to a lack of information in the evaluation and any other associated materials.

If a criterion was partially or not at all addressed, or if there was insufficient information in the evaluation to assess the criterion, we considered it a deficiency. We did not consider study protocols or design documents that indicated plans for a particular evaluation step as sufficient evidence that such a step was performed unless the evaluation report also provided evidence that it had.

The descriptive questions in the DCI about evaluation types and methodology were based on prior GAO work and asked about designs that examined net impacts of interventions, outcomes of interventions, and processes. The questions on net impact evaluations asked about the type of design using four categories: (1) randomized controlled trials or groups, (2) comparison groups, (3) time series that would allow for trends to be determined pre- and post- intervention, and (4) quasi-experimental statistical modelling techniques. The questions on outcome and process evaluations asked whether baselines and targets had been established for the outcomes and whether criteria had been established to assess processes. From these types of evaluation, we created two broad categories to use in our analysis of evaluation cost and quality: net impact evaluations, and performance evaluations. The net impact category included all four impact design types, while the performance evaluation category included outcome and process evaluations as well as a few evaluations that did not fall within the outcome and process categories. The performance evaluations included some that had established targets or baselines and others that had not, while the process evaluations included some that had established criteria for assessment and others that had not. We noted some overlap between the net impact, performance, and process categories. For example, net impact evaluations often considered outcomes or processes, and performance evaluations often considered both outcomes and processes. This overlap was a key reason we decided to develop and analyze two broad types of evaluation categories rather than attempt to develop more refined types.

A key assumption underlying our analysis was that different types of evaluations were appropriate for different types of study questions. The evaluative questions in the DCI were relative, rather than absolute, with respect to the study questions and were intended to be applicable across evaluation types. We did not assess the study questions in terms of their scope or rigor. We instead took the study questions as given, thereby giving every evaluation an equal chance to receive a high score if its design and implementation were appropriate for the study questions. Evaluation reviewers were instructed to consider design, implementation, and reporting in terms of the study questions the evaluations set out to answer rather than against an absolute standard. In this way, we determined that it would be as possible for a qualitative midterm evaluation that considered program implementation to achieve high scores as it would for a final net impact evaluation that considered effects attributable to the program.

The main criteria questions in the DCI asked about the appropriateness of the design, implementation, and conclusions in light of the study objectives. We did not determine a single definition of appropriateness because we recognized that it is dependent on the study objectives and data collection conditions. For example, the standards for appropriateness of a final net impact evaluation of a pilot health care program that seeks to establish whether the intervention is achieving positive outcomes are different from the standards for a mid-term performance evaluation of a well-established water and sanitation program supported by a solid evidence base and focused on whether the program was implemented as planned. Given the variation in agency goals and programs, evaluation types, and evaluation timing, we determined that we would rely on expert professional judgment rather than attempt to use a single definition of appropriateness for every situation.

While we designed our DCI to apply broadly across agencies and evaluation types, differences in agency evaluation practices and areas of responsibility may limit comparisons between the agencies. For example, the target audience of an evaluation may determine whether it includes certain reporting elements. HHS’s PEPFAR evaluations are generally produced for dissemination in research publications and journals, while USAID evaluations evaluate a wide range of programs and are generally directed to an audience of program officials and managers. In addition, the evaluations we reviewed assessed a wide range of foreign assistance programs with varying characteristics. These characteristics include the nature of the foreign assistance intervention, the type of program responsible for the intervention, whether the program was designed to be evaluated, and the timing of the evaluation. For example, some evaluations consider ongoing development assistance in areas such as education or health, while others consider emergency responses to humanitarian crises. Agency officials noted that it could be harder to ensure quality in an evaluation of a program that had to respond quickly to a crisis and therefore did not have the opportunity to plan for an evaluation. They also noted that if an evaluation is not started until after the program has begun, there may not be any baseline data available.

The evaluation review consisted of multiple reviews by a team of GAO staff with experience and familiarity with research methods as well as with reviewing studies and evaluations across a wide range of subject areas and disciplines. After completing his or her initial review, the first reviewer notified the second reviewer that the evaluation report was available for his or her review. The second reviews were not independent; as the second reviewer saw the decisions made by the first reviewer and could review the first reviewer’s notes on sources and justifications for his or her decisions. The second reviewers read the evaluation and indicated whether he or she agreed with the first reviewers’ decisions or whether he or she proposed another decision. The first and second reviewers subsequently met to reconcile any differences. After the reconciliations were completed, a supervisor then reviewed the work of the two reviewers for internal consistency and completeness according to a standard protocol but did not re-review the evaluation documents. The supervisor related the identified issues as needed to the first and second reviewers, who addressed them before the supervisor recorded the review as final.

We took several steps to ensure consistency among the reviewers. We conducted two pretests of the DCI on sample evaluations. The first pretest included members of the engagement team as well as GAO staff with experience in the design of survey instruments or in the review of foreign assistance evaluations. The second pretest included members of the evaluation review team. After each round of pretests, we made appropriate revisions to the DCI. To help ensure consistency of interpretation, we created a guidance document where reviewers recorded questions about certain decision rules to follow in specific instances. Answers to the questions were then posted after discussion among the review and engagement team members. Additionally, the engagement and review team held regular weekly meetings to discuss any methodological issues that arose and preliminary tabulations of the review data.

To analyze the responses to the DCI, we examined how evaluations’ quality varied by the eight quality criteria, by agency, by timing of the evaluation relative to the stage of program implementation (midterm or interim vs. final), and type of evaluation (net impact vs. performance). While there were some differences between the final and midterm evaluations, these were not statistically significant. However, a higher percentage of evaluations that attempted to assess net impacts were of high quality than those that did not attempt to assess net impacts. We determined that these differences were due primarily to specific weaknesses in the implementation of the design of the evaluations. For example, performance evaluations used nonrandom sampling more often than net impact evaluations, which used at least some random sampling. While our DCI and subsequent analysis treated both methods of sampling equally, we focused our assessment on the extent to which each method had been appropriate for the study questions. We found that the performance evaluations’ nonrandom sampling was carried out appropriately less often than the random sampling typically used in net impact evaluations. In the body of our report, therefore, we focus on the specific weaknesses that we found, such as the one regarding nonrandom sampling, rather than differences at the level of the two broad evaluation types.

Our analysis found a reasonably high degree of overlap between several approaches that we considered for categorizing the evaluations. For example, we categorized the evaluations into three groups: high quality, acceptable quality, and lower quality based on the number of quality criteria that were generally or partially met for each evaluation, as well as instances when quality areas were either not met or there was insufficient information to determine if a certain criterion was met by a particular evaluation. Those evaluations that fell into the lowest and middle categories based on these three categories also generally fell into the lowest and middle categories using another approach that we examined, as table 7 shows. This comparison also shows that some evaluations in the highest category had a relatively higher number of criteria generally met, while some in the middle category had a relatively lower number of criteria generally met.

To determine the cost of foreign aid evaluations, we reviewed contracts, invoices, and related documents to determine the cumulative cost of final evaluations that were conducted by an outside evaluator. We defined the cumulative cost as the cost of the final evaluation and any related activities that informed the evaluation’s findings, such as a separate data collection effort or a midterm or baseline evaluation. We did not determine the cost of evaluations that had a midterm evaluation but not a final evaluation because midterm evaluation costs do not reflect the total cumulative cost of evaluating a program. State and USAID officials noted that some of their midterm evaluations may ultimately be a program’s only evaluation. We did not review the State and USAID fiscal year 2015 midterm evaluations to determine if the agency intends to conduct an additional, final evaluation or include these midterms in our cost analysis. Table 8 shows the total number of evaluations in GAO’s quality sample, the number of those evaluations whose costs we reviewed, and the number of evaluations whose costs we reviewed that we included in the statistical analysis by agency.

We used contracts and invoices to determine the cost of 42 of the 76 MCC, State, USAID, and USDA final evaluations. To determine an evaluation’s cost, we used either the final invoiced amount or the contract’s total obligations. For each evaluation, we also read the statement of work to determine if the contract covered only the evaluation in our sample or if it covered additional activities as well. In some cases, while the contract covered additional activities, the evaluation’s cost was clearly identifiable in a separate line item or invoice. We identified the evaluation start and end dates using the period of performance in the contract or statement of work. In some cases we determined the period of performance using dates in the evaluation report if the contract’s period of performance covered a broader time period than the evaluation in our sample.

For six USAID evaluations and nine State Department evaluations, we determined the evaluation cost using data from the Federal Procurement Data System – Next Generation (FPDS-NG). FPDS-NG provides a contract’s obligations, start and end dates, and other descriptive data. In each case, we confirmed that the contract covered only the evaluation in our sample by reviewing the statement of work or by confirming with agency officials. We used total obligations to determine the evaluation’s cost and also used the date signed and end date listed in FPDS-NG to determine the period of performance. To assess the reliability of the FPDS-NG data, we (1) reviewed related documentation, (2) traced to or from source documents, and (3) confirmed FPDS-NG data with knowledgeable agency officials. We determined that the FPDS-NG data were sufficiently reliable for the purposes of this engagement.

For MCC, State, and USAID evaluations without clearly identifiable cost information from contract documents or FPDS-NG, we estimated the cost based on budget documents or cost estimates provided by the agency or contractor, where available. We relied on budgets or cost estimates to determine the cost of 5 MCC evaluations, 4 State Department evaluations, and 10 USAID evaluations. We excluded one MCC evaluation from the cost sample because MCC provided a wide range for the estimated cost, and we concluded that this range was not sufficiently reliable to report. We could not determine the cost of one State Department evaluation and one USAID evaluation that were each procured under large agreements that did not separately track evaluation costs. Additionally, USAID officials did not provide cost information for one evaluation.

We report only limited data on the cost of DOD’s GT&E and HHS’s PEPFAR evaluations because the evaluation contracts or implementing partner agreements did not separately track evaluation costs, and we concluded that the available estimates were too limited to include in our statistical analysis. To estimate the cost of DOD’s GT&E evaluations, we reviewed the associated contract and invoices, which included the evaluations as well as additional services. Since the contract and related documents did not contain a separate line item for the evaluations, we requested a cost estimate from agency officials and the contractor. The contractor was able to provide only the broad estimate that we include in our report with appropriate caveats but which we concluded was not sufficiently reliable to include in our statistical analysis. The costs of the HHS PEPFAR evaluations were also not separately tracked by the agency and implementing partners. Evaluation costs were instead estimated by HHS country teams or implementing partners based on their review of previous years’ financial records, budgets, or cooperative agreements. Because of the volume of records involved, we judgmentally selected a subsample of 10 HHS evaluations to review the cost estimates provided by HHS officials. To review these estimates, we traced the estimates to source documentation and spoke with knowledgeable agency officials to understand the methodology used to prepare the source estimates. Because of the uncertainty of these cost estimates, we include them in our report with appropriate caveats but concluded that they were not sufficiently reliable to include in our statistical analysis.

To determine the factors that are associated with the costs of foreign aid evaluations, we analyzed the costs of MCC, State, USDA, and USAID evaluations in relation to the data that we collected on these evaluations’ quality scores, duration, and other characteristics. We then produced summary statistics showing the cost differences of various characteristics. For example, we compared the average cost of evaluations with a survey to those without surveys. We conducted difference-in-means tests to determine if any of the characteristics were statistically significant at the 95-percent confidence level and reported characteristics that were significantly related to costs. We also reviewed the evaluations to obtain insights into other likely cost factors, such as unstable locations and the number of sites, for which systematic data were not available for difference-in-means tests. We included location among the characteristics we considered after observing that evaluations that were more costly than others that were of the same type, or required the same performance period to complete, tended to be conducted in unstable or multiple locations.

To assess our third objective, we identified leading practices for the dissemination of evaluation findings. We identified these leading practices using federal guidance, including the President’s Open Government Directive and Office of Management and Budget (OMB) guidance, which encourages or requires the timely public posting of agency information on a searchable website, as well as plans and additional efforts to actively disseminate agency information. In addition to the federal guidance, we also used the American Evaluation Association’s (AEA) An Evaluation Roadmap for a More Effective Government (AEA Roadmap); the Organization for Economic Co-operation and Development, Development Assistance Committee’s (OECD DAC) Quality Standards for Development Evaluation and Evaluating Development Activities: 12 Lessons from the OECD DAC; and HHS and the General Services Administration’s (GSA) Research-Based Web Design and Usability Guidelines that cite timely public posting, dissemination planning, and additional active efforts to disseminate results as important communication tools for evaluations. We used these sources to identify six practices that agencies should use in order to successfully disseminate the results of foreign aid evaluations. We reviewed the dissemination of all evaluations from fiscal year 2015 for five of the agencies and a sample of USAID evaluations. In total, we examined the dissemination of 193 evaluations: 4 at DOD, 49 at HHS, 16 at MCC, 23 at State, 63 at USAID, and 38 at USDA.

To assess the availability and timeliness of the dissemination of evaluation reports, we reviewed agency policies and websites and interviewed agency officials. We reviewed agency evaluation websites to determine if the evaluation reports in agency evaluation lists had been publicly posted. If an evaluation report had not been posted, we followed up with agency officials regarding the reasons it had not been. We also reviewed the evaluation reports to ensure the documents contained the information necessary for a user to determine if the findings were valid. For example, we reviewed evaluations to ensure that any related annexes had been included when the document had been posted. We examined each agency website to determine whether it provided a search engine that could be used to locate evaluations. We also checked whether the search engine included additional search filters such as the year the evaluation was completed or its location. To assess timeliness, we reviewed agency policies and guidance to determine how soon it required evaluation reports to be posted after the completion of the report. We compared the date an evaluation was considered complete by the agency to the date that it was posted online to determine whether it had been posted within the timeframe required by the agency. To determine whether sensitive evaluations were made available to identified stakeholders via an internal digital system, we reviewed agency lists of sensitive evaluations and interviewed agency officials about agency processes for making sensitive evaluations available internally. We also received an in-person demonstration of the internal posting of USAID’s sensitive evaluations and documentation of the internal systems that DOD and USDA use to post evaluations.

To assess agency dissemination planning and its use of additional means for dissemination, we interviewed agency officials and reviewed agency policies, practices, and evaluation documents. To determine if the agency required dissemination planning, we reviewed the dissemination requirements in its evaluation guidance. If the agency required dissemination plans, we reviewed its evaluation reports, contracts, and related documents to determine if they included an identification of the potential users of an evaluation and a description of the approach that will provide users with the evaluation results. If the agency guidance did not require dissemination plans, we asked the agencies if dissemination planning had occurred without the policy in place. If such ad hoc planning had occurred, we asked that the agencies provide examples. We also provided written questions to agency officials regarding additional agency practices for disseminating evaluations other than posting the evaluation online. If agency officials identified additional means of dissemination, we reviewed additional documentary evidence that evaluation findings had been disseminated using these means.

We conducted this performance audit from October 2015 to March 2017 in accordance with generally accepted government auditing standards. Those standards require that we plan and perform the audit to obtain sufficient, appropriate evidence to provide a reasonable basis for our findings and conclusions based on our audit objectives. We believe that the evidence obtained provides a reasonable basis for our findings and conclusions based on our audit objectives.

Appendix II: Evaluation Review Data, by Agency

Agencies varied in the extent to which they met the applicable quality criteria for evaluations that we identified. Tables 9 through 26 below provide further detail on the characteristics and quality of the design, implementation, and conclusions of fiscal year 2015 evaluations we reviewed summarized for all six agencies and then individually for (1) the President’s Emergency Plan for AIDS Relief (PEPFAR) programs implemented by the Centers for Disease Control and Prevention (CDC) of the Department of Health and Human Services (HHS), (2) the Millennium Challenge Corporation (MCC), (3) the Department of State (State), (4) the U.S. Agency for International Development (USAID) , and (5) the U.S. Department of Agriculture’s (USDA) Foreign Agricultural Service’s food aid programs.

Appendix III: Comments from the Department of Defense

GAO Comment

DOD partially concurs with our recommendation and notes that in many cases certain methodologies are not well suited for security assistance evaluation. DOD observed that, for example, it would be unethical for DOD to establish a randomized control group for security assistance evaluation and that some foreign military organizations may be unwilling to provide significant access to military units solely for the purpose of an evaluation. We recognize that certain methodologies are not appropriate in every context, and we do not advocate the use of randomized control groups in the evaluations we reviewed for DOD. Our main concerns about the DOD evaluations focus on implementation of the methods used. In particular, we found limitations in sampling methods, including descriptions of the target population; data collection methods; and data analysis. We adjusted pertinent wording in our report to clarify these points.

Appendix IV: Comments from the Department of Health and Human Services

Appendix V: Comments from the Millennium Challenge Corporation

GAO Comments

1. MCC notes that it has required independent third-party evaluation of all its projects since 2009 and that, in 2013, it standardized the language in its independent evaluation contracts to explicitly define an independent role for evaluators. While these are positive steps, we believe that including in MCC’s published evaluations explicit statements about the evaluators’ independence and any potential conflicts of interest would bolster the evaluations’ credibility and the usefulness. 2. MCC states that it had not anticipated the length of time required by the review process for all evaluations that it implemented beginning in 2013. MCC notes that its forthcoming revised policy on monitoring and evaluation will state that “MCC expects to make each interim and final evaluation report publicly available as soon as practical after receiving the draft report.” This revised guidance does not set a specific time frame for the reviews. While agency review efforts may help ensure quality, a specific target for the length of time for the reviews would provide a metric for assessing whether reports are being published in a timely fashion that maximizes their usefulness.

Appendix VI: Comments from the Department of State

Appendix VII: Comments from the U.S. Agency for International Development

Appendix VIII: Comments from the U.S. Department of Agriculture

Appendix IX: GAO Contact and Staff Acknowledgments

GAO Contact

Staff Acknowledgments

In addition to the contact named above, James Michels, Assistant Director; Thomas Beall, Miranda Berry, Anthony Costulas, Gergana Danailova-Trainor, Martin De Alteriis, Neil Doherty, Mark Dowling, Laurie Ekstrand, Justin Fisher, Georgette Hagans, Kay Halpern, Reid Lowe, Luann Moy, Barry Seltser, Stephanie Shipman, Michael Simon, Douglas Sloane, and Gregory Wilmoth made key contributions to this report.