President's Emergency Plan for AIDS Relief: Agencies Can Enhance Evaluation Quality, Planning, and Dissemination
Published: 2012-05-31T00:00:00

Fast Facts:

Highlights:
section_title: Why GAO Did This Study
paragraphs: PEPFAR, reauthorized by Congress in fiscal year 2008, supports HIV/AIDS prevention, treatment, and care overseas. The reauthorizing legislation, as well as other U.S. law and government policy, stresses the importance of evaluation for improving program performance, strengthening accountability, and informing decision making. OGAC leads the PEPFAR effort by providing funding and guidance to implementing agencies, primarily CDC and USAID. Responding to legislative mandates, GAO (1) identified PEPFAR evaluation activities and examined the extent to which evaluation findings, conclusions, and recommendations were supported and (2) examined the extent to which PEPFAR policies and procedures adhere to established general evaluation principles. GAO reviewed these principles as well as agencies policies and guidance; surveyed CDC and USAID officials in 31 PEFAR countries and 3 regions; and analyzed evaluations provided by OGAC, CDC, and USAID.
section_title: What GAO Found
paragraphs: The Department of States (State) Office of the U.S. Global AIDS Coordinator (OGAC), the Department of Health and Human Services (HHS) Centers for Disease Control and Prevention (CDC), and the U.S. Agency for International Development (USAID) have evaluated a wide variety of Presidents Emergency Plan for AIDS Relief (PEPFAR) program activities, demonstrating a clear commitment to evaluation. However, GAO found that the findings, conclusions, and recommendations were not fully supported in many PEPFAR evaluations. Agency officials provided nearly 500 evaluations addressing activities ongoing in fiscal years 2008 through 2010 in all program areas relating to HIV/AIDS treatment, prevention, and care. GAOs assessment of a selected sample of seven OGAC-managed evaluations found that they generally adhered to common evaluation standards, as did most of a selected sample of 15 evaluations managed by CDC and USAID headquarters. Based on this assessment, GAO determined that these evaluations generally contained fully supported findings, conclusions, and recommendations. However, based on a similar assessment of a randomly selected sample taken from 436 evaluations provided by PEPFAR country and regional teams, GAO estimated that 41 percent contained fully supported findings, conclusions, and recommendations, while 44 percent contained partial support and 15 percent were not supported.
State, OGAC, CDC, and USAID have established detailed evaluation policies, as recommended by the American Evaluation Association (AEA). However, PEPFAR does not fully adhere to AEA principles relating to evaluation planning, independence and qualifications of evaluators, and public dissemination of evaluation results. Specifically, OGAC does not require country and regional teams to include evaluation plans in their annual operational plans, limiting its ability to ensure that evaluation resources are appropriately targeted. Further, although OGAC, CDC, and USAID evaluation policies and procedures provide some guidance on how to ensure evaluator independence and qualifications, they do not require documentation of these issues. GAO found that most PEPFAR program evaluations did not fully address whether evaluators had conflicts of interest and some did not include detailed information on the identity and makeup of evaluation teams. Finally, although OGAC, CDC, and USAID use a variety of means to share evaluation findings, not all evaluation reports are available online, limiting their accessibility to the public and their usefulness for PEPFAR decision makers, program managers, and other stakeholders.
section_title: What GAO Recommends
paragraphs: GAO recommends that State work with CDC and USAID to (1) improve adherence to common evaluation standards, (2) develop PEPFAR evaluation plans, (3) provide guidance for assessing and documenting evaluators independence and qualifications, and (4) increase online accessibility of evaluation results. Commenting jointly with HHSs CDC and USAID, State agreed with these recommendations and noted steps it will take to implement them.

section_title: Letter
paragraphs: Through the multibillion-dollar President’s Emergency Plan for AIDS  Relief (PEPFAR), the United States has supported significant advances in  global HIV/AIDS prevention, treatment, and care. Since the program was  first authorized in 2003, the estimated number of new HIV infections and  AIDS-related deaths has steadily declined while millions of people in low-  and middle-income countries have received antiretroviral treatment. Yet  for every person placed on treatment, an estimated two people are newly  infected with HIV, and the number of people living with HIV expanded  from about 28 million in 2001 to 34 million in 2010.
Congress reauthorized PEPFAR in 2008 through passage of the Tom  Lantos and Henry J. Hyde United States Global Leadership Against  HIV/AIDS, Tuberculosis, and Malaria Reauthorization Act of 2008 (2008  Leadership Act), which sets multiyear targets for prevention, treatment,  care, and health systems strengthening programs supported through  PEPFAR through fiscal year 2013. The 2008 Leadership Act stated,  among other things, that assistance provided to combat HIV/AIDS shall  expand impact evaluation and other research and analysis efforts to  improve accountability, increase transparency, measure the outcomes  and impacts of interventions, ensure the delivery of evidence-based  services, and identify and replicate effective models. The Government  Performance and Results Act of 1993, amended in 2010 as the  Government Performance and Results Modernization Act, also  encourages evaluation of federal programs. Moreover, since 2002, the  Office of Management and Budget has set expectations for agencies to  conduct program evaluations as essential tools for improving program  design and operations, determining whether intended outcomes are  achieved effectively, and informing decision making.
Responding to requirements in the Consolidated Appropriations Act of  2008 and the 2008 Leadership Act to review global HIV/AIDS program  monitoring, this report (1) identifies PEPFAR evaluation activities and  examines the extent to which evaluation findings, conclusions, and  recommendations are supported and (2) examines the extent to which  PEPFAR policies and procedures adhere to established general  principles for the evaluation of U.S. government programs.
To address these objectives, we reviewed the American Evaluation  Association’s (AEA) An Evaluation Roadmap for a More Effective  Government (AEA Roadmap) as well as policies and guidance  developed by the Department of State (State), State’s Office of the U.S.  Global AIDS Coordinator (OGAC), the Department of Health and Human  Services’ (HHS) Centers for Disease Control and Prevention (CDC), and  the U.S. Agency for International Development (USAID). We conducted  interviews with officials at OGAC, USAID, and CDC. We also surveyed  CDC and USAID headquarters officials as well as CDC and USAID  officials in the 31 countries and 3 regions that had PEPFAR annual  operational plans in fiscal year 2010 about which of their PEPFAR- funded activities operating in fiscal years 2008 through 2010 had ongoing  or completed evaluations. In addition, we obtained electronic copies of  completed evaluations for programs operating during this time period  from CDC and USAID officials at headquarters and in the PEPFAR  countries and regions. Using a standard assessment tool, we  systematically assessed the level of support for findings, conclusions, and  recommendations in samples of these evaluations, as indicated by the  degree to which they were conducted in adherence with selected  common evaluation standards. We assessed judgmental samples of  evaluations submitted by OGAC and by CDC and USAID headquarters.  We assessed a randomly selected sample of the evaluations submitted  by PEPFAR country and regional teams, in order to generalize our  assessment results to all of the submitted evaluations. Finally, we  assessed State, OGAC, CDC, and USAID policies and practices against  selected general principles of evaluation defined in the AEA Roadmap.  (See app. I for a detailed description of our objectives, scope, and  methodology.)
We conducted this performance audit from October 2011 to May 2012 in  accordance with generally accepted government auditing standards.  Those standards require that we plan and perform the audit to obtain  sufficient, appropriate evidence to provide a reasonable basis for our  findings and conclusions based on our audit objectives. We believe that  the evidence obtained provides a reasonable basis for our findings and  conclusions based on our audit objectives.
subsections: section_title: Background
paragraphs: OGAC establishes overall PEPFAR policy and program strategies and  coordinates PEPFAR program activities. In addition, OGAC allocates  PEPFAR resources from the Global Health and Child Survival account to  PEPFAR implementing agencies, primarily CDC and USAID. The  agencies execute PEPFAR program activities through agency  headquarters offices and interagency teams consisting of PEPFAR  implementing agency officials in the countries and regions with PEPFAR- funded programs (PEPFAR country and regional teams). OGAC  coordinates these activities through its approval of operational plans,  which serve as annual work plans and document planned investments in,  and the anticipated results of, HIV/AIDS-related programs. OGAC  provides annual guidance on how to develop and submit operational  plans.
In fiscal years 2009 through 2011, OGAC approved operational plans  representing $11.7 billion in PEPFAR program activities. These activities  fall primarily in three broad program areas—prevention, treatment, and  care—and 18 related program areas. Program activities aimed at  preventing HIV infection and at treating those infected each represented  about 30 percent of approved PEPFAR funding, while activities aimed at  caring for AIDS patients represented about 20 percent. The remaining  approximately 20 percent funded a variety of other program areas, such  as health systems strengthening and building laboratory infrastructure.  Figure 1 summarizes approved funding for these program areas in fiscal  years 2009 through 2011.
To carry out activities in these program areas, CDC and USAID use  implementing mechanisms—grants, cooperative agreements, and  contracts—with a variety of implementing partners. These partners  include partner country governments, nongovernmental and international  organizations, and academic institutions. CDC and USAID used more  than 3,000 implementing mechanisms in fiscal years 2008 through 2010.
CDC and USAID offices employ a wide variety of individuals and  organizations to conduct PEPFAR evaluations, including implementing  agency officials, consultants, and academic institutions as well as partner  government organizations and implementing partners. Evaluation teams  sometimes comprise representatives from several of these organizations.  OGAC coordinates, and PEPFAR implementing agencies also engage in,  several related activities that support evaluation, such as oversight of  implementing partners, routine performance planning and reporting,  biological and behavioral health surveillance, baseline studies and  needs assessments, and development of health management information  systems.
PEPFAR evaluations are subject to common evaluation standards  defined in various agency-specific and governmentwide guidance. This  guidance includes CDC’s Framework for Program Evaluation in Public  Health and USAID’s evaluation policy and Automated Directives  System guidance. In addition, GAO published guidance on designing  evaluations and assessing social program impact evaluations.
Also, in September 2010, the AEA published a framework to guide the  development and implementation of federal agency evaluation programs  and policies. The framework offers a set of general principles intended to  facilitate the integration of evaluation activities with program  management. These principles include developing evaluation policies and  procedures; developing evaluation plans; ensuring independence of  evaluators in designing, conducting, and determining findings of their  evaluations; ensuring professional competence of evaluators; and  disseminating evaluation results publicly and in a timely fashion.
section_title: PEPFAR Agencies Have Evaluated a Broad Range of PEPFAR Programs, but Results Are Not Fully Supported in Many Evaluations
paragraphs: OGAC, CDC, and USAID managed and conducted evaluations of a wide  variety of PEPFAR programs that were ongoing during fiscal years 2008  through 2010. However, we found that many of these evaluations— particularly evaluations managed by PEPFAR country and regional  teams—did not consistently adhere to common evaluation standards, in  many cases calling into question the evaluations’ support for their  findings, conclusions, and recommendations.
subsections: section_title: OGAC, CDC, and USAID Evaluated a Broad Range of Programs
paragraphs: OGAC, CDC, and USAID provided 496 evaluations addressing programs  ongoing during fiscal years 2008 to 2010 in all PEPFAR program areas  relating to HIV/AIDS treatment, prevention, and care. Of these 496  evaluations, 18 were public health evaluations (PHE), managed by  OGAC; 42 were program evaluations provided by CDC and USAID  headquarters officials; and 436 were program evaluations provided by  CDC and USAID country and regional team officials. (For more  information about these evaluations, see app. III.)    OGAC-managed evaluations. OGAC provided 18 PHEs that CDC  and USAID had completed as of November 2011 under an OGAC- managed approval, implementation review, and reporting process.  The completed PHEs addressed the following program areas:  prevention of mother-to-child transmission, testing and counseling,  adult care and support, adult treatment, sexual prevention, and  pediatric care and support. In addition, OGAC indicated that 82  other PHEs had been initiated as of November 2011. According to  OGAC, PHEs are intended to assess the effectiveness and impact of  PEPFAR programs; compare evidence-based program models in  complex health, social, and economic contexts; and address  operational questions related to program implementation within  existing and developing health systems infrastructures. OGAC  guidance states that these evaluations focus on strategies to increase  program efficiency and impact to guide program development and  inform the public, using rigorous quantitative or qualitative methods  that permit broad generalization. For all PHEs, OGAC requires  PEPFAR country and regional teams to submit evaluation concepts or  protocols for approval by an interagency subcommittee and requires  periodic progress and closeout reports.    CDC and USAID headquarters-managed evaluations. CDC  headquarters officials provided 20 evaluations in the following  program areas: blood safety, injection safety, adult treatment,  pediatric treatment, and strategic information. USAID headquarters  officials provided 22 evaluations in the following program areas:  abstinence/be faithful, sexual prevention, orphans and vulnerable  children, strategic information, and health systems strengthening  programs. Four CDC and USAID headquarters evaluations addressed  more than one program area.    Country and regional team-managed evaluations. CDC and  USAID officials representing 31 PEPFAR country and 3 regional  teams provided a total of 436 evaluations; CDC officials provided 185  evaluations, and USAID officials provided 251 evaluations. The  evaluations addressed 18 program areas related to PEPFAR  prevention, treatment, and care, with about one-fifth of the evaluations  addressing activities in more than one program area (see fig. 2). CDC  and USAID officials also provided copies of evaluation protocols and  statements of work, indicating that additional evaluations had been  initiated. Further, based on our analysis of a randomly selected  sample of 78 evaluations, we estimate that 51 percent of the  evaluations used qualitative methods, 35 percent used quantitative  methods, and 14 percent used a mix of quantitative and qualitative  methods. In addition, evaluations provided by USAID tended to  employ qualitative methods (32 of 48 evaluations), while those  provided by CDC tended to use quantitative methods (20 of 30  evaluations). (See app. III for additional results of our analysis.)
Our assessments of judgmental and randomly selected samples of  PEPFAR evaluations indicate that many—particularly those managed by  PEPFAR country and regional teams—contain findings, conclusions, and  recommendations that are not fully supported. To determine the extent to  which these elements are supported, we synthesized our assessments of  the extent to which evaluations generally adhered to several common  evaluation standards defined in guidance issued by CDC, USAID, and  GAO. Specifically, we considered whether the evaluations describe the  program to be evaluated and its objectives, the purpose of the evaluation,  and the criteria used to reach conclusions about the achievement of the  program’s objectives. We also considered the extent to which evaluations  incorporate appropriate designs, sample selection methods, measures,  and data collection and analysis methods.
All OGAC-managed PHEs that we reviewed generally adhered to these  standards and thus their findings, conclusions, and recommendations  were fully supported. We found similar results for most CDC and USAID  headquarters’ program evaluations we reviewed. However, PEPFAR  country and regional teams’ evaluations did not consistently adhere to  common evaluation standards, and thus, in most cases, their findings,  conclusions, and recommendations were not fully supported.
OGAC-managed evaluations. Our assessment of seven OGAC- managed PEPFAR PHEs indicates that they all generally adhered to  common evaluation standards, and thus their findings, conclusions, and  recommendations were fully supported. All of the evaluations that we  reviewed identified program and evaluation objectives and used  appropriate measures, and most used appropriate evaluation designs and  data collection and analysis methods. Three of the evaluations employed  fully appropriate sampling methods. Table 1 summarizes our  assessments of these evaluations.
CDC and USAID headquarters-managed evaluations. Our assessment  of 15 CDC and USAID headquarters-managed evaluations indicates that  most generally adhere to common evaluation standards. As a result, we  found that findings, conclusions, and recommendations were fully  supported in 9 evaluations and partially supported in 6 evaluations. Most  of the evaluations employed appropriate evaluation designs, measures,  and data collection and analysis methods. However, 7 evaluations did not  fully identify the evaluation criteria, and 8 did not employ fully appropriate  sampling methods. Table 2 summarizes our assessments of these  evaluations.
Country and regional team-managed evaluations. We found that  evaluations managed by country and regional teams, which make up the  bulk of all PEPFAR program evaluations, did not consistently adhere to  common evaluation standards. Based on our analysis of a randomly  selected sample of country and regional team evaluations, we estimate  that findings, conclusions, and recommendations were fully supported in  41 percent of all evaluations provided to us by country and regional  teams, partially supported in 44 percent of these evaluations, and not  supported in 15 percent of these evaluations. We estimate that 24  percent of these evaluations did not identify any evaluation criteria, and  more than half did not employ evaluation designs, sampling methods,  measures, or data collection and analysis methods that were fully  appropriate. For example, an evaluation of activities for providing care to  orphans and vulnerable children drew conclusions about results and  made recommendations, based almost exclusively on favorable  anecdotal information collected from selected program participants and  beneficiaries. As a result, the objectivity and credibility of these  evaluations’ findings, conclusions, and recommendations are in question.  Table 3 summarizes our assessments of these evaluations.
Further analysis of the results of our assessments showed that  evaluations using qualitative methods were more likely to contain results  that were partially supported or not supported than evaluations using  quantitative methods. (See app. III for additional results of our analysis.)
section_title: PEPFAR Policies and Procedures Do Not Fully Adhere to AEA Evaluation Principles Relating to Planning, Independence, and Dissemination
paragraphs: State, OGAC, CDC, and USAID have developed policies and procedures  that apply to evaluations of PEPFAR programs, as called for in the AEA  Roadmap. However, they have not fully adhered to other AEA Roadmap  principles regarding evaluation planning, independence and competence  of evaluators, and dissemination of evaluation results. First, OGAC has  not developed PEPFAR evaluation plans at the program level or required  the development of such plans in individual countries and regions, limiting  its own ability to ensure that evaluation resources are appropriately  targeted. Second, State, OGAC, CDC, and USAID guidance does not  specify how to document the independence and competency of  evaluators, and almost half of the evaluations we reviewed did not  provide sufficient information to fully determine whether evaluators were  free of conflicts of interest. Finally, not all evaluation reports are available  online, thus limiting their accessibility and usefulness to PEPFAR decision  makers and other stakeholders.
subsections: section_title: State, OGAC, and PEPFAR Implementing Agencies Have Issued Evaluation Policies and Procedures
paragraphs: In accordance with AEA principles, State, OGAC, CDC, and USAID have  issued policies and procedures that are applicable to PEPFAR program  evaluation.    State evaluation policy. In February 2012, State’s Bureau of  Resource Management issued an evaluation policy that applies to all  State bureaus and OGAC. The policy provides a framework for  implementing evaluations of State’s various programs and projects  and encourages evaluations for programs and projects at all funding  levels.    OGAC operational plan guidance. According to OGAC officials,  OGAC generally has deferred to implementing agency policies.  OGAC also issues annual guidance to PEPFAR implementing  agencies for preparation of their operational plans. OGAC’s fiscal year  2012 operational plan guidance to PEPFAR country and regional  teams, issued in August 2011, addresses some elements of  evaluation. The guidance differentiates three types of evaluation and  research: basic program evaluation, which focuses on descriptive and  normative evaluation questions; operations research, which focuses  on program delivery and optimal allocation of resources; and impact  evaluation, which measures the change in an outcome attributable to  a particular program.    CDC evaluation framework. In September 1999, the Program  Evaluation Unit at CDC’s Office of the Associate Director for Program  issued an evaluation framework for CDC programs. The framework  summarizes essential elements of program evaluation, clarifies  program evaluation steps, and reviews standards for effective  program evaluation, among other things. According to CDC’s Chief  Evaluation Officer, as of May 2012, CDC plans to issue evaluation  guidelines and recommendations as well as additional guidance for  using the evaluation framework.    USAID evaluation policy. In January 2011, USAID’s Bureau for  Policy, Planning, and Learning revised evaluation policy to  supplement existing evaluation guidance in USAID’s Automated  Directive System. According to USAID, this revised policy was  intended to address a decline in the quantity and quality of evaluation  practice within the agency in the recent past. The policy clarifies for  USAID staff, partners, and stakeholders the purposes of evaluation;  the types of evaluations that are required and recommended; and  USAID’s approach for conducting, disseminating, and using  evaluations. Among other things, the policy sets forth the purposes of  evaluation, the roles and responsibilities of USAID operating units,  and evaluation requirements and practices for all USAID programs  and projects. The policy requires all USAID operating units to consult  with program office experts to ensure that scopes of work for external  evaluations meet evaluation standards. The policy also states that  operating units, in collaboration with the program office, must ensure  that evaluation draft reports are assessed for quality by management  and through an in-house peer technical review.
section_title: PEPFAR Lacks Evaluation Plans
paragraphs: OGAC has not yet developed a program-level PEPFAR evaluation plan or  required implementing agencies or country and regional teams to develop  evaluation plans as called for by the AEA Roadmap.    OGAC. State’s recently issued evaluation policy requires that each  State bureau, including OGAC, develop and submit a bureauwide  evaluation plan that encompasses major policy initiatives and new  programs as well as existing programs and projects. According to a  senior OGAC official, at the time of our review, OGAC was discussing  with State’s Bureau of Resource Management how it will comply with  this new requirement.    CDC and USAID headquarters. OGAC defers to implementing  agencies to plan evaluations of their headquarters-managed PEPFAR  program activities, but CDC and USAID have not developed  evaluation plans for such activities included in recent headquarters  operational plans. OGAC’s 2011 guidance for developing the  headquarters operational plan requires a plan for technical area  program priorities but does not address evaluation planning. Similarly,  the fiscal year 2012 guidance does not include a requirement for an  evaluation plan.    Country and regional teams. OGAC defers to PEPFAR country and  regional teams to plan evaluations of their program activities, but does  not require that the teams develop and submit annual evaluation  plans. OGAC’s 2011 guidance on developing country and regional  operational plans urges country and regional teams to prioritize  program evaluation in order to make PEPFAR programs more  effective and sustainable. In addition, OGAC’s fiscal year 2012  guidance calls for country and regional teams to address monitoring  and evaluation in describing individual implementing partners’  activities. However, neither the 2011 guidance nor the 2012 guidance  instructs all country teams to develop evaluation plans. We reviewed  PEPFAR country and regional operational plans for fiscal year 2011  and found that they did not include evaluation plans. Instead, these  documents generally included (1) descriptions of ongoing or planned  evaluations and related activities (e.g., surveillance) in program area  narrative summaries and (2) descriptions of monitoring and evaluation  activities in implementing partner activity narratives.
In our analysis of information provided by country and regional teams, as  well as CDC and USAID headquarters, we did not detect an evaluation  rationale or strategy. Based on responses to our survey of CDC and  USAID officials in 31 PEPFAR country and 3 regional teams, we  calculated that evaluations had been conducted or were ongoing for  about one-third of these countries’ program activities in fiscal years 2008  through 2010. In addition, based on these officials’ responses, we found  similar percentages of ongoing and completed evaluations across the  broad program areas of prevention, treatment, and care. We also  analyzed CDC and USAID headquarters officials’ responses to our survey  and found that evaluations had been conducted or were ongoing for  about half of the PEPFAR program activities managed by agencies’  headquarters and implemented during fiscal years 2008 to 2010.  However, we found no relationships between the percentages of program  activities with ongoing or completed evaluations and budgets at the  country, program area (i.e., prevention, treatment, or care), or program  activity levels.
section_title: Evaluator Independence and Qualifications Are Not Consistently Documented
paragraphs: State, CDC, and USAID policies and procedures address the  independence of evaluators but do not consistently require that evaluation  reports identify the evaluation team or address whether there are any  potential conflicts of interest. In addition, some agency policies and  procedures address the need to ensure that evaluators have appropriate  qualifications, but none require that evaluations document those  qualifications or certify that they are adequate.    State. State’s recently issued evaluation policy addresses evaluator  independence and integrity, stating that evaluators should be free  from program managers and not subject to their influence. This policy  does not address evaluator qualifications.    OGAC. OGAC’s operational plan guidance to country and regional  teams does not address the independence or professional  qualifications of evaluators. According to OGAC officials, OGAC  defers to implementing agency evaluation policies.    CDC. CDC’s evaluation framework addresses the need to assemble  an evaluation team with the needed competencies, highlighting the  importance of ensuring that evaluators have no particular stake in the  results of the evaluation. The CDC evaluation framework also  discusses appropriate ways to assemble an evaluation team.    USAID. USAID’s evaluation policy recommends that most evaluations  be external and requires a disclosure of conflicts of interest for all  evaluation team members. In addition, USAID’s evaluation policy  requires that evaluation-related competencies be included in staffing  selection policies.
Our analysis of a randomly selected sample of evaluations submitted by  31 PEPFAR country and 3 regional teams found that the evaluations  often did not address whether evaluators have potential conflicts of  interest, as called for by the AEA Roadmap. We estimate that 27 percent  of the evaluations fully addressed potential conflicts of interest, 59  percent partially addressed the issue, and 14 percent did not address the  issue. In addition, while we were unable to determine whether potential  conflicts of interest existed with the information provided in some of the  evaluation reports, it appeared that there were evaluations in which  potential conflicts of interest existed but were not addressed. For  example, one evaluation report, relating to strengthening a partner  country’s nongovernmental HIV/AIDS organizations, indicated that the  evaluation team was employed by the program activity’s implementing  partner, but the report did not address potential conflict of interest.
Furthermore, some country and regional program evaluations sometimes  did not provide enough identifying information about evaluators to allow  an assessment of evaluator independence or qualifications. We estimate  that 86 percent of the evaluations fully identified the evaluators, while 14  percent provided either partial or no information. For example, an  evaluation report we reviewed relating to HIV prevention program  activities in one region named the organization that conducted the  evaluation but did not provide any information on the evaluation team  members. Moreover, we were unable to find any information about this  organization in an online search based on the limited information  available in the report.
section_title: PEPFAR Evaluations Are Not All Publicly Disseminated
paragraphs: Agency policies and procedures generally support dissemination of  evaluation results, but OGAC, CDC, and USAID have not ensured that  evaluation methods, data, and evaluation results are made fully and  easily accessible to the public.    State. State’s newly released evaluation policy requires bureaus to  submit evaluations to a central repository.    OGAC. OGAC officials told us that the office supports dissemination  of the results of important global HIV/AIDS research and evaluations  to a variety of stakeholders. For example, OGAC officials noted that  the PEPFAR website contains information on PEPFAR results as well  as monitoring and evaluation guides. OGAC officials also noted that  dissemination strategies are a common component of evaluation  protocols and the procurement mechanisms that fund them. In  addition, OGAC maintains an intranet site, which is accessible to  PEPFAR implementing agency officials and contains information  about evaluation. However, OGAC does not have a mechanism for  publicly and systematically disseminating evaluation results.    CDC. CDC policy advises that effort is needed to ensure that  evaluation findings are disseminated appropriately but does not  require online dissemination of evaluation reports. CDC officials told  us that they recently made changes to CDC’s public website, which,  as of April 2012, includes some information on program evaluations.  In addition, CDC’s Division of Global HIV/AIDS (DGHA) Science  Office maintains a catalog of published journal articles coauthored by  DGHA officials. However, CDC does not maintain a complete online  inventory of evaluations.    USAID. USAID’s policy states that evaluation findings should be  shared as widely as possible with a commitment to full and active  disclosure. USAID requires submission of completed evaluations to  the Development Experience Clearinghouse (DEC), the agency’s  online repository of research documentation, but does not enforce  this requirement. In 2010, USAID reported that practices for  disseminating evaluation results were generally limited, that  dissemination practices varied across the agency, and that the  requirement to submit completed evaluations to the DEC had not  been fully enforced. Additionally, USAID found that documents in the  DEC were sometimes difficult to find. In February 2012, USAID also  found that missions had reported submitting only 20 percent of their  evaluations to the DEC in fiscal year 2009.
Although documents submitted by 31 PEPFAR country and 3 regional  teams showed that CDC and USAID have disseminated evaluation  findings within these countries and regions in several ways, we found no  publicly accessible and easily searchable Internet source for PEPFAR  program evaluations. We received abstracts from annual meetings and  conferences, presentations to partner government officials and  stakeholders, published journal articles, and periodic agency reports,  which may be publicly accessible via the Internet. However, as of the  time of our review, our searches of five key websites generated far fewer  PEPFAR evaluations than the 496 evaluations we received from country  teams, CDC and USAID headquarters, and OGAC. We searched  PubMed, the U.S. National Library of Medicine’s online database, but a  search using “PEPFAR” and “evaluation” as search terms generated  seven results. Likewise, as of April 2012, our search of USAID’s DEC,  using “HIV/AIDS” and “evaluation” as search terms, generated 87 results,  including some that were not evaluations, but USAID officials, in response  to our request, later provided us nearly 300 evaluations. We also found  some evaluations at two USAID-maintained websites, OVCsupport.net  and AIDStar-One, but neither site was comprehensive or fully  searchable. In addition, a website called Global HIV M&E Information  provides a repository of voluntarily submitted monitoring and evaluation  resources; however, we found few evaluations of PEPFAR programs.
section_title: Conclusions
paragraphs: PEPFAR’s authorizing legislation emphasizes the importance of program  evaluation as a tool for OGAC to ensure, among other things, that funds  are spent on programs that show evidence of success. State, CDC, and  USAID have demonstrated a clear commitment to program evaluation by  conducting a wide variety of program evaluations that address at least  one activity in each PEPFAR program area. However, many evaluations  managed by PEPFAR country and regional teams lack fully supported  findings, conclusions, and recommendations, evidenced by a lack of  general adherence to common evaluation standards. Without fully  supported findings, conclusions, and recommendations, these PEPFAR  program evaluations have limited usefulness as a basis for decision  making and may supply incomplete or misleading information for  managers’ and stakeholders’ efforts to direct PEPFAR funding to  programs that produce the desired outcomes and impacts.
State, CDC, and USAID have demonstrated their commitment to program  evaluation by developing policies and procedures that apply to  evaluations, in accordance with established general principles. However,  without a requirement that country and regional teams prepare and  submit annual evaluation plans—for example, as a component of  operational plans—OGAC is unable to ensure that program activities are  subject to appropriate levels of evaluation. Moreover, without  documentation of the independence and competence of PEPFAR  program evaluators, OGAC, agency program managers, and other  stakeholders have limited assurance that evaluation results are unbiased  and credible. Finally, unless evaluation results are publicly and  systematically disseminated and made easily searchable online, program  officials and public health researchers may be unable to assess the  credibility of their findings or use them for program decision making.
section_title: Recommendations for Executive Action
paragraphs: We recommend that the Secretary of State direct the U.S. Global AIDS  Coordinator to take the following four actions in collaboration with CDC  and USAID to enhance PEPFAR evaluations:  1.  develop a strategy to improve PEPFAR implementing agencies’ and  country and regional teams’ adherence to common evaluation  standards;  2.  require implementing agency headquarters and country and regional  teams to include evaluation plans in their annual operational plans;  3.  provide detailed guidance for implementing agencies and country and  regional teams on assessing, ensuring, and documenting the  independence and competence of PEPFAR program evaluators; and  4.  increase the online accessibility of PEPFAR program evaluation  results.
section_title: Agency Comments
paragraphs: We provided a draft of this report to State, HHS’s CDC, and USAID.  Responding jointly with CDC and USAID, State OGAC provided written  comments (see app. IV). CDC and USAID also provided technical  comments, which we incorporated as appropriate.
In its written comments, State agreed with our recommendations and,  emphasizing the interagency nature of the PEPFAR program, indicated  that it will coordinate with PEPFAR agencies to implement our  recommendations. First, State explained that it will work with PEPFAR  implementing agencies to carry out the agencies’ evaluation policies and  practices, which State deemed generally consistent with AEA principles,  and will develop strategies to ensure the appropriate application of  common evaluation standards. Second, State responded that it will work  through PEPFAR interagency processes to develop PEPFAR program  evaluation plans, which it noted could be included in annual PEPFAR  operational plans. Third, State will work with PEPFAR implementing  agencies to put in place guidance to document program evaluators’  independence and qualifications. Fourth, State affirmed that OGAC will  collaborate with PEPFAR implementing agencies to develop strategies for  improving dissemination of evaluation results and will use PEPFAR’s  public website to link to agencies’ online resources.
We are sending copies of this report to the Secretary of State, the Office  of the U.S. Global AIDS Coordinator, U.S. Agency for International  Development’s Office of HIV/AIDS, the Department of Health and Human  Services’ Office of Global Affairs, the Centers for Disease Control and  Prevention’s Division of Global HIV/AIDS, and appropriate congressional  committees. In addition, the report is available at no charge on the GAO  website at http://www.gao.gov.
If you or your staffs have any questions about this report, please contact  me at (202) 512-3149 or gootnickd@gao.gov. Contact points for our  Offices of Congressional Relations and Public Affairs may be found on  the last page of this report. GAO staff who made major contributions to  this report are listed in appendix VI.
section_title: Appendix I: Objectives, Scope, and Methodology
paragraphs: This report (1) identifies President’s Emergency Plan for AIDS Relief  (PEPFAR) evaluation activities and examines the extent to which  evaluation results are supported and (2) examines the extent to which  PEPFAR policies and procedures adhere to established principles for the  evaluation of U.S. government programs. To identify PEPFAR program  evaluations and examine the extent to which they generated supported  evaluation results, we collected and analyzed program evaluation  documents provided by Centers for Disease Control and Prevention  (CDC) and U.S. Agency for International Development (USAID) officials in  the 31 PEPFAR countries and 3 regions with PEPFAR country or regional  operational plans in fiscal year 2010, as well as the Department of State’s  (State) Office of the U.S. Global AIDS Coordinator (OGAC) and CDC and  USAID headquarters officials. To examine the extent to which PEPFAR  program evaluation policies and procedures adhered to principles in the  American Evaluation Association’s (AEA) An Evaluation Roadmap for a  More Effective Government (AEA Roadmap), we reviewed the general  principles for conducting federal government program evaluations, as well  as OGAC, State, USAID, and CDC policies and guidance. In addition, we  surveyed CDC and USAID officials in the 31 PEPFAR countries and 3  regions with PEPFAR annual country or regional operational plans in  fiscal year 2010, as well as CDC and USAID headquarters officials,  regarding ongoing and completed evaluations. Finally, we conducted  interviews with OGAC, CDC, and USAID officials in Washington, D.C.,  and Atlanta, Georgia.
subsections: section_title: Survey of PEPFAR Officials
paragraphs: To survey PEPFAR country and regional team officials, we took the  following steps:  1.  We consulted with OGAC and CDC and USAID headquarters officials  and decided to use implementing mechanism as a proxy for a  program activity. We determined that using implementing  mechanisms was the only viable unit of analysis to estimate the  percentage of PEPFAR programs with evaluations because (1) OGAC  officials maintained updated data on implementing mechanisms and  (2) PEPFAR officials regularly used and understood data on  implementing mechanisms. However, in some of these cases, if the  broader program was evaluated, not all implementing mechanisms  under the larger program were necessarily evaluated. We also  recognized that evaluations may not be appropriate for all  implementing mechanisms (such as those that provide funding for  staffing costs). To the extent possible, we eliminated these  implementing mechanisms from our analysis.  2.  We obtained lists of program activities for fiscal years 2008 through  2010 from OGAC for each country and region. We then analyzed  program activities by country (or region) and agency; the lists included  identification numbers, names, and partner names for each of the  program activities. Each survey tool then contained a list of program  activities relevant to the country or regional team.  3.  Based on GAO and OGAC guidance, we developed the following  working definition of evaluation: Evaluations are systematic studies to  assess how well a program is working. Evaluations are often  conducted by experts external to the program, either inside or outside  the agency. Types of evaluations include process, outcome, impact,  or cost-benefit analysis.  4.  We developed a survey tool for ongoing and completed evaluations of  PEPFAR programs. We consulted with OGAC and CDC and USAID  headquarters officials about the survey tool and made revisions as  appropriate. For example, based on input from CDC and USAID  headquarters officials, we determined that some PEPFAR evaluations  could address several implementing mechanisms. In addition, in some  of these cases, if a broader program (e.g., national treatment  program) was evaluated, not all implementing mechanisms under the  broader program were necessarily evaluated. In response, we  included questions in our survey prompting PEPFAR officials to  indicate whether an implementing mechanism has been evaluated as  part of a broader evaluation of several implementing mechanisms.  5.  We tested the survey tool with officials in two PEPFAR countries— Angola and Ethiopia—and finalized the survey tool based on  discussions with these officials.  6.  We sent the final survey tool to PEPFAR country contacts (PEPFAR  coordinators and CDC and USAID officials) identified by OGAC and  CDC and USAID headquarters. The survey tool instructed CDC and  USAID country or regional team officials to provide “yes” or “no”  responses to the following questions for each implementing  mechanism in the country’s (or region’s) agency-specific lists:  Is this one of your agency’s fiscal year 2008-2010 country or regional  operational plan program activities?    Has at least one evaluation specific to this implementing mechanism  been completed?
Is at least one evaluation specific to this implementing mechanism  ongoing?    Has at least one evaluation covering, but broader than, this  implementing mechanism been completed?
Is at least one evaluation covering, but broader than, this  implementing mechanism ongoing?
We also prompted the country or regional officials to provide additional  information for each implementing mechanism, such as explanations for  program activities that do not belong to the agency and identification of  duplicate program activities. Officials were instructed to either e-mail the  completed surveys to GAO or upload them to a website regularly used by  OGAC and country and regional teams for submitting and sharing  planning and reporting documents.
In some cases, we met with country or regional team officials via  telephone, or corresponded via e-mail, to clarify the purpose of the  survey, the questions themselves, and the evaluation document request  as well as to correct anomalies and ask follow-up questions. One GAO  analyst also attended the May 2011 PEPFAR implementing agency  annual meeting in Johannesburg, South Africa, to provide information  about the survey and evaluation document request to PEPFAR country  and regional team officials also attending the annual meeting. We  received responses from all 31 PEPFAR countries and 3 regions with  fiscal year 2010 operational plans.
Using a similar survey tool, we also conducted surveys of CDC and  USAID headquarters officials regarding program activities managed by  agency headquarters and listed in PEPFAR headquarters operational  plans for 2008 through 2010.
section_title: Analysis of Survey Responses
paragraphs: To analyze country and regional teams’ survey responses, we made the  following assumptions regarding the survey responses:  If officials did not provide a response to the question “Is this one of  your agency’s fiscal year 2008-2010 country or regional operational  plans program activities?” we included that implementing mechanism  in the analysis. Program activities with responses of “no” or “duplicate”  were eliminated from the analysis.
If officials did not respond to any of the four questions regarding  ongoing or completed evaluations, we assumed that there were no  ongoing or completed evaluations for that implementing mechanism.
In addition, we reviewed narrative comments provided by country and  regional team officials. We recognized that evaluations may not be  appropriate for all implementing mechanisms (such as those that provide  funding for staffing costs). To the extent possible, we eliminated these  implementing mechanisms from our analysis. Based in part on our review  of the narrative comments, we flagged and eliminated implementing  mechanisms with evidence indicating that the implementing mechanism  was either “to be determined” (i.e., the agency had yet to make an award  to an implementing partner), related to staffing costs, related to strategic  information and monitoring and evaluation, recently begun, a duplicate of  another implementing mechanism, or listed in error.
Once the survey responses were ready for analysis, we calculated the  summary statistics that are reported in the body of the report. We also  included the survey responses provided by officials in CDC and USAID  headquarters in the analysis. To check the reliability of the data analysis,  a second independent analyst reviewed the statistical programs used to  analyze the data for accuracy.
section_title: Program Evaluation Document Collection
paragraphs: In addition to our survey of CDC and USAID officials in the 31 countries  and 3 regions with fiscal year 2011 operational plans, we requested  program evaluation documents. To do this, the survey tool instructions  prompted CDC and USAID officials to provide documentation of  completed and ongoing evaluations. Specifically, for implementing  mechanisms where officials indicated that at least one evaluation had  been completed, we requested documentation—such as an evaluation  report—of all such completed evaluations. For implementing mechanisms  where officials indicated that at least one evaluation was ongoing, we  requested documentation—such as terms of work or an evaluation plan.
We generally advised country and regional team officials to err on the  side of inclusion when in doubt about whether to submit documentation of  ongoing and completed evaluations. We instructed these officials to e- mail, or, in some cases, mail electronic versions of the program  evaluation documents to GAO, or to upload them to a website regularly  used by OGAC and country and regional teams for submitting and  sharing planning and reporting documents.
In response to this document request, we received more than 1,350  documents. For example, we received documentation of ongoing or  planned evaluations, such as statements of work or evaluation protocols  and protocol approval forms. We also received meeting minutes, trip  reports, financial review and audit documents, presentation slides,  abstracts, and conference posters. To determine which documents met  our definition of evaluation, we reviewed each of these documents and  categorized them as meeting the definition of evaluation or not, following  a set of decision rules. For example, we included data quality  assessments, costing studies that compared costs and explained cost  differences, and analyses of surveillance data pre- and postintervention.  We excluded surveillance studies that simply reported the results of a  surveillance activity (but did not link it to a specific program or  intervention); needs assessments, baseline studies, and situation  analyses; trip and site visit reports; and pre- and postevent (e.g.,  workshop) questionnaires or surveys. We identified and eliminated  duplicate documents. This categorization was checked by a second  analyst and yielded 436 program evaluations. We believe that this final  set of evaluations constitutes an essentially full universe of PEPFAR  country and regional program evaluation documents.
In addition to the program evaluation documents collected from CDC and  USAID officials in PEPFAR countries and regions, we requested  documents from OGAC related to PEPFAR public health evaluations. We  also requested evaluation documents related to PEPFAR program  managed by CDC and USAID headquarters from officials at each  agency’s headquarters. OGAC provided copies of 18 completed public  health evaluations, CDC headquarters provided copies of 22 completed  evaluations, and USAID headquarters provided copies of 24 completed  evaluations.
We reviewed the program evaluation documents submitted by PEPFAR  country and regional teams as well as CDC and USAID headquarters  officials. We identified whether each program evaluation was ongoing or  completed as well as which program area or areas (e.g., prevention,  treatment, care, or other) were evaluated. To do this, we used program  categories defined by OGAC’s fiscal years 2011 operational plan  guidance, resulting in the program areas and related areas reported in the  report. This categorization was checked by a second analyst. Table 4  provided descriptions of the PEPFAR program areas.
To determine the degree to which these evaluations were conducted in  adherence with common evaluation standards, we used an assessment  tool to systematically conduct in-depth analyses of a probability sample of  the evaluations submitted by the PEPFAR country and regional teams  and a nonprobability sample of the evaluations submitted by OGAC and  CDC and USAID headquarters officials. Our PEPFAR evaluation  assessment tool was based on an assessment tool used for a prior GAO  report, which we updated using guidance on evaluation from USAID,  CDC, the Organization for Economic Cooperation and Development  (OECD), and GAO. We piloted the assessment tool with three PEPFAR  program evaluation documents provided by CDC and USAID  headquarters officials and revised the evaluation assessment as  appropriate. After piloting and revising the tool, we finalized the tool and  used it to conduct the in-depth analyses of program evaluation  documents. Table 5 lists the questions and supporting questions included  in the assessment tool.
To allow us to generalize to the entire set of evaluations provided by  PEPFAR country and regional teams, we randomly selected a sample of  84 of 436 evaluations submitted by CDC and USAID officials in 31  PEPFAR countries and 3 regions. The list of all evaluations was sorted by  total approved operational plan budgets for each country or region for  fiscal years 2008 through 2010, so that a systematic sample would  ensure representation of countries with relatively large, medium, and  small budgets for fiscal years 2008 through 2011.
After sampling, 6 evaluations—including, for example, baseline and  feasibility studies—were found to be out of scope, resulting in a final  sample of 78. Results based on random probability samples are subject  to sampling error. The sample we drew for our survey is only one of a  large number of samples we might have drawn. Because different  samples could have provided different estimates, we express our  confidence in the precision of our particular sample results as a 95  percent confidence interval. This is the interval that would contain the  actual population values for 95 percent of the samples we could have  drawn. The margin of error associated with proportion estimates is no  more than plus or minus 11 percentage points at the 95 percent level of  confidence and estimates of totals have a margin of error no larger than  44 evaluations.
For the 18 public health evaluations submitted by OGAC, as well as the  20 and 22 evaluations submitted by CDC and USAID headquarters,  respectively, we selected a nonprobability sample based on the type of  program (e.g., prevention, treatment, care, or other) evaluated as well as  country or countries addressed by each evaluation. Because this is a  nonprobability sample, the results of our assessments of these  evaluations cannot be used to make inferences about all evaluations  managed by OGAC and CDC and USAID headquarters. However, they  do represent a mix of the types of evaluations managed by OGAC and  CDC and USAID headquarters.
section_title: Assessing Program Evaluation Documents
paragraphs: Using our evaluation assessment tool, we conducted in-depth analyses of  the evaluation documents submitted by the PEPFAR country and regional  teams and also those submitted by OGAC, USAID, and CDC  headquarters. To do so, one analyst conducted an initial review of the  evaluation document and then completed the evaluation assessment tool.  The analyst also recorded basic information about each evaluation,  including title, author, date of publication, and the country or countries  included in the evaluation. For each of the questions in the assessment  tool (see table 1), analysts were instructed to (1) respond using “yes,”  “no,” “partial,” “not sure,” or “not applicable” and (2) summarize or cite  relevant information from the evaluation documents. Analysts then were  instructed to weigh the evidence and answers to these questions and  provide “yes,” “no,” “partial,”, “not sure,” or “not applicable” responses for  each category. Based on the analysis of the elements addressed in the  assessment tool, analysts determined the extent to which each  evaluation’s findings, conclusions, and recommendations were supported  using “yes,” “no,” “partial,” or “not sure” as their responses. This overall  determination was not based on a tally of responses to individual  elements in the evaluation assessment tool, but rather a synthesis of  these responses and an assessment of the contribution of each element  to the overall support for the evaluation’s findings, conclusions, and  recommendations. To help ensure consistency in the application of the  standards and questions, the assessors met weekly during the  assessment period to clarify the instructions and discuss their  observations. After each assessment was complete, a second analyst  independently verified the results of the analysis by reviewing the  program evaluation document and the completed evaluation assessment  tool. In cases where the two analysts did not concur on the results, or  where there was a “not sure” response, they met to discuss the evidence  and documented a final determination. All the results for the evaluation  assessment tools were then entered into a spreadsheet and analyzed.
section_title: Analyzing Data Generated by Evaluation Assessments
paragraphs: To assess potential associations between key attributes of the sample of  78 evaluations we randomly selected, we calculated chi-square tests and  the associated odds ratios for all pairs of the following variables: agency,  methods used, evaluation type, and program type. Key results from these  analyses are presented in the report. Additional results can be found in  appendix III. We also employed logistic regressions to assess which of  these variables (i.e., agency, methods used, evaluation type, and  program type) had the strongest effects on the extent to which sampled  evaluations contained support for findings, conclusions, and  recommendations.
section_title: Assessing State, OGAC, CDC, and USAID Evaluation Policies
paragraphs: To assess State, OGAC, CDC, and USAID evaluation policies, we  developed an assessment tool based on nine AEA Roadmap principles.  For each principle, we developed a question or series of questions asking  how the policies addressed the AEA Roadmap principles. One analyst  reviewed each agency’s policy and filled out the tool by citing evidence  that would support the policy’s consistency with the AEA Roadmap  principle, or a conclusion that no evidence could be found to support  adherence to the principle. The analyst then concluded whether the policy  was consistent with each principle assessed. A second analyst conducted  a review of the completed assessment tools and either concurred with or  disputed the conclusion for each principle. In cases where the two  analysts did not concur, they met to discuss the evidence and made a  final determination.
To determine the extent to which operational plans contained evaluation  plans, we reviewed OGAC’s fiscal year 2011 and 2012 annual guidance  to implementing agency headquarters regarding development of the  annual PEPFAR headquarters operational plan. We documented  instances where the guidance addressed program evaluation and  determined whether it constituted instructions to develop an evaluation  plan. We conducted similar analysis of OGAC’s fiscal year 2011 and 2012  annual guidance to PEPFAR country and regional teams to identify  instances where the guidance addressed evaluation and, finally, to  determine whether the guidance constituted instructions for developing  evaluation plans. In addition, we assessed 11 of the 33 country  operational plans and 2 of the 3 regional operational plans submitted to  OGAC for fiscal year 2011, the most recent year in which plans were  available. We documented instances where these operational plans  discussed evaluation and whether they contained evaluation plans.
To determine the extent to which the program evaluations documented  potential conflicts of interest and the identity of evaluators, we included  questions on these two elements in our evaluation assessment tool.  Analysts were instructed to respond using “yes,” “no,” or “partial” to these  questions and to cite relevant evidence. After each assessment was  complete, a second analyst verified the results of the analysis by  reviewing the program evaluation document and the completed  evaluation assessment tool. In cases where the two analysts did not  concur on the results, they met to discuss the evidence and documented  a final determination. All the results for the evaluation assessment tools  were then entered into a spreadsheet and analyzed.
We searched five Internet databases referenced by OGAC, CDC, and  USAID officials to determine the public accessibility of PEPFAR program  evaluations. These five sites included the Development Experience  Clearinghouse (http://dec.usaid.gov/index.cfm), PubMed  (http://www.ncbi.nlm.nih.gov/pubmed/), OVCsupport.net  (http://www.ovcsupport.net/s/), AIDSTAR-One (http://www.aidstar- one.com/), and Global HIV M&E Info  (https://www.globalhivmeinfo.org/Pages/HomePage.aspx). For each of  these websites, we conducted searches using keywords that would  capture any PEPFAR-related program evaluations or documentation,  such as “PEPFAR,” “evaluation,” and “HIV/AIDS.” Where applicable, we  then captured the results and counted the number of documents that  could reasonably be considered documentation of a PEPFAR program  evaluation.
We conducted this performance audit from August 2011 to May 2012 in  accordance with generally accepted government auditing standards.  Those standards require that we plan and perform the audit to obtain  sufficient, appropriate evidence to provide a reasonable basis for our  findings and conclusions based on our audit objectives. We believe that  the evidence obtained provides a reasonable basis for our findings and  conclusions based on our audit objectives.
section_title: Appendix II: GAO Evaluation Definitions and Standards
paragraphs: Past GAO work has emphasized evaluation as a key source of  information to help agency officials and Congress make decisions about  the programs they oversee. GAO distinguishes performance  measurement—the ongoing monitoring and reporting of program  accomplishments—from evaluation, which is defined as individual,  systematic studies conducted periodically or on an ad hoc basis to assess  how well a program is working. Further, according to GAO guidance,  experts external to the program, program managers, or both conduct  evaluations to examine the performance of a program within a given  context to understand not only whether a program works but also how to  improve results. GAO guidance identifies four types of evaluation:    Process evaluation. This type of evaluation assesses the degree to  which a program is operating as it was intended. It typically assesses  program activities’ conformance to statutory or regulatory  requirements, program design, and professional standards or  customer expectations.    Outcome evaluation. This type of evaluation assesses the degree to  which a program achieves its outcome-oriented objectives. It focuses  on outputs and outcomes (including unintended effects) to judge  program effectiveness, but may also assess program process to  understand how outcomes are produced.
Impact evaluation. This is a form of outcome evaluation that assesses  the net effect of a program by comparing program outcomes with an  estimate of what would have happened in the absence of the  program. Impact evaluation is used when external factors are known  to influence the program’s outcomes, in order to isolate the program’s  contribution to achievement of its objectives.    Cost-benefit or cost-effectiveness analysis. This type of evaluation  compares a program’s outputs or outcomes with the costs to produce  them. Cost-effectiveness analysis assesses the cost of meeting a  single objective and can be used to identify the least costly alternative  for meeting that goal.
In addition, GAO guidance provides basic information about the more  commonly used evaluation methods; introduces key issues in planning  evaluation studies of federal programs to best meet decision makers’  needs; and describes different types of evaluations for answering varied  questions about program performance, the process of designing  evaluation studies, and key issues to consider in ensuring overall study  quality. Further, the guidance recommends standards for evaluation  design, including establishing evaluation objectives, identifying  constraints, and assessing the appropriateness of the evaluation design.
section_title: Appendix III: Statistical Comparison of PEPFAR Evaluations
paragraphs: We conducted a statistical analysis of the adequacy of support for  findings in evaluations provided to us by CDC and USAID, to determine  whether the adequacy of support differed by agency, by methods used, or  by type of evaluation. Our analysis indicated that fully supported findings  were more likely in CDC’s evaluations than in USAID’s evaluations; in  evaluations that used quantitative methods than in evaluations that used  qualitative or mixed methods; and in cost-benefit or impact evaluations,  as well as outcome evaluations, than in process evaluations. However,  while CDC’s evaluations’ findings were more likely to be fully supported  than USAID’s evaluations’ findings, the difference was not statistically  significant after we accounted for the method used in the evaluations.  This lack of statistical significance suggests that the difference was driven  partly by the agencies’ choice of evaluation method.
Table 6 shows technical details of our statistical analysis of the level of  support for findings in CDC and USAID evaluations.
In table 6, the chi-square statistics at the base of each of the three panels  show that the adequacy of support for findings varied significantly  between the two agencies and differed significantly based on the  methods used and type of evaluations. The odds ratios in the far-right  column show that the odds of evaluations’ being fully supported were 3.6  times greater for CDC than for USAID; 18 times greater for quantitative  evaluations than for qualitative or mixed-methods evaluations; 23 times  greater for cost-benefit or impact evaluations than for process  evaluations; and 3.7 times greater for outcome evaluations than for  process evaluations.
In addition, we estimated binary logistic regression models to determine  whether the difference in adequacy of support for findings in CDC’s and  USAID’s evaluations resulted from differences in the methods used or  differences in the types of evaluations conducted. Table 7 shows the  odds ratios that result from fitting logistic regression models to estimate  the effects of the three different factors (agency, methods used, and type  of evaluation) on the adequacy of support for findings. Models 1, 2, and 3  are bivariate models, which regress “support” on dummy variables for  agency, methods used, and type of evaluation, with each variable  considered one at a time. These produce the same odds ratios that we  obtained from the observed data in table 6. In contrast, model 4 estimates  the effects of agency and methods simultaneously, and model 5  estimates the effects of agency and type of evaluation. In comparing  these models, we found that controlling for the methods used (model 4)  rendered insignificant the differences between agencies in adequacy of  support for findings, whereas controlling for type of evaluation (model 5)  did not.
section_title: Appendix IV: Comments from the Department of State
section_title: Appendix V: GAO Contact and Staff Acknowledgments
subsections: section_title: GAO Contact
section_title: Staff Acknowledgments
paragraphs: In addition to the contact named above, Jim Michels, Assistant Director;  Todd M. Anderson; Chad Davenport; David Dornisch; Lorraine Ettaro;  Justin Fisher; Brian Hackney; Kay Halpern; Fang He; Reid Lowe; Grace  Lui; and Erika Navarro made key contributions to this report. In addition to  these staff, the following GAO staff assisted by conducting in-depth  assessments of selected evaluations: Sada Aksartova, Gergana  Danailova-Trainor, Leah DeWolf, Rachel Girshick, Jordan Holt, Kara  Marshall, Jeff Miller, Steven Putansu, Mona Sehgal, and Doug Sloane.  Sushmita Srikanth and Katy Crosby assisted with quality assurance  reviews.
section_title: Related GAO Products
paragraphs: President’s Emergency Plan for AIDS Relief: Program Planning and  Reporting. GAO-11-785. Washington, D.C.: July 29, 2011.
Global Health: Trends in U.S. Spending for Global HIV/AIDS and Other  Health Assistance in Fiscal Years 2001-2008. GAO-11-64. Washington,  D.C.: October 8, 2010.
President’s Emergency Plan for AIDS Relief: Efforts to Align Programs  with Partner Countries’ HIV/AIDS Strategies and Promote Partner  Country Ownership. GAO-10-836. Washington, D.C.:   September 20, 2010.
President’s Emergency Plan for AIDS Relief: Partner Selection and  Oversight Follow Accepted Practices but Would Benefit from Enhanced  Planning and Accountability. GAO-09-666. Washington, D.C.:   July 15, 2009.
Global HIV/AIDS: A More Country-Based Approach Could Improve  Allocation of PEPFAR Funding. GAO-08-480. Washington, D.C.:   April 2, 2008.
Global Health: Global Fund to Fight AIDS, TB and Malaria Has Improved  Its Documentation of Funding Decisions but Needs Standardized  Oversight Expectations and Assessments. GAO-07-627. Washington,  D.C.: May 7, 2007.
Global Health: Spending Requirement Presents Challenges for Allocating  Prevention Funding under the President’s Emergency Plan for AIDS  Relief. GAO-06-395. Washington, D.C.: April 4, 2006.
Global Health: The Global Fund to Fight AIDS, TB and Malaria Is  Responding to Challenges but Needs Better Information and  Documentation for Performance-Based Funding. GAO-05-639.  Washington, D.C.: June 10, 2005.
Global HIV/AIDS Epidemic: Selection of Antiretroviral Medications  Provided under U.S. Emergency Plan Is Limited. GAO-05-133.  Washington, D.C.: January 11, 2005.
Global Health: U.S. AIDS Coordinator Addressing Some Key Challenges  to Expanding Treatment, but Others Remain. GAO-04-784. Washington,  D.C.: June 12, 2004.
Global Health: Global Fund to Fight AIDS, TB and Malaria Has Advanced  in Key Areas, but Difficult Challenges Remain. GAO-03-601. Washington,  D.C.: May 7, 2003.