Water and Sanitation Assistance: USAID Has Increased Strategic Focus but Should Improve Monitoring
Published: 2015-10-06T00:00:00

Fast Facts:

Highlights:
section_title: Why GAO Did This Study
paragraphs: Millions of people in developing countries lack access to safe water and improved sanitation. Congress passed the Senator Paul Simon Water for the Poor Act of 2005 to improve access to safe water and sanitation for developing countries. In 2013, USAID released its first Water and Development Strategy , which includes the objective of improving health through sustainable WASH.
GAO was asked to review USAID's WASH efforts. Focusing on WASH activities in 9 selected countries, this report (1) describes recent activities and funding, (2) assesses USAID missions' efforts to plan and implement activities, and (3) assesses USAID's monitoring of activities. GAO selected a nongeneralizable sample of 9 countries from USAID's list of 22 priority WASH countries. These 9 countries received about 53 percent of funding attributed to WASH for fiscal years 2012 and 2013. GAO also selected 16 activities for detailed review in the 9 countries, primarily on the basis of levels of funding. GAO analyzed USAID WASH funding data for fiscal years 2012 through 2014 and reviewed agency documents, interviewed mission officials, and visited sites in 2 African countries.
section_title: What GAO Found
paragraphs: U.S. Agency for International Development (USAID) missions in the 9 countries GAO selected for its review reported implementing a variety of water supply, sanitation, and hygiene (WASH) activities in fiscal years 2012 through 2014. WASH activities included capacity building, behavior-change communication, infrastructure construction, technical assistance, policy and governance, and financing. The missions' funding for WASH activities in these countries ranged from $4.4 million to $53.4 million.
Note: Funding shown generally represents allocations for activities through Sept. 2014.
USAID missions in these 9 countries are taking steps to develop and implement plans for WASH activities, with some missions making more progress than others. These missions are also generally taking steps to address long-term sustainability when planning WASH activities, as directed by USAID guidance, including the Water and Development Strategy . USAID is in the process of developing additional guidance to help all its missions address the sustainability of WASH activities.
The completeness and accuracy of USAID's monitoring of WASH activities varied in the 9 selected countries. GAO found that, inconsistent with agency guidance, these missions did not (1) consistently set annual targets for 6 of 16 WASH activities, (2) disaggregate beneficiaries by gender for 6 of 10 water supply and sanitation activities, (3) verify the accuracy of beneficiary data for 3 of 10 water supply activities, and (4) report accurate numbers of beneficiaries for 6 of 8 sanitation activities. Mission officials cited a variety of reasons for adhering inconsistently with agency guidance in some instances and in others the reasons for inconsistent adherence were not clear. These limitations in the completeness and accuracy of monitoring information for WASH activities may inhibit the effectiveness of USAID's oversight of such activities and affect its ability to accurately report on progress in increasing access to safe water and sanitation.
section_title: What GAO Recommends
paragraphs: GAO recommends that USAID take steps to improve monitoring and reporting of WASH activities, by identifying and addressing reasons for missions' inconsistent adherence with agency guidance. USAID generally concurred with the recommendations and, in particular, outlined steps it is taking to address the report's second recommendation.

section_title: Letter
paragraphs: In many developing countries, a lack of access to safe water and basic  sanitation severely limits economic growth and development and leads to  suffering and death for millions each year. In 2005, Congress passed the  Senator Paul Simon Water for the Poor Act of 2005 (2005 Act), making  access to safe water and basic sanitation for developing countries a U.S.  foreign policy objective. The 2005 Act required, among other things, that  the Department of State (State), in consultation with the U.S. Agency for  International Development (USAID), develop and implement a U.S. water  and sanitation strategy that included designation of high-priority countries  where need is greatest. In 2010, we reported on State’s implementation  of U.S. water and sanitation assistance. In 2014, Congress passed the  Senator Paul Simon Water for the World Act of 2014 (2014 Act) to  strengthen implementation of the 2005 Act by improving the capacity of  the U.S. government to implement, monitor, leverage, and evaluate  programs to provide first-time or improved access to safe drinking water,  sanitation, and hygiene to the world’s poorest on an equitable and  sustainable basis. The United States has primarily provided water and  sanitation assistance through USAID. In fiscal years 2005 through 2014,  USAID reported allocating about $3.5 billion for water supply, sanitation,  and hygiene (WASH).
You asked us to review the U.S. government’s international water and  sanitation assistance. This is the second of two reports responding to  your request. In July 2014, we provided information about U.S. agencies’  funding for international water-related assistance and the extent to which  the agencies complied with congressional spending requirements. In this  report, we (1) describe the types of WASH activities USAID has  implemented in selected countries and funding it has provided for these  activities, (2) assess the extent to which USAID guidance has informed  the agency’s efforts to plan and implement WASH activities in these  countries, and (3) assess USAID’s monitoring and evaluation of selected  WASH activities.
We focused our review on USAID’s WASH activities in 9 selected  countries: the Democratic Republic of the Congo (DRC), Ethiopia, Haiti,  Indonesia, Kenya, Tanzania, Senegal, Uganda, and Zambia. We selected  these countries from 22 that USAID designated as priorities for WASH  assistance in fiscal year 2014. The 9 selected countries accounted for  $155 million—about 53 percent—of funding attributed to congressional  spending requirements for WASH by all tier 1 and tier 2 priority countries  in fiscal years 2012 and 2013. For all three reporting objectives, we  interviewed USAID and State officials and conducted semi-structured  telephone interviews with officials at the USAID missions in the 9 selected  countries. Because we judgmentally selected these 9 countries, our  findings from these interviews cannot be generalized to all USAID  missions. We also conducted fieldwork in Tanzania and Ethiopia to meet  with U.S. and host country officials, implementing partners, and  beneficiaries and to conduct site visits for USAID water and sanitation  activities. To describe the types of WASH activities USAID has  implemented and funding it has provided for these activities, we obtained  funding data and descriptive information from each mission regarding its  ongoing or planned WASH activities in fiscal years 2012 through 2014. To  assess the reliability of the data and information we obtained, we  reviewed documentation and interviewed agency officials to identify and  correct any missing data and any errors. We determined that the data and  information were sufficiently reliable to provide general information on  funding and types of WASH activities.
To assess the extent to which USAID guidance informed planning and  implementation of WASH activities in the 9 selected countries, we  reviewed USAID guidance, mission-level WASH plans, and other  documents. We assessed the status of the USAID missions’ efforts to  develop WASH plans, steps they have taken to implement these plans  and to incorporate USAID guidance into their recent or planned WASH  activities, and steps they have taken to address sustainability. We  conducted a detailed review of documents and information for 16  activities that we selected on the basis of factors such as levels of  funding and types of activities implemented in the 9 countries in fiscal  years 2012 through 2014.
To assess monitoring of selected WASH activities in the 9 countries, we  reviewed award agreements and modifications, performance  management plans, monitoring and evaluation plans, and monitoring  reports for the 16 activities that we selected for detailed review. In  addition, to assess USAID evaluations for activities in the 9 countries, we  assessed the quality of 14 evaluations for WASH activities completed in  fiscal years 2012 through 2014 against established principles of  evaluation, which correspond with USAID’s requirements for evaluations.  We did not assess the selected USAID missions’ compliance with the  requirement to conduct evaluations. Appendix I provides additional  information about our objectives, scope, and methodology.
We conducted this performance audit from July 2014 to October 2015 in  accordance with generally accepted government auditing standards.  Those standards require that we plan and perform the audit to obtain  sufficient, appropriate evidence to provide a reasonable basis for our  findings and conclusions based on our audit objectives. We believe that  the evidence obtained provides a reasonable basis for our findings and  conclusions based on our audit objectives.
subsections: section_title: Background
subsections: section_title: Legislation and Key Agency Roles Related to WASH Assistance
paragraphs: The 2005 Act made access to safe water and sanitation a U.S. foreign  policy objective. Specifically, the act aimed to promote the provision of  access to safe water and sanitation to countries, locales, and people with  greatest need, including the very poor, women, and other vulnerable  populations.
Congress passed the 2014 Act to strengthen the 2005 Act by improving  the capacity of the U.S. government to implement, monitor, and evaluate  programs that increase access to safe water, sanitation, and hygiene.  The 2014 Act requires, among other things, that USAID ensure that  WASH projects are designed to promote maximum impact and long-term  sustainability. The 2014 Act also calls for rigorous monitoring and  evaluation to assess improvements in WASH.
State’s Office of Conservation and Water within the Bureau of Oceans  and International Environmental and Scientific Affairs is responsible for  the development and implementation of U.S. foreign policy on  international water and sanitation assistance. USAID’s Office of Water  within the Bureau for Economic Growth, Education and Environment is  responsible for coordinating, managing, and overseeing USAID’s  response to water policy initiatives, including the 2014 Act.
section_title: Annual Spending Requirement for WASH Assistance
paragraphs: Annual appropriations acts funding foreign operations have established a  spending requirement for WASH. These spending requirements have  ranged from $315 million to $365 million since 2012. USAID and State  have provided annual guidance to USAID missions regarding activities for  which they may attribute funding to the annual spending requirement. The  guidance notes that for attribution, proposed activities must be able to  demonstrate an impact on water supply, sanitation, or hygiene through  objectively verifiable indicators to measure progress. The 2005 Act  required State to submit an annual report to Congress detailing the status  of WASH efforts; these reports included information from USAID about  funds allocated to WASH activities to meet the spending requirements for  WASH. This reporting requirement was repealed by the 2014 Act.
section_title: USAID Water and Development Strategy
paragraphs: Improving health outcomes through the provision of sustainable WASH is  the first strategic objective of USAID’s Water and Development Strategy,  2013-2018 (Water Strategy). To achieve this objective, the Water  Strategy includes three key WASH-related goals:  Increase first-time and improved access to sustainable water  supply. First-time access refers to access to an improved water  source that is gained by previously unserved populations. Improved  access refers to enhancing existing access to, and the quality of, an  already improved water supply.
Increase first-time and improved access to sustainable  sanitation. First-time access to improved sanitation generally refers  to access to a pit latrine with a slab, septic system, or similar type of  improved sanitary facility. Improved access to sanitation generally  refers to improvement of an existing sanitation facility.
Increase adoption of key hygiene behaviors. The Water Strategy  recommends the promotion of three hygiene practices with the  greatest demonstrated impact on health: (1) hand washing with soap  at critical times; (2) safe disposal and management of excreta; and (3)  improving household water storage, handling, and treatment.
Figure 1 shows USAID’s definitions for improved and unimproved drinking  water and sanitation.
In March 2014, USAID issued the Water Strategy Implementation Field  Guide (Field Guide) as a tool to help missions understand and implement  the Water Strategy. The Field Guide requires that USAID missions track  progress toward the three WASH-related goals using standard indicators  and some custom indicators, as shown in table 1.
State’s annual reports to Congress detailing the status of WASH projects  have included country-level results for two of the standard WASH  indicators: (1) number of people gaining access to an improved drinking  water source and (2) number of people gaining access to an improved  sanitation facility. This information has also been reported in USAID’s  annual report on international water-related assistance, Safeguarding the  World’s Water.
The Water Strategy projected that during the next 5 years, at least 10  million persons would receive sustainable access to improved water  supply and 6 million persons would receive sustainable access to  improved sanitation. These projections included persons receiving first- time and improved access to water supply and sanitation.
According to the Water Strategy and the Field Guide, to achieve the  greatest impact, WASH projects should include the following elements: expanded access to “hardware” (e.g., water and sanitation  infrastructure and hygiene commodities); required “software” activities to promote behavior changes for  sustained improvements in water and sanitation access/service and  hygiene practices; and an improved enabling policy and institutional environment, including  strengthened financial frameworks and public-private partnerships for  WASH.
The Field Guide states that the level of effort in each area may vary  depending on local context and other factors. In addition, the Water  Strategy and the Field Guide highlight gender issues in the water sector  as a key focus. They note that the burden of inadequate access to water  and sanitation often falls heavily on women and girls and that WASH  activities should promote gender equality and female empowerment to  address the needs and opportunities of both men and women.
section_title: WASH Sustainability
paragraphs: The Water Strategy emphasizes the importance of sustainable WASH  services. USAID policy defines sustainable services as public services in  which host country partners and beneficiaries take ownership of  development processes and maintain project results beyond the life of a  USAID project. The strategy notes that the pillars of sustainability for  water projects include integrated water resource management, sound  governance, and appropriate environmental design, among other factors.  In particular, regarding projects to provide first-time and improved access  to quality water supply services, the strategy states that methods for  ensuring the sustainability of water quality in the long term should be  incorporated into project design and that this may include developing  monitoring systems to ensure that water quality and supply are sustained  at acceptable levels. Further, regarding sanitation services, the strategy  notes that it supports the development and testing of improved, low-cost  sanitation and waste management technologies, as well as innovative  management and financing approaches, to ensure sustainability and to  facilitate more rapid expansion of basic sanitation solutions.
section_title: USAID Priority Countries for WASH Assistance
paragraphs: In fiscal year 2014, USAID designated 22 countries as tier 1 or tier 2  priorities for WASH assistance on the basis of their need and the  opportunity to achieve significant impact. The 6 tier 1 countries are  those where USAID found an opportunity to have a transformative impact  on national-level policies and to leverage host country resources for the  development and implementation of assistance. The 16 tier 2 countries  are those where USAID determined that relatively small investment levels  were likely to generate a significant impact in at least one dimension of  WASH. USAID reported attributing about $463.1 million to the annual  spending requirements for WASH in fiscal years 2012 through 2014 for  these priority countries. Figure 2 shows the locations of the tier 1 and 2  priority countries.
section_title: USAID Missions’ WASH Activities and Funding Varied
paragraphs: In the nine countries we selected for our review, USAID missions reported  $214.5 million in WASH funding for 74 activities, with 6 key focus areas.  Each mission’s total funding for WASH activities in these nine countries  ranged from about $53.4 million, in Indonesia, to about $4.4 million, in  Haiti. Missions reported implementing from as many as 19 WASH  activities, in Kenya, to as few as 3 activities, in both Tanzania and  Zambia. The most frequent key focus areas for these activities were  capacity building and behavior-change communication, followed by  infrastructure construction, technical assistance, policy and governance,  and financing. Figure 3 provides additional detail about the activities  implemented by each selected mission in fiscal years 2012 through 2014.
Table 2 describes the 74 activities’ key focus areas and provides  examples of the types of WASH activities in each focus area for the nine  selected countries.
Missions noted that some of the 74 reported activities included WASH as  one of several other components, such as maternal-child health, nutrition,  or natural resources management. For example, the Senegal mission’s  Yaajeende Agriculture activity is an agricultural activity that aims to  increase nutritional status by diversifying foods produced and eaten. The  activity includes a WASH component to expand access to clean drinking  water and improved sanitation and also includes water resources  management efforts to promote effective small irrigation technologies. In  another example, the DRC mission’s Integrated Health Project aims to  improve the enabling environment and increase the availability of  services, products, and practices for family planning; maternal, newborn,  and child health; nutrition, malaria, and tuberculosis; and WASH in  targeted health zones.
Additionally, the nine missions reported implementing WASH activities  targeting three types of geographic areas: rural, peri-urban (e.g., small  towns), and urban. For example, the Tanzania mission’s Integrated  Water, Sanitation, and Hygiene activity targeted rural and peri-urban  areas (e.g., small towns) to provide support including community piped  water schemes, rehabilitated wells, training for community groups to  operate and maintain water systems, and school latrines, among other  efforts. In contrast, the Indonesia mission’s Indonesia Urban Water,  Sanitation, and Hygiene activity aimed to increase access to clean water  and improve sanitation facilities for people in urban settings, where about  half of Indonesia’s population lives. According to the Indonesia mission,  the activity focused on fostering demand for WASH, building capacity,  and strengthening the policy and financing environment for WASH  through its work with central and local governments, including entities  responsible for delivering water and sanitation services.
Table 3 includes information about the geographic focus of the 74  activities in the nine selected countries.
section_title: USAID Is Taking Steps to Enhance Strategic WASH Approach and Address WASH Sustainability
paragraphs: In the nine countries we selected for our review, we found that USAID  missions are taking steps to develop and implement WASH plans and are  incorporating the Water Strategy’s principles and approach into recent  and planned WASH activities. The missions are considering sustainability  as part of WASH project planning, and USAID’s Office of Water is  developing guidance for missions specific to WASH sustainability.
subsections: section_title: Missions in Selected Countries Have Taken Steps to Develop and Implement WASH Plans and Enhance Strategic WASH Approach
paragraphs: We found that the missions in the nine selected countries have made  varying degrees of progress in developing WASH plans. In addition, we  found that these missions have generally taken steps to implement  WASH plans and enhance the strategic approach in recent WASH  projects.
Our review of documents describing these missions’ strategic approach to  WASH, as well as interviews with officials at the nine missions, found that  five of the missions had completed WASH plans. Missions’ WASH plans  may consist of one or more project appraisal documents (PAD), which  document a project’s design and expected results. The remaining four  missions were in the process of developing or finalizing such plans.
Five of the nine missions—in Ethiopia, Indonesia, Kenya, Tanzania,  and Zambia, respectively—had completed WASH plans and taken  steps to implement these plans as of July 2015. The Indonesia and  Tanzania missions initially developed WASH plans in fiscal year  2009. The Zambia, Kenya, and Ethiopia missions completed WASH  plans more recently, in the period from fiscal year 2013 to fiscal year  2015.
Three of the nine missions—in the DRC, Senegal, and Uganda,  respectively—were in the process of finalizing WASH plans as of July  2015.
One mission, in Haiti, had begun developing a plan for a more  strategic approach to its WASH activities as of February 2015.  According to mission officials, the mission had focused its recent  water-related activities on agriculture, water resources management,  and water productivity. In response to the Water Strategy, the mission  has begun to develop a plan for future WASH activities that will  consider findings from a fiscal year 2015 mission WASH sector  assessment.
Our review of missions’ WASH project documents, as well as semi- structured interviews with officials at the nine missions, also found that  these missions have generally taken steps to implement WASH plans and  incorporate the Water Strategy’s strategic principles and approach in  recent WASH projects, such as by implementing activities to support  expanded access to water and sanitation infrastructure along with  activities to support capacity building and behavior change. Although four  of the nine missions acknowledged that their prior WASH projects lacked  strategic focus, all nine missions cited ongoing efforts to develop more  focused, strategic, and impactful WASH projects in accordance with the  Water Strategy. government support and leadership for WASH were critical to  sustainability and continued improvement in the sector. The Tanzania  mission’s primary WASH activity, the Tanzania Integrated Water,  Sanitation, and Hygiene Program, focused on increasing water supply  through a multiple-use water services approach, which the mission’s  WASH plan describes as considering domestic and productive water  needs to maximize benefits and increase the cost-effectiveness of WASH  investments (see sidebar).
Missions that recently completed, or are in the process of finalizing,  WASH plans have also taken steps to incorporate the Water Strategy’s  principles and approach into recent activities, such as by implementing  activities to support expanded access to “hardware” (e.g., water and  sanitation infrastructure), as well as to support “software” activities (e.g.,  efforts to build capacity and promote behavior change). For example, the  Zambia mission’s Schools Promoting Learning Achievement through  Sanitation and Hygiene activity provided support for latrine construction,  rehabilitated water points, and handwashing facilities, as well as hygiene  education and capacity building related to operations and maintenance of  WASH infrastructure. In addition, missions have supported community-led  total sanitation efforts to create demand for improved sanitation. For  instance, the Senegal mission’s Millennium Water and Sanitation activity  included support for community-led total sanitation, among other  components.
Additionally, missions cited examples of steps they have planned to  enhance the strategic focus and potential impact of WASH projects going  forward, such as by increasing investments in sanitation, encouraging  local government ownership of efforts, supporting public-private  partnerships and WASH financing options, and enhancing activities’ focus  on gender issues. For example, six of the nine missions—in Ethiopia,  Indonesia, Senegal, Tanzania, Uganda, and Zambia, respectively—cited  plans to increase the focus of their WASH activities on sanitation. The  Uganda mission noted that although it had previously implemented  several activities that addressed sanitation on a small scale, it plans to  focus exclusively on sanitation in its upcoming primary WASH activity.  According to the mission, the planned activity will seek to improve access  to affordable and acceptable sanitation through public-private  partnerships, support for affordable sanitation financing options, and  subsidization schemes to reach the poor.
Figure 4 summarizes our findings regarding the status of the nine  selected missions’ WASH plans and their recent and planned WASH  activities as of July 2015. Appendix II provides additional information  about the selected missions’ WASH activities.
section_title: Missions Are Considering Sustainability as Part of WASH Planning
paragraphs: Our review of mission documents and information showed that the nine  selected missions are generally taking steps to address sustainability as  part of their WASH project planning, as described below:
WASH PADs for all five of the missions that had completed these  documents—in Indonesia, Kenya, Tanzania, Uganda, and Zambia— included sustainability analyses. For example, the Kenya mission’s  WASH PAD included an annex with a two-page sustainability analysis  that described the mission’s planned steps to address challenges to  sustainable WASH, including challenges related to choice of  technologies and technical approaches, weaknesses in governance,  long-term financing needs, and the need for sustained behavior  changes. The Uganda and Zambia missions’ PADs sustainability  analyses did not address WASH specifically but described  sustainability considerations in the context of broader project efforts  (e.g., health or education).
Missions’ award documents for the 16 activities we reviewed in detail  generally described sustainability considerations. For example, the  award agreement for a WASH activity in Senegal noted that decades  of development experience have shown that merely building water  and sanitation infrastructure is not sufficient to deliver adequate  service or to ensure that such services are sustainable. The  agreement added that WASH efforts would incorporate principles  such as optimal balance between hardware and software activities,  local ownership and decentralized management of WASH  infrastructure and service delivery, and use of appropriate and  affordable technologies.
Five of the nine missions conducted more in-depth WASH  sustainability assessments to inform planning and design of WASH  activities. The Ethiopia mission completed two studies between April  2012 and February 2013 to assess existing water supply schemes  and potential strategies to improve sustainability for the mission’s  primary ongoing WASH activity, which was implemented in several  regions. In addition, in 2013 and 2014, USAID piloted a tool that it  developed with Rotary International to provide an in-depth  assessment of the sustainability of several WASH activities  implemented by missions in three of the nine countries we selected  for our review—Indonesia, Kenya, and Tanzania. The assessments  included an overview of the WASH sector in each country; a  description of steps taken to apply the sustainability tool; results of  using the tool, including scores for institutional, management,  financial, technical, and environmental sustainability; key findings; and  priority areas for action and recommendations by intervention  category (i.e., water supply, sanitation, or hygiene). A USAID official  noted that for this type of tool to be effective, USAID would need to  use it systematically and incorporate a feedback loop to ensure that  results are incorporated into future activities. The Indonesia mission  incorporated the results of using the tool into its latest WASH plan,  while the Kenya and Tanzania missions completed their latest WASH  plans before the results of the tool were available. However, Kenya  and Tanzania mission officials stated that they are considering results  of the assessments in planning their upcoming activities.
Mission officials in the selected countries generally told us that they plan  to enhance the focus of future activities on sustainability. For example,  officials at the Senegal mission noted that they conducted a sustainability  analysis in the process of developing the mission’s forthcoming PAD  related to WASH. To enhance its focus on sustainability, the mission is  planning, among other efforts, to implement private sector reform  activities and more directly engage with the government of Senegal to  better ensure ownership of water and sanitation projects. Mission officials  in the DRC said that they are in the process of determining the cost of  monitoring all of their WASH activities after implementation is completed  to better assess sustainability. The officials noted that UNICEF, which is  implementing one of the mission’s WASH activities, is using its own  resources to monitor the activity to assess sustainability 1 year after  completion and that the mission would like to standardize the process for  other activities.
section_title: USAID’s Office of Water Is Developing Guidance Specific to WASH Sustainability
paragraphs: USAID’s Office of Water has begun to develop guidance to address  WASH sustainability. The March 2014 Field Guide states that USAID’s  Office of Water will develop guidance for missions to program effectively  for sustainable WASH services, including sustainability indicators,  monitoring options, and tools for assessing sustainability. To inform  future WASH sustainability guidance, in December 2014, USAID’s Office  of Water developed a draft technical paper that included  recommendations for designing projects to achieve sustainable WASH  service and for confirming sustained results through longer-term  monitoring. The draft paper described, among other things, the types of  indicators that could be used to assess sustainability, including the  functionality and reliability of WASH services. Additionally, the draft paper  recommended that USAID’s Water Office should  supplement its sustainability guidance with questions and resources  for missions to consider in WASH project design, adopt sustainability indicators to assess the functionality and reliability develop an approach to post-project monitoring, and  conduct an evaluation on the sustainability of WASH services in a  sampling of countries.
According to USAID officials, USAID’s Office of Water intends to finalize  its guidance for addressing the sustainability of WASH activities in 2015.
The officials stated that the guidance will likely be broad and include a  menu of options, tools, and resources that can be tailored to the context  in which each mission implements WASH activities.
section_title: Monitoring and Reporting for USAID’s WASH Activities in Selected Countries Had Limitations, although Evaluations Were Generally Sound
paragraphs: Our detailed review of 16 WASH activities in the nine selected countries  found limitations in the monitoring and reporting of some activities’  performance, although most evaluations we reviewed were sufficiently  reliable and methodologically sound for their intended purposes.  Monitoring plans for 6 activities did not consistently include annual targets  for key WASH indicators in accordance with USAID requirements, limiting  the missions’ ability to measure progress toward WASH goals. Also,  while the annual reports submitted by the 16 activities’ implementers  generally included performance data for the key WASH indicators, the  reports for 6 activities did not present data disaggregated by gender as  required by USAID policy. Moreover, contrary to agency guidance,  missions did not verify beneficiaries for at least 3 activities aimed at  increasing access to improved drinking water sources and overstated  beneficiaries for 6 activities aimed at increasing access to improved  sanitation facilities, calling into the question the accuracy of USAID’s  annual reporting about progress toward these WASH goals. The reasons  for the missions’ inconsistent adherence to agency guidance regarding  annual targets, gender-disaggregated data, and verification of  beneficiaries were generally unclear, while USAID officials provided  differing reasons for the inaccurate reporting for sanitation activities. In  contrast, 12 of the 14 performance evaluations that we reviewed were  sufficiently reliable and methodologically sound for their intended  purposes.
subsections: section_title: Monitoring Plans and Reports for Several Activities Lacked Annual Targets, and Reports Did Not Consistently Disaggregate Beneficiaries by Gender
paragraphs: For more than one-third of the 16 activities we reviewed, the monitoring  plans did not consistently include annual targets for key WASH indicators,  such as the number of people gaining access to an improved drinking  water source. USAID requires that at the start of each activity, the  implementer establish a monitoring plan with indicators and associated  targets to assess progress on an activity. Additionally, State guidance to  USAID missions requires that proposed activities demonstrate impact  through objectively verifiable indicators to measure progress toward  WASH goals, if funds are attributed to the annual congressional spending  requirement for international water and sanitation assistance. We found  that, of the 16 activities, 3 lacked annual targets for key WASH indicators  for the entire duration of the activity and 1 lacked annual targets for key  WASH indicators for the remaining 2 years of the activity’s 5-year  duration. Additionally, 1 activity included annual targets for the drinking  water and sanitation component but not for the hygiene component; for  another activity, annual targets were clearly identified for 2 years of its 5- year duration.
While annual reports for the 16 activities we reviewed generally included  performance data for key WASH indicators, the reports for more than  half of activities with beneficiary indicators did not disaggregate data for  these indicators by gender as USAID policy requires. Ten activities had  indicators to measure numbers of beneficiaries gaining increased access  to an improved water source or improved sanitation facility. However, for  6 of these activities, the missions did not disaggregate performance data  by beneficiaries’ gender, making it difficult for USAID to assess the  activity’s contributions to gender equality and female empowerment.
Reasons for the inconsistency that we observed in the selected missions’  adherence to agency guidance regarding annual targets and gender- disaggregated data were unclear. USAID officials in Washington, D.C.,  noted the absence of consistent reasons among the nine missions for a  lack of regular reporting on annual targets and of gender-disaggregated  data. Officials at the DRC mission noted that staffing constraints had  generally limited their ability to monitor their WASH activities and that they  planned to hire additional staff to improve monitoring.
Figure 5 summarizes our findings regarding the nine selected missions’  compliance with USAID requirements in documenting targets and  reporting performance data for the 16 activities that we reviewed.
section_title: Missions Did Not Verify Results for Some Water Activities and Overstated Results for Most Sanitation Activities
paragraphs: For some of the 10 activities with a water component that we reviewed,  the missions did not verify the numbers of beneficiaries as required, and  for most of the 8 activities with a sanitation component, the missions  overstated the numbers of beneficiaries. As a result, because activity  performance data contribute to USAID’s annual public reporting of WASH  results, the data that USAID uses to report the numbers of people gaining  access to an improved drinking water source and to an improved  sanitation facility annually may not be accurate. The reasons for lack of  verification for water beneficiaries were generally unclear, and mission  officials, as discussed later, generally provided varying reasons for the  inaccurate reporting for sanitation activities. Figure 6 summarizes our  findings regarding the nine selected USAID missions’ compliance with  State and USAID requirements for verifying beneficiaries of water  activities and reporting beneficiaries of sanitation facilities.
For 7 of the 10 activities that we reviewed, the missions and  implementers did not undertake efforts to verify reported beneficiaries or  did not document such efforts, calling into question the accuracy of the  reported results. State and USAID guidance for the standard indicator of  access to an improved drinking water source requires that the  implementer or evaluator verify these estimates by assessing factors  such as the amount of time the user spent in collecting water and the  quantity of water produced by the new or rehabilitated water source.  The reasons for the lack of verification, or documentation of verification,  of beneficiaries of the 7 activities were generally unclear.
Documents for only 3 of the 10 activities—in Tanzania, Ethiopia, and  Indonesia, respectively—evidenced verification efforts and, in one case,  corrective actions related to verifying results of activities to provide  access to improved drinking water. For 4 other activities, it was unclear,  on the basis of documents that the missions provided and mission  officials’ responses to our queries, whether the reported results were  verified. For example, although officials at the Senegal mission stated that  they undertook regular site visits and data quality assessments to verify  estimated beneficiaries, they did not provide documentation of such  verification. Moreover, the midterm evaluation for this activity noted that  estimating the number of beneficiaries on the basis of the Senegalese  government’s assumptions was feasible at the planning stage but not  during implementation, when the number of beneficiaries could be  verified. The evaluation also noted that, absent precise numbers of  beneficiaries, it was difficult to determine the cost-effectiveness of the  investment.
For the remaining 3 activities—2 in the DRC and 1 in Zambia—mission  officials informed us that they had not verified the reported results and  cited varying reasons.
In the DRC, annual reports for 2 of the WASH activities we reviewed  indicated that more than 900,000 people gained access to an  improved drinking water source between fiscal years 2012 and  2014. In USAID’s annual report for fiscal year 2013, the number of  beneficiaries reported as gaining access to an improved drinking  water source in the DRC constituted more than 13 percent of  worldwide beneficiaries (446,989 out of 3,509,090). Officials at the  DRC mission informed us that they had been unable to conduct a  data quality assessment planned for October 2014 because of  challenges that included security concerns and difficulties of traveling  in a country with limited roads. Mission officials also noted that  estimating numbers of people who gained access to an improved  drinking water source was complicated by the displacement of  population in the areas where USAID was implementing its WASH  activities.
In Zambia, the mission reported in fiscal years 2012 and 2013 that  82,606 and 62,098 people, respectively, gained access to an  improved drinking water source as a result of the mission’s two school  WASH activities. Officials at the Zambia mission informed us that  although they undertook verification of data for the ongoing school  WASH activity’s sanitation component, they did not verify all reported  beneficiaries of the water component, which included the school  population as well as people from the surrounding communities. The  officials noted that while beneficiaries gaining access to improved  drinking water in schools could be verified, it was not possible to verify  beneficiaries from communities surrounding the school, who also  have access to water points constructed or rehabilitated under this  activity.
subsections: section_title: Inaccurate Reporting of Beneficiaries Gaining Access to Improved Sanitation Facilities
paragraphs: For six of the eight activities aimed at increasing access to improved  sanitation facilities that we reviewed, the activities’ implementers reported  beneficiaries for facilities that did not meet USAID’s definition of an  improved facility. As a result, the data reported to track progress toward  the goal of increasing access to improved sanitation are likely overstated.  USAID uses WHO and UNICEF definitions of an improved sanitation  facility, which state that a pit latrine without a slab or platform is an  unimproved sanitation facility and that only facilities that are not shared or  are not public are considered improved.
For six of the eight sanitation activities that we reviewed, implementers  tracked and reported numbers of people gaining access to sanitation  facilities that included unimproved latrines and shared facilities. Mission  officials generally provided differing reasons for the inaccurate reporting  for these activities, such as perceived agency emphasis on reporting  beneficiaries and adherence to host-government policy or practice.  USAID officials in Washington, D.C., noted that, to some extent, the  missions’ inconsistency in accurately reporting on the sanitation indicator  resulted from inadequate understanding of USAID’s definition of improved  sanitation facilities among some mission staff overseeing these activities.
In the DRC, more than 520,000 people gained access to an improved  sanitation facility in fiscal years 2012 through 2014 as a result of the  mission’s primary WASH activity, according to activity annual  reports. However, according to mission officials, the activity’s  reported beneficiaries included those who gained access to  household-built latrines, including basic pit latrines. USAID officials in  Washington, D.C., stated that community-led total sanitation, which  reflects the DRC mission’s approach, involves changing people’s  behavior regarding use of sanitation facilities by encouraging them to  build latrines; however, the officials said that these latrines generally  do not meet USAID’s definition of improved sanitation facilities. These  officials noted that the results of community-led total sanitation efforts  should be reported for USAID’s WASH indicator for a community  becoming open-defecation free rather than for the WASH indicator for  increased access to an improved sanitation facility. According to  officials at the DRC mission, the mission reported results of  community-led total sanitation efforts for the indicator for first-time  access to an improved sanitation facility in part because of perceived  USAID headquarters emphasis on reporting numbers of beneficiaries  of WASH assistance. Mission officials noted that headquarters’  emphasis on numbers of beneficiaries had, to some extent, led the  mission to focus on activities to increase direct access to water and  sanitation rather than on efforts to improve institutions or governance.
In Ethiopia, according to the final report for one of the activities we  reviewed, 385,909 people gained access to improved sanitation in  fiscal years 2009 through 2013 as a result of this activity. However,  the latrines built through this activity included pit latrines without slabs  (see fig. 7 for an example), which do not meet USAID’s definition of  improved sanitation facilities; consistent with WHO and UNICEF  definitions, USAID categorizes a basic pit latrine without a slab as an  unimproved sanitation facility. According to Ethiopia mission officials,  the activity did not fund the construction of household latrines but  instead encouraged households to build their latrines with locally  available materials, in consistency with Ethiopian government policy.  Furthermore, mission officials noted that the reported results for this  activity were based on data from the Ethiopian government’s Health  Management Information System, which uses an official Ethiopian  government definition of improved sanitation that is not consistent with  USAID’s definition.
In Indonesia, Kenya, Tanzania, and Zambia, respectively, the  reported beneficiaries of activities to increase access to improved  sanitation facilities included people who gained access to shared  facilities, such as school toilets. For example, the Zambia mission  reported that more than 133,000 people gained access to an  improved sanitation facility in fiscal years 2012 through 2014 as a  result of the activity focused on increasing WASH access in schools.  However, according to USAID guidance, shared sanitation facilities,  such as those in schools and hospitals, cannot be included in the  results to track progress on the number of people gaining access to  improved sanitation facilities, because this indicator is assessed at the  household level. According to officials at the Zambia mission, the data  they reported included the entire population of schools where  sanitation facilities were built; moreover, the ratio of population to  sanitation facility exceeded the national standards. Mission officials  noted that they focused on schools rather than households because  the mission’s WASH efforts are aimed at improving the environment in  schools to improve education outcomes.
section_title: Evaluations of Selected USAID WASH Activities Were Generally Sound
paragraphs: While we found limitations in the monitoring of several of the 16 activities  we reviewed, the WASH activity evaluations that we assessed were, in  general, sufficiently reliable and methodologically sound for their intended  purposes. Ten of the 14 evaluations in our review assessed 9 of the 16  activities that we selected to examine monitoring. As noted previously in  this report, performance monitoring and evaluation are separate activities;  evaluations entail collection of additional information from sources that  may be different from sources of monitoring data. For example,  evaluators of the Ethiopia mission’s primary WASH activity selected a  random sample of 6 out of 41 activity sites and obtained information  through focus groups with potential beneficiaries, interviews with local  government officials, and personal observation of new or rehabilitated  water sources. The evaluators used monitoring data where it was  available for one indicator (final target and final performance data for  beneficiaries gaining access to an improved water source) but  independently collected data via surveys to assess the results of hygiene  and sanitation component for which the implementer established a final  target but did not report on annual targets or performance data. This  example also illustrates that while evaluations are distinct from  monitoring, they can fill some gaps in monitoring information. (See app. III  for USAID’s evaluation findings related to monitoring, outcomes, and  sustainability.)
We assessed 14 evaluations of USAID’s WASH-related activities—2  baseline evaluations, 6 midterm evaluations, and 7 final evaluations—that  were conducted in fiscal years 2012 through 2014 in the nine selected  countries. We assessed the methodological quality of these evaluations  on the basis of established evaluation principles, including the  appropriateness of the evaluation design; clarity of population selection;  clarity of data collection; and adequacy of support for findings,  conclusions, and recommendations.
Of the 14 evaluations, 7 had clearly supported findings, conclusions,  and recommendations, while 5 had certain limitations in their support  for some findings, conclusions, or recommendations. These  limitations included insufficient details about data collection and  unclear or inappropriate criteria for population selection, given  research objectives. However, we determined that, while such  limitations can lead to unsupported findings and limit the usefulness of  findings, conclusions, and recommendations, the limitations in these 5  evaluations were either clearly stated or otherwise did not  substantially detract from the evaluations’ overall purpose or utility.
The 2 remaining evaluations had significant design limitations that  suggested a lack of appropriate support for at least one finding,  conclusion, or recommendation. For example, the sampling  approaches for these evaluations were problematic. Both evaluations  selected locations or participants on the basis of convenience, which  is a nongeneralizable sampling method. This approach was not  appropriate, because the evaluations’ research questions were aimed  at generalizing to the entire population or establishing a baseline for  the future.
Figure 8 shows the results of our assessment of the 14 evaluations of  USAID’s WASH activities.
We found that in some cases, USAID incorporated evaluation results to  improve activity monitoring. Our interviews with mission officials and  reviews of monitoring documents indicated that the selected USAID  missions modified ongoing or planned activities—for at least 6 of the 10  we reviewed that had relevant evaluations—on the basis of evaluation  results. For example, the midterm evaluation of the primary WASH  activity in Indonesia recommended reducing the activity’s target of 40,000  households willing to pay for sanitation improvements, because of the  challenges involved in meeting this goal. As a result, the Indonesia  mission modified its agreement with the implementer and reduced the  target to 15,000 households. Mission officials noted that the midterm  evaluation also provided specific recommendations for future USAID  projects that would be taken into consideration during the design process  for follow-on WASH investments in Indonesia.
Of the 7 final performance evaluations of WASH activities that we  reviewed, 3 were conducted before the activity’s completion, 3 were  conducted within 1 month of the activity’s completion, and 1 was  conducted within 3 months of the activity’s completion. As a result, these  studies were not set up to allow for an assessment or the longer-term  sustainability of these projects. To enable assessments of activities’  impact and sustainability, USAID has identified plans to conduct  evaluations for some WASH activities several years after project  completion. Specifically, USAID’s Water Strategy notes that the agency  plans to conduct assessments of WASH sustainability beyond the typical  USAID project cycle and to provide support for issues that arise  subsequent to completion of WASH activities. Additionally, in an April  2014 document, USAID noted plans to increase its focus on sustaining  development outcomes by, among other things, conducting evaluations 3  to 5 years after project conclusion. According to USAID, such long-term  evaluations provide opportunities to explore the impact of interventions  and may contribute to a deeper understanding of programmatic risk.
section_title: Conclusions
paragraphs: Since 2005, USAID has reported that millions have gained access to  improved drinking water and sanitation facilities as a result of its  assistance. In 2013, USAID issued its first Water Strategy, articulating  goals for the provision of sustainable WASH assistance. In response,  USAID missions in a number of priority countries are developing a more  strategic approach to WASH, and USAID’s Office of Water and missions  have begun taking steps to address sustainability of WASH investments.  Nevertheless, these efforts are still in the early stages.
However, limitations in some missions’ monitoring and reporting for  WASH activities that we reviewed call into question USAID’s ability to  reliably assess and report progress toward its strategic WASH goals.  Unless USAID identifies and addresses factors contributing to missions’  inconsistent adherence to guidance regarding establishing annual targets  for key WASH indicators for all WASH activities and disaggregating  activity data by gender, USAID cannot reliably measure these activities’  contributions to achieving WASH goals or toward gender equality and  women’s empowerment. Moreover, unless USAID identifies and  addresses factors contributing to missions’ inconsistent adherence to  guidance for verifying beneficiaries of water activities and accurately  reporting beneficiaries of sanitation activities, USAID cannot ensure the  accuracy of its annual reports regarding progress in increasing access to  safe water and sanitation.
section_title: Recommendations for Executive Action
paragraphs: To effectively address limitations in missions’ monitoring and reporting of  USAID’s WASH activities, we are making the following two  recommendations to the USAID Administrator. Specifically, with respect  to inconsistent adherence to agency guidance for establishing annual  targets, for reporting gender disaggregated data, for verifying  beneficiaries of water activities, and for accurately reporting beneficiaries  of sanitation activities, USAID should  identify factors contributing to missions’ inconsistent adherence to  agency guidance and  take steps to address these factors.
section_title: Agency Comments and Our Evaluation
paragraphs: We provided a draft of this report to State and USAID. USAID provided  written comments, which appear in appendix IV, as well as technical  comments that we incorporated as appropriate. State did not provide  comments.
In its written comments, USAID generally concurred with our  recommendations and outlined steps it is taking to address our second  recommendation. Following are highlights of USAID’s comments, with our  evaluation:  1.  USAID noted that our definition of “obligations” excludes bilateral  obligations that have not yet been sub-obligated at the mission-level  for WASH activities. As our report states, for the purposes of our  review, we defined obligations as orders placed, contracts awarded,  and similar transactions during a given period that will require  payments during the same or a future period. USAID categorizes  these as “sub-obligations,” because it considers these funds to have  been obligated through a bilateral agreement with the host country.
In reporting funding for nine selected missions’ WASH activities, we  generally reported total allocations for WASH activities ongoing at  each mission in fiscal years 2012 through 2014, including, when  applicable, funding for years before fiscal year 2012 and through fiscal  year 2014. (State and USAID define allocations as the distribution of  resources to bureaus and operating units by foreign assistance  account.) We included obligations when USAID data did not show  WASH activity allocations.  2.  USAID stated that it develops targets through multiple processes and  that it had provided us with performance plans and reports, which  include future-year targets for missions. On the basis of USAID’s  technical comments on our draft report, we revised our assessment to  acknowledge that the mission-wide performance management plans  and reports for fiscal years 2012 and 2013 included annual targets for  the two activities we had selected for Zambia (“Schools Promoting  Learning Achievement through Sanitation and Hygiene” and  “Partnership for Integrated Social Marketing”). Although we included  this information in response to USAID’s comments, it is important to  note that neither the monitoring plans nor the annual reports for these  two activities included annual targets. According to USAID policy  (ADS ch. 203), activity-level monitoring plans feed into project-level  monitoring plans and mission-wide performance management plans.  Therefore, the absence of annual targets in activity-level monitoring  plans calls into question the completeness of the Zambia mission- wide performance management plan. Additionally, given that mission- wide performance plans and reports provide only aggregate target  and performance data, it may not be possible to identify activity- specific targets or performance data from these reports.  3.  USAID stated that our report does not reference certain key  documents that USAID had provided. Although USAID’s letter does  not specify which key documents it is referring to, in its technical  comments, USAID refers to mission-wide performance plans and  reports and to State’s annual Water Key Issue Definition guidance. In  our report, we refer to performance plans and reports as mission-level  annual reporting. In appendix I of our report, we also note that to  assess data reliability, we compared implementer reporting on an  activity with mission-level annual reporting for the nine selected  USAID missions. In addition, our report notes that we reviewed  State’s annual Water Key Issue Definition guidance and assessed the  extent to which WASH activities in the nine missions generally  adhered to this guidance’s requirement regarding activities for which  funds can be attributed to the congressional spending requirement for  water and sanitation. We described the content of State’s guidance in  the background section of our report, where we added a footnote, in  response to USAID comments, to more clearly identify the guidance  documents we referred to.  4.  USAID stated that the inconsistencies identified in our draft report  pertaining to the inaccurate categorization of results were the  byproduct of isolated incidences of reporting against incorrect  indicators. We found inaccuracies in six of eight activities we reviewed  that reported on the standard indicator “number of people gaining  access to an improved sanitation facility.” While USAID’s comments  confirmed these inaccuracies, the agency has not provided any  documentation or other support for its statement that they represent  isolated incidences.  5.  USAID stated that missions are allowed to use custom indicators to  track results against water-directive funded activities. We did not  intend to imply that use of custom indicators was not in compliance  with State and USAID requirements. While we assessed the selected  missions’ use of a number of WASH indicators to track progress for  16 selected activities, our review focused in particular on the standard  indicators for access to improved water source and improved  sanitation. As our report notes, State and USAID have reported  overall progress on WASH using these two indicators, and USAID’s  Water and Development Strategy (2013-2018) uses these two  indicators to project numbers of beneficiaries during the strategy’s 5- year period (i.e., at least 10 million persons would receive sustainable  access to improved water supply and 6 million persons would receive  sustainable access to improved sanitation). Nevertheless, in our  report’s background section, we include a table of standard and  custom indicators that USAID allows missions to use in reporting on  results of WASH activities. Additionally, on the basis of a follow-up  discussion with USAID, we understand that the agency was  particularly concerned that our report might seem to imply that its  Indonesia mission’s use of a custom indicator for verifying households  for the mission’s Water Hibah activity was not in compliance with  agency guidance. As a result, we have added a note to figure 6 to  clarify that USAID guidance allows missions the flexibility to report  using custom indicators.  6. USAID noted that it informed us that verification information could be found in data quality assessments and site visit reports from missions such as Senegal. During our audit work, after the mission informed us that it had conducted verification of beneficiaries through these assessments, we requested data quality assessments from the USAID mission in Senegal. However, the mission did not provide these assessments to us. In the absence of documentation of verification of beneficiaries, we maintain that it is unclear whether the mission verified the number of beneficiaries gaining access to an improved water source.
We are sending copies of this report to the appropriate congressional  committees, the USAID Administrator, and the Secretary of State.  In  addition, the report is available at no charge on the GAO website at  http://www.gao.gov.
If you or your staff have any questions about this report, please contact  me at (202) 512-3149 or gootnickd@gao.gov. Contact points for our  Offices of Congressional Relations and Public Affairs may be found on  the last page of this report. GAO staff who made major contributions to  this report are listed in appendix V.
section_title: Appendix I: Objectives, Scope, and Methodology
paragraphs: In this report, we (1) describe the types of water supply, sanitation, and  hygiene (WASH) activities the U.S. Agency for International Development  (USAID) has implemented in selected countries and the funding it has  provided for these activities; (2) assess the extent to which USAID  guidance has informed the agency’s efforts to plan and implement WASH  activities in these countries; and (3) assess USAID’s monitoring and  evaluation of selected WASH activities.
We focused our review on WASH activities that USAID missions had  implemented in 9 selected countries: the Democratic Republic of the  Congo (DRC), Ethiopia, Haiti, Indonesia, Kenya, Tanzania, Senegal,  Uganda, and Zambia. We selected these countries from the list of 22  countries that USAID designated as priority countries for WASH  assistance in fiscal year 2014. We based our country selection primarily  on the amounts of funding that the missions in these countries attributed  to the congressional spending requirement for international water and  sanitation assistance in fiscal years 2012 through 2013. During that  period, WASH assistance in the 9 selected countries accounted for 53  percent—$155 million—of the funding attributed to the spending  requirement by the missions in all tier 1 and tier 2 priority countries. To  address our three reporting objectives, we interviewed USAID and  Department of State (State) officials and conducted semi-structured  telephone interviews with officials at the nine USAID missions regarding  their efforts to plan, monitor, and evaluate WASH activities. Because we  judgmentally selected the nine USAID missions for our review, our  findings from these interviews cannot be generalized to all USAID  missions. In addition, we conducted fieldwork in Tanzania and Ethiopia  from January 26, 2015, to February 6, 2015. We selected these countries  for fieldwork based on factors such as level of funding and type of WASH  activities implemented in each country. For example, the Ethiopia mission  attributed the most funding to the congressional spending requirement in  fiscal years 2012 through 2013 for a range of WASH activities, and the  Tanzania mission has implemented one long-standing (since 2009)  primary activity focused on WASH.
To describe the types of WASH activities that USAID has implemented  and the funding it has provided for these activities, we developed a data  collection instrument to obtain funding data and descriptive information  from each mission regarding its WASH activities that were ongoing or  planned during fiscal years 2012 through 2014. Each mission’s total  funding for WASH activities generally represents allocations for WASH  components of activities that were ongoing at the mission at any time in  fiscal years 2012 through 2014, including, when applicable, WASH  funding for years before fiscal year 2012 and through fiscal year 2014.  We included obligations in cases for which USAID was not able to provide  allocations for WASH activities, such as when a USAID mission obligated  unplanned funding to an activity for WASH. To assess the reliability of  the data and information we obtained, we reviewed documentation and  interviewed agency officials to identify and correct any missing data and  any errors. We determined that the data and information we gathered  were sufficiently reliable to provide general information about the types of  activities implemented and approximate funding provided for these  activities.
To assess the extent to which USAID guidance informed planning and  implementation of WASH activities, we reviewed USAID guidance,  mission-level WASH plans, and other documents. We reviewed USAID  guidance including: (1) USAID’s Automated Directives System (ADS),  chapter 201, which contains agency policies and procedures and includes  guidance on strategic planning and project and activity design; (2)  USAID’s Water and Development Strategy, 2013-2018 (Water Strategy);  and (3) USAID’s Water and Development Strategy Implementation Field  Guide (Field Guide). We assessed the status of the USAID missions’  efforts to develop WASH plans, steps that they have taken to implement  these plans, and steps that they have taken to adopt the Water Strategy’s  principles and approach for their recent or planned WASH activities.  Missions’ WASH plans may consist of one or more project appraisal  documents (PAD), which document a project’s design and expected  results, as well as a WASH sector assessment or other related  documents that describe the mission’s strategic approach to WASH. To  assess steps that missions have taken to address sustainability, we  reviewed documents that included PADs and sustainability assessments;  we also reviewed award documents for selected activities.
We generally selected two activities per country on the basis of factors  such as the level of funding allocated and types of WASH activities  implemented in fiscal years 2012 through 2014. We based our activity  selection on funding data reported in State’s Foreign Assistance  Coordination and Tracking System for fiscal years 2012 and 2013, data  included in missions’ operational plans for fiscal year 2014, and  discussions with mission officials about their WASH activities between  fiscal years 2012 through 2014. The activities we selected included those  to which the missions in the selected countries had allocated the largest  amounts of WASH funding, with one exception: In Uganda, the selected  activities did not include the Uganda mission’s activity that received the  largest allocations of WASH funding in fiscal years 2012 through 2014,  because the mission had allocated the majority of funding for its largest  WASH activity before fiscal year 2012, did not initially inform us of this  activity, and did not provide the prior-year funding data until after we had  made our activity selection. We selected one activity in Tanzania,  because the Tanzania mission implemented only one primary WASH  activity during the period we reviewed. We initially selected three WASH  activities in Haiti. However, the Haiti mission subsequently informed us  that two of the selected activities did not have a WASH component and  that the mission had incorrectly attributed WASH funds to these activities  and planned to take corrective action to ensure use of attributed funds for  WASH. As a result, we reviewed one WASH activity for Haiti.
To assess monitoring of WASH activities in the nine countries, we  obtained and analyzed documentation for selected WASH activities,  including award agreements and modifications, performance  management plans, monitoring and evaluation plans, quarterly and  annual monitoring reports, and annual funding data. We compared  monitoring plans with quarterly and annual reports to determine whether  specific WASH indicators had been identified for the activity. Since  several activities had multiple WASH indicators, we focused our analysis  on standard indicators related to water, sanitation, or hygiene that were  used to monitor the activity. When an activity’s monitoring plans included  no standard WASH indicators, we identified at least one activity indicator  related to access to water, access to sanitation, or hygiene improvement,  as relevant. For the purposes of this report, we refer to the indicators that  we identified for our review as key WASH indicators. We reviewed each  activity’s monitoring plan and monitoring reports to determine whether (1)  annual targets were identified for the key WASH indicators and (2) results  or performance data for each key WASH indicator were reported on an  annual basis, if applicable. Further, we reviewed monitoring reports to  assess whether they included gender-disaggregated data for indicators  on number of people gaining access to an improved drinking water  source or number of people gaining access to an improved sanitation  facility. We compared the reported performance data for three key WASH  indicators—number of people gaining access to improved drinking water  source, number of people gaining access to an improved sanitation  facility, and number of liters of drinking water disinfected with point-of-use  treatment products as a result of U.S. government assistance—with  USAID and State’s guidance for these indicators to assess the extent to  which the reported data conformed to the definitions in the guidance  document. Although USAID and State guidance required verification of  estimated beneficiaries for the indicator for access to an improved water  source, the guidance did not require verification of estimated numbers of  liters of water purified. In addition to assessing activity performance data  against agency guidance, we conducted internal consistency checks to  assess the reliability of reported data. For example, to the extent feasible,  we compared implementer reporting on an activity with mission-level  annual reporting and USAID-wide annual reporting on WASH indicators.  Because we found data inconsistencies for several activities, as noted in  the report, we did not use the performance data to report the extent to  which activities met intended targets.
To assess USAID evaluations for WASH activities in the nine countries,  we selected all evaluations for WASH activities completed in fiscal years  2012 through 2014. We identified 14 completed evaluations, including  evaluations for 10 of the 16 activities we selected to assess monitoring.  To assess the soundness of the evaluations, we reviewed background  information about programs and evaluation questions, assessed  evaluation design and process, and considered the evaluation results and  limitations of each study. Two GAO specialists conducted these  assessments independently, using a tool that incorporated key elements  of USAID’s scope-of-work checklist for evaluations and considered  various aspects of these issues. The two specialists compared the results  of their independent assessments and came to agreement about all  conclusions. We did not assess selected USAID missions’ compliance  with the requirement to conduct evaluations.
To review background information about the evaluated programs, we  considered whether the evaluations considered evaluator  independence, program objectives and mechanisms, and evaluation  goals. We also assessed whether the relationship between the  evaluation objectives and program design was clear and appropriate.
To assess evaluation design and process, we considered whether the  evaluations clearly described their design; whether appropriate  methods were used to select the locations and people covered by the  study, including whether the evaluations provided sufficient detail  about sampling methods; whether measures used were clearly related  to evaluation questions; and whether data collection and analysis  were sufficient and appropriate. We also assessed whether selection  methods, sample sizes, criteria, measures, data collection, analysis,  and overall design were appropriate, given the evaluation objectives.
To consider evaluation results and limitations, we determined whether  findings, conclusions, recommendations, and lessons learned were  clearly stated, whether stakeholders were given an opportunity to  comment on the results, and whether evaluations provided  information about how results should be used. We also assessed  whether the evaluations clearly and sufficiently described  assumptions and limitations of their design and results, including  potential biases, confounding variables, unintended consequences,  alternative explanations, and methodological limitations. We assessed  whether any findings, conclusions, recommendations, or lessons  learned were appropriately supported and caveated, given the  evaluation design. To summarize evaluation results, we determined  whether the sections of evaluations related to program monitoring,  outcomes, or sustainability contained descriptive information or  appropriately supported findings, conclusions, recommendations, or  lessons learned.
We conducted this performance audit from July 2014 to October 2015 in  accordance with generally accepted government auditing standards.  Those standards require that we plan and perform the audit to obtain  sufficient, appropriate evidence to provide a reasonable basis for our  findings and conclusions based on our audit objectives. We believe that  the evidence obtained provides a reasonable basis for our findings and  conclusions based on our audit objectives.
section_title: Appendix II: USAID WASH Activities in Nine Selected Countries
paragraphs: Following are funding data and descriptive information about water,  sanitation, and hygiene (WASH) activities implemented in fiscal years  2012 through 2014 in the nine countries that we selected for our review— the Democratic Republic of the Congo, Ethiopia, Haiti, Indonesia, Kenya,  Tanzania, Senegal, Uganda, and Zambia.
subsections: section_title: Democratic Republic of the Congo
paragraphs: The U.S. Agency for International Development (USAID) mission in the  Democratic Republic of the Congo implemented seven WASH activities in  fiscal years 2012 to 2014, with WASH activity funding totaling  $26,125,075 (see table 4). The mission reported two primary WASH  activities—the Integrated Health Project and Sustainable WASH  Interventions: Healthy Villages Program in Two Health Zones—focused  on improving access to water, sanitation, and hygiene services in target  locations. Allocations of WASH funding for these two activities totaled  $22,518,470.
section_title: Ethiopia
paragraphs: The USAID mission in Ethiopia implemented 11 WASH activities in fiscal  years 2012 through 2014, with WASH activity funding totaling  $24,055,770 (see table 5). The mission’s largest WASH activity, called  Water, Sanitation, and Hygiene Transformation for Enhanced Resilience,  included $10,984,723 in funding for WASH and focused on providing  water infrastructure in pastoralist areas.
Figure 9 shows a drinking water reservoir and a water point constructed  through the Ethiopia mission’s largest WASH activity.
section_title: Haiti
paragraphs: The USAID mission in Haiti implemented six WASH activities in fiscal  years 2012 through 2014, with WASH activity funding totaling $4,436,481  (see table 6). The mission’s largest WASH activity, called Santé pour le  Développement et la Stabilité d’Haiti, included $1,711,000 in funding for  the WASH component of a broader activity that focused on improving the  health status of Haitians through improved primary care, referral  networks, and management practices at health facilities and in  communities.
section_title: Indonesia
paragraphs: The USAID mission in Indonesia implemented five WASH activities in  fiscal years 2012 through 2014, with WASH activity funding totaling  $53,401,700. The mission’s largest WASH activity, called Indonesia  Urban Water, Sanitation, and Hygiene, included $38,696,403 in WASH  funding and focused on providing access to water and sanitation facilities  in urban areas.
section_title: Kenya
paragraphs: The USAID mission in Kenya implemented 19 WASH activities in fiscal  years 2012 through 2014, with WASH activity funding totaling  $18,245,655. The mission’s largest WASH activity, called AIDS,  Population and Health Integrated Assistance Program (APHIA) Plus  Northern Arid Lands Service Delivery, included $3,024,887 in funding for  WASH. The activity focuses on HIV/AIDS, maternal and child health,  WASH, and nutrition for orphaned and vulnerable children. The mission  reported five similar activities targeting other regions of the country.
section_title: Senegal
paragraphs: The USAID mission in Senegal implemented five WASH activities in fiscal  years 2012 through 2014, with WASH activity funding totaling  $27,616,000. The mission’s largest WASH activity, called Senegal  Millennium Water and Sanitation Program, included $20,866,000 in  funding for WASH. The activity focused on governance and management,  the creation of local business opportunities, increasing demand for clean  water and sanitation, the construction of infrastructure, and hygiene.
section_title: Tanzania
paragraphs: The USAID mission in Tanzania implemented three WASH activities in  fiscal years 2012 through 2014, with WASH activity funding totaling  $17,753,586. The mission’s primary WASH activity, called Tanzania  Integrated Water, Sanitation and Hygiene Program, included $17,443,586  in funding for WASH. The activity supported community piped and  gravity-fed water schemes, rehabilitated wells with rope pumps, training  for community groups responsible for operations and maintenance of  water schemes, and school latrines. Activity also included a water  resource management component.
Figure 10 shows a demonstration rope pump, which the Tanzania  Integrated Water, Sanitation and Hygiene Program supported as a cost- effective, easy-to-maintain technology, and a piped water point in the  village of Mvumi, Wami-Ruvu River Basin, Tanzania.
section_title: Uganda
paragraphs: The USAID mission in Uganda implemented 15 WASH activities in fiscal  years 2012 through 2014, with WASH activity funding totaling  $15,284,415. The mission’s largest WASH activity, called Northern  Uganda Development of Enhanced Local Governance, Infrastructure, and  Livelihoods, included $6,225,000 in funding to support local government  efforts in northern Uganda to expand basic WASH services. Another  activity, called WASHPlus: Supportive Environments for Healthy  Communities, included $500,000 in funding for WASH and aimed to build  the capacity of district government and USAID implementing partners for  WASH efforts to support community-led total sanitation, promoting  handwashing in villages, and integrating WASH with nutrition and  HIV/AIDS services and programs.
section_title: Zambia
paragraphs: The USAID mission in Zambia implemented three WASH activities in  fiscal years 2012 through 2014, with WASH activity funding totaling  $27,600,000. The mission’s largest WASH activity, called Schools  Promoting Learning Achievement through Sanitation and Hygiene,  included $15,200,000 in funding for WASH and focused on providing  WASH services in schools. Efforts included hygiene education, capacity  building for operations and maintenance, and support to establish private- sector spare-parts supply.
section_title: Appendix III: USAID Evaluations’ Assessment of Monitoring, Outcomes, and Sustainability
paragraphs: Our review of 14 U.S. Agency for International Development (USAID)  evaluations of water, sanitation, and hygiene (WASH) activities in nine  selected countries found that the evaluations assessed monitoring,  outcomes, and sustainability to varying extents.
Monitoring. Evaluations reported on various issues, including monitoring,  WASH indicators, indicators’ limitations, and data related to gender.
All 14 evaluations reported on monitoring of WASH activities.
Ten evaluations reported on one or more of USAID’s indicators  related to access to drinking water, sanitation, and hygiene, including  whether WASH activities were on track to meet their targets.
Five evaluations discussed limitations in the quality of one or more  indicators used to monitor WASH activities. For example, an  evaluation of Senegal’s Yaajeende activity noted limitations of the  indicator for the number of individuals trained on improved hygiene  behaviors. The evaluation stated that, according to USAID’s guidance,  the success of training and other interventions related to human and  organizational capacity building is to be measured by improvement in  organizational output and performance, not simply by the number of  individuals trained.
Five evaluations presented and discussed data related to gender,  thereby filling some gaps that we had identified related to a lack of  disaggregated gender data in monitoring reports. For example, the  evaluations for the primary WASH activities in Ethiopia and Senegal  assessed women’s participation in community water management  committees, which were generally responsible for operations and  maintenance (see fig. 11).
Outcomes. Six of the 14 evaluations that we reviewed provided insights  into WASH activity outcomes. Specifically, these evaluations had findings  related to outcomes such as disease incidence, health expenses, school  attendance, time spent getting water, economic impacts, and beneficiary  satisfaction. For example, the evaluation of Ethiopia’s Water Sanitation  and Hygiene Transformation for Enhanced Resiliency found that the  activity resulted in access to safe water at a much closer distance than  before and also increased access to safe latrines and improved health  practices (such as handwashing). In addition, the evaluation found that  activity results included increased time for beneficiaries to participate in  other productive and income-generating activities, including more time at  school, as well as reduced health expenses.
Sustainability. Twelve of the 14 evaluations that we reviewed addressed  the sustainability of WASH activities. Specifically, these evaluations  broadly discussed WASH sustainability issues, and 7 of the 12 had  findings related to WASH sustainability challenges. These challenges  included limitations related to capacity building, a lack of spare parts, and  a lack of funding for operations and maintenance. For example, the  evaluation of Tanzania’s Integrated Water, Sanitation, and Hygiene  Program reported on factors that improved sustainability, such as the  ease of repairing a rope pump. The Tanzania evaluation also noted  challenges related to capacity building for community water committees.  The evaluation assessed 10 of 26 committees as having “fair” usage and  maintenance, where community fees were generally not collected or  maintenance was spotty, and rated 6 of the 26 as “poor” for  underperformance relative to the rest of the project.
section_title: Appendix IV: Comments from the U.S. Agency for International Development
section_title: Appendix V: GAO Contact and Staff Acknowledgments
subsections: section_title: GAO Contact
section_title: Staff Acknowledgments
paragraphs: In addition to the contact named above, Emil Friberg (Assistant Director),  Mona Sehgal (Assistant Director), Lisa Helmer, Mitchell Delaney, Jesse  Elrod, Reid Lowe, Bethany Patten, Steven Putansu, and Monica Savoy  made significant contributions to this report. Mark Dowling, Jon Melhus,  and Ozzy Trevino provided technical assistance.