Foreign Assistance: USAID Venture Capital Approach Relies on Evidence of Results but Could Strengthen Collaboration among Similar Programs
Published: 2015-12-21T00:00:00

Fast Facts:

Highlights:
section_title: Why GAO Did This Study
paragraphs: USAID established the DIV program in 2010 with a goal of creating a portfolio of innovations that contribute to reducing global poverty. Borrowing from the venture capital model, DIV seeks to identify and test innovative development solutions based on three core principles: rigorous evidence, cost-effectiveness, and potential to scale up. As of 2014, India was the largest recipient of DIV funding, representing approximately one-third of the program's portfolio.
In this report, GAO examines the DIV program's (1) distribution of funding and (2) efforts to measure progress toward achieving its goals, and for DIV's activities in India, GAO examines (3) the extent to which DIV uses evidence to make funding decisions and assess results and (4) DIV's collaboration with similar U.S. development assistance innovation programs. GAO reviewed and analyzed DIV documents and data for fiscal year 2010 to 2015, and interviewed agency officials and grant recipients. GAO selected India as a nongeneralizable case study and conducted fieldwork in that country.
section_title: What GAO Found
paragraphs: From fiscal years 2010 to 2015, the U.S. Agency for International Development's (USAID) Development Innovation Ventures (DIV) program obligated approximately $72.5 million for innovation projects to reduce poverty across a range of sectors, including energy, health, and education. In India, for example, DIV funded intensive learning camps that group children by reading and math abilities rather than by grade level, and a solar micro-grid service providing lighting to off-grid customers for approximately $0.27 per week. While DIV has a global focus and is open to applications regardless of source, approximately 52 percent of its funding is concentrated in two countries, India and Kenya, and 40 percent of its funding is concentrated with four grantees.
DIV is collecting data for several program-level performance measures, which show some positive outcomes, but has not established targets for these measures, making it difficult to assess DIV's progress. GAO's review of DIV's draft framework indicates that it does not include performance targets.
DIV has applied evidence-based requirements for awarding grants and assessing results, emphasizing rigorous evaluations. Specifically, applications that DIV funded in India have generally met the program's evidence requirements, such as including evaluation plans. DIV grantees in India have also provided final reports and evaluations that generally met DIV's requirements. In addition, DIV recently has taken action to ensure that the final reports and evaluations of its projects are publicly disseminated, as generally required by USAID policy.
DIV's limited collaboration with similar U.S.-funded innovation programs in India has contributed to missed opportunities to share information and leverage resources. DIV and several other U.S.-funded programs in India support similar objectives and beneficiaries. For example, the strategy of the USAID mission in India focuses, in part, on innovation and modeled its Millennium Alliance program after DIV. Such programs have funded some similar innovations, such as “clean” cook stoves and low-cost eyewear. Although DIV has begun implementing a plan to improve collaboration, it does not yet reflect a joint approach among similar programs, including those of other agencies. Without such an approach, DIV may not be capitalizing on opportunities to gain efficiencies and maximize the impact of its innovation programs.
section_title: What GAO Recommends
paragraphs: GAO recommends that USAID establish (1) performance targets to assess DIV's progress toward its goal and (2) a joint approach to collaboration for similar programs in India, while considering such an approach in other countries, as appropriate. USAID agreed with these recommendations and noted steps it is taking to implement them.

section_title: Letter
paragraphs: In 2010, the U.S. Agency for International Development (USAID) initiated  the USAID Forward reform agenda to strengthen USAID by pursuing a  more results-oriented approach, developing new partnerships, and  investing in innovative solutions to development challenges. In October  2010, USAID established the Development Innovation Ventures (DIV)  program as a key component of this initiative. DIV’s goal is to create a  portfolio of innovations that contribute to reducing global poverty.  Borrowing from the venture capital model, DIV seeks to test new  approaches to development assistance and identify those that can deliver  more impact per dollar spent. DIV also aims to manage risks by investing  relatively small amounts of funding in new and innovative ideas for  solving development problems, and investing larger amounts in ideas that  have proven to work through rigorous evaluation. In 2012, the Office of  Management and Budget (OMB) cited DIV’s model as an example of  transforming a traditional competitive grant program into an innovative,  evidence-based program. DIV’s planned fiscal year 2015 budget is  approximately $21.8 million, according to a DIV official.
Five years after its initial experiment with this new funding model and  approach, DIV has gained experience implementing projects in real-world  settings and has an opportunity to determine what results have been  achieved and what lessons have been learned thus far to inform future  planning for the program. You asked us to review various aspects of  DIV’s program. As of 2014, when we began reviewing DIV, India was the  largest recipient of DIV funding, representing approximately one-third of  the program’s portfolio. In this report, we examine the DIV program’s (1)  distribution of funding and (2) efforts to measure progress toward  achieving its goals, and for DIV’s activities in India, we examine (3) the  extent to which DIV uses evidence to make funding decisions and assess  results and (4) DIV’s collaboration with other similar U.S. development  assistance innovation programs.
To address these objectives, we reviewed and analyzed USAID program  documents and data, including DIV project data for fiscal years 2010  through 2015. To assess the reliability of these data, where possible, we  cross-checked the data with other sources, evaluated the data for internal  consistency, and interviewed agency officials knowledgeable about the  data sources. We determined the data presented in this report to be  sufficiently reliable for our purposes. To examine the distribution of DIV  funding and DIV’s efforts to measure progress toward achieving its goals,  we also interviewed USAID officials in Washington, D.C. We selected  India as a nongeneralizable case study based on our analysis of USAID  project data. Of the 39 countries DIV had supported through 2014, India  had received the largest amount of funding and had the largest number of  projects. To examine the extent to which DIV uses evidence to make  funding decisions and assess results in India, we reviewed and analyzed  applications, final reports, and final evaluations for DIV projects in India,  for fiscal years 2010 to 2015. To examine DIV’s collaboration with similar  U.S. development assistance innovation programs in India, we  interviewed officials from USAID, the Departments of State (State) and  Energy, and the Department of Agriculture’s Foreign Agricultural Service  (FAS) in Washington, D.C. We also traveled to the USAID mission in New  Delhi, India, to interview officials and obtain data and information, from  USAID, State, FAS, the U.S. Trade and Development Agency (USTDA),  and the Department of Commerce, regarding development assistance  and innovation programs managed by the USAID mission and offices  within the U.S. embassy. In addition, we interviewed DIV grant recipients  that implemented projects across several sectors.
We conducted this performance audit from July 2014 to December 2015  in accordance with generally accepted government auditing standards.  Those standards require that we plan and perform the audit to obtain  sufficient, appropriate evidence to provide a reasonable basis for our  findings and conclusions based on our audit objectives. We believe that  the evidence obtained provides a reasonable basis for our findings and  conclusions based on our audit objectives.
subsections: section_title: Background
paragraphs: DIV’s goal is to support USAID’s mission and improve the lives of  millions of people around the world within 10 years, especially those living  in poverty or extreme poverty. To achieve this global development goal,  DIV awards grants and cooperative agreements (in this report, collectively  referred to as grants) to academic institutions, nongovernmental  organizations, and businesses, among other types of organizations, to  advance development-related innovations. DIV defines such innovations  as novel business or organizational models, operational or production  processes, or products or services that could lead to substantial  improvements in addressing development challenges.
According to DIV officials, the program does not outline specific problems  to be solved or propose specific solutions but is intentionally open-ended,  funding grants on the basis of three core principles:
Evidence: rigorous evaluation of what works and what does not,  scaling up only those solutions proven to produce demonstrable  impact
Cost-effectiveness: potential to deliver greater development impacts  per dollar than traditional development assistance
Potential to scale up: a plan to deliver and maintain widespread  impact by increasing the geographic scope of operations and reaching  financial sustainability beyond DIV’s support through private or public  funding  Managed at USAID headquarters, DIV takes a venture capital approach  to investing in innovations, by awarding grants through a three-stage  funding model. The model is intended to identify, evaluate, and scale up  development innovations that demonstrate widespread impact and cost- effectiveness. According to DIV officials, the program developed this  three-stage model as a risk mitigation approach.
Stage 1—Proof of Concept. In stage 1, DIV provides small grants for  testing the viability of an innovation in a real-world setting. Grantees  must assess whether the innovation will yield results through  evaluation or performance monitoring. Innovations that have  demonstrated results and satisfied stage 1 criteria are eligible for  stage 2 funding to support evaluations that will test for impact.
Stage 2—Testing and Positioning for Scale. In stage 2, grantees  determine, through rigorous assessments including impact  evaluations, whether the solution can achieve larger-scale impact and  can also be implemented successfully at a larger scale. Innovations  that have credible evidence of development impact at stage 2  standards are eligible for stage 3 funding.
Stage 3—Transitioning Proven Solutions to Scale. In stage 3, DIV  funding supports innovations that seek to transition a solution from  large-scale implementation to widespread adoption in one country or  to replication in an additional country.
DIV has established maximum funding amounts and project durations for  each funding stage of this model, as shown in table 1.
According to DIV officials, the staged funding model was not envisioned  as an inevitable progression of projects from stage 1 to stage 3. For  example, only 7 of 56 projects received stage 2 funding after having  previously received a stage 1 grant. Grantees can receive funding at any  stage without necessarily implementing a DIV project at an earlier stage,  provided they meet the established requirements for that stage. For  example, in many cases, stage 2 grantees have demonstrated a proof of  concept prior to receiving a DIV grant.
In addition, DIV places a particular emphasis on using rigorous evaluation  methods, such as randomized controlled trials (RCTs), to determine  impact. DIV officials noted that because DIV has set a maximum funding  level of $150,000 for stage 1 grants, grantees conducting evaluation  studies often require additional funding from a source other than USAID.  DIV officials also noted that other funding sources may be required  because the time frame for completing a study may extend beyond the  duration of the DIV grant. In the case of projects at stage 2 or stage 3, the  grantee has already tested its idea and requires funding to bring the  project to scale.
section_title: DIV Has Funded Innovation Projects across Nine Sectors and Has Concentrated Funding Largely in Two Countries and among Four Grantees
paragraphs: In fiscal years 2010 through 2015, DIV obligated approximately $72.5  million for innovation projects across nine sectors, including energy,  economic growth, health, and education. DIV directed approximately 52  percent of project funding to two countries where it funded projects and  40 percent of funding to four grantees, which supported projects in India  and Kenya, as well as other countries.
subsections: section_title: DIV Has Funded $72.5 Million for Innovation Projects across Nine Sectors
paragraphs: In fiscal years 2010 through 2015, USAID obligated approximately $72.5  million for DIV grants, with annual funding rising from nearly $1 million in  fiscal year 2010 to approximately $19 million in fiscal year 2015 (see fig.  1). The number of DIV grants increased from 8 in fiscal year 2010 to 41 in  fiscal year 2013 and declined to 33 in fiscal year 2015.
Overall, DIV has awarded 142 grants from over 7,500 applications  submitted since July 2010. These awards consisted of 83 grants for stage  1 (Proof of Concept), 56 grants for stage 2 (Testing and Positioning for  Scale), and three grants for stage 3 (Transitioning Proven Solutions to  Scale). The three stage 3 grants represented 21 percent of overall DIV  funding, as shown in figure 2.
Overall, DIV has funded a range of projects across nine sectors, with the  largest share of funding supporting projects in the energy sector ($16.6  million) and the economic growth sector ($15.8 million), as shown in  figure 3.
Two DIV projects in India, in the education and training sector and the  energy sector, respectively, provide illustrative examples of the types of  grantee organizations and innovations that DIV has funded.
Education and training sector. In 2013, DIV awarded a $927,000  stage 2 grant to the Pratham Education Foundation to expand its  evaluation of intensive learning camps in selected villages using a  randomized evaluation, and determine if this model is a good  candidate for implementation at scale (see fig. 4). The intensive  learning camps were intended to improve learning outcomes for  children in grades 3 through 5. The project organized and grouped  students by ability, rather than grade level, to provide more focused  instruction tailored to students’ learning needs. For example, for  reading instruction, Pratham grouped students by their ability to  recognize Hindi characters, words, and sentences, while for math  instruction, it grouped students by their ability to recognize numbers.  The project’s evaluation showed that the reading and math scores of  students who participated in the learning camps increased by as  much as 22 percent over the scores of students who did not  participate in the learning camps.
Energy sector. In 2011, DIV provided a $300,000 stage 2 grant to  Mera Gao Power (MGP) to test whether its solar micro-grid system  providing low-cost electricity to off-grid villages in India was  commercially viable. After micro grids were installed, MGP’s  customers paid a weekly subscription fee of approximately $0.27 for  the use of two LED lights and one phone charger. An MGP staff  member came to subscribing villages each week at prearranged times  to collect customer payments in cash (see fig. 5). By the conclusion of  the grant in March 2013, MGP had installed the service in  approximately 180 villages, reaching 4,480 households—exceeding  its targets of 40 villages and 4,000 customers. MGP officials told us in  March 2015 that they had further expanded the service to  approximately 17,000 customers and had secured external financing  from an impact investment firm.
section_title: DIV Funding Is Concentrated in Two Countries and with Four Grantees
paragraphs: While DIV has funded projects in 43 countries since 2010, over half of its  project funding—roughly $37.8 million—is concentrated in India and  Kenya where it has awarded 64 of its 142 grants (45 percent). Figure 6  provides information on the number of DIV projects by country, including  those that were part of multicountry projects. In India, DIV funded 18  stage 1 projects and 21 stage 2 projects. In Kenya, DIV funded 14 stage  1 projects, 9 stage 2 projects, and 2 stage 3 projects.
According to DIV officials, to support its global development goal, DIV  awards grants through an open-ended process to applicants that best  meet its criteria, regardless of geographic location. DIV officials stated  that the program did not target or prioritize India and Kenya for the  implementation of its projects, and that the concentration of funding in  these two countries reflects the relative strength of their applications.  These officials indicated that in recent years DIV had expanded its  geographic distribution of projects and made efforts to work with USAID  missions to promote the program in other countries. In 2015, for example,  DIV funded 33 projects in 22 countries. However, our analysis of DIV data  showed that the distribution of DIV projects remained concentrated—at  over 40 percent of all DIV projects—in India and Kenya in fiscal years  2013 through 2015.
DIV funding is also concentrated among its grantees, awarding roughly  $29 million (40 percent of program funding) to four grantees—the Abdul  Latif Jameel Poverty Action Lab at the Institute for Financial Management  and Research (J-PAL/IFMR), Innovations for Poverty Action (IPA), Off  Grid Electric Limited, and Georgetown University (see fig. 7). J- PAL/IFMR and IPA—research organizations that focus on evaluating  development interventions using RCTs—have received a total of 31 (22  percent) of DIV’s 142 grants. J-PAL/IFMR received 13 grants, totaling  $4.13 million, and IPA received 18 grants, totaling $14.1 million—the  largest total amount awarded to a DIV grantee. In 2015, Off Grid Electric  received a stage 3 grant totaling $5 million and is the only grantee to have  received a grant for each of the three stages. These three grants  supported the testing and expansion of an innovation to provide electricity  to households in Tanzania with limited access to the electric grid.  Georgetown University has received three DIV grants totaling $4.28  million, the third largest total dollar amount awarded to a DIV grantee.  One of the grants to Georgetown University, a $3 million stage 3 grant,  supports the expansion of an innovation to reduce traffic accidents in  Kenya and other countries in East Africa. In a previous evaluation,  researchers tested the effects of placing behavior change messages on  stickers in buses that urged passengers to speak up against dangerous  driving and encourage their bus drivers to slow down. They found that  accident insurance claims for buses with stickers fell by half compared  with claims for buses without stickers.
As with the concentration by country, DIV officials stated that the  concentration of funding among four grantees reflects the relative  strength of their applications and that DIV did not target these  organizations for awards.
section_title: Lack of DIV Performance Targets Makes It Difficult to Determine Progress
paragraphs: DIV has developed and is collecting data for several program-level  performance measures, which show some positive outcomes, but has not  established specific targets for these measures, making it difficult to  assess DIV’s progress. DIV is in the process of developing a new results  framework; however, our review of a draft version of the framework  shows that it does not include performance targets.
subsections: section_title: DIV Collects Data on Performance Measures but Has Not Established Corresponding Targets
paragraphs: DIV has used various program-level measures to track performance since  the beginning of the program, and DIV officials provided data that they  have collected for these performance measures through October 2015  (see table 2). These range from process-oriented measures related to  overall DIV program management to measures that capture results from  specific grantees. Examples of process-oriented measures include  tracking the length of time between receiving an application and the final  decision on whether or not to make an award, and the length of time  between approving an application and issuing the award.
Examples of outcome-oriented performance measures include the  percentage of projects conducting RCTs and the number of grantees that  connect to outside sources of funding after the award of the DIV grant.  DIV’s outcome-oriented measures, including those focused on bringing  awards to scale, are similar to measures used by some venture capital  firms. For example, DIV reported that as of October 2015, 11 of its  grantees had obtained outside sources of funding after the award of the  DIV grant, and 5 of its awards had scaled up through the public sector.
Although DIV established performance measures and is collecting data  that correspond to these measures, DIV officials stated in November  2015 that they have not established targets for these measures as a  means for assessing DIV’s performance. Therefore, it is difficult to  determine the level of performance that DIV is intending to achieve and to  determine DIV’s actual progress against targets. Our past work has  shown that, although agencies collect a significant amount of  performance information, they have not consistently used that information  to improve management and results. The GPRA Modernization Act of  2010 (GPRAMA) requires agencies to establish performance measures  to assess progress toward goals. Moreover, we have previously identified  practices for enhancing agency use of performance information, including  communicating performance against targets. Without related targets,  DIV may be unable to demonstrate to key stakeholders, including  Congress and the public, that it is making progress in achieving agency  goals.
During the course of our review, DIV officials were in the process of  developing a results framework for the program and provided us with a  draft version of the framework. Although the draft framework provided an  expanded list of indicators for measuring results, it did not include targets  for these indicators. DIV officials did not provide a specific time frame for  completing this results framework but stated that they expect to finalize  the results framework in 2015. These officials stated that they are  developing the results framework as part of an effort to better articulate  and measure the program’s goals and to link to other related USAID  programs. DIV officials also stated that, as part of this effort, they are  outlining a monitoring and evaluation plan to inform learning objectives for  the program.
section_title: DIV Has Established Evidence-Based Requirements for Funding and Evaluating Grants and Has Applied These Requirements to Grants in India
paragraphs: DIV has established and applied evidence-based requirements when  awarding grants and assessing results, emphasizing the use of rigorous  evaluation. For example, applications that DIV funded in India generally  met the program’s evidence requirements, such as including evaluation  plans. In addition, most completed DIV projects in India provided final  reports and evaluations that corresponded with DIV’s evidence  requirements. DIV also has recently taken action to ensure that the final  reports and evaluations of its projects are publicly disseminated, as  generally required by USAID policy.
subsections: section_title: DIV Has Established Specific Evidence Requirements and Emphasizes Rigorous Evaluations
paragraphs: DIV emphasizes the testing of potential development solutions and  rigorously evaluating impact, to scale up only those solutions with proven  results. DIV outlines specific evidence requirements for each funding  stage, with greater evidence of impact required, the higher the stage  applied for. For example, while applicants for stage 1 grants are not  required to provide evidence of prior testing of the proposed solution, they  are required to present a plan for assessing results or impact, including  specific metrics for success. Applicants for higher-stage grants are  required to discuss prior experiences implementing or testing their  solutions and evidence of successful development impact. Additionally,  stage 3 applicants are required to discuss specific evaluation  methodologies and findings.
DIV encourages grantees to utilize rigorous evaluation methods, including  RCTs, in their projects, while recognizing that RCTs are not appropriate in  all cases. Our review of DIV project data showed that approximately 43  percent of its global portfolio, and 54 percent of projects in India, included  an RCT to assess development impact (see table 3). Of the 103 DIV  projects in countries other than India, 39 percent of these projects  conducted RCTs. We found that DIV projects across all three stages  conducted RCTs. DIV officials also stated that, although many awards  use an RCT, this type of evaluation design is not applicable for every  question being examined. However, these officials added that each grant  does include a test of some sort and analysis of the data.
Our review of DIV documents and meetings with India-based grantees  found a number of examples of DIV projects in India that are employing  RCTs as part of the DIV award. For example, two DIV grantees that we  met with were evaluating the use of biometric fingerprinting technologies  as a tool for making improvements in different aspects of India’s health  care system.
J-PAL/IFMR stage 2 DIV grant to evaluate the problem of  absenteeism among medical staff in India through an RCT.  Awarded in 2010, this study evaluated the impact of an intervention  using a digital attendance and medical information system to monitor  attendance of medical staff in government health centers in the state  of Karnataka. J-PAL/IFMR randomly assigned primary health centers  to treatment and control groups, and the treatment health centers  were equipped with fingerprint reader devices and a mobile device for  uploading attendance and patients’ data. The preliminary results of  the program showed a modest effect on the attendance of nurses,  pharmacists, and lab technicians, but no effect on the attendance of  medical officers.
Operation ASHA stage 2 DIV grant to evaluate the effectiveness  of a fingerprint identification system in preventing the  occurrence and lapses in the treatment of Multidrug-resistant  tuberculosis (MDR-TB) through an RCT. Awarded in 2012, this  project evaluated the effectiveness of fingerprint reader devices in  preventing lapses in treatment of MDR-TB patients. These devices  register the presence of patients and staff at treatment centers in  receiving MDR-TB treatments, and Operation ASHA’s system informs  TB counselors when a patient misses a treatment. Operation ASHA is  carrying out the RCT involving about 12,000 patients across  approximately 200 health centers to establish the effectiveness of the  fingerprint identification system as a tool for ensuring patients’  compliance with their MDR-TB treatments.
section_title: Applications DIV Funded in India Generally Met Evidence Requirements
paragraphs: Our review of the applications for grants DIV awarded in India found that  the applications generally met evidence and other requirements. We  examined 33 of the applications to determine the nature and types of  evidence and other information that they contained, as well as the extent  to which they met DIV’s evidence requirements (see table 4). For  example, we found that 31 of the 33 applications fully or partially provided  information on how the innovation would be evaluated for impact. In  addition, we determined that 16 of 19 applications that were awarded  stage 2 grants provided evidence that the innovation had previously been  tested for impact. We also found that 29 of the 33 applicants included  an analysis or information on the innovation’s cost-effectiveness,  although the level of information on cost-effectiveness varied among the  applications. In some applications, for example, we found that the  applicants discussed cost-effectiveness, but did not provide supporting  data or analysis to compare the costs of their solution to competing  solutions or traditional methods of delivering development assistance.
section_title: DIV Grantees in India Provided Final Reports and Evaluations That Generally Met DIV’s Requirements
paragraphs: Based on our review of DIV documents, we found that most DIV grantees  in India provided final reports and evaluations that met DIV’s  requirements. We reviewed grant agreements for 33 of the DIV projects in  India, and these agreements contained specific requirements linked to  estimated completion dates and funding amounts that would be disbursed  upon completion of each milestone. The DIV grants we reviewed  contained requirements for delivering final reports and evaluations,  although these requirements varied across projects. For example, some  grant agreements required specific elements to be included in the final  report, such as an analysis of the cost-effectiveness of the project, while  other grants generally required that a final report be delivered without  requiring that the report include specified elements.
We analyzed 18 DIV final reports and found that almost all of the reports  provided information that met DIV’s milestone requirements for a final  report or final evaluation (see table 5). Specifically, we found that 16 of  the 18 projects met DIV’s requirements for completing a final report to  close the grant agreement. In addition, we found that 10 of 13 projects  completed the evaluation requirement.
We also found that 13 of the 18 final reports we reviewed provided data  on results and outcomes consistently with the methodologies discussed in  the respective applications. In some cases, we found inconsistencies  between the evaluation methodology discussed in the application and the  corresponding information in the final report. When asked for clarification,  DIV officials explained that, in some cases, during the award process they  negotiated changes to grantees’ methodologies or implementation plans  from what was initially discussed in the application, based on updates or  contextual changes.
In addition, we found that the 18 final reports for India generally  addressed DIV’s three core principles: (1) evidence, (2) cost- effectiveness, and (3) potential to scale up (see table 6). For example, 17  of 18 final reports fully or partially provided evidence of the project’s  development outcomes. We also found that 13 of the 18 final reports fully  or partially provided information on DIV’s core principle of cost- effectiveness, although they varied in the levels and types of information  provided. For example, the final report for a project testing a metered  pricing system for off-grid power provided a detailed cost breakdown of  the solar micro-grid system in comparison to the costs of competing  energy sources, such as diesel generators and household solar panels. In  another example, a grantee reported that undertaking a cost-benefit  analysis was difficult because of the challenge of quantifying the  intervention’s economic benefits, but it did report the cost of conducting  the intervention. Finally, in five other cases, final reports did not include  any discussion of cost-effectiveness.
section_title: DIV Has Begun to Publicly Disseminate Final Reports and Evaluations
paragraphs: DIV has recently taken action to ensure that its final reports and  evaluations are publicly disseminated, as is generally required by USAID  policy. USAID’s policy states that evaluation findings should be shared as  widely as possible with a commitment to full and active disclosure. A  standard requirement for USAID grants, including DIV’s grants, is that  grantees post final reports and evaluations from completed projects on  USAID’s Development Experience Clearinghouse (DEC), the agency’s  online repository of research information. This requirement was specified  in DIV grant agreements we reviewed that were awarded after the  requirement came into effect. However, when we initially reviewed the  DEC in July 2015, it did not contain any DIV final reports or evaluations.  After raising the issue with DIV officials, we searched the DEC again in  November 2015 and found that it contained 87 total documents from DIV  projects, including 46 final reports. These included 22 total documents  and 13 final reports from projects based in India, and 64 total documents  and 33 final reports from DIV projects in other countries. Appendix II  provides a list of published studies by DIV grantees in India.
section_title: DIV’s Limited Collaboration with Similar Innovation Programs in India Has Contributed to Missed Opportunities to Share Information and Leverage Resources
paragraphs: DIV and other U.S.-funded innovation programs in India support similar  objectives and beneficiaries, and in several cases these programs have  funded the same types of innovations. Collaboration among these  overlapping programs in India has been limited, which has contributed to  missed opportunities to share information and leverage resources. During  the course of our review, DIV began implementing an action plan  intended to improve its collaboration with the USAID mission in India  (“USAID India”) and other missions. However, the plan does not establish  a joint approach to development among these programs. Without such an  approach, USAID may not be capitalizing on opportunities to gain  efficiencies and maximize the impact of its innovation programs.
subsections: section_title: DIV and Other U.S Innovation Programs in India Support Similar Objectives and Beneficiaries
paragraphs: We identified several U.S.-funded innovation programs in India with  similar objectives and beneficiaries, as shown in table 7. These  overlapping programs award grants to promote proven innovations that  will benefit poor, underserved populations in India. USAID India’s 2012- 2016 country development strategy has two objectives focused on  innovation, including supporting innovations that impact those living in  extreme poverty, and supporting innovations proven in India and  disseminating them to other countries. To support these objectives,  USAID India created several innovation programs similar to DIV. For  example, in 2012 USAID India established the Millennium Alliance, an  innovation grant program modeled on DIV. This program provides funding  to Indian grantees that demonstrate cost-effective solutions that address  the needs of the extreme poor in India. Like DIV, the program uses a  staged funding model to make relatively small initial investments, test  more developed solutions, and scale up those that have proven  development impact through rigorous evaluations. USAID India also  created the U.S.-India Partnerships program to overcome critical  development challenges through new technologies and other innovations  that can be rigorously tested, shared, and scaled up in India and abroad.  State also funds innovations to support economic growth and clean  energy for underserved populations in India through the U.S.-India  Science and Technology Endowment Fund and the PACESetter fund,  respectively.
While these programs have similar objectives and beneficiaries, they also  have some differences in the way they are implemented. For example,  the Millennium Alliance and the U.S.-India Science and Technology  Endowment Fund are managed in cooperation with the Indian  government, while DIV is funded and managed solely by USAID. In  addition, the USAID India Partnerships Program requires partners to  share the costs and contribute at least $500,000 in cash and in-kind  resources for a one-to-one match totaling at least $1 million, while DIV  has no requirements for cost-sharing.
section_title: DIV and Other USAID and State Innovation Programs Fund Some Similar Projects in India
paragraphs: Our review of project data for innovation programs in India identified  examples in which these programs have funded innovations similar to  those funded by DIV, as shown in figure 8. For example, both DIV and  another program supported projects to test the viability of “clean” cook  stoves in rural markets—that is, stoves designed to reduce air pollution  and firewood consumption compared with traditional cook stoves. DIV  and two other programs supported projects to provide inexpensive eye  care and eyewear for poor and underserved populations. DIV and two  other programs funded projects to support the development, testing, or  implementation of micro grids for people living in rural areas who are  unconnected to the power grid.
USAID and State officials we interviewed in India stated that they  generally supported the implementation of similar innovation programs in  India by different organizations if the programs resulted in additional  resources being made available to poor and underserved populations.  According to these officials, there is a vast need for innovations, such as  clean energy and off-grid electricity, which improve the lives of the poor in  that country. In addition, State officials commented that in a country of  1.25 billion people, with significant diversity in cultural, linguistic, and  religious norms, as well as considerable geographic diversity, different  solutions to the same problem may produce varying levels of success in  different contexts. These officials said that, as a result, it may be in some  cases necessary to fund several similar or competing solutions in an  effort to identify the few that demonstrate widespread impact and cost- effectiveness. USAID India officials added that because of the variety of  difficult and intractable problems in India, the mission does not see it as  problematic that there would be more than one activity aimed at  addressing the same problem.
section_title: Collaboration among U.S.- Funded Innovation Programs in India Has Been Limited and Opportunities Have Been Missed to Share Information and Leverage Resources
paragraphs: We have previously found that several key practices that enhance  collaboration, including articulating a joint strategy and common  outcomes, agreeing on roles and responsibilities, and identifying and  addressing needs by leveraging resources, can help manage programs  with similar objectives and beneficiaries. Collaboration among U.S.- funded innovation programs in India has not routinely or systematically  included these practices and, with some exceptions, has been limited to  USAID India’s providing initial input to DIV regarding grant award  decisions.
During USAID India’s technical reviews of DIV applications, DIV routinely  communicated with USAID India officials, requesting that the mission  review applications for projects that DIV subsequently funded in India.  USAID India officials rated the proposals, provided narrative information  on the strengths and weaknesses of the projects, and in some cases  raised concerns.
However, according to DIV officials and a wide range of USAID India and  State officials we spoke with in India, collaboration among programs  beyond these examples has been limited and has not routinely included  the key practices we identified. For example, these officials indicated that  after award decisions were made, DIV and the other programs did not  systematically share information about project results, or reach  agreements on their respective roles and responsibilities, such as roles in  coordinating planning for the use of grant funds or in monitoring the  implementation of grants. State officials we interviewed in India who  manage other innovation programs—such as the U.S.-India Science and  Technology Endowment Fund and the PACESetter fund—stated that,  while they were aware of some of the activities that DIV has supported in  India, DIV had not communicated with these other agency programs to  collaborate on ongoing or upcoming efforts.
USAID India and State officials we interviewed in India who manage  innovation programs under the India Partnerships program and the  PACESetter Fund, among others, told us that limited collaboration among  these programs and DIV had resulted in missed opportunities to share  information and leverage USAID India resources by providing outreach  and monitoring of project implementation and marketing DIV innovations.  We have previously found that, without engaging in collaboration  practices such as agreeing on roles and responsibilities or identifying  common outcomes, overlap can have a negative effect in that limited  resources may not be used in the most efficient and effective manner and  opportunities may be missed to leverage resources. The following  provide some examples of missed opportunities to share information and  leverage the mission’s resources.
Missed opportunities to provide outreach to DIV grantees and  monitor project implementation. USAID India officials stated that in  many cases, they did not know DIV grants had been awarded and  thus missed opportunities to provide outreach and establish  productive working relationships with DIV grantees in India because  roles and responsibilities were not clarified. For example, USAID India  officials from the India Partnerships program, and the health sector,  cited examples in which DIV grantees had contacted USAID India  seeking assistance but found that the mission was unaware that DIV  had awarded grants to these organizations, limiting the effectiveness  of USAID India’s outreach to them. In addition, USAID India officials  stated that the mission’s and DIV’s respective roles and  responsibilities were not always clear to USAID India officials or to  DIV grantees, which negatively affected some grantees’ perceptions  of USAID. USAID India officials also discussed missed opportunities  to conduct project monitoring on DIV’s behalf, and DIV officials stated  that they had not sought assistance from USAID India in monitoring  DIV projects.
Missed opportunities to market DIV innovations. USAID India and  State officials responsible for promoting U.S.-funded clean energy  projects in India, including those of the PACEsetter Fund, indicated  that they had missed opportunities to share those results with the  government of India or other stakeholders with the means to scale  them up, if appropriate. For example, during a recent high-level U.S.  government delegation’s visit to India, USAID India officials stated that  they had missed an opportunity to highlight promising DIV projects  focused on clean energy activities, because they were unaware of the  projects’ results.
Although collaboration among innovation programs in India has been  limited, DIV officials provided some additional examples of  communication and consultation with USAID India beyond the initial  consultation on DIV applications. For example, USAID India and DIV  jointly funded a project to rigorously test an innovation to increase full  immunization rates in rural areas. This project is testing the viability of  implementing the innovation in cooperation with the government system  in the Indian state of Haryana. USAID India and DIV officials also  discussed DIV’s consultations with the USAID India energy team and  corresponding interactions between DIV grantees and the USAID India  energy program, which was in contrast to more limited consultations  between DIV and USAID India officials from other programs.
section_title: DIV Has Begun Implementing an Action Plan to Improve Collaboration but Has Not Agreed on a Joint Approach with Relevant Programs in India
paragraphs: During the course of our review, DIV officials, acknowledging that  collaboration could be improved, began implementing an action plan to  improve collaboration with missions and bureaus within USAID. The plan  outlines steps to share information on DIV’s activities across the agency,  including establishing DIV points of contact for outreach with missions  and bureaus and developing tools for providing more frequent updates on  DIV projects with the missions. The plan also discusses time frames for  collaboration activities, including identifying opportunities for joint  management and co-investment with missions on DIV projects. During  our visit to India, a DIV official briefed staff at the mission about the DIV  program and about its broader plans to increase outreach to USAID  missions. DIV officials also provided us with additional examples of  briefings and outreach that it conducted with other bureaus and missions.  In addition, in 2015, DIV collaborated with the USAID mission in Jordan to  hold a regional competition for innovators to pitch their ideas for  development solutions to be considered for grants in the Middle East and  North Africa region, where DIV has made the fewest awards thus far.
We previously found that developing a joint approach among related  programs, including reaching agreement on a joint strategy, common  outcomes, roles and responsibilities, and leveraging resources, can  improve collaboration and generate greater results than the programs  could achieve independently. However, DIV’s action plan, while a  promising step toward improving collaboration, does not yet represent a  joint approach among the overlapping innovation programs that we  identified in India. Thus, DIV cannot ensure that the benefits of its initial  outreach efforts will be realized. DIV has not extended its action plan to  include the similar innovation programs we identified outside USAID  India, for example. Also, DIV has not harmonized its award selection  processes with those of the other innovation programs to help ensure that  funding similar projects is appropriate and not unnecessarily duplicative.  Furthermore, DIV has not reached agreement with the other U.S.-funded  innovation programs on a common approach to monitoring and evaluation  of the projects they select to fund to help ensure that results from similar  projects are being assessed consistently. Consequently, it may be difficult  to determine which version of a technology or innovation has the greatest  potential to scale up and where further U.S. support would have the most  impact.
section_title: Conclusions
paragraphs: USAID created DIV to demonstrate a new model of U.S. development  assistance, and DIV’s approach has shown promise, especially through  the rigorous evaluation of an innovation’s results and outcomes before  determining whether to increase the agency’s investment to bring these  innovations to scale. After 5 years of experience in implementing projects  and testing its model in real-world situations, DIV has an opportunity to  assess what results have been achieved and what lessons have been  learned from these initial experiences. However, because DIV lacks  clearly identified performance targets for its program, and its draft results  framework does not contain targets, it is difficult to assess DIV’s overall  progress toward achieving its goal of promoting global development for  the poor through its portfolio of innovations.
Since DIV was established in 2010, several other U.S. grant programs  have emerged that overlap with DIV and in some cases have funded the  same types of innovations. Although USAID’s mission in India prioritized  innovation and modeled one of its programs after DIV, DIV and the  mission collaborated to a limited extent during the implementation of  DIV’s projects. As a result, DIV and the USAID mission in India have  missed opportunities to share information and leverage resources. While  DIV has begun implementing an action plan to improve collaboration with  other USAID missions and bureaus, the action plan does not establish a  joint approach, including reaching agreement on a joint strategy, common  outcomes, or roles and responsibilities among all relevant programs and  agencies. Thus, USAID may not be capitalizing on potential synergies  among these innovation programs or maximizing their efficiency and  impact.
section_title: Recommendations for Executive Action
paragraphs: To help ensure that DIV is making progress toward achieving its global  development goal, we recommend that the Administrator of USAID take  the following two actions:  1.  Establish performance targets that will allow periodic assessment of  DIV’s progress toward achieving its goal.  2.  Establish a joint approach to collaboration reflecting agreement with  the USAID mission in India and with other related U.S. agency  programs in India, and consider where such a joint approach would be  beneficial in other countries.
section_title: Agency Comments and Our Evaluation
paragraphs: We provided a draft of this report to USAID and State for comment.  USAID and State provided technical comments, which we incorporated  into the report as appropriate. USAID also provided written comments,  which are reprinted in appendix III.
USAID agreed with our recommendations, stating that our review had  helped identify areas for improvement. USAID also discussed steps it is  taking to respond to the recommendations. With regard to the  recommendation to establish performance targets that will allow periodic  assessment of DIV’s progress toward its goal, USAID noted that it had  recently established a results framework that includes targets and  performance milestones to be assessed on a semiannual basis. With  regard to the recommendation to establish a joint approach to  collaboration with the USAID mission in India and with other agency  programs in India, USAID discussed collaboration and coordination  between DIV and the USAID mission in India that had occurred during the  course of our review. USAID also stated that it would build on these  collaboration efforts, discuss where improvements could be made, and  take action to formalize them.
We are sending copies of this report to the appropriate congressional  committees, the Administrator of USAID, the Secretary of State, and other  interested parties. In addition, the report is available at no charge on the  GAO website at http://www.gao.gov.
If you or your staff have any questions about this report, please contact  me at (202) 512-3149 or gootnickd@gao.gov. Contact points for our  Offices of Congressional Relations and Public Affairs may be found on  the last page of this report. GAO staff who made key contributions to this  report are listed in appendix IV.
section_title: Appendix I: Objectives, Scope, and Methodology
paragraphs: We were asked to review the U.S. Agency for International  Development’s (USAID) Development Innovation Ventures (DIV)  program. In this report, we examine the DIV program’s (1) distribution of  funding and (2) efforts to measure progress toward achieving its goals,  and for DIV’s activities in India, we examine (3) the extent to which DIV  uses evidence to make funding decisions and assess results and (4)  DIV’s collaboration with other similar U.S. development assistance  innovation programs.
To examine the distribution of DIV funding, we obtained and analyzed  funding data for projects from fiscal years 2010 to 2015. These data  included information on the stage of the project, the fiscal year, and the  award start and end dates of the projects, among other things. Projects  implemented in more than one country were counted as one project per  each country. For example, by 2015 DIV had funded 33 individual  projects in India and 6 projects that were implemented in multiple  countries—of which India was one—for a total of 39 projects for India. To  assess the reliability of these data, where possible, we cross-checked the  data with other sources, evaluated the data for internal consistency, and  interviewed agency officials knowledgeable about the data sources. We  determined the data presented in this report to be sufficiently reliable for  our purposes.
To examine DIV’s efforts to measure progress toward achieving the  program’s goals, we reviewed and analyzed DIV’s performance data and  interviewed USAID officials in Washington, D.C. Specifically, we analyzed  performance data publicly available on the USAID website, reviewed data  and documentation on the DIV program provided by USAID, and  interviewed officials in the DIV program office at USAID. To assess the  reliability of these data, where possible, we cross-checked the data with  other sources, evaluated the data for internal consistency, and  interviewed agency officials knowledgeable about the data sources. We  determined the data presented in this report to be sufficiently reliable for  our purposes.
Based on our analysis of USAID project data, we selected India as a  nongeneralizable case study for two objectives. Of the 33 countries DIV  had supported through 2014, India had received the largest amount of  funding and had the largest number of projects. To examine the extent to  which DIV uses evidence to make funding decisions and assess the  results of projects it funded in India, we reviewed and analyzed DIV  documents, including applications, final reports, and final evaluations. We  focused our analysis on projects implemented in India, including  multicountry projects where India was one of the countries where the  project was being implemented, for a total of 33 projects. For the  applications analysis, we reviewed USAID applications for funding  through the DIV program, for multiple rounds, and developed a data  collection instrument based on criteria in the application. We  independently reviewed the applications and rated the extent to which the  applicant responded to the question, supplying data and citations when  necessary. We then reconciled any instances where the ratings of the  initial and secondary reviewer did not concur. We aggregated and  reported the outcomes of several questions we developed in the data  collection instrument. For the final reports and evaluations, we reviewed  the award letters for all 18 projects in India that had completed the grant  agreement and submitted a final report or final evaluation. The award  letters contain the terms and conditions of the grant agreement, including  required tasks to be completed with the supporting documentation, to be  submitted periodically to DIV by the grant recipient. We developed a data  collection instrument based on DIV’s principles of testing and scaling  innovations that demonstrate widespread impact and cost-effectiveness.  In addition, we analyzed the award letters, the applications, and the final  reports and evaluations submitted to DIV to determine the extent to which  the methodologies in final reports and evaluations corresponded to the  proposed methodologies in the application.
Finally, we examined DIV’s collaboration with other similar U.S.  development assistance innovation programs, using India as a  nongeneralizable case study. We interviewed officials at USAID, the  Department of State (State), the Department of Agriculture’s Foreign  Agricultural Service (FAS), and the Department of Energy, in Washington,  D.C. We traveled to the USAID mission in New Delhi, India, to interview  officials from USAID, State, FAS, the U.S. Trade and Development  Agency (USTDA), and the Department of Commerce, and to obtain  information on any development and assistance innovation programs  managed by the USAID India mission or the U.S. embassy. During our  fieldwork, we observed DIV projects and interviewed grant recipients. We  also identified programs at the USAID mission and embassy in New Delhi  that had innovation and development components. We interviewed  program officials from these agencies, including the Chief of Mission and  the Deputy Chief of Mission, to obtain information on the programs and  their experiences in collaborating with the DIV office in Washington, D.C.  In addition, we analyzed program data that we obtained from DIV, and  from USAID India and other agency officials in New Delhi, to determine  the extent to which the DIV program overlaps with programs from USAID  and other U.S. agencies in India, and has funded projects that could  overlap or duplicate projects funded by U.S. agencies in India.
section_title: Appendix II: Published DIV Studies for Projects in India
paragraphs: DIV officials provided us with the following list of published studies  relating to DIV projects implemented in India.
Ashraf, Nava, Oriana Bandiera, and Scott S. Lee. “Do-gooders and  Go-getters: Career Incentives, Selection, and Performance in Public  Service Delivery.” Harvard Business School Working Paper, March  2015.
Banerjee, Abhijit, Donald Green, Jennifer Green, and Rohini Pande.  “Can Voters be Primed to Choose Better Legislators? Experimental  Evidence from Rural India.” Working Paper, 2010.
Borkum, Evan, Anitha Sivasankaran, Swetha Sridharan, Dana Rotz,  Sukhmani Sethi, Mercy Manoranjini, Lakshmi Ramakrishnan, and Anu  Rangarajan. “Evaluation of the Information and Communication  Technology (ICT) Continuum of Care Services (CCS) Intervention in  Bihar.” Mathematica Policy Research Report, May 8, 2015.
Callen, Michael, and James Long. “Institutional Corruption and  Election Fraud: Evidence from a Field Experiment in Afghanistan.”  American Economic Review, vol. 105, no. 1 (2015): 354-381.
Dhaliwal, Iqbal, and Rema Hanna. “Deal with the Devil: The  Successes and Limitations of Bureaucratic Reform in India.” NBER  Working Paper No. 20482, September 2014.
Duflo, Esther, Michael Greenstone, Rohini Pande, and Nicholas Ryan.  “The Value of Regulatory Discretion: Estimates from Environmental  Inspections in India.” NBER Working Paper No. 20590, October 2014.
Duflo, Esther, Michael Greenstone, Rohini Pande, and Nicholas Ryan.  “Truth-Telling by Third-Party Auditors and the Response of Polluting  Firms: Experimental Evidence from India.” The Quarterly Journal of  Economics, vol. 128, no. 4 (2013): 1499-1545.
Duflo, Esther, Michael Greenstone, Rohini Pande, and Nicholas Ryan.  “What Does Reputation Buy? Differentiation in a Market for Third- party Auditors.” American Economic Review: Papers & Proceedings,  vol. 103, no. 3 (2013): 314–319.
Habyarimana, James, and William Jack. Results of a Large-scale  Randomized Behavior Change Intervention on Road Safety in Kenya.  Proceedings of the National Academy of Sciences, 2015.
Karlan, Dean, and Leigh L. Linden. Loose Knots: Strong versus Weak  Commitments to Save for Education in Uganda. No. w19863. National  Bureau of Economic Research, 2014.
Maitra, Pushkar, Sandip Mitra, Dilip Mookherjee, Alberto Motta, and  Sujata Visaria. “Financing Smallholder Agriculture: An Experiment  with Agent-Intermediated Microloans in India.” Hong Kong University  of Science & Technology Institute for Emerging Market Studies  Working Paper No. 2015-23, April 2015.
section_title: Appendix III: Comments from the U.S. Agency for International Development
section_title: Appendix IV: GAO Contact and Staff Acknowledgments
subsections: section_title: GAO Contact
paragraphs: David B. Gootnick, (202) 512-3149 or gootnickd@gao.gov.
section_title: Staff Acknowledgments
paragraphs: In addition to the contact named above, James Michels (Assistant  Director), Jeremy Latimer (Analyst-in-Charge), Debbie Chung, Daniel  Kuhn, Jill Lacey, Christopher J. Mulkins, Kyerion Printup, and Ozzy  Trevino made key contributions to this report.