Foreign Assistance: USAID Venture Capital Approach Relies on Evidence of Results but Could Strengthen Collaboration among Similar Programs
Released: 2016-01-20T00:00:00

Letter

Background

DIV’s goal is to support USAID’s mission and improve the lives of millions of people around the world within 10 years, especially those living in poverty or extreme poverty. To achieve this global development goal, DIV awards grants and cooperative agreements (in this report, collectively referred to as grants) to academic institutions, nongovernmental organizations, and businesses, among other types of organizations, to advance development-related innovations. DIV defines such innovations as novel business or organizational models, operational or production processes, or products or services that could lead to substantial improvements in addressing development challenges.

According to DIV officials, the program does not outline specific problems to be solved or propose specific solutions but is intentionally open-ended, funding grants on the basis of three core principles:

Evidence: rigorous evaluation of what works and what does not, scaling up only those solutions proven to produce demonstrable impact

Cost-effectiveness: potential to deliver greater development impacts per dollar than traditional development assistance

Potential to scale up: a plan to deliver and maintain widespread impact by increasing the geographic scope of operations and reaching financial sustainability beyond DIV’s support through private or public funding Managed at USAID headquarters, DIV takes a venture capital approach to investing in innovations, by awarding grants through a three-stage funding model. The model is intended to identify, evaluate, and scale up development innovations that demonstrate widespread impact and cost- effectiveness. According to DIV officials, the program developed this three-stage model as a risk mitigation approach.

Stage 1—Proof of Concept. In stage 1, DIV provides small grants for testing the viability of an innovation in a real-world setting. Grantees must assess whether the innovation will yield results through evaluation or performance monitoring. Innovations that have demonstrated results and satisfied stage 1 criteria are eligible for stage 2 funding to support evaluations that will test for impact.

Stage 2—Testing and Positioning for Scale. In stage 2, grantees determine, through rigorous assessments including impact evaluations, whether the solution can achieve larger-scale impact and can also be implemented successfully at a larger scale. Innovations that have credible evidence of development impact at stage 2 standards are eligible for stage 3 funding.

Stage 3—Transitioning Proven Solutions to Scale. In stage 3, DIV funding supports innovations that seek to transition a solution from large-scale implementation to widespread adoption in one country or to replication in an additional country.

DIV has established maximum funding amounts and project durations for each funding stage of this model, as shown in table 1.

According to DIV officials, the staged funding model was not envisioned as an inevitable progression of projects from stage 1 to stage 3. For example, only 7 of 56 projects received stage 2 funding after having previously received a stage 1 grant. Grantees can receive funding at any stage without necessarily implementing a DIV project at an earlier stage, provided they meet the established requirements for that stage. For example, in many cases, stage 2 grantees have demonstrated a proof of concept prior to receiving a DIV grant.

In addition, DIV places a particular emphasis on using rigorous evaluation methods, such as randomized controlled trials (RCTs), to determine impact. DIV officials noted that because DIV has set a maximum funding level of $150,000 for stage 1 grants, grantees conducting evaluation studies often require additional funding from a source other than USAID. DIV officials also noted that other funding sources may be required because the time frame for completing a study may extend beyond the duration of the DIV grant. In the case of projects at stage 2 or stage 3, the grantee has already tested its idea and requires funding to bring the project to scale.

DIV Has Funded Innovation Projects across Nine Sectors and Has Concentrated Funding Largely in Two Countries and among Four Grantees

In fiscal years 2010 through 2015, DIV obligated approximately $72.5 million for innovation projects across nine sectors, including energy, economic growth, health, and education. DIV directed approximately 52 percent of project funding to two countries where it funded projects and 40 percent of funding to four grantees, which supported projects in India and Kenya, as well as other countries.

DIV Has Funded $72.5 Million for Innovation Projects across Nine Sectors

In fiscal years 2010 through 2015, USAID obligated approximately $72.5 million for DIV grants, with annual funding rising from nearly $1 million in fiscal year 2010 to approximately $19 million in fiscal year 2015 (see fig. 1). The number of DIV grants increased from 8 in fiscal year 2010 to 41 in fiscal year 2013 and declined to 33 in fiscal year 2015.

Overall, DIV has awarded 142 grants from over 7,500 applications submitted since July 2010. These awards consisted of 83 grants for stage 1 (Proof of Concept), 56 grants for stage 2 (Testing and Positioning for Scale), and three grants for stage 3 (Transitioning Proven Solutions to Scale). The three stage 3 grants represented 21 percent of overall DIV funding, as shown in figure 2.

Overall, DIV has funded a range of projects across nine sectors, with the largest share of funding supporting projects in the energy sector ($16.6 million) and the economic growth sector ($15.8 million), as shown in figure 3.

Two DIV projects in India, in the education and training sector and the energy sector, respectively, provide illustrative examples of the types of grantee organizations and innovations that DIV has funded.

Education and training sector. In 2013, DIV awarded a $927,000 stage 2 grant to the Pratham Education Foundation to expand its evaluation of intensive learning camps in selected villages using a randomized evaluation, and determine if this model is a good candidate for implementation at scale (see fig. 4). The intensive learning camps were intended to improve learning outcomes for children in grades 3 through 5. The project organized and grouped students by ability, rather than grade level, to provide more focused instruction tailored to students’ learning needs. For example, for reading instruction, Pratham grouped students by their ability to recognize Hindi characters, words, and sentences, while for math instruction, it grouped students by their ability to recognize numbers. The project’s evaluation showed that the reading and math scores of students who participated in the learning camps increased by as much as 22 percent over the scores of students who did not participate in the learning camps.

Energy sector. In 2011, DIV provided a $300,000 stage 2 grant to Mera Gao Power (MGP) to test whether its solar micro-grid system providing low-cost electricity to off-grid villages in India was commercially viable. After micro grids were installed, MGP’s customers paid a weekly subscription fee of approximately $0.27 for the use of two LED lights and one phone charger. An MGP staff member came to subscribing villages each week at prearranged times to collect customer payments in cash (see fig. 5). By the conclusion of the grant in March 2013, MGP had installed the service in approximately 180 villages, reaching 4,480 households—exceeding its targets of 40 villages and 4,000 customers. MGP officials told us in March 2015 that they had further expanded the service to approximately 17,000 customers and had secured external financing from an impact investment firm.

DIV Funding Is Concentrated in Two Countries and with Four Grantees

While DIV has funded projects in 43 countries since 2010, over half of its project funding—roughly $37.8 million—is concentrated in India and Kenya where it has awarded 64 of its 142 grants (45 percent). Figure 6 provides information on the number of DIV projects by country, including those that were part of multicountry projects. In India, DIV funded 18 stage 1 projects and 21 stage 2 projects. In Kenya, DIV funded 14 stage 1 projects, 9 stage 2 projects, and 2 stage 3 projects.

According to DIV officials, to support its global development goal, DIV awards grants through an open-ended process to applicants that best meet its criteria, regardless of geographic location. DIV officials stated that the program did not target or prioritize India and Kenya for the implementation of its projects, and that the concentration of funding in these two countries reflects the relative strength of their applications. These officials indicated that in recent years DIV had expanded its geographic distribution of projects and made efforts to work with USAID missions to promote the program in other countries. In 2015, for example, DIV funded 33 projects in 22 countries. However, our analysis of DIV data showed that the distribution of DIV projects remained concentrated—at over 40 percent of all DIV projects—in India and Kenya in fiscal years 2013 through 2015.

DIV funding is also concentrated among its grantees, awarding roughly $29 million (40 percent of program funding) to four grantees—the Abdul Latif Jameel Poverty Action Lab at the Institute for Financial Management and Research (J-PAL/IFMR), Innovations for Poverty Action (IPA), Off Grid Electric Limited, and Georgetown University (see fig. 7). J- PAL/IFMR and IPA—research organizations that focus on evaluating development interventions using RCTs—have received a total of 31 (22 percent) of DIV’s 142 grants. J-PAL/IFMR received 13 grants, totaling $4.13 million, and IPA received 18 grants, totaling $14.1 million—the largest total amount awarded to a DIV grantee. In 2015, Off Grid Electric received a stage 3 grant totaling $5 million and is the only grantee to have received a grant for each of the three stages. These three grants supported the testing and expansion of an innovation to provide electricity to households in Tanzania with limited access to the electric grid. Georgetown University has received three DIV grants totaling $4.28 million, the third largest total dollar amount awarded to a DIV grantee. One of the grants to Georgetown University, a $3 million stage 3 grant, supports the expansion of an innovation to reduce traffic accidents in Kenya and other countries in East Africa. In a previous evaluation, researchers tested the effects of placing behavior change messages on stickers in buses that urged passengers to speak up against dangerous driving and encourage their bus drivers to slow down. They found that accident insurance claims for buses with stickers fell by half compared with claims for buses without stickers.

As with the concentration by country, DIV officials stated that the concentration of funding among four grantees reflects the relative strength of their applications and that DIV did not target these organizations for awards.

Lack of DIV Performance Targets Makes It Difficult to Determine Progress

DIV has developed and is collecting data for several program-level performance measures, which show some positive outcomes, but has not established specific targets for these measures, making it difficult to assess DIV’s progress. DIV is in the process of developing a new results framework; however, our review of a draft version of the framework shows that it does not include performance targets.

DIV Collects Data on Performance Measures but Has Not Established Corresponding Targets

DIV has used various program-level measures to track performance since the beginning of the program, and DIV officials provided data that they have collected for these performance measures through October 2015 (see table 2). These range from process-oriented measures related to overall DIV program management to measures that capture results from specific grantees. Examples of process-oriented measures include tracking the length of time between receiving an application and the final decision on whether or not to make an award, and the length of time between approving an application and issuing the award.

Examples of outcome-oriented performance measures include the percentage of projects conducting RCTs and the number of grantees that connect to outside sources of funding after the award of the DIV grant. DIV’s outcome-oriented measures, including those focused on bringing awards to scale, are similar to measures used by some venture capital firms. For example, DIV reported that as of October 2015, 11 of its grantees had obtained outside sources of funding after the award of the DIV grant, and 5 of its awards had scaled up through the public sector.

Although DIV established performance measures and is collecting data that correspond to these measures, DIV officials stated in November 2015 that they have not established targets for these measures as a means for assessing DIV’s performance. Therefore, it is difficult to determine the level of performance that DIV is intending to achieve and to determine DIV’s actual progress against targets. Our past work has shown that, although agencies collect a significant amount of performance information, they have not consistently used that information to improve management and results. The GPRA Modernization Act of 2010 (GPRAMA) requires agencies to establish performance measures to assess progress toward goals. Moreover, we have previously identified practices for enhancing agency use of performance information, including communicating performance against targets. Without related targets, DIV may be unable to demonstrate to key stakeholders, including Congress and the public, that it is making progress in achieving agency goals.

During the course of our review, DIV officials were in the process of developing a results framework for the program and provided us with a draft version of the framework. Although the draft framework provided an expanded list of indicators for measuring results, it did not include targets for these indicators. DIV officials did not provide a specific time frame for completing this results framework but stated that they expect to finalize the results framework in 2015. These officials stated that they are developing the results framework as part of an effort to better articulate and measure the program’s goals and to link to other related USAID programs. DIV officials also stated that, as part of this effort, they are outlining a monitoring and evaluation plan to inform learning objectives for the program.

DIV Has Established Evidence-Based Requirements for Funding and Evaluating Grants and Has Applied These Requirements to Grants in India

DIV has established and applied evidence-based requirements when awarding grants and assessing results, emphasizing the use of rigorous evaluation. For example, applications that DIV funded in India generally met the program’s evidence requirements, such as including evaluation plans. In addition, most completed DIV projects in India provided final reports and evaluations that corresponded with DIV’s evidence requirements. DIV also has recently taken action to ensure that the final reports and evaluations of its projects are publicly disseminated, as generally required by USAID policy.

DIV Has Established Specific Evidence Requirements and Emphasizes Rigorous Evaluations

DIV emphasizes the testing of potential development solutions and rigorously evaluating impact, to scale up only those solutions with proven results. DIV outlines specific evidence requirements for each funding stage, with greater evidence of impact required, the higher the stage applied for. For example, while applicants for stage 1 grants are not required to provide evidence of prior testing of the proposed solution, they are required to present a plan for assessing results or impact, including specific metrics for success. Applicants for higher-stage grants are required to discuss prior experiences implementing or testing their solutions and evidence of successful development impact. Additionally, stage 3 applicants are required to discuss specific evaluation methodologies and findings.

DIV encourages grantees to utilize rigorous evaluation methods, including RCTs, in their projects, while recognizing that RCTs are not appropriate in all cases. Our review of DIV project data showed that approximately 43 percent of its global portfolio, and 54 percent of projects in India, included an RCT to assess development impact (see table 3). Of the 103 DIV projects in countries other than India, 39 percent of these projects conducted RCTs. We found that DIV projects across all three stages conducted RCTs. DIV officials also stated that, although many awards use an RCT, this type of evaluation design is not applicable for every question being examined. However, these officials added that each grant does include a test of some sort and analysis of the data.

Our review of DIV documents and meetings with India-based grantees found a number of examples of DIV projects in India that are employing RCTs as part of the DIV award. For example, two DIV grantees that we met with were evaluating the use of biometric fingerprinting technologies as a tool for making improvements in different aspects of India’s health care system.

J-PAL/IFMR stage 2 DIV grant to evaluate the problem of absenteeism among medical staff in India through an RCT. Awarded in 2010, this study evaluated the impact of an intervention using a digital attendance and medical information system to monitor attendance of medical staff in government health centers in the state of Karnataka. J-PAL/IFMR randomly assigned primary health centers to treatment and control groups, and the treatment health centers were equipped with fingerprint reader devices and a mobile device for uploading attendance and patients’ data. The preliminary results of the program showed a modest effect on the attendance of nurses, pharmacists, and lab technicians, but no effect on the attendance of medical officers.

Operation ASHA stage 2 DIV grant to evaluate the effectiveness of a fingerprint identification system in preventing the occurrence and lapses in the treatment of Multidrug-resistant tuberculosis (MDR-TB) through an RCT. Awarded in 2012, this project evaluated the effectiveness of fingerprint reader devices in preventing lapses in treatment of MDR-TB patients. These devices register the presence of patients and staff at treatment centers in receiving MDR-TB treatments, and Operation ASHA’s system informs TB counselors when a patient misses a treatment. Operation ASHA is carrying out the RCT involving about 12,000 patients across approximately 200 health centers to establish the effectiveness of the fingerprint identification system as a tool for ensuring patients’ compliance with their MDR-TB treatments.

Applications DIV Funded in India Generally Met Evidence Requirements

Our review of the applications for grants DIV awarded in India found that the applications generally met evidence and other requirements. We examined 33 of the applications to determine the nature and types of evidence and other information that they contained, as well as the extent to which they met DIV’s evidence requirements (see table 4). For example, we found that 31 of the 33 applications fully or partially provided information on how the innovation would be evaluated for impact. In addition, we determined that 16 of 19 applications that were awarded stage 2 grants provided evidence that the innovation had previously been tested for impact. We also found that 29 of the 33 applicants included an analysis or information on the innovation’s cost-effectiveness, although the level of information on cost-effectiveness varied among the applications. In some applications, for example, we found that the applicants discussed cost-effectiveness, but did not provide supporting data or analysis to compare the costs of their solution to competing solutions or traditional methods of delivering development assistance.

DIV Grantees in India Provided Final Reports and Evaluations That Generally Met DIV’s Requirements

Based on our review of DIV documents, we found that most DIV grantees in India provided final reports and evaluations that met DIV’s requirements. We reviewed grant agreements for 33 of the DIV projects in India, and these agreements contained specific requirements linked to estimated completion dates and funding amounts that would be disbursed upon completion of each milestone. The DIV grants we reviewed contained requirements for delivering final reports and evaluations, although these requirements varied across projects. For example, some grant agreements required specific elements to be included in the final report, such as an analysis of the cost-effectiveness of the project, while other grants generally required that a final report be delivered without requiring that the report include specified elements.

We analyzed 18 DIV final reports and found that almost all of the reports provided information that met DIV’s milestone requirements for a final report or final evaluation (see table 5). Specifically, we found that 16 of the 18 projects met DIV’s requirements for completing a final report to close the grant agreement. In addition, we found that 10 of 13 projects completed the evaluation requirement.

We also found that 13 of the 18 final reports we reviewed provided data on results and outcomes consistently with the methodologies discussed in the respective applications. In some cases, we found inconsistencies between the evaluation methodology discussed in the application and the corresponding information in the final report. When asked for clarification, DIV officials explained that, in some cases, during the award process they negotiated changes to grantees’ methodologies or implementation plans from what was initially discussed in the application, based on updates or contextual changes.

In addition, we found that the 18 final reports for India generally addressed DIV’s three core principles: (1) evidence, (2) cost- effectiveness, and (3) potential to scale up (see table 6). For example, 17 of 18 final reports fully or partially provided evidence of the project’s development outcomes. We also found that 13 of the 18 final reports fully or partially provided information on DIV’s core principle of cost- effectiveness, although they varied in the levels and types of information provided. For example, the final report for a project testing a metered pricing system for off-grid power provided a detailed cost breakdown of the solar micro-grid system in comparison to the costs of competing energy sources, such as diesel generators and household solar panels. In another example, a grantee reported that undertaking a cost-benefit analysis was difficult because of the challenge of quantifying the intervention’s economic benefits, but it did report the cost of conducting the intervention. Finally, in five other cases, final reports did not include any discussion of cost-effectiveness.

DIV Has Begun to Publicly Disseminate Final Reports and Evaluations

DIV has recently taken action to ensure that its final reports and evaluations are publicly disseminated, as is generally required by USAID policy. USAID’s policy states that evaluation findings should be shared as widely as possible with a commitment to full and active disclosure. A standard requirement for USAID grants, including DIV’s grants, is that grantees post final reports and evaluations from completed projects on USAID’s Development Experience Clearinghouse (DEC), the agency’s online repository of research information. This requirement was specified in DIV grant agreements we reviewed that were awarded after the requirement came into effect. However, when we initially reviewed the DEC in July 2015, it did not contain any DIV final reports or evaluations. After raising the issue with DIV officials, we searched the DEC again in November 2015 and found that it contained 87 total documents from DIV projects, including 46 final reports. These included 22 total documents and 13 final reports from projects based in India, and 64 total documents and 33 final reports from DIV projects in other countries. Appendix II provides a list of published studies by DIV grantees in India.

DIV’s Limited Collaboration with Similar Innovation Programs in India Has Contributed to Missed Opportunities to Share Information and Leverage Resources

DIV and other U.S.-funded innovation programs in India support similar objectives and beneficiaries, and in several cases these programs have funded the same types of innovations. Collaboration among these overlapping programs in India has been limited, which has contributed to missed opportunities to share information and leverage resources. During the course of our review, DIV began implementing an action plan intended to improve its collaboration with the USAID mission in India (“USAID India”) and other missions. However, the plan does not establish a joint approach to development among these programs. Without such an approach, USAID may not be capitalizing on opportunities to gain efficiencies and maximize the impact of its innovation programs.

DIV and Other U.S Innovation Programs in India Support Similar Objectives and Beneficiaries

We identified several U.S.-funded innovation programs in India with similar objectives and beneficiaries, as shown in table 7. These overlapping programs award grants to promote proven innovations that will benefit poor, underserved populations in India. USAID India’s 2012- 2016 country development strategy has two objectives focused on innovation, including supporting innovations that impact those living in extreme poverty, and supporting innovations proven in India and disseminating them to other countries. To support these objectives, USAID India created several innovation programs similar to DIV. For example, in 2012 USAID India established the Millennium Alliance, an innovation grant program modeled on DIV. This program provides funding to Indian grantees that demonstrate cost-effective solutions that address the needs of the extreme poor in India. Like DIV, the program uses a staged funding model to make relatively small initial investments, test more developed solutions, and scale up those that have proven development impact through rigorous evaluations. USAID India also created the U.S.-India Partnerships program to overcome critical development challenges through new technologies and other innovations that can be rigorously tested, shared, and scaled up in India and abroad. State also funds innovations to support economic growth and clean energy for underserved populations in India through the U.S.-India Science and Technology Endowment Fund and the PACESetter fund, respectively.

While these programs have similar objectives and beneficiaries, they also have some differences in the way they are implemented. For example, the Millennium Alliance and the U.S.-India Science and Technology Endowment Fund are managed in cooperation with the Indian government, while DIV is funded and managed solely by USAID. In addition, the USAID India Partnerships Program requires partners to share the costs and contribute at least $500,000 in cash and in-kind resources for a one-to-one match totaling at least $1 million, while DIV has no requirements for cost-sharing.

DIV and Other USAID and State Innovation Programs Fund Some Similar Projects in India

Our review of project data for innovation programs in India identified examples in which these programs have funded innovations similar to those funded by DIV, as shown in figure 8. For example, both DIV and another program supported projects to test the viability of “clean” cook stoves in rural markets—that is, stoves designed to reduce air pollution and firewood consumption compared with traditional cook stoves. DIV and two other programs supported projects to provide inexpensive eye care and eyewear for poor and underserved populations. DIV and two other programs funded projects to support the development, testing, or implementation of micro grids for people living in rural areas who are unconnected to the power grid.

USAID and State officials we interviewed in India stated that they generally supported the implementation of similar innovation programs in India by different organizations if the programs resulted in additional resources being made available to poor and underserved populations. According to these officials, there is a vast need for innovations, such as clean energy and off-grid electricity, which improve the lives of the poor in that country. In addition, State officials commented that in a country of 1.25 billion people, with significant diversity in cultural, linguistic, and religious norms, as well as considerable geographic diversity, different solutions to the same problem may produce varying levels of success in different contexts. These officials said that, as a result, it may be in some cases necessary to fund several similar or competing solutions in an effort to identify the few that demonstrate widespread impact and cost- effectiveness. USAID India officials added that because of the variety of difficult and intractable problems in India, the mission does not see it as problematic that there would be more than one activity aimed at addressing the same problem.

Collaboration among U.S.- Funded Innovation Programs in India Has Been Limited and Opportunities Have Been Missed to Share Information and Leverage Resources

We have previously found that several key practices that enhance collaboration, including articulating a joint strategy and common outcomes, agreeing on roles and responsibilities, and identifying and addressing needs by leveraging resources, can help manage programs with similar objectives and beneficiaries. Collaboration among U.S.- funded innovation programs in India has not routinely or systematically included these practices and, with some exceptions, has been limited to USAID India’s providing initial input to DIV regarding grant award decisions.

During USAID India’s technical reviews of DIV applications, DIV routinely communicated with USAID India officials, requesting that the mission review applications for projects that DIV subsequently funded in India. USAID India officials rated the proposals, provided narrative information on the strengths and weaknesses of the projects, and in some cases raised concerns.

However, according to DIV officials and a wide range of USAID India and State officials we spoke with in India, collaboration among programs beyond these examples has been limited and has not routinely included the key practices we identified. For example, these officials indicated that after award decisions were made, DIV and the other programs did not systematically share information about project results, or reach agreements on their respective roles and responsibilities, such as roles in coordinating planning for the use of grant funds or in monitoring the implementation of grants. State officials we interviewed in India who manage other innovation programs—such as the U.S.-India Science and Technology Endowment Fund and the PACESetter fund—stated that, while they were aware of some of the activities that DIV has supported in India, DIV had not communicated with these other agency programs to collaborate on ongoing or upcoming efforts.

USAID India and State officials we interviewed in India who manage innovation programs under the India Partnerships program and the PACESetter Fund, among others, told us that limited collaboration among these programs and DIV had resulted in missed opportunities to share information and leverage USAID India resources by providing outreach and monitoring of project implementation and marketing DIV innovations. We have previously found that, without engaging in collaboration practices such as agreeing on roles and responsibilities or identifying common outcomes, overlap can have a negative effect in that limited resources may not be used in the most efficient and effective manner and opportunities may be missed to leverage resources. The following provide some examples of missed opportunities to share information and leverage the mission’s resources.

Missed opportunities to provide outreach to DIV grantees and monitor project implementation. USAID India officials stated that in many cases, they did not know DIV grants had been awarded and thus missed opportunities to provide outreach and establish productive working relationships with DIV grantees in India because roles and responsibilities were not clarified. For example, USAID India officials from the India Partnerships program, and the health sector, cited examples in which DIV grantees had contacted USAID India seeking assistance but found that the mission was unaware that DIV had awarded grants to these organizations, limiting the effectiveness of USAID India’s outreach to them. In addition, USAID India officials stated that the mission’s and DIV’s respective roles and responsibilities were not always clear to USAID India officials or to DIV grantees, which negatively affected some grantees’ perceptions of USAID. USAID India officials also discussed missed opportunities to conduct project monitoring on DIV’s behalf, and DIV officials stated that they had not sought assistance from USAID India in monitoring DIV projects.

Missed opportunities to market DIV innovations. USAID India and State officials responsible for promoting U.S.-funded clean energy projects in India, including those of the PACEsetter Fund, indicated that they had missed opportunities to share those results with the government of India or other stakeholders with the means to scale them up, if appropriate. For example, during a recent high-level U.S. government delegation’s visit to India, USAID India officials stated that they had missed an opportunity to highlight promising DIV projects focused on clean energy activities, because they were unaware of the projects’ results.

Although collaboration among innovation programs in India has been limited, DIV officials provided some additional examples of communication and consultation with USAID India beyond the initial consultation on DIV applications. For example, USAID India and DIV jointly funded a project to rigorously test an innovation to increase full immunization rates in rural areas. This project is testing the viability of implementing the innovation in cooperation with the government system in the Indian state of Haryana. USAID India and DIV officials also discussed DIV’s consultations with the USAID India energy team and corresponding interactions between DIV grantees and the USAID India energy program, which was in contrast to more limited consultations between DIV and USAID India officials from other programs.

DIV Has Begun Implementing an Action Plan to Improve Collaboration but Has Not Agreed on a Joint Approach with Relevant Programs in India

During the course of our review, DIV officials, acknowledging that collaboration could be improved, began implementing an action plan to improve collaboration with missions and bureaus within USAID. The plan outlines steps to share information on DIV’s activities across the agency, including establishing DIV points of contact for outreach with missions and bureaus and developing tools for providing more frequent updates on DIV projects with the missions. The plan also discusses time frames for collaboration activities, including identifying opportunities for joint management and co-investment with missions on DIV projects. During our visit to India, a DIV official briefed staff at the mission about the DIV program and about its broader plans to increase outreach to USAID missions. DIV officials also provided us with additional examples of briefings and outreach that it conducted with other bureaus and missions. In addition, in 2015, DIV collaborated with the USAID mission in Jordan to hold a regional competition for innovators to pitch their ideas for development solutions to be considered for grants in the Middle East and North Africa region, where DIV has made the fewest awards thus far.

We previously found that developing a joint approach among related programs, including reaching agreement on a joint strategy, common outcomes, roles and responsibilities, and leveraging resources, can improve collaboration and generate greater results than the programs could achieve independently. However, DIV’s action plan, while a promising step toward improving collaboration, does not yet represent a joint approach among the overlapping innovation programs that we identified in India. Thus, DIV cannot ensure that the benefits of its initial outreach efforts will be realized. DIV has not extended its action plan to include the similar innovation programs we identified outside USAID India, for example. Also, DIV has not harmonized its award selection processes with those of the other innovation programs to help ensure that funding similar projects is appropriate and not unnecessarily duplicative. Furthermore, DIV has not reached agreement with the other U.S.-funded innovation programs on a common approach to monitoring and evaluation of the projects they select to fund to help ensure that results from similar projects are being assessed consistently. Consequently, it may be difficult to determine which version of a technology or innovation has the greatest potential to scale up and where further U.S. support would have the most impact.

Conclusions

USAID created DIV to demonstrate a new model of U.S. development assistance, and DIV’s approach has shown promise, especially through the rigorous evaluation of an innovation’s results and outcomes before determining whether to increase the agency’s investment to bring these innovations to scale. After 5 years of experience in implementing projects and testing its model in real-world situations, DIV has an opportunity to assess what results have been achieved and what lessons have been learned from these initial experiences. However, because DIV lacks clearly identified performance targets for its program, and its draft results framework does not contain targets, it is difficult to assess DIV’s overall progress toward achieving its goal of promoting global development for the poor through its portfolio of innovations.

Since DIV was established in 2010, several other U.S. grant programs have emerged that overlap with DIV and in some cases have funded the same types of innovations. Although USAID’s mission in India prioritized innovation and modeled one of its programs after DIV, DIV and the mission collaborated to a limited extent during the implementation of DIV’s projects. As a result, DIV and the USAID mission in India have missed opportunities to share information and leverage resources. While DIV has begun implementing an action plan to improve collaboration with other USAID missions and bureaus, the action plan does not establish a joint approach, including reaching agreement on a joint strategy, common outcomes, or roles and responsibilities among all relevant programs and agencies. Thus, USAID may not be capitalizing on potential synergies among these innovation programs or maximizing their efficiency and impact.

Recommendations for Executive Action

To help ensure that DIV is making progress toward achieving its global development goal, we recommend that the Administrator of USAID take the following two actions: 1. Establish performance targets that will allow periodic assessment of DIV’s progress toward achieving its goal. 2. Establish a joint approach to collaboration reflecting agreement with the USAID mission in India and with other related U.S. agency programs in India, and consider where such a joint approach would be beneficial in other countries.

Agency Comments and Our Evaluation

We provided a draft of this report to USAID and State for comment. USAID and State provided technical comments, which we incorporated into the report as appropriate. USAID also provided written comments, which are reprinted in appendix III.

USAID agreed with our recommendations, stating that our review had helped identify areas for improvement. USAID also discussed steps it is taking to respond to the recommendations. With regard to the recommendation to establish performance targets that will allow periodic assessment of DIV’s progress toward its goal, USAID noted that it had recently established a results framework that includes targets and performance milestones to be assessed on a semiannual basis. With regard to the recommendation to establish a joint approach to collaboration with the USAID mission in India and with other agency programs in India, USAID discussed collaboration and coordination between DIV and the USAID mission in India that had occurred during the course of our review. USAID also stated that it would build on these collaboration efforts, discuss where improvements could be made, and take action to formalize them.

We are sending copies of this report to the appropriate congressional committees, the Administrator of USAID, the Secretary of State, and other interested parties. In addition, the report is available at no charge on the GAO website at http://www.gao.gov.

If you or your staff have any questions about this report, please contact me at (202) 512-3149 or gootnickd@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. GAO staff who made key contributions to this report are listed in appendix IV.

Appendix I: Objectives, Scope, and Methodology

We were asked to review the U.S. Agency for International Development’s (USAID) Development Innovation Ventures (DIV) program. In this report, we examine the DIV program’s (1) distribution of funding and (2) efforts to measure progress toward achieving its goals, and for DIV’s activities in India, we examine (3) the extent to which DIV uses evidence to make funding decisions and assess results and (4) DIV’s collaboration with other similar U.S. development assistance innovation programs.

To examine the distribution of DIV funding, we obtained and analyzed funding data for projects from fiscal years 2010 to 2015. These data included information on the stage of the project, the fiscal year, and the award start and end dates of the projects, among other things. Projects implemented in more than one country were counted as one project per each country. For example, by 2015 DIV had funded 33 individual projects in India and 6 projects that were implemented in multiple countries—of which India was one—for a total of 39 projects for India. To assess the reliability of these data, where possible, we cross-checked the data with other sources, evaluated the data for internal consistency, and interviewed agency officials knowledgeable about the data sources. We determined the data presented in this report to be sufficiently reliable for our purposes.

To examine DIV’s efforts to measure progress toward achieving the program’s goals, we reviewed and analyzed DIV’s performance data and interviewed USAID officials in Washington, D.C. Specifically, we analyzed performance data publicly available on the USAID website, reviewed data and documentation on the DIV program provided by USAID, and interviewed officials in the DIV program office at USAID. To assess the reliability of these data, where possible, we cross-checked the data with other sources, evaluated the data for internal consistency, and interviewed agency officials knowledgeable about the data sources. We determined the data presented in this report to be sufficiently reliable for our purposes.

Based on our analysis of USAID project data, we selected India as a nongeneralizable case study for two objectives. Of the 33 countries DIV had supported through 2014, India had received the largest amount of funding and had the largest number of projects. To examine the extent to which DIV uses evidence to make funding decisions and assess the results of projects it funded in India, we reviewed and analyzed DIV documents, including applications, final reports, and final evaluations. We focused our analysis on projects implemented in India, including multicountry projects where India was one of the countries where the project was being implemented, for a total of 33 projects. For the applications analysis, we reviewed USAID applications for funding through the DIV program, for multiple rounds, and developed a data collection instrument based on criteria in the application. We independently reviewed the applications and rated the extent to which the applicant responded to the question, supplying data and citations when necessary. We then reconciled any instances where the ratings of the initial and secondary reviewer did not concur. We aggregated and reported the outcomes of several questions we developed in the data collection instrument. For the final reports and evaluations, we reviewed the award letters for all 18 projects in India that had completed the grant agreement and submitted a final report or final evaluation. The award letters contain the terms and conditions of the grant agreement, including required tasks to be completed with the supporting documentation, to be submitted periodically to DIV by the grant recipient. We developed a data collection instrument based on DIV’s principles of testing and scaling innovations that demonstrate widespread impact and cost-effectiveness. In addition, we analyzed the award letters, the applications, and the final reports and evaluations submitted to DIV to determine the extent to which the methodologies in final reports and evaluations corresponded to the proposed methodologies in the application.

Finally, we examined DIV’s collaboration with other similar U.S. development assistance innovation programs, using India as a nongeneralizable case study. We interviewed officials at USAID, the Department of State (State), the Department of Agriculture’s Foreign Agricultural Service (FAS), and the Department of Energy, in Washington, D.C. We traveled to the USAID mission in New Delhi, India, to interview officials from USAID, State, FAS, the U.S. Trade and Development Agency (USTDA), and the Department of Commerce, and to obtain information on any development and assistance innovation programs managed by the USAID India mission or the U.S. embassy. During our fieldwork, we observed DIV projects and interviewed grant recipients. We also identified programs at the USAID mission and embassy in New Delhi that had innovation and development components. We interviewed program officials from these agencies, including the Chief of Mission and the Deputy Chief of Mission, to obtain information on the programs and their experiences in collaborating with the DIV office in Washington, D.C. In addition, we analyzed program data that we obtained from DIV, and from USAID India and other agency officials in New Delhi, to determine the extent to which the DIV program overlaps with programs from USAID and other U.S. agencies in India, and has funded projects that could overlap or duplicate projects funded by U.S. agencies in India.

Appendix II: Published DIV Studies for Projects in India

DIV officials provided us with the following list of published studies relating to DIV projects implemented in India.

Ashraf, Nava, Oriana Bandiera, and Scott S. Lee. “Do-gooders and Go-getters: Career Incentives, Selection, and Performance in Public Service Delivery.” Harvard Business School Working Paper, March 2015.

Banerjee, Abhijit, Donald Green, Jennifer Green, and Rohini Pande. “Can Voters be Primed to Choose Better Legislators? Experimental Evidence from Rural India.” Working Paper, 2010.

Borkum, Evan, Anitha Sivasankaran, Swetha Sridharan, Dana Rotz, Sukhmani Sethi, Mercy Manoranjini, Lakshmi Ramakrishnan, and Anu Rangarajan. “Evaluation of the Information and Communication Technology (ICT) Continuum of Care Services (CCS) Intervention in Bihar.” Mathematica Policy Research Report, May 8, 2015.

Callen, Michael, and James Long. “Institutional Corruption and Election Fraud: Evidence from a Field Experiment in Afghanistan.” American Economic Review, vol. 105, no. 1 (2015): 354-381.

Dhaliwal, Iqbal, and Rema Hanna. “Deal with the Devil: The Successes and Limitations of Bureaucratic Reform in India.” NBER Working Paper No. 20482, September 2014.

Duflo, Esther, Michael Greenstone, Rohini Pande, and Nicholas Ryan. “The Value of Regulatory Discretion: Estimates from Environmental Inspections in India.” NBER Working Paper No. 20590, October 2014.

Duflo, Esther, Michael Greenstone, Rohini Pande, and Nicholas Ryan. “Truth-Telling by Third-Party Auditors and the Response of Polluting Firms: Experimental Evidence from India.” The Quarterly Journal of Economics, vol. 128, no. 4 (2013): 1499-1545.

Duflo, Esther, Michael Greenstone, Rohini Pande, and Nicholas Ryan. “What Does Reputation Buy? Differentiation in a Market for Third- party Auditors.” American Economic Review: Papers & Proceedings, vol. 103, no. 3 (2013): 314–319.

Habyarimana, James, and William Jack. Results of a Large-scale Randomized Behavior Change Intervention on Road Safety in Kenya. Proceedings of the National Academy of Sciences, 2015.

Karlan, Dean, and Leigh L. Linden. Loose Knots: Strong versus Weak Commitments to Save for Education in Uganda. No. w19863. National Bureau of Economic Research, 2014.

Maitra, Pushkar, Sandip Mitra, Dilip Mookherjee, Alberto Motta, and Sujata Visaria. “Financing Smallholder Agriculture: An Experiment with Agent-Intermediated Microloans in India.” Hong Kong University of Science & Technology Institute for Emerging Market Studies Working Paper No. 2015-23, April 2015.

Appendix III: Comments from the U.S. Agency for International Development

Appendix IV: GAO Contact and Staff Acknowledgments

GAO Contact

David B. Gootnick, (202) 512-3149 or gootnickd@gao.gov.

Staff Acknowledgments

In addition to the contact named above, James Michels (Assistant Director), Jeremy Latimer (Analyst-in-Charge), Debbie Chung, Daniel Kuhn, Jill Lacey, Christopher J. Mulkins, Kyerion Printup, and Ozzy Trevino made key contributions to this report.