Pay for Success: Collaboration among Federal Agencies Would be Helpful as Governments Explore New Financing Mechanisms
Released: 2015-09-09T00:00:00

Letter

Background

OMB Guidance Encourages Agencies to Use Evidence-Based Approaches, Including PFS

According to OMB’s 2013 Memo on the Next Steps in the Evidence and Innovation Agenda …evidence includes evaluation results, performance measures, and other relevant data analytics and research studies, with a preference for high-quality experimental and quasi-experimental studies.

Since 2010, OMB has used its annual budget guidance to broadly encourage federal agencies to use evidence to consider the effectiveness of their programs and to institutionalize the use of evidence to foster innovation rooted in research and rigorous evaluation. In a 2013 memo, OMB identified areas in which agencies could improve their use of evidence, including: (1) strengthening evaluation capacity and proposing new evaluations; (2) developing high-quality, low-cost evaluations and rapid, iterative experimentation; (3) using innovative outcome-focused design; and (4) increasing agency capacity to use evidence. In 2015, OMB required agencies to include an evidence template with their fiscal year 2017 budget submissions. The template should provide an overview of evidence-building strategies and identify related priorities. The six federal agencies in our study that typically fund social programs said they used evidence to inform a range of program proposals or decisions. See appendix 2 for illustrative examples of these agencies’ evidence-based programs.

As part of its agenda to base management and policy decisions on evidence, the White House and OMB have encouraged agencies to explore PFS, where appropriate, as one tool to incorporate evidence into grant making. As of July 2015, four federal agencies have awarded PFS grants or are developing PFS grant proposals (see table 1).

PFS Is a Mechanism for Contracting for Social Outcomes

PFS is a program funding mechanism that some U.S. state and local and foreign governments are using to contract for prevention services to address entrenched social problems. The first PFS project began in the United Kingdom in 2010. In a PFS contract, private investors provide upfront capital that a service provider uses to implement an evidence- based prevention program. An intermediary organization can solicit funding from the investors, facilitates the flow of capital from the investors to the service provider, and oversees the provider’s work throughout the project. If the program is successful, the government entity makes payments to the intermediary, which in turn may repay the investors their capital along with interest. These payments are, in theory, based on the savings that result from decreased use of government services. PFS is intended to allow government to transfer the risk of paying for a program that does not achieve desired outcomes from the government to other participating organizations. If the program is successful, the government obtains improved outcomes for vulnerable populations. In addition, PFS can potentially capture cost savings by reducing the need to provide costly remediation services to the individuals in vulnerable populations. However, stakeholders said that PFS can also be used to contract for outcomes that a government deems worthwhile, even if the outcomes do not generate savings. Because the government makes payments based on the service provider meeting outcome targets, all parties to the contract have a vested interest in the success of the service provider.

An individual PFS project comprises a series of contracts and other vehicles, such as loan and grant agreements, that defines entities’ roles in the project and identifies the risks they each agree to assume and the benefits they each stand to receive by virtue of their participation.

Although individual PFS projects may vary in terms of roles and responsibilities for the parties to the contract, there are five main types of organizations that commonly participate in a PFS project: government, intermediaries, investors, service providers, and evaluators (see figure 1).

Government originates a contract for defined outcomes and pays on the basis of those outcomes being achieved. In the United States, state and local governments have initiated PFS projects to date, while in the UK both national and local-level governments have initiated them. Examples of government entities that have been involved in PFS projects include state or local programmatic agencies and central management offices, which often work together collaboratively on a project. As mentioned above, DOL and CNCS have awarded grants to support state and local PFS projects.

Investors provide capital to finance a program. They provide capital on the condition that the government will repay the investment if outcomes are achieved, usually with a rate of return. In U.S. projects to date, capital has been provided by a mix of investor types, such as banks, foundations, and high net worth individuals, whereas in UK projects, capital has been provided solely by social impact investors.

Intermediaries enter into a government contract in a PFS project and usually receive a fee for the services they provide. Intermediaries can take on a variety of roles in a PFS project, such as providing technical support to governments on the feasibility of a project, identifying the service provider, raising operating funds from investors, negotiating the terms of contracts or agreements with governments, service providers, and investors; disbursing payments to service providers; and overseeing day-to-day performance management of the project. Examples of intermediaries that have been involved in PFS projects include nationally focused nonprofit organizations with public policy and financial expertise and a locally focused nonprofit organization with experience implementing programs within local communities.

Service providers enter into contracts with an intermediary or a government to administer the evidence-based prevention program. Examples of service providers that have been involved in PFS projects include nonprofit or charitable organizations with established track records of implementing programs in specific policy areas, such as job training or homelessness prevention.

Evaluators measure a program’s outcomes or evaluate a program’s effect on the target population. The evaluator’s work assures the other participating organizations that defined outcomes were achieved and are likely due to the program, rather than to other factors, including chance. Outcome payments are based on these findings. In some cases, a validator, independent of the evaluator, is responsible for certifying the evaluator’s findings. Examples of evaluators that have been involved in existing PFS projects include a research office within a government department and an organization with experience evaluating programs within a specific public policy area, such as early childhood education.

The PFS projects we reviewed generally had three phases: a feasibility assessment, project design, and program implementation (see figure 2).

Practices from Federal Experience with Complex Service Acquisitions Can Inform PFS Projects

As described previously, a PFS project is a complex set of contracts through which multiple organizations work together to deliver social service programs. PFS projects are similar to complex acquisitions. Complex acquisitions are defined, in part, by the difficulty in defining precise terms of exchange for all the characteristics of the product or service at the time of the contract negotiation. We have previously identified key factors that government entities should incorporate into complex acquisitions to achieve improved service acquisition outcomes. Key literature on complex acquisitions we reviewed reinforces these factors. Taken together, this understanding can inform the design and management of PFS contracts. To address the inherent ambiguity in complex acquisitions, parties can include processes in the contract to manage decision making and to make ongoing adjustments during implementation. Specifically, providing clear standards in the following areas can facilitate PFS contracts:

Valid and well-defined requirements. We have previously reported that establishing a valid need for a service, and then translating that need into a service acquisition requirement, is essential for obtaining the right outcome. Literature on complex contracting underscores the need for well-defined requirements, but states that it can be challenging to define all requirements upfront in a complex acquisition. Undefined requirements leave room for interpretation that, if not carefully managed, can lead to outcomes that are inconsistent with expectations.

Properly structured business arrangements. We have previously reported that once requirements have been defined, it becomes necessary to develop a business arrangement to meet the requirements and protect the government’s interests. At a basic level, properly structuring business arrangements includes defining a clear scope of expected contractor performance and developing an objective means of assessing the contractor’s performance. There should be clearly established relationships among the tasks the contractor is expected to perform, the contract terms and conditions, and performance evaluation factors and incentives. Literature on complex contracting underscores the importance of properly structured business arrangements, including governance rules, which should define how the government and contractor will interact to make decisions during the contract’s implementation.

Proactively managed outcomes. We have previously reported that managing and assessing post-award performance entails various activities performed by government officials to ensure that the delivery of services meets the terms of the contract, including adequate oversight resources, proper incentives, and a capable workforce for overseeing contractor activities. Each of these activities requires metrics and tools to encourage contractors to provide superior performance and to manage and document acceptable contractor performance. Literature on complex contracting also underscores the importance of rewarding successful outcomes, particularly in cases where it is difficult to define requirements, such as efforts to acquire services.

PFS Projects Were Structured in a Variety of Ways and Sought to Achieve Potential Benefits, Such as Improved Social Outcomes and Cost Savings

PFS Projects Were Structured to Implement Prevention Programs to Address Entrenched Social Problems

In our selected case illustrations, governments used PFS to address entrenched social problems. The individuals who face these problems have complex, multi-faceted needs and are often poorly served by existing interventions or programs. The organizations participating in the PFS projects in our study said that these individuals can be better served through prevention programs focused on defined outcomes. Stakeholders told us that PFS is just one tool among many for addressing social problems and that PFS is not a panacea.

Stakeholders said PFS is most appropriate in policy areas in which populations face complex, entrenched problems. One stakeholder also noted that PFS may work best on projects that supplement, not supplant, existing services, such as prevention programs for which existing services have been insufficient in achieving desired outcomes. Projects in our study used PFS to implement programs intended to reduce recidivism among an adolescent jail population, gaps in academic achievement among preschoolers, homelessness for difficult-to-serve individuals, wait times for adoption, and to provide preventive health care to address poor birth outcomes and asthma.

Massachusetts: Reducing Recidivism and Increasing Employment among Ex- Offenders

Massachusetts Juvenile Justice PFS Initiative The “Roca Model,” which the project is funding, includes 4 basic elements: outreach to young men; intensive case management; life skills, educational, prevocational, and employment programming; and a connection to work opportunities. Project stakeholders believe that the Roca Model will be able to help young men exiting the criminal justice system to reduce the average 2.4 years they spend in prison for new convictions after they are released.

The Massachusetts Juvenile Justice PFS Initiative, which began in October 2013, was designed to address high rates of recidivism among young men. Every year approximately 4,000 high-risk young men age out of the state’s juvenile justice system or are released from adult probation. Sixty-four percent of these young men are incarcerated at least once within 5 years of release. Since its initial design, the project has been expanded to serve young men connected to the adult corrections system who are at high risk of future adult incarceration. The project is being used to fund an intervention provided by Roca, Inc., which has been shown to reduce incarceration rates for the young men it serves by 33 percent.

Massachusetts will make payments on the basis of reduced incarceration, increased employment, and for each individual program participant who meets 9 or more times per quarter with a Roca staff member to discuss job readiness issues. The project’s target impact is a 40 percent reduction in participants’ days in incarceration compared to a control group during the 5 years following their enrollment in the program. At that level, Massachusetts expects to make $21 million in outcome payments and realize $22 million in gross savings during the approximately 6-year lifespan of the project. If participants’ days in incarceration are reduced by 55 percent, Massachusetts expects to make the maximum $28 million in outcome payments and realize $33 million in gross savings. The U.S. Department of Labor (DOL) awarded Massachusetts an $11.7 million grant for the project. Up to $10.8 million of this amount is available for outcome payments associated with the first 2 years of service for up to 535 young men. Massachusetts made its first payments related to the job readiness metric in May 2015. The state will make its first payments related to the incarceration and employment metrics in the fall of 2017. See figure 2 for an example of the roles of the organizations participating in the project and the size of the project’s investment and potential savings.

Adolescents released from Rikers Island jail in New York City have a 47 percent chance of returning to jail within 1 year. The New York City Rikers Island Project, which began in September 2012, was designed to address this recidivism rate. To do so, Osborne Association and Friends of Island Academy administered a form of cognitive behavioral therapy called Moral Reconation Therapy to adolescents at Rikers Island. Project participants estimated that the city could save $1.7 million in net long-term costs if participants’ future days in jail are reduced by 11 percent compared to a similar group that did not receive the therapy. They estimated the city could save $20.5 million if participants’ future days in jail are reduced by 20 percent. In July 2015, the project’s evaluator determined that the program did not lead to reductions in recidivism rates for participants in its 1-year assessment of the project. The New York City government did not make outcome payments to investors and the program was discontinued as of August 31, 2015. See figure 3 for a description of the roles of the organizations participating in the project and the size of the project’s investment and potential savings.

The New York PFS project, which began in October 2013, was designed to reduce the high recidivism rate and increase the low employment rate of high-risk individuals recently released from state prison each year. According to a report on the project, these individuals spend an average of 460 days in prison or jail in the 5 years following their release. The project is being used to fund an intervention by the Center for Employment Opportunities (CEO). A random control trial (RCT) evaluation showed CEO’s intervention reduced recidivism for recently released participants by 16 to 22 percent and days incarcerated for high- risk participants by 30 percent. New York will make payments on the basis of reduced recidivism, increased employment, and engagement in transitional jobs for recently released, high-risk individuals. New York estimates that it can capture savings and benefits ranging from $13 million if recidivism is reduced by 10 percent to $37 million if recidivism is reduced by 40 percent over a 5-year period. The U.S. Department of Labor (DOL) awarded New York a $12 million grant for the project. Up to $11.1 million is available for outcome payments associated with the first 2 years of the project. The first outcome payment would occur in 2017. See figure 4 for a description of the roles of the organizations participating in the project and the size of the project’s investment and potential savings and benefits.

In 2011, a 3-year study conducted by Utah’s Granite School District showed that between 25 percent and 33 percent of low-income students in the district would likely be placed in special education without a preschool intervention. The Utah PFS project was designed to decrease the use of special education and to address academic achievement gaps for low-income students who enter kindergarten without having attended high-quality preschool. The project began in school year 2013-2014, with outcome payments being supported by Salt Lake County and the United Way of Salt Lake. Beginning in school year 2014-2015, Utah became the outcome payor.

The project aims to expand Granite School District’s high-quality preschool program statewide, which evidence has shown reduces by 95 percent the need for special education among the low-income students who would otherwise likely be placed in special education programs. Economically disadvantaged students who are eligible to receive a free or reduced price lunch are eligible to be enrolled in the program. Of these students, those with the most extreme negative scores (as reflected by their scoring two standard deviations below the mean) on a test taken at the beginning of preschool are tracked from kindergarten through sixth grade to verify whether they are enrolled in special education. Beginning in the summer of 2016, Utah will make annual payments to investors that equal 95 percent of cost savings for each year from kindergarten through sixth grade that each designated child is not enrolled in special education. Utah estimates it can save approximately $2,700 per year if a child is not enrolled in special education. See figure 5 for a description of roles of the organizations participating in the project and the size of the project’s investment and potential savings and benefits.

The Peterborough Prison Project, which began in September 2010, was designed to reduce the recidivism rate among prisoners released from Peterborough Prison after serving a sentence of less than a year by improving their access to supportive social services. The project is being used to fund a range of services for the prisoners, including housing assistance, drug or alcohol treatment, and mental health support. The UK Ministry of Justice agreed to make outcome payments on the basis of reductions in recidivism of 10 percent for each program cohort or group of prisoners served during a 2-year time frame, or a 7.5 percent reduction for the three cohorts as a whole. In April 2014, the Ministry of Justice announced that it would close the Peterborough Project 2 years early because of a shift to the Transforming Rehabilitation program, which will provide statutory services similar to those provided in the Peterborough Project for all short-sentenced prisoners. See figure 6 for a description of roles of the organizations participating in the project and the size of the project’s investment and potential savings and benefits.

The London Homelessness Project, which began in November 2012, was designed to improve outcomes for a cohort of entrenched homeless individuals who do not respond to traditional models of resettlement and who are not targeted by other initiatives. The project funds two providers to deliver services tailored to the specific needs of the individuals within the cohort. The Greater London Authority will pay the providers for achieving outcomes including a reduction in the number of individuals sleeping on the street; an increase in individuals who sustain stable accommodation and gain employment; a reduction in average use of the healthcare system for accident or emergency episodes; and reconnecting individuals to a country outside of the UK if he or she is a foreign national.

Targets for each of these outcomes are based on historical data on homeless individuals within London. The project does not include an evaluation for the purposes of determining outcome payments because it was not possible to create a matched comparison group for the cohort. This is because other cities in the UK do not have as complete data on homeless individuals as does London and the cohort is unique within London. A feasibility study estimated the net present value of providing services across the five outcome areas to the entire cohort over 5 years to be approximately £30 million (approximately $49 million). If all of the project outcomes are achieved, the maximum outcome payments and administrative costs will total £5 million (approximately $8 million), thus resulting in substantial public sector savings. See figure 7 for a description of roles of the project’s participating organizations.

The It’s All About Me (IAAM) adoption project, which began in October 2013, was designed to find adoptive homes for children that otherwise would not find them. The project funds adoption agencies to recruit parents to adopt hard-to-place children. The agencies also provide these parents with support services. The program’s goal is for up to 140 of these children to be adopted each year. A UK local government pays a service provider defined amounts at different points in the adoption process, such as when a service provider places a child with an adoptive family, and after 1 and 2 years of the child remaining with the adoptive family. According to the project intermediary, while the maximum cost is greater than what a local authority usually would pay adoption agencies, it is also estimated to be about half of the cost of a child remaining in foster care for 2 years. Evaluations are being conducted on the health and psychological outcomes for the adopted children; outcomes for families who adopt the children; and how decision making changes within local governments as a result of the project. See figure 8 for a description of the roles of the organizations participating in the project.

In addition to the case illustrations identified above, we examined 3 additional PFS initiatives that have yet to be implemented or were not implemented.

South Carolina Maternal Health PFS Initiative

Stakeholders in South Carolina are developing a PFS initiative focused on controlling costs and improving health and other outcomes of mothers and newborns in South Carolina’s Medicaid program. The project is expected to expand the Nurse-Family Partnership program, which, according to stakeholders, is an evidence-based nurse home visiting program for first-time, low-income mothers and their children, to rural and under-served parts of the state. Trained professionals provide services and support to pregnant women and families with young children, primarily during visits to families’ homes. According to a feasibility study prepared for the project, multiple trials and evaluations have demonstrated NFP’s effectiveness, which includes reduced emergency room visits and better health outcomes in later years.

Asthma Impact Model for Fresno (AIM4Fresno)

In Fresno, California, a nonprofit organization is conducting a demonstration project to determine whether a PFS project is feasible. The project, which began in March 2013, was designed to improve the health of low-income children with asthma and reduce the costs that result from emergency treatments. Fresno, according to the stakeholders, has one of the highest asthma rates in the United States, and approximately 20 percent of children there have been diagnosed with the disease. The stakeholders are conducting an evaluation to demonstrate the social and financial benefits of a comprehensive in-home asthma management program for children at high risk of asthma emergencies. The program includes in-home education, environmental assessment, and remediation, such as mold removal to help manage asthma. Based on the outcomes of the demonstration project, stakeholders plan to pursue PFS financing to scale up the program to more children who can benefit.

Maryland Review of a Potential PFS Initiative

In January 2013, the Maryland Department of Legislative Services (DLS) published the findings of its review of the feasibility and potential benefits and risks of a PFS project for financing reentry programs for state prisoners. The review considered, among other things, whether a PFS project could generate cost savings for the state. In its review, DLS assumed that the PFS project would fund a pilot program to serve 1,250 participants over 5 years and reduce recidivism by 10 percent or 20 percent. DLS concluded that even if the pilot program reduced recidivism by 20 percent, that result would be insufficient to close a wing of a prison or an entire prison facility, which DLS stated are the largest cost savings associated with avoided reimprisonment. Therefore, the state would be unable to realize cost savings through the project. Maryland did not pursue a PFS project to finance reentry programs after the report was released.

Governments, Service Providers, and Investors Pursued PFS for Potential Performance and Financial Benefits

Potential Benefits for Governments

Stakeholders in PFS projects stated that governments face significant challenges in implementing programs that effectively address entrenched social problems. They also stated that government can benefit from using PFS as a tool to address these challenges. The benefits of PFS include the following: Funding for prevention programs and potential cost savings. Government stakeholders said constrained budgets in recent years have made it difficult for governments to fund new prevention programs, even though they would potentially lead to cost savings resulting from a reduced demand for remediation services. Because investors provide capital that a government only has to repay if the specified outcomes are achieved, a PFS project helps governments implement programs that they might not otherwise be able to fund. In addition, if successful, a PFS project reduces the need for remediation services and associated budget pressures and, therefore, frees up additional resources that government can reinvest in priority programs. For example, an official from the Massachusetts Executive Office of Administration and Finance told us that any savings attributable to the Massachusetts Juvenile Justice PFS project will likely be reinvested in the same or similar programs. Further, the recipients of the prevention program are expected to benefit from improved outcomes, such as better education outcomes resulting from high-quality preschool.

The ability to implement innovative programs by transferring risk. PFS is designed so that the government only pays for a program if its outcomes are achieved, thus shifting the risk of failure to investors.

Stakeholders said that PFS creates incentives to both scale prevention programs where strong evidence exists and to innovate where it does not exist. For example, the Massachusetts project expanded a proven program to reduce recidivism rates. The service provider had extensive experience working with young men in Massachusetts at risk of recidivating and had successfully reduced their incarceration rates by 33 percent. The Massachusetts project expanded the service to more than 900 young men in Boston and Springfield, areas where the service provider had extensive experience. If successful, Massachusetts will pay for the program and expects to benefit from improved social outcomes and cost savings in its budget. If the service provider is unsuccessful, Massachusetts does not pay.

PFS also creates incentives to innovate and apply evidence-based approaches to new or different populations. In the New York City project the intervention was administered in a new environment, a jail, with a service provider with prior experience working in prisons, but without prior experience with the specific intervention. New York City and the project intermediary chose Moral Reconation Therapy for the intervention because of the available evidence base and its flexibility in implementation that could be adapted to a jail setting. The intermediary selected a provider to administer the service. The PFS structure allowed the project’s stakeholders to implement the intervention at Rikers Island for the first time while providing the city with assurance that investors will absorb financial loss if the intervention does not reduce participants’ recidivism. When the intervention did not reduce recidivism at the target rates, New York City did not pay for the program.

A potential benefit of PFS is the opportunity for governments to implement innovative programs without an established evidence base by transferring risk to investors. In practice, investors told us they prefer to back programs that already have a rigorous evidence base because these programs have a known likelihood for success. In Maryland, the DLS concluded that PFS may limit the implementation of innovative programs and government’s opportunity to apply evidence-based approaches to new populations because investors would be most likely to back projects with well-established records of success in order to minimize their risk. Therefore, PFS may exclude new service providers and program types because of the perceived risk.

One of the organizations involved in the PFS projects in our review did not think that risks would be transferred from a government to investors. For example, in its review, the Maryland DLS concluded that in practice there are significant challenges to transferring risk to investors, such as identifying investors who are willing to invest in PFS projects and designing contract mechanisms to fully transfer risk. This was a factor in the DLS recommendation against using a PFS contract for a prisoner reentry program. The Maryland DLS stated that it may be difficult to design contract provisions to provide an enforcement mechanism to prevent investors from terminating the contract early. In addition, during the implementation of the project, the costs to administer the intervention may be greater than the investors’ upfront funding and it is not clear whether the government or the investors would be responsible for those additional costs.

Collaboration across government agencies. Government stakeholders told us that many social services are “siloed” within governments. In other words, individual agencies, largely on their own, design, budget for, and implement narrowly targeted social services, such as job training courses or temporary accommodation for homeless individuals. These services are usually not integrated with others that could collectively help vulnerable individuals overcome complicated issues and are potentially not as effective as a more integrated approach. PFS projects target outcomes for populations or individuals rather than the output of specific services, which would be the responsibility of a specific agency. PFS projects bring together multiple government agencies to work collaboratively to find solutions for difficult social problems. For example, the New York project targets improved employment and recidivism outcomes for program participants and therefore brought together multiple state agencies to oversee their achievement. Officials from the New York Department of Labor and the Department of Corrections and Community Supervision are members of an executive steering committee that provides strategic direction and represents the project across agency boundaries. Similarly, a stakeholder in Massachusetts said that PFS can help break down silos because it can bring together multiple agencies to work together on an important social problem.

Sustained focus for entrenched social problems. Government stakeholders told us that governments find it challenging to sustain focus on difficult social problems over multiple years. Governments generally operate on annual budget cycles, making it difficult to set resource allocations over multiple years. In addition, changes in political leadership may shift policy priorities and support for levels of funding or program continuation. PFS contracts include upfront agreements about investor funding levels over multi-year time frames, providing sufficient time to collect, track, and analyze data, and measure outcomes that take multiple years to achieve.

A focus on outcomes and building an evidence base. Government stakeholders told us that governments sometimes are not focused on their programs’ outcomes because for social services they tend to contract for inputs instead of outcomes. As a result, governments have difficulty knowing whether a particular social program is having the desired effect on its target population. For PFS projects, governments contract for outcomes, and the contracts generally include a requirement that the program’s impact be rigorously evaluated, which allows governments to demonstrate the effect the program has as well as to add to the evidence base to inform future programs.

Potential Benefits for Service Providers

Stakeholders told us that PFS offers benefits to service providers, including the following: Stable funding for operations. Stakeholders told us that service providers must often pursue multiple grant and fundraising efforts to fund their services and enter into contracts in which they are not reimbursed for the full cost of services provided. PFS provides an opportunity for stable funding spanning multiple years. Officials from the service providers in New York and Massachusetts told us that upfront, multiyear funding for their programs is one of the key reasons why their organizations were excited about participating in a PFS project. An official from Roca in Massachusetts said that without the consistent funding provided under PFS, the organization typically has had to piece funding together from multiple grants to fund its programs.

Stakeholders also told us that PFS can be preferable for service providers compared to traditional performance-based contracts. In a traditional performance-based contract, the service provider is not paid all or a portion of its fees until it meets defined outcomes. In such a model, the service provider assumes financial risk that could damage it if it does not succeed, therefore potentially excluding new or small service providers. For example, UK government officials told us that in a performance-based contract, the service provider must cover the initial costs of delivering services, which is difficult for many nonprofit service providers because they do not have the capital available to provide services in advance of being paid. A benefit of PFS is that investors offer service providers full upfront funding, or cost reimbursement, for service provision while assuming most or all of the financial risk.

Flexibility in implementing service delivery. Service providers identified two ways PFS projects can give service providers flexibility compared to a traditional government contract or grant. In some of our cases, service providers were using PFS to administer their proven approaches on a larger scale. For example, officials from the service provider in New York told us that participation in the PFS project has allowed the organization to implement its service delivery model, for which the organization believes it has a solid evidence base, on a larger scale rather than conform to government-determined service delivery requirements. PFS can also offer flexibility to service providers in their day-to-day operations because the government contracts for outcomes, rather than how the service provider spends the funding on the activities it undertakes. Service providers said they have more flexibility to tailor their approaches to meet the needs of the target population.

Potential Benefits for Investors

Stakeholders told us that PFS also offers benefits to investors, including the following: Potential for return on investment. Stakeholders told us that investors benefit from potential financial returns. Investors conduct due diligence on projects and invest in projects where they believe there is potential for return on their investment. Investors and intermediaries noted that the level of potential returns for PFS projects has so far been below-market, compared to the level of risk.

Opportunity for social investment. Stakeholders said that certain investors are interested in making investments with positive social impacts. Officials from the bank that helped to facilitate investments in the New York project told us that the individuals who invested in the project are interested in aligning their investments with their values and creating positive financial and social impacts. Social investors, such as nonprofit philanthropic organizations in the United States and UK, said PFS is a tool to support social programs consistent with their missions. For example, an official from a philanthropic organization that invested in the Peterborough Project told us that the organization had previously provided grants to support services for short-sentenced prisoners and decided to invest in the project, in part, because it offered a sustainable solution to reducing recidivism among such prisoners. The organization reasoned that if the project turned out to be successful and could be shown to deliver cost savings to government, the UK government would be incentivized to directly fund services for short-sentenced prisoners in the future. In addition, social investors told us that because they will be repaid if a service provider meets outcome measures in a PFS project, they will have the opportunity to recycle the funds into other projects that align with their missions.

Participating Organizations Sought to Manage Risks in the Feasibility Assessment, Design, and Implementation Phases of Complex PFS Contracts

Governments, Service Providers, and Investors Recognize that PFS Projects Present Risk

PFS is predicated on the idea that governments shift the risk associated with developing and implementing a prevention program to investors. Stakeholders told us that the various parties to PFS contracts face a range of risks that are inherent to contracts:

Financial risk. The risk that one or more parties to the contract could lose money by participating in a PFS project. For example, a PFS project may not generate savings that are greater than its costs, which can include management fees for an intermediary, evaluation costs, and investors’ contracted outcome payments. Critics of PFS have argued that, given the extra costs associated with intermediaries and investors, using PFS to implement a program is inherently more expensive than providing the prevention program as a direct government service. Because interest rates are low for government borrowing, it could still cost less to borrow funds to pay for the service. However, some investors and intermediaries said they believe the complexity and associated costs of implementing PFS projects may decrease over time if the projects become more replicable. PFS proponents in general have argued that such direct government funding of new prevention programs has been difficult to do in the current budget environment.

At the same time, investors risk losing their investment if the service provider fails to achieve the desired outcomes. Officials from a bank that has invested in multiple U.S. projects told us that the organization carefully considers the evidence of the intervention that the service provider will deliver in order to determine whether it is willing to take the financial risk associated with an investment and how that risk should be priced.

In addition, organizations participating in PFS projects told us that governments, service providers, and investors have dedicated a significant amount of time and resources to developing PFS projects, at times without being compensated for their work. There is a risk that the parties are unable to agree on a project’s complex details and fail to launch the project, in which case participants lose the time and resources that they devoted to developing the project.

Stakeholders identified other risks that could potentially result in increased financial risk for parties to the PFS contract:

Perverse incentives. The risk that outcome payment structures will create perverse incentives for stakeholders to take actions at the expense of the individuals receiving treatment in the project. For example, creaming is the risk that service providers focus on individuals that are easiest to serve to achieve the defined outcomes. As a result of the perverse incentive to focus on those that are easiest to serve, those that are most in need and most difficult to serve could receive fewer services or a diminished focus. If creaming occurs, the government is potentially paying outcome payments when the target population is not being served.

Termination risk. The risk that one or more parties to the contract decide to terminate their participation in a PFS project, requiring the project to end earlier than initially planned. The early termination of a PFS project presents a risk that the target population may be disadvantaged or hurt by losing access to the intervention. Officials from an investor in multiple U.S. projects said that the organization understands that the early termination of a PFS project has the potential to negatively affect the programs’ target populations. Therefore, the organization worked with other parties during contract negotiations to create rules that would allow for the gradual termination of unsuccessful projects while taking into account the needs of current participants.

A variant of termination risk is policy risk, or the risk that a government initiates a policy change that prevents a PFS project from operating as initially intended. A policy change could disrupt a service provider’s program delivery, putting the achievement of outcomes— and the investors’ investment—at risk. For example, the UK Ministry of Justice terminated the Peterborough Project 2 years early. The UK government implemented a new initiative called Transforming Rehabilitation that provides a similar set of services for all short- sentenced prisoners in the UK. Officials from the UK Ministry of Justice said maintaining the PFS program at Peterborough until 2017 for the third and final cohort as planned was not possible, as the majority of prisoners within that group would already be receiving 12 months of supervision and rehabilitation as a result of the wider reforms to probation. According to officials from the intermediary, it would not have been possible to maintain a control group for the evaluation of the last cohort in the project. The Ministry of Justice and other project participants agreed to transition arrangements for the project that, among other things, established that investors could receive outcome payments if the project evaluation shows recidivism across the first two cohorts was reduced by 7.5 percent and that the Ministry of Justice would pay termination costs. Officials from the intermediary for the project said that investors are pleased that the needs of short-sentenced prisoners are being met under Transforming Rehabilitation. However, because the Peterborough Project ended early, information and learning about the effectiveness of the program was not fully realized.

Performance risk. The risk that an intervention fails to achieve defined outcomes. A government does not make payments if defined outcomes are not achieved. However, if outcomes are not achieved, then the government has lost time developing the program and failed to meet mission-related goals and objectives. Performance risk includes how the prevention program is operationalized, managed, and implemented. This risk can be heightened if the intervention is poorly structured or poorly tailored to the needs of a particular participant population, or if service delivery is poorly executed, which could lead to potential harm to program recipients. Investors also risk losing some or all of their investment. Stakeholders from the government and an investor in New York City told us that they considered the implications of potential performance risks in the project during its conception and development. For example, an official from the New York City government told us that the parties involved had extensive discussions, including about program outcomes and timelines. In addition, officials from an organization that invested in the project told us that they had to decide how likely, based on existing evidence, the program was to achieve the established outcome.

Appropriation risk. From the investors’ perspective, the risk that a government fails to appropriate funds necessary to make outcome payments. If a government does not appropriate necessary funds, investors could lose some or all of their investment, making them less likely to invest in PFS projects in the future. Investors’ perception of appropriations risk affects their decision on whether to participate in a PFS project and on how they view the risk of a project. For example, officials from a bank that has invested in multiple U.S. projects identified appropriations risk as one of the primary risks it sees for investors considering PFS projects. The officials told us that since investors assume substantial performance risk in the projects, they must have an assurance that, in the end, the government is going to make the outcome payments to which it initially committed. Stakeholders in our UK case illustrations did not view appropriations risk as a concern in the United Kingdom PFS market.

Reputational risk. The risk that a party’s reputation will be affected negatively by events that occur in a PFS project. Stakeholders in New York City told us that they are assuming reputational risk by participating in the Rikers Island project. For example, a former official from the New York City government told us that the city took a large reputational risk by launching the first PFS project in the United States, and the city is therefore very invested in its success. In addition, an official from one of the service providers working to implement the New York City project told us that the organization has assumed reputational risk by working as the service provider for the project. If the service provider fails to achieve outcomes for the project, it could make it challenging to get future contracts. Furthermore, an official from the organization that is evaluating the New York City project told us that since there are individuals and organizations within the criminal justice sector who are skeptical of PFS or believe that it is a flawed concept, the organization is potentially risking its reputation within the criminal justice sector by participating in the project.

During Project Feasibility Assessments, Governments Reported Seeking to Maximize Cost Savings While Managing Financial and Performance Risk

To assess project feasibility, governments and other stakeholders identified and evaluated potential projects in an effort to maximize cost savings and also to manage potential financial and performance risk. Stakeholders from the selected case studies stated that, in the first phase of PFS project development, they evaluated the feasibility of using PFS to address a particular social problem within their jurisdiction. We found that in practice PFS contracts were complex and governments relied extensively on outside technical assistance to assess the feasibility of their projects.

Attention to cost-benefit analysis. Governments estimate the potential effect the PFS project will have on future budgets to determine if the project is feasible. Using cost-benefit analysis and cost modeling, governments evaluate whether the prevention program is likely to generate cost savings, cost avoidance, or other benefits over the course of the project that exceed the project’s costs, which include the total amount of outcome payments and associated administrative costs. Based on this analysis, governments determine whether a project is feasible. For example, as noted earlier, the Maryland Department of Legislative Services determined that the state should not pursue a PFS project because even relatively large reductions in recidivism among program participants were unlikely to result in the closure of a wing of a prison facility, the type of large, fixed-cost savings that it believed would be needed to cover the program’s costs.

Governments’ cost-benefit analyses may also include indirect costs and social benefits. For example, New York assessed both the direct cost savings of reduced prison expenses to the government as well as quantifiable public sector benefits of savings from reduced crime resulting from the project. The New York project’s outcome payments are derived from these estimated public sector savings and benefits. According to a detailed summary of the project, the state estimated and included an amount for the cost savings from reduced crime for victims based on a study that estimated the intangible victim costs. To estimate these costs, the state used other costs such as medical costs and lost earnings for crime victims. The state also used a jury-compensation approach that used the money awarded to victims by juries to estimate the intangible victim costs of crime. Figure 10 shows the potential savings and benefits, as well as the outcome payment costs that New York identified for its PFS project. For example, a 10 percent reduction in recidivism would result in potential savings and benefits that equal outcome payment costs. As the rate of reduction in recidivism increases, the state’s potential savings and benefits (after making outcome payments) increase.

Identification of entities that capture financial benefits and budget savings. Stakeholders said that in order to maximize cost savings to the entity who is sponsoring the PFS project, governments must know which government budget will capture and benefit from the savings. Stakeholders said that a barrier to potential PFS projects occurs when expected savings accrue to a different government entity than the one that is leading the project. This is referred to as the “wrong pockets problem.” If the savings resulting from a successful intervention by one government entity accrue to a different government entity, without agreement from the other government entity to contribute to outcome payments in proportion to the savings, the project may not be cost- beneficial to implement. For example, stakeholders in South Carolina have been developing a PFS project to fund a program designed to improve maternal health outcomes. Since potential savings generated by the program would accrue to Medicaid, those savings would be split between the state and the federal government. Stakeholders told us that the state would find it challenging to agree to make all of the project’s potential outcome payments since it would only be capturing approximately 30 percent of the project’s savings, with the federal government capturing the rest. A stakeholder in New York City said the city chose to pursue the Rikers Island project specifically because the city bears all of its own criminal justice costs and therefore would not have to involve the state or federal government in developing the project or paying for the outcomes achieved.

Access to, and capacity for, robust data collection and analytics. To determine project feasibility, governments must have access to complete and accurate cost and benefit data and must have the capacity to analyze it. However, stakeholders said that the availability of government data on social programs and the capacity to analyze the data is a challenge to the growth of PFS. Stakeholders said that governments need data systems in place that allow agencies to collect data and share it between agencies, as needed. However, in some cases, the necessary data can be difficult to obtain or share. For example, government stakeholders in Massachusetts told us that differences in the ways in which jurisdictions within the state maintained files on individuals on probation made it difficult to collect and share data on potential program participants at the beginning of the project’s implementation. Stakeholders said legal barriers can prevent governments from sharing data across multiple systems. For example, we previously reported that matching education and workforce data is challenging in states where collecting a Social Security number in education data is prohibited by state law or agency policy.

Stakeholders also expressed concerns that many governments do not have sufficient data collection and analysis capacity for PFS projects. Similarly, we have previously reported that it is critical to define the relevant types of information required for service acquisitions and then develop the appropriate data systems to collect and provide reliable data. Governments will need staff with expertise in data analysis and financial modeling to be able to conduct feasibility studies, identify future savings, evaluate service delivery, determine outcome payments, and negotiate PFS contracts. Government officials from both New York and New York City said that their government’s robust data systems facilitated their cost-benefit analyses. For example, according to a lessons learned report from the intermediary in the New York City Rikers Island project, the city’s strong financial planning structure allowed the city to gain a solid understanding of the amount of money that it could save based on changes in the size of its jail population. The city used this information to establish its willingness to pay for reductions in recidivism attributable to the intervention being funded by the PFS project. The cost information, together with historical criminal justice data, allowed the city to identify a specific reduction in recidivism that would make the PFS project beneficial for the city. To address challenges with government capacity, all of the U.S. PFS projects in our study included provisions for technical assistance from experts affiliated with universities or intermediary organizations to analyze project data and structure financial models needed to determine project feasibility and outcome payments.

Sufficient evidence of an intervention’s success. In addition to cost- benefit analysis and access to data, stakeholders told us that during the feasibility phase it is important to consider the evidence of an intervention’s past success to help manage the performance risk of the project. While some stakeholders stressed the importance of sufficient evidence of an intervention’s success, others noted that there is variation in the PFS field in the level of evidence and how it relates to the individual project. For example, evidence supporting an intervention applied to one population could provide the basis for testing the intervention with a different population. A PFS project could also be used to implement an innovative intervention with limited but promising evidence. Participating organizations—including governments, investors, and service providers— may differ in their tolerance of risk and uncertainty associated with the evidence of any intervention. Stakeholders from government, intermediaries, and investors said they also assessed the availability of service providers who could implement the intervention to determine whether the project would be feasible. Officials from an investor in multiple U.S. projects told us that the bank has looked to invest in PFS projects that feature a service provider that has a track record of success.

Stakeholders said the availability of evidence-based social programs and service providers with the capacity to implement them is a challenge to growing the PFS field. Stakeholders expressed concerns that there is not a sufficient pipeline of nonprofit service providers whose interventions have been rigorously evaluated. We have previously reported on challenges nonprofits face related to funding indirect costs, such as program evaluation. In addition, stakeholders told us that in order to be successful in an established PFS project, service providers need experience implementing the evidence-based intervention, the capacity to implement the intervention on a larger scale, and the ability to collect and report data for evaluation. The enacting legislation for the Utah PFS Project includes a grant program to build the capacity of preschool providers to implement the high-quality preschool model, which is based on a prior evaluation, so these providers are able to participate in potential future PFS projects.

Leadership support. Stakeholders said that support from top government leaders is essential to moving a PFS project forward to the design and implementation phases of the project. This support can provide the momentum that is necessary to get a PFS project off the ground and sustain the project through implementation. State officials in New York told us that PFS was part of Governor Andrew Cuomo’s policy agenda during his gubernatorial campaign, as it aligned with his interest in taxpayer efficiency and a focus on outcomes. In addition, a report on the project stated that it was conceived when Governor Cuomo outlined a state priority to use PFS to reduce recidivism and increase employment among high-risk ex-prisoners. Furthermore, stakeholders in Utah told us that the long-term prospects of the project were boosted by the Board of the United Way of Salt Lake and the Mayor of Salt Lake County’s agreement, during the project’s proof of concept year, to dedicate funds to support the project’s outcome payments. Stakeholders also said that leadership from a central government agency can help maximize collaboration across government agencies. They said that it is important that a central agency coordinates across multiple departments in a situation where an intervention in one department will accrue future savings in a different department or where participation from multiple departments is needed to implement the project. We have previously reported that top leadership commitment can improve outcomes and set the direction, pace, and tone for implementing a significant policy change. Stakeholders also said it is critical to have a dedicated government official to manage the development of a PFS project. For example, the governments in Massachusetts, New York, and Utah had a full-time position in their budget offices to manage their PFS efforts.

During Contract Design, Parties Reported Defining Requirements to Manage Risk

Stakeholders said contract negotiations for PFS projects are complex, lengthy, and resource intensive. Stakeholders from government, service providers, intermediaries, and investors told us that they required significant legal and technical capacity to support the development of these complex contracts because PFS contracts were new and existing templates were not available to use as a guide. While the same types of parties were present, the PFS projects in our study had different contractual arrangements. In some cases, the PFS contract was between the government and the intermediary, which held additional contracts with investors and service providers. In other cases, the contract was between the government, intermediary, and service provider. The intermediary holds separate agreements with the investors in the project. While the contractual arrangements varied, stakeholders said that all parties were involved with negotiating the terms of the contracts.

Defining Requirements in Outcome Payment Structures

Parties to the PFS contracts in our study defined the requirements for the service provider, in part, by using performance targets for specific outcomes that were rewarded with outcome payments. These performance targets and associated payouts make up an outcome payment structure. See table 2 for examples of four different types of outcome payment structures we observed in the PFS projects we reviewed. For example, in the Utah High Quality Preschool project, the outcome payment structure was defined as a set payment for each student participant who avoided being enrolled in special education each year from kindergarten through sixth grade. Stakeholders from governments and service providers said that the outcome payment structure aligned the incentives of all parties to be invested in the service provider’s successful delivery of the prevention program. Further, as suggested in practices for complex service acquisitions, because the parties to the contract agree in advance to outcome measures and associated payments, there is less room for interpretation once the contract is being implemented.

We have previously reported that a balanced set of performance goals and measures can help organizations address varied aspects of program performance. In some of the PFS cases in our study, the parties included a range of performance measures in the outcome payment structure to more effectively serve the target population. For example, the New York project included performance measures on both reducing recidivism and increasing employment for individuals who were recently released from prison. According to a detailed summary of the project, formerly incarcerated individuals face numerous challenges, including barriers to finding a job upon release. Stable employment can be an important factor in success upon release from prison.

The outcome payment schedules also create financial incentives for investors and service providers to improve the outcomes for the target population. Stakeholders told us that multiple outcome measures can be a tool to incentivize service providers to address various participant needs. For example, the London Homelessness Project includes outcome measures to target the different causes of homelessness of individuals in the cohort. One of the five measures is related to supporting individuals’ progress toward employment, and another outcome measure is related to moving individuals into stable housing. Outcome payment structures with higher payouts at the end of the project create incentives for investors and service providers not to terminate the contract early because they will receive the majority of their return on investment at the end of the contract period.

Challenge of Securing Investors

Stakeholders said that the rates of return offered by PFS projects likely will not meet market returns available on traditional investments with similar risk profiles, therefore reducing investor demand. Lower returns compared to traditional investments could make it more difficult for governments to attract sufficient capital for PFS projects. Investors assess the risks of a project to determine whether or not they will invest. Investors’ valuation of risk is primarily based on several factors related to managing their financial risk, including evidence of the intervention and the ability of the service provider to implement it, the outcome payment structure, and the level of appropriations risk. For example, commercial investors prefer interventions backed by rigorous evidence and service providers who have a history of strong performance because it lowers their financial risk. Governments also manage their financial risk by determining the outcome payments and rate of return they are willing to pay to investors. In Utah, the legislation that authorizes the High Quality Preschool Project included a cap on the return for investors. A Utah legislator said the cap helps to ensure that the state captures more of the cost avoidance from reduced use of special education services. The legislature wanted to ensure they were being good government stewards because they would not be at the table negotiating PFS transactions. In other projects, the return on investment was negotiated on a case-by- case basis. For example, in the New York project, there is a cap on the total outcome payments the state can make over the life of the project. In the UK, government officials told us that some PFS project outcome payments and associated rates of return funded by the Department for Work and Pensions’ (DWP) Innovation Fund were not subject to negotiation. Instead, DWP developed a rate card, where it defined the maximum it is willing to pay for specific outcomes (see appendix IV for an example of a rate card).

In the U.S. projects in our study, parties to the contracts used financing structures to address challenges in attracting investors, such as a tiered capital structure or a loan guarantee to secure the upfront capital needed for the project.

Tiered capital structure. A tiered capital structure, where some investors take on more risk, can attract bigger pools of capital because it lowers risk for some investors. In three of our U.S. case illustrations a combination of commercial and philanthropic investors made up the capital structure. Stakeholders said that investors’ willingness to accept risk can vary and that philanthropic investors may be willing to accept more risk at a lower rate of return than commercial investors. For example, according to the intermediary in the Massachusetts Juvenile Justice project, to encourage investment the project capital structure was split between philanthropic and commercial investors. Commercial funding partners contributed senior debt, which will be paid out first; foundations contributed junior debt, which will be paid out following the senior debt. Philanthropic grants from foundations also served as first- loss capital, which will bear the first loss in the event that the service provider does not meet the outcome measures. According to the intermediary in the project, in the first-loss role philanthropic donors can combine and leverage their grant making with commercial capital.

Loan guarantee. In a different model, a philanthropic investor can provide a loan guarantee to the primary investor to reduce its risk by assuming the loan if the service provider does not meet the outcome targets. For example, in the New York City Rikers Island Project, the city agreed to a maximum return of $2.1 million which was not high enough for Goldman Sachs to justify the level of risk it perceived regarding repayment of its $9.6 million loan. In an effort to balance the risk/reward trade-off, Bloomberg Philanthropies provided a $7.2 million grant to MDRC to act as a loan guarantee that would reduce Goldman Sachs’ capital at risk in the event that outcome targets were not achieved. An investor in the project further explained that although the service provider had a track record of success, its risk of failure in the NYC Rikers Island project was higher because it would be implementing a new curriculum in a difficult environment. The combination of the performance risk and its status as the first PFS project in the United States were the primary reasons for the loan guarantee.

During Project Design, Parties Reported Using PFS Contracts to Establish Governance Structures and Rules for Project Implementation to Manage Risk

Governance rules that define interactions and the authority to make decisions after the contract has been signed can encourage cooperation and manage the risk of unexpected termination by any party. For the projects in our study, the contracts also defined how parties will interact during the implementation of the project. PFS contracts in our study included outcome payment structures, oversight structures, governance rules, and project evaluation mechanisms to manage risk.

Oversight structures. PFS contracts included oversight structures to manage performance risk during contract implementation. All of the projects in our study had a governance structure, which defined how parties would make decisions during implementation of the contract, including how the project would be managed and overseen. For example, the New York project had a three-tier governance structure made up of the following:

An executive steering committee: This committee meets semi- annually to provide strategic direction, monitor compliance with the agreement, and review reports. Members include the Deputy Secretary for Civil Rights; the Deputy Secretary for Public Safety; the Commissioner of Department of Labor (DOL); the Commissioner of the Department of Corrections and Community Supervision (DOCCS); CEO of the intermediary; and the Executive Director of the service provider.

A management committee: The committee meets monthly to monitor reports and program implementation and provides updates to the executive steering committee. Members include the Associate Commissioner of DOCCS; two Deputy Commissioners of DOCCS; the Deputy Commissioner of DOL; the Executive Director of the service provider’s regional office; the Director of DOCCS Research; a representative from the Harvard Kennedy School Social Impact Bond Technical Assistance Lab; and a director from the intermediary.

Working groups: Working groups meet to discuss operational and research issues that arise over the course of the project. They are composed of staff and subject-matter experts from DOCCS; DOCCS Research; DOL; the service provider; the Harvard Kennedy School Social Impact Bond Technical Assistance Lab; and the intermediary.

Contract governance rules. Complex contracting practices suggest that including provisions for how decisions will be made when changes are needed during service delivery can help address implementation challenges. To manage various risks and inherent uncertainties, PFS contracts included governance rules to define how decisions would be made over the course of the project. PFS contracts included rules to define decision making after the contract was in place as well as provisions to address how parties to the contract could be replaced in the case of underperformance. For example, the New York contract included governance rules that defined when and how the service provider could be replaced. The contract specified that any replacement service provider is subject to approval from the New York Department of Labor and Department of Corrections and Community Supervision, among others.

PFS contracts also defined the circumstances under which investors could decide to terminate the project based on the service provider’s performance. A Massachusetts official said that determining the provisions that would allow investors to terminate the contract was one of the most difficult parts of the negotiations. Because the PFS project was designed for the service provider’s model, Massachusetts wanted to support the model, which requires a number of years of participation to achieve outcomes. Massachusetts also wanted to ensure that program participants would not be left without services as a result of the program’s early termination. Investors did not want to continue their investment if the service provider was underperforming. In the contract, the parties agreed that the lead investor has a right to exit the project at the end of the second and third years if the service provider’s participant attrition rate exceeded defined thresholds. The contract also included provisions, such as the development of a remediation plan, designed to improve the service provider’s performance if it fell below a defined threshold. These provisions gave investors the opportunity to reduce their financial risk if the service provider was underperforming. Allowing for 2 years of implementation and outcome measurement before investors can exit the contract also helped mitigate the government’s performance risk because it allowed enough time for service providers to collect data to measure outcomes and make adjustments if necessary to improve performance. Termination provisions also defined what would happen if governments terminated the contract early. For example, in the NYC Rikers Island Project, the City of New York could terminate without cause. However, if the city terminated the contract before the entire evaluation cohort receives services, it would lose the pay-for-success features of the project and must reimburse the intermediary for program costs to date.

Stakeholders said that building flexibility into contracts to allow for negotiations to make operational changes without amending the contract could help address implementation challenges. Some changes to the projects required updates to the PFS contract, which were costly and time intensive because all parties to the contract had to negotiate and agree to the changes. Stakeholders said that it is important to find a balance between specifically defining requirements and building sufficient flexibility into the contract to address challenges during program implementation. A Massachusetts official said the contract must be detailed enough to ensure that the program participants are protected and that risk is properly transferred from the government to other parties. However, the contract’s terms must also offer the parties a degree of flexibility to make changes as problems arise, because it is very time consuming and difficult to amend the contract.

Government budgeting structure. Because appropriations risk was a key concern of investors, the U.S. cases in our study managed this risk by establishing different mechanisms to secure funds for outcome payments:

Massachusetts enacted legislation authorizing the backing of PFS contract payments with the full faith and credit of the commonwealth and establishing a Social Innovation Financing Trust Fund into which annual appropriations may be made.

New York City, if the program had been successful, would have made outcome payments as it would for any other contractual obligation. There were two evaluation points that would trigger a possible payment. The final payment was scheduled to occur 1 year after the final results, giving the city more time to realize savings.

New York pledged to obtain appropriations to make outcome payments. The contract includes provisions for an orderly wind down if the state does not renew the appropriation.

Utah enacted legislation that authorizes a School Readiness Restricted Account which consists of funds appropriated by the legislature, federal grant funds, and private donations. Funds in this account may be used for contracts with independent evaluators, future outcome payments, and grant awards.

Project evaluation mechanisms. The PFS case studies we examined used evaluations to manage the governments’ various risks and to ensure the defined outcomes are achieved. In each case, outcome payments were contingent upon the evaluator certifying that the agreed-upon outcomes were met. Four of the evaluations in our case illustrations were impact evaluations, such as a random control trial (RCT) or a quasi- experimental design, where outcomes for individuals receiving the intervention were compared to those from a similar control group. Three of the evaluations used a process to verify that outcomes were being achieved for the target population. For example, in the Utah project’s evaluation, the evaluator tests each student participating in the high- quality preschool program at the beginning and end of the school year. The students who score two standard deviations below the mean on the pre-test are tracked from kindergarten through sixth grade to verify whether they are enrolled in special education. The evaluation was derived from an earlier analysis that showed a decrease in special education use among students attending the high-quality preschool program. In the London Homelessness project, service providers submit evidence for some of the measures to the Greater London Authority, which is responsible for verifying the attainment of those measures. For example, for the outcome measure on helping clients into stable housing, providers submit a form from an individual’s landlord, which the Greater London Authority verifies and determines whether to make the associated outcome payment.

Stakeholders said that evaluations can manage potential perverse incentives, such as creaming. Some projects in our study included a mechanism to address creaming in the evaluation. For example, the Utah project on early childhood education addressed the risk of creaming by only including the children who score two standard deviations below the mean on the pre-test in the evaluation for outcome payments.

Stakeholders said that evaluations can also help manage perverse incentives for stakeholders to take actions at the expense of the individuals receiving treatment in the project. For example, because outcome payments in the Utah project are made based on students avoiding special education, there is potentially an incentive to withhold special education services to increase outcome payments. The Utah project manages this risk by keeping the identity of the students in the evaluation cohort confidential, so the teachers who determine special education needs are not aware of a student’s status in the project. In another example, a report on the London Homelessness Project stated that well-defined target populations can reduce the risk of creaming because the service provider cannot manipulate the population. This project targets 830 entrenched homeless individuals who had been seen sleeping rough or who have stayed in a London rough sleeping hostel in the last 3 months, and who had been rough sleeping at least six times over the last 2 years.

Stakeholders said that there are challenges to implementing RCT evaluations for PFS projects. We have previously reported that well- conducted randomized experiments are best suited for assessing effectiveness when multiple causal influences create uncertainty about what caused results. Such experiments are often expensive because they are among the most rigorous forms of evaluation, as well as difficult or sometimes impossible to carry out. For example, a risk of RCTs is that the evaluation may be inconclusive if the control group cannot be maintained and because a large sample size is required to yield statistically significant results. One stakeholder said that a RCT can increase the reputational risk to service providers if it is poorly designed and misses the real outcomes of the program. To manage the risk of these challenges, stakeholders from several projects identified the need for an alternative methodology in case the evaluation could not be implemented due to unanticipated implementation challenges. For example, in the NYC Rikers Island Project, the initial evaluation plan was based on a random control trial, but in the jail environment it was not possible to keep the control and treatment groups separate. Therefore, project stakeholders designed an alternative methodology, a quasi- experimental design, which provided a sufficient degree of confidence that the outcomes achieved could be attributed to the program.

Stakeholders also addressed potential ethical concerns associated with RCTs, such as withholding the service intervention from a population that could potentially benefit from the services in order to maintain a control group for an impact evaluation. While there could be ethical concerns associated with using a control group for a random control trial evaluation of a service, limited availability of funding often prevents treating the entire population. Therefore, stakeholders indicated they could possibly draw from patients on a wait list for the program to serve as a control group. The Massachusetts project addressed ethical concerns with RCTs by including a “no denial of service policy” if an individual in the nonparticipating control group asked to participate in the program. This design required a significant increase in the evaluation’s sample size to address potential contamination between the treatment group and the control group in the sample.

Ongoing PFS Contracts Show Benefits of Data- Driven Performance Management, but Final Outcomes Are Not Yet Known

During the third phase of a PFS project, service providers implement the prevention program. To manage performance risk, participating organizations provide oversight, monitor performance, and evaluate the extent to which outcomes are achieved, based on the structures and governance roles established in PFS contracts.

Implement Prevention Program Using Data-Driven Performance Management

Actively managing performance to achieve outcomes during contract implementation is vital to achieving results. While the structures and membership varied, stakeholders said PFS oversight bodies regularly reviewed performance data during service delivery. According to stakeholders, intermediaries and investors can bring performance management expertise to service providers and provide a rigorous focus on performance management and accountability. Officials from the intermediary for the New York City project said they continued to monitor the implementation of the service delivery using metrics that predict whether the project will be successful. They said that these metrics and targets allowed them to troubleshoot project implementation and intervene early to modify elements of the program that were not working. For example, there is a community service element to the Moral Reconation Therapy program that was modified to work in the Rikers Island jail environment. In another example, officials from the intermediary for the Peterborough Project said they used data analytics and provided program management support to help improve service delivery. In 2014 the UK Department of Work and Pensions evaluated its Innovation Fund PFS projects and found that for its first 10 PFS projects the strong focus on monitoring and managing project performance built capacity among service providers and helped to increase performance levels and improve efficiency of service delivery.

Social service providers often collect and report input and output data in response to multiple grant requirements, but these data may not always be useful for monitoring the performance of a PFS project in achieving outcomes. Service providers said the PFS model may require new methods to collect and use performance information that shows the program is being implemented as planned and with fidelity to the model that was shown to be successful. Some service providers may also need to invest in building their capacity to collect and report outcome data to participate in PFS projects. An official from one service provider noted that her organization invested in data entry and data analyst positions and has a team that collects, analyzes, and processes data that it submits to the intermediary.

Projects in our study faced unanticipated challenges in delivering services and stakeholders said oversight mechanisms helped to identify issues quickly and facilitated decision making to address problems. For example, stakeholders in our study reported difficulty in identifying cases for referrals to the service provider. Maintaining the planned level of referrals was important because the evaluation and outcome payments were dependent on the level of participation in the project. An official from the service provider in the New York project said that approximately 3 to 4 months into program implementation, there was a dip in the number of potential participants referred to the program. He said the cause was that the service provider was seeking referrals from parole bureaus where it had traditionally drawn few participants in order to maintain the project’s randomized referral and evaluation plan. However, he said this strategy depressed the overall number of referrals. As a result, they decided to expand the number of parole bureaus that could serve as a source of referrals.

Stakeholders said that pilot periods for service delivery before the evaluation starts could reduce unanticipated challenges during PFS implementation and reduce risks that service providers will not meet outcome targets. According to a report on the New York City project, it may be beneficial to pilot-test a program at full scale and only evaluate it after it achieves a steady state of operation. On the other hand, pilot programs require time and money, both of which are likely to be constrained for PFS projects. The New York City project included a pilot test conducted at half scale. Stakeholders said that although the experience was valuable, certain issues came to light only after the program expanded to full scale. A pilot project can also be used to build evidence on an intervention to develop a future PFS project. For example, the Asthma Impact Model for Fresno is implementing a pilot project from spring 2013 through early 2016. The pilot project includes a random control trial evaluation on the intervention to demonstrate the outcomes of a home-based asthma management program. If successful, the results of the pilot will be used to develop a PFS strategy.

Evaluate Outcomes and Next Steps

Following program implementation, the parties to a PFS project must evaluate whether the service provider achieved outcome measures and determine whether the government must make outcome payments based on the results. PFS contracts we examined spanned 3 to 10 years and one project—New York City—had been completed as of the time of our review. Results for the New York City project were reported in July 2015 and indicated that the program did not reduce recidivism among adolescents at Rikers Island and therefore did not meet the predetermined threshold of success: a 10 percent reduction in recidivism. As a result, New York City did not pay for the program and the intervention was discontinued as of August 31, 2015. Results of the evaluation of the initial cohort of the Peterborough Project were reported in August 2014 and indicated that reoffending rates for participants were reduced by 8.39 percent compared to a control group. A reduction of at least 10 percent within the cohort was necessary to trigger an outcome payment to investors, so no payment was made. However, the project’s terms state that outcome payments will be made if reoffending rates are reduced for each individual cohort by 10 percent or across all cohorts by 7.5 percent. Therefore, if the evaluation to be conducted following the second cohort finds that reoffending rates across both cohorts were reduced by at least 7.5 percent, the Ministry of Justice will make outcome payments to investors.

We asked participating organizations what will happen to their programs at the end of their PFS contracts. Some told us that governments may consider funding the service delivery directly. In another case, the It’s All About Me adoption project has been designed so that if local governments use the service in large numbers over 10 years, the project will amass sufficient reserves after repaying investors to self-finance the service in the future. While PFS financial models were designed based on future government savings, some government stakeholders indicated they were not planning to reduce future agency budgets if PFS outcomes are achieved, but will reallocate the savings to address other unmet needs in the same agency budget.

The Federal Government Could Play a Role in Funding, Building Capacity for, and Managing Investor Risk in PFS, but a Formal Mechanism to Collaborate and Share Lessons Learned Does Not Exist

The Federal Government Could Play Several Roles to Address PFS Challenges at the State and Local Level

While the effectiveness of the PFS model remains to be seen, stakeholders cited three potential roles the federal government could play to help address challenges at the state and local levels of government and further develop the PFS field: providing outcome payments, building capacity, and providing loan guarantees.

Providing outcome payments. State and federal officials and other stakeholders said the federal government could provide outcome payments in PFS projects to address the wrong pockets problem, which could arise when an intervention results in savings to the federal government as well as the government commissioning the project. This issue makes it challenging for state and local governments to have sufficient funds available to make outcome payments. In our case studies, some officials said that a key consideration when assessing project feasibility was whether cost savings to their jurisdiction were sufficient to justify the cost of implementing the project when other jurisdictions, such as the federal government, shared in the costs and savings as well. To date, the federal government has not become a formal party to any PFS contract in which the federal government would share in the project’s savings. However, the federal government has provided grants to support outcome payments for two of the case studies we examined. To assist New York and Massachusetts with their recidivism PFS projects, DOL awarded nearly $24 million in grant funding to support outcome payments for these projects. A stakeholder said that the DOL grant catalyzed the New York project, allowing them to fund outcome payments for the first cohort of 1,000 participants. Having support for outcome payments helped Massachusetts officials increase the number of participants in the project, thus allowing for a stronger evaluation of the program’s outcomes.

The UK government also took steps to mitigate challenges related to the wrong pockets issue. For example, the £20 million (approximately $31 million) Social Outcomes Fund, managed by the Cabinet Office, was created to address challenges related to public sector spending silos that occur between agencies in central and local governments trying to address complex social issues. The Social Outcomes Fund provides outcome payments for government departments, local authorities, and other organizations to implement innovative projects to address social problems where savings could be spread among several government agencies. This approach addresses the difficulty faced by a single office attempting to justify the cost of implementing the program. The federal government could also commission PFS projects and provide outcome payments for programs where it would capture all of the savings, such as programs targeting Social Security Disability Insurance.

Similar to state and local governments, as an outcome payor the federal government could face many of the same potential benefits, risks, and decisions regarding project feasibility identified by stakeholders in our study. Playing the role of outcome payor in PFS projects could provide benefits from potential future savings to the federal government if outcomes are achieved. For example, a stakeholder said a potential area where the federal government could play a role as outcome payor is in the provision of nonmedical asthma management services including in- home education, environmental assessment, and remediation such as mold removal, which could potentially reduce the number of asthma attacks and emergency visits by Medicaid recipients. Reducing emergency room visits paid by the Medicaid program could yield savings to both the state and federal governments. However, as an outcome payor, federal agencies would need to manage financial and performance risks similar to state and local governments. Federal agencies considering entering a PFS contract as an outcome payor would need to assess the feasibility of individual projects, including the factors stakeholders in our study identified, such as evidence on the intervention, cost-benefit analysis of the project, and the commitment of government leadership.

Another potential challenge for federal agencies to consider is whether their existing grant program authority can accommodate outcome payments. PFS projects can span 3 to 10 years, while grant program authorities may limit the duration of a grant project. For example, DOJ officials told us that the reentry grant program authorities under the Second Chance Act limited the grant performance period to 12 months. This restriction would have prohibited the use of a PFS project as no outcomes could be achieved in such a short project period. However, this time limit was later eliminated for grants funded by the appropriation made available for DOJ’s PFS efforts for fiscal years 2014 and 2015. Another issue for agencies to consider is the period of availability of the funds and what happens to funds obligated for outcome payments if they are not made because the service provider does not meet the outcome targets. DOL officials told us that the grant funds obligated for outcome payments will be deobligated and unavailable for new obligations if outcome targets are not met in the New York or Massachusetts PFS projects. In another example, CNCS’s fiscal years 2014 and 2015 appropriations include authority that allows CNCS to reobligate funds to other authorized projects if outcome targets are not met.

Building capacity for PFS projects. According to stakeholders, the federal government could make grants for feasibility studies and project development to build the capacity of state and local governments, intermediaries, and service providers, and offset administrative costs. In fiscal year 2014, CNCS awarded $12 million in grants to eight intermediaries to help governments and nonprofits build their capacity to implement and structure PFS projects. As of July 2015, the initial eight intermediaries had made awards to a total of 48 state and local governments and nonprofit organizations to fund technical assistance on PFS project development. According to CNCS, the grants address the limited availability of funding for planning, feasibility studies, and deal structuring, which has constrained growth in the field. In another example, DOJ and HUD are jointly pursuing a PFS demonstration project on permanent supportive housing for the prisoner reentry population. The project will make grants to intermediary organizations that will work with state and local governments and service providers to implement PFS projects and will include an evaluation to determine if this model of delivering supportive housing can achieve savings by decreasing government spending on corrections systems, homeless services, Medicaid, and crisis care services, some of which are federally funded.

The UK central government addressed capacity issues by creating organizations to manage grant making and technical assistance for organizations involved in PFS projects. Through its Commissioning Better Outcomes Fund, the Big Lottery Fund provides up to £6 million (approximately $9 million) in grants (up to a maximum of £150,000 (approximately $229,000) per project) for technical assistance to assess the feasibility of PFS projects. It also provides outcome payments to encourage and sustain government engagement. The Better Outcomes Fund seeks to increase the capacity of service providers, enhance understanding of how to organize and implement a successful PFS project, and increase the knowledge base of local government officials working with PFS projects in the field. A stakeholder from the Centre for Social Impact Bonds said these grants have helped improve the technical and financial capacity of key players in PFS projects.

State and federal officials also said that the federal government is uniquely positioned to help build capacity by encouraging knowledge sharing among parties that are implementing PFS projects. For example, to share knowledge with its grantees, DOL leads a working group for the state and local government recipients of PFS grant awards in an effort to increase knowledge and share best practices. DOL officials said that the monthly meetings have included speakers with PFS expertise, such as officials from Social Finance, the Peterborough Project, and other federal agencies. In addition, the White House Office of Social Innovation and Civic Participation co-hosted, with the Arnold Foundation and the Nonprofit Finance Fund, a series of PFS Summits in late 2014 and early 2015 to highlight and advance regional activity underway across the country, build the longer term Pay for Success infrastructure to catalyze future projects, and facilitate an open, transparent dialogue among public, private, and philanthropic stakeholders. Officials from one state government said that knowledge sharing by the federal agencies has been helpful to the state as it develops and implements PFS projects.

UK government officials also said that knowledge sharing was an important component of the growth of PFS. As part of the annual reporting requirements for PFS grants, the Big Lottery Fund requires that grantees inform officials of plans to communicate information on their learning and experience as the project progresses. Where the Big Lottery Fund supports outcome payments, projects are also required to evaluate their own performance, contribute to an independent program evaluation, and support dissemination of learning from these evaluations. In the UK, officials from the Centre for Social Impact Bonds said the Centre has also addressed knowledge sharing issues by coordinating meetings with other government organizations, holding workshops, networking, and making connections between interested parties.

Providing loan guarantees. A challenge to PFS projects that could limit growth is obtaining capital from investors. Investors said that the federal government could provide loan guarantees or credit enhancements for PFS projects to increase the availability of capital by reducing risk for investors. Stakeholders said that having the backing of the federal government would provide greater incentives for investment in PFS projects and help establish the market. For example, Bloomberg Philanthropies provided a $7.2 million loan guarantee for the New York City Riker’s Island project to reduce some of the risk to investors of losing their investment if outcome targets were not achieved. However, some stakeholders said a government loan guarantee would negate a primary benefit of PFS to transfer the risk of program implementation away from the government. Such guarantees provided by the federal government would potentially leave the federal government liable for paying for a project that did not achieve agreed-upon outcomes.

There Is No Formal Mechanism for Federal Agencies to Collaborate on PFS

In our prior work, we stated that collaborative mechanisms, such as interagency groups or collaboration technology, can be used to develop policies, implement programs, and share information. The agencies in our study participated in an interagency PFS working group that met between 2012 and 2014 to share information on agency efforts related to PFS. Federal officials said the working group is no longer meeting because of staff turnover in leadership positions. Officials who participated in the working group said it was beneficial and furthered their agencies’ efforts related to PFS. For example, a federal agency official said the PFS working group helped officials network with their peers and collaborate on specific efforts. When CNCS developed its fiscal year 2014 PFS grant notice of funding availability, it consulted with other federal agency officials with PFS expertise. Though officials found these meetings useful, they were not maintained with a regular schedule and information resulting from meetings was not documented or broadly disseminated. In May 2015, OMB officials noted that the White House and federal agency officials are working to restart interagency meetings on PFS.

A mechanism for agencies to share knowledge and information on PFS could give agencies interested in potential PFS projects access to leading practices and current information on how federal participation is evolving.

In addition, a collaborative mechanism would allow agencies to share information on more technical aspects of PFS, such as practices on how to evaluate savings that accrue to federal agencies as the result of successful PFS efforts. Officials from one federal agency said that information about the various roles that federal agencies have played in the PFS field would be informative, given that the approach to supporting PFS has varied by agency. Having a central mechanism for sharing information on PFS would also decrease the risk of duplicative efforts in projects that affect multiple agencies.

Conclusions

The PFS model is evolving as one among many tools for policymakers to consider when seeking to provide solutions for some of the nation’s most vexing social problems. According to stakeholders, PFS allows governments to shift the focus of social service interventions from costly remediation to lower cost prevention, while simultaneously mitigating risks associated with scale-up by funding projects with private investment. For service providers, the PFS model provides a source of upfront capital for operations, as well as an opportunity to scale-up promising programs and refine their capacity for sophisticated data collection and analysis.

As organizations consider pursuing PFS contracts, they will need to think about how they will manage the risks inherent in PFS contracting. For example, a government’s costs of implementing a PFS project, which can include investor returns, management fees, and evaluation costs, can be high, so the government must decide whether potential benefits outweigh these costs. Flexibilities built into PFS contracts can help all parties to the project manage risks by allowing them to make changes as implementation and evaluation methodologies evolve. In addition, the parties to a project will need to manage the potential risks that unintended consequences will negatively affect the population receiving treatment. Managing these risks requires rigorous design of project implementation and evaluation methodology to minimize perverse incentives for organizations to focus treatment on the easiest cases while avoiding the more challenging ones that could negatively affect outcomes. PFS is most likely an appropriate tool for prevention programs that address entrenched social problems. It is important to consider key project feasibility factors, such as the availability of sufficient evidence on an intervention’s effectiveness and the capacity of PFS stakeholders to collect and analyze the data required by these complex projects.

The PFS model is evolving, and early programs are just beginning to evaluate results. It is too soon to tell if governments will realize the benefits that induced them to try PFS. For example, a potential benefit of PFS is that it allows a government to shift the risk of implementing an innovative program to investors. In practice, investors whose return on investment is contingent on positive results may prefer projects that are based on rigorous evidence of success and may avoid innovative approaches that have not been rigorously tested. If this potential flight to programs with a strong evidence base turns out to become reality, it may not make sense for governments to rely on PFS projects. Instead, they may consider funding these types of programs directly, through traditional performance-based contracts that incorporate features of PFS projects that reduce the government’s risk, such as independent evaluation and governance rules that allow for strong management and oversight. By undertaking a PFS project to implement a program that is known to be successful, a government could be taking on extraneous costs for little or no benefit.

As the PFS field continues to grow and as Congress and federal agencies consider new investments in these projects, important considerations remain about the appropriate role for the federal government. Similar to state and local governments, the federal government faces potential benefits and challenges stemming from participation in PFS programs. As federal agencies consider expanding their involvement in PFS, it becomes increasingly important for officials at all levels of government to collaborate to share knowledge and experiences.

OMB led an informal working group on PFS for a limited period. Despite the stated usefulness of this group, notes and findings from the meetings were not shared among agencies, and the meetings were eventually discontinued. Having a mechanism for federal collaboration on PFS projects as the field grows would allow agencies to leverage the experience of early federal actors in the PFS field and would decrease the potential for missteps in developing projects due to information gaps and failure to learn from experience with this evolving tool of government.

Recommendation for Executive Action

To identify and broadly disseminate information on leading practices and lessons learned, the Director of OMB should establish a formal means for federal agencies to collaborate on PFS. This could include creating a formal working group and providing collaboration technologies, such as shared databases or web portals.

Agency Comments and Our Evaluation

We provided a draft of this report to the Director of the Office of Management and Budget, the Secretaries of the Treasury, Labor, Housing and Urban Development, Justice, Health and Human Services, and Education, and the Chief Operating Officer of the Corporation for National and Community Service.

The Office of Management and Budget (OMB) provided written comments concurring with our recommendation and is working with agencies to explore options for continued collaboration on Pay for Success. The response from OMB is reproduced in appendix V. The Departments of Labor, Housing and Urban Development, Health and Human Services, Education, and the Corporation for National and Community Service provided technical comments that were incorporated into the draft as appropriate. The Departments of Treasury and Justice had no comments on the draft report.

We are sending copies of this report to the Director of OMB, the Secretaries of the Treasury, Labor, Housing and Urban Development, Justice, Health and Human Services, and Education, and the Chief Executive Officer of CNCS.

If you or your staff have any questions about this report, please contact me at (202) 512-6806 or sagerm@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. GAO staff who made key contributions to this report are listed in appendix VI.

Appendix I: Objectives, Scope, and Methodology

This report responds to a request that we examine Pay for Success (PFS) to help Congress gain a better understanding of how the mechanism has been defined in theory and practice and what is known about its implementation and results to date. We examined (1) how selected PFS projects have been structured and what potential benefits these projects can provide; (2) how selected PFS contracts have been structured to address potential project risks; and (3) the potential roles for the federal government’s involvement in PFS projects.

To address these objectives, we began by identifying and reviewing relevant literature on PFS, including work produced by public sector, nonprofit, academic, and private sector organizations. We also interviewed individuals with public management expertise who had examined PFS and published work on or related to PFS. We identified some of these individuals through our literature review, while others were referred to us by other experts. These individuals were Rob Duggar of ReadyNation; Mildred Warner of Cornell University; Leonard Gilroy of the Reason Foundation; John Roman of the Urban Institute; Jitinder Kohli of Deloitte Consulting LLP; and Shar Habibi of In the Public Interest. During these interviews we asked the interviewees, among other things, about the potential benefits and challenges of PFS; the advantages and disadvantages that PFS has compared to other program funding mechanisms; and the potential roles that the federal government could play in PFS projects.

Based on our literature review and our interviews with individuals knowledgeable about PFS, we identified jurisdictions in the United States and abroad that were implementing, planning to implement, or studying the potential of PFS projects. We then selected seven U.S. and three United Kingdom (UK) case illustrations to review in depth. We selected cases from the UK in order to capture the country’s experience as the first to develop and implement PFS projects as well as because it is the location with the longest experience with this financing mechanism. We selected four PFS projects that were being implemented in the United States at the time we began our engagement in July 2014. Those projects were the Massachusetts Juvenile Justice PFS Initiative; the New York City Rikers Island Project; the New York PFS project: Increasing Employment and Improving Public Safety; and the Utah High Quality Preschool Program. Because those cases focused heavily on reducing recidivism and improving early childhood education, we selected two additional U.S. projects that were in development to reflect an additional policy area (improving health outcomes). Those projects were the South Carolina Maternal Health PFS Initiative and the Asthma Impact Model for Fresno (AIM4Fresno). Finally, we selected one case in which a state considered but ultimately decided not to move forward with a PFS project. That case was Maryland’s review of a potential PFS initiative focused on reducing recidivism. For the UK projects, we selected three projects that were also in implementation and varied in issue area, structure, and the participating organization that conceived them. Those cases were the Peterborough Prison Project; the London Homelessness Project; and the It’s All About Me Adoption Project, on reducing adoption wait times.

To obtain information about each of these case illustrations, we conducted site visits to Boston, Massachusetts; New York City, New York; Albany, New York; Annapolis, Maryland; Salt Lake City, Utah; and London, United Kingdom. During these visits, we requested and reviewed documents relevant to the case illustrations, including feasibility studies, contracts, and evaluation reports. We conducted in-depth interviews with representatives of organizations participating in the PFS projects in these locations. For each case, as appropriate, we interviewed representatives from government entities, intermediaries, service providers, investors, evaluators, and validators. We conducted some of the interviews by telephone before or after site visits due to scheduling conflicts. During the interviews, we asked the interviewees, among other things, about the potential benefits and risks of PFS; the advantages and disadvantages that PFS has compared to other program funding mechanisms; the specific roles their organizations play in PFS projects; the PFS project contract negotiations, including the establishment of outcome payment and governance structures; and the potential roles that the U.S. federal government or UK national government could play in PFS projects.

The interviewees for our case illustrations were representatives from: Bank of America Merrill Lynch; Bates Wells Braithwaite; Big Lottery Fund; Big Society Capital; Bloomberg Philanthropies; The Boston Foundation; Bridges Ventures; Center for Employment Opportunities; Chesapeake Research Associates; Collective Health; Esmee Fairbairn Foundation; Goldman Sachs; Greater London Authority; Granite School District Preschool Services; Harvard University’s Social Impact Bond Technical Assistance Lab; Imprint Capital; Maryland Department of Legislative Services; Massachusetts Executive Office of Administration and Finance; MDRC; New Profit Inc.; New York City Mayor’s Office; New York Department of Corrections and Community Supervision; New York Division of the Budget; The Osborne Association; Public Consulting Group; Roca Inc.; The Robin Hood Foundation; Rockefeller Foundation; Salt Lake County Mayor’s Office; Sibayltics LLC; Social Finance (UK); Social Finance (U.S.); St. Giles Trust; Thames Reach; Third Sector Capital Partners; UK Cabinet Office; UK Department for Communities and Local Government; UK Ministry of Justice; United Way of Salt Lake; Utah Governor’s Office of Management and Budget; Utah House of Representatives; Utah School Readiness Board; Utah State University’s Early Intervention Research Institute; Vera Institute of Justice; and Voices for Utah Children.

To assess the PFS contract structure, we reviewed our prior reports and contracting literature to identify contracting practices from the federal experiences relevant to PFS contracts, which are complex service acquisitions. We compared the contracting practices to the PFS contracts in our study to establish their relevance to the design and implementation of PFS contracts.

Because the scope of our review was to assess the potential benefits and risks of PFS and the structure of PFS projects, we did not assess whether the outcomes established for specific projects and any cost savings that were identified with their attainment were appropriate, ambitious, or met other dimensions of quality. In addition, since most of the case illustrations we reviewed were in early stages of implementation, we did not request any performance information related to them and did not assess the likelihood of the projects successfully achieving outcomes.

To obtain information about federal agencies’ involvement in PFS, we reviewed relevant documentation, including the President’s Budget for fiscal years 2012 through 2016. Through this review, we identified eight federal agencies that were involved in an informal working group and played different roles vis-à-vis PFS. The Office of Management and Budget, working with the White House Office of Social Innovation and Civic Participation, has provided informal guidance on PFS efforts, and participated in the informal working group. The Department of the Treasury participated in the informal working group. The Departments of Labor (DOL), Housing and Urban Development (HUD), and Justice (DOJ), and the Corporation for National and Community Service (CNCS) have funded or plan to fund PFS projects. The Departments of Health and Human Services (HHS) and Education (Education) have been exploring opportunities to pursue the PFS concept. We interviewed officials from each of these agencies about their involvement in PFS, and asked them, among other things, about their strategic approaches for using PFS; the advantages and disadvantages that PFS presents compared to other financing tools; the amount of funding they allocated to PFS; interagency collaboration on PFS; and early lessons learned from their involvement in PFS.

Furthermore, of the eight agencies involved in the informal working group on PFS, we interviewed officials and reviewed budget documents from six agencies that typically fund social programs—CNCS, DOJ, DOL, Education, HHS, and HUD—to identify illustrative examples of how these agencies have used evidence-based approaches in their decision making more generally.

We conducted our work from July 2014 to September 2015 in accordance with generally accepted government auditing standards. Those standards require that we plan and perform the audit to obtain sufficient, appropriate evidence to provide a reasonable basis for our findings and conclusions based on our audit objectives. We believe that the evidence obtained provides a reasonable basis for our findings and conclusions based on our audit objectives.

Appendix II: Illustrative Examples of Evidence-Based Programs from Selected Federal Agencies

To provide examples of how federal agencies are using evidence to inform decision making, we asked the agencies in our study to provide us with examples of existing programs or program proposals that were consistent with Office of Management and Budget (OMB) guidance on areas in which agencies could improve their use of evidence.

Appendix III: Overview of PFS Projects in Our Study

Investment

$9.6 million loan from Goldman Sachs. 75 percent of the loan is guaranteed by Bloomberg Philanthropies. $16.11 million, including a $8 million loan from Goldman Sachs and $8.11 million in funding from various philanthropic organizations. $13.5 million equity investment from 40+ investors and foundations. 10 percent of the investment is guaranteed by the Rockefeller Foundation. $7 million, including a $4.6 million loan from Goldman Sachs and a $2.4 million subordinate loan from J.B. Pritzker Foundation.

Size of target population Stakeholders •

5 government offices (U.S. Department of Labor, NY Governor’s Office, the NY Division of the Budget, NY Department of Labor, and NY Department of Corrections and Community Supervision); Intermediary (Social Finance) Service provider (Center for Employment Opportunities)

2 government agencies (NYC Mayor’s Office and Department of Corrections) Intermediary (MDRC) 2 service providers (Osborne Association and Friends of Island Academy) 1 private-sector investor (Goldman Sachs) and 1 philanthropic loan guarantor (Bloomberg Philanthropies) Evaluator (Vera Institute of Justice)

12 government agencies; (U.S. Department of Labor, MA Executive Office of Administration and Finance, MA Office of the Commissioner of Probation, and MA Department of Youth Services, MA Department of Corrections; the MA Parole Board; the Suffolk County Sheriff’s Department; the Middlesex County Sheriff’s Department; the Hampden County Sheriff’s Department; and the Essex County Sheriff’s Department. Also, data for evaluation and validation come from the Executive Office for Labor and Workforce Development (wage data) and the Department of Criminal Justice Information Services (criminal records) investors) and 3 philanthropic investors (Robin Hood Foundation, Rockefeller Foundation, and Laura and John Arnold Foundation)

2 government offices (Salt Lake County and the Utah School Readiness Board) Intermediary (United Way of Salt Lake) 6 service providers 1 private-sector investor (Goldman Sachs) and 1 philanthropic investor (J.B. Pritzker Foundation) Evaluator (Utah State University)

New York City (NYC) Rikers Island Project

Evaluator (NY Department of Labor Research and NY Department of Corrections and Community Supervision Research) Validator (Chesapeake Research Associates)

Intermediary (Third Sector Capital Partners) Service provider (Roca) 1 private-sector investor (Goldman Sachs) and 5 philanthropic investors Fiscal services provider (New Profit) Evaluator (Urban Institute) Validator (Public Consulting Group)

Number of outcome metrics Evaluation method

Range of outcome payments First outcome payment due

Government budget structure

No new budget structure for outcome payments.

The state requests an annual appropriation for outcome payments and backs future payments with its full faith and credit.

The state is renewing a 2- year appropriation every year.

Legislation created a special account into which an appropriation to cover outcome payments was placed.

Oversight structure

Intermediary was responsible for project management, including making adjustments to service delivery and coordinating with and reporting to other program partners.

Provided by operating and oversight committees, composed of the project’s stakeholders.

Provided by the executive steering committee, which provides strategic guidance, the management committee, which provides operational management, and working groups, which provide implementation support.

Provided by the Utah School Readiness Board, which receives and reviews regular operational reporting from the intermediary.

Appendix IV: Example of United Kingdom Rate Card for Social Impact Bonds

Government officials from the United Kingdom (UK) told us that the rate of return for Social Impact Bonds funded with the Department for Work and Pensions’ (DWP) Innovation Fund are not subject to individual negotiations. The Innovation Fund is a pilot initiative that has commissioned Social Impact Bonds to help young people who are disadvantaged participate and succeed in education or training, therefore improving their employability and reducing their dependency on benefits. The Innovation Fund is 100 percent outcome based. Investors or intermediaries work with service providers to bid on Social Impact Bond projects based on the predetermined outcome payments published in DWP’s rate card (see figure 11). For example, if a service provider meets an outcome on an individual’s sustained employment, DWP pays £2,000 ($3,000). DWP makes awards for Social Impact Bond projects through a competitive bidding process. DWP will pay the investors or intermediaries solely on the basis of outcomes achieved. The service providers receive funding from the investors or intermediaries to cover their delivery costs, so they do not have to take the risk of depending on outcome payments.

Appendix V: Comments from the Office of Management and Budget

Appendix VI: GAO Contact and Staff Acknowledgments

GAO Contact

Staff Acknowledgments

In addition to the above contact, Elizabeth Curda (Acting Director) and Barbara Lancaster supervised this review and the development of the resulting report. Virginia Chanley, Karin Fangman, Sherrice Kerns, Donna Miller, and Daniel Webb made significant contributions to this report. In addition, Joy Booth, Alicia Cackley, Jeffrey DeMarco, Deirdre Duffy, Gale Harris, Katherine Iritani, Shelby Kain, Adam Miles, Susan Offutt, Erin Saunders-Rath, Paul Schmidt, and William Woods made key contributions.