Pay for Success: Collaboration among Federal Agencies Would be Helpful as Governments Explore New Financing Mechanisms
Published: 2015-09-09T00:00:00

Fast Facts:

Highlights:
section_title: Why GAO Did This Study
paragraphs: Federal, state, and local agencies play an important role in improving social outcomes for society's most vulnerable populations. A small number of state, local, and foreign governments are employing PFS to fund efforts designed to better serve these vulnerable populations.
GAO was asked to provide information about PFS. This report examines (1) how selected PFS projects have been structured and what potential benefits these projects can provide; (2) how selected PFS contracts have been structured to address potential project risks; and (3) the potential roles for the federal government's involvement in PFS projects.
To address these objectives, GAO reviewed relevant literature on PFS; selected 10 PFS projects that reflected a variety of policy areas and were in different stages of implementation in state and local governments in the United States and the United Kingdom, where PFS originated; reviewed key documents and interviewed stakeholders from organizations that played a role in the selected projects; and reviewed documents and interviewed officials from the eight federal agencies that participated in an informal PFS working group.
section_title: What GAO Found
paragraphs: Pay for Success (PFS), also known as Social Impact Bonds, is a new contracting mechanism to fund prevention programs, where investors provide capital to implement a social service— for example, to reduce recidivism by former prisoners. If the service provider achieves agreed upon outcomes, the government pays the investor, usually with a rate of return, based on savings from decreased use of more costly remedial services, such as incarceration. Stakeholders from the 10 PFS projects in GAO's study said that PFS offers potential benefits to all parties in the project. For example, governments can implement prevention programs that potentially lead to reduced spending on social services and transfer the risk of failing to achieve outcomes to investors.
The PFS projects in GAO's review sought to manage potential risks in the feasibility assessment, design, and implementation of complex PFS contracts. For example, without safeguards, there is a risk that paying for outcomes could create perverse incentives, such as focusing on individuals who are easiest to serve rather than those most in need. To address this risk, PFS contracts included various provisions, such as only including those with the greatest need in the evaluation that determines if the government makes payments.
To date, federal government involvement in PFS has been limited. The Office of Management and Budget (OMB) has encouraged agencies to explore the use of PFS as appropriate. Potential roles federal agencies could play in PFS projects include making outcome payments or helping build capacity. We have previously reported that collaborative mechanisms, such as interagency groups, can be used to implement programs and share information. However, a formal mechanism for federal agencies to collaborate on PFS does not exist. Given the evolving nature of PFS, a mechanism for federal agencies to collaborate on PFS would increase access to leading practices.
section_title: What GAO Recommends
paragraphs: To identify and broadly disseminate information on leading practices and lessons learned, the Director of OMB should establish a formal means for federal agencies to collaborate on PFS. OMB concurred with this recommendation and is working with agencies to explore options for collaboration.

section_title: Letter
paragraphs: Federal, state, and local agencies play an important role in improving  outcomes for our society’s most vulnerable populations, who often face  complex social or economic problems. Agencies at all levels of  government can improve the lives of vulnerable populations and make  effective use of available resources by allocating them to prevention  programs that demonstrate the promise to innovatively solve intractable  social challenges. These challenges include reducing recidivism rates of  recently released prisoners, helping homeless individuals find stable  accommodation, and expanding access to quality early childhood  education for economically disadvantaged children. Moreover, by defining  and achieving prevention outcomes, agencies can potentially reduce  future budgetary pressures by lessening the need for costly remediation  programs.
A small number of foreign and U.S. state and local governments are  employing a contracting mechanism known as a Social Impact Bond  (SIB) or a Pay for Success (PFS) program to fund preventive efforts  designed to more effectively serve vulnerable populations. In a PFS  contract, private or philanthropic capital is used to fund an intervention  that targets the achievement of specific outcomes, such as a reduction of  the recidivism rate and an increase in the employment rate of a target  population. After confirming that outcomes have been achieved, a  government entity repays the other funders, usually with a rate of return.
Proponents of PFS believe that this contracting mechanism allows  government entities to implement innovative, evidence-based social  services that better serve vulnerable populations, and that it does so at a  reduced level of risk to agency budgets because government payments  are contingent on achieved outcomes. On the other hand, skeptics of  PFS believe that the mechanism is unlikely to reduce the level of risk to  agency budgets and could actually increase a government entity’s cost of  implementing a program as a result of the higher administrative costs  associated with the complexity of the project design.
To date, four federal agencies have allocated resources to support PFS  projects. The President and members of Congress have made proposals  to expand federal support for PFS projects. Since fiscal year 2014, the  President’s budgets have proposed a $300 million appropriation for a new  PFS Incentive Fund in the Department of the Treasury to expand the role  of the federal government and support state and local governments and  other intermediaries seeking to establish PFS projects. Members of  Congress have also introduced legislation during the 113th and 114th  Congresses on PFS to expand federal support for PFS projects.
You asked us for information to help Congress have a better  understanding of PFS. This report examines (1) how selected PFS  projects have been structured and what potential benefits these projects  can provide; (2) how selected PFS contracts have been structured to  address potential project risks; and (3) the potential roles for the federal  government’s involvement in PFS projects.
To address these objectives, we identified and reviewed relevant  literature on PFS, including work produced by public sector, nonprofit,  academic, and private sector organizations, and interviewed individuals  with public management expertise who had examined PFS and published  work on or related to PFS. Based on the literature review and the  interviews, we identified jurisdictions in the United States and abroad that  were implementing, planning to implement, or studying the potential of  PFS projects. For illustrative purposes, we selected seven U.S. and three  United Kingdom (UK) cases to review in depth. We selected cases from  the UK to capture the country’s experience as the first to develop and  implement PFS projects as well as the location with the longest  experience with this financing mechanism. For the U.S. projects we  selected the four PFS projects that were being implemented in the United  States at the time we began our review in July 2014. Those projects were  the Massachusetts Juvenile Justice PFS Initiative; the New York City  Rikers Island Project; the New York PFS project: Increasing Employment  and Improving Public Safety; and the Utah High Quality Preschool  Program. Because those projects focused heavily on reducing recidivism  and improving early childhood education, we selected two additional U.S.  projects that were in development to reflect an additional policy area  (improving health outcomes). Those projects were the South Carolina  Maternal Health PFS Initiative and the Asthma Impact Model for Fresno  (AIM4Fresno). Finally, we selected one case in which a state considered  but ultimately decided not to move forward with a PFS project. That case  was Maryland’s review of a potential PFS initiative focused on reducing  recidivism. For the UK projects, we selected three projects that varied in  policy area, structure, and the participating organization that conceived  them. Those cases were the Peterborough Prison Project; the London  Homelessness Project; and the It’s All About Me Project on reducing wait  times for adoption.
We reviewed documents relevant to the case illustrations, including  feasibility studies, contracts, and evaluation reports. We conducted in- depth interviews with representatives of organizations participating in  each of these projects, including representatives from government  entities, intermediaries who help manage PFS projects, service providers,  investors, evaluators, and validators, who validate evaluation results. We  refer to individuals from these organizations by using particular categories  of stakeholders, such as “investors,” when the distinction is meaningful.  We refer to them collectively as “stakeholders” when all organization  representatives provided a similar point of view.
To assess the PFS contract structure, we reviewed our prior reports and  contracting literature to identify contracting practices from the federal  experiences relevant to PFS contracts, which are complex service  acquisitions. We compared these contracting practices to the PFS  contracts in our study to establish their relevance to the design and  implementation of PFS contracts.
We also examined federal agencies’ involvement in PFS by reviewing  relevant documents, including the President’s Budget for fiscal years  2012 through 2016. We identified eight federal agencies that were  involved in an informal PFS working group and have played different roles  vis-á-vis PFS. The Office of Management and Budget (OMB), working  with the White House Office of Social Innovation and Civic Participation,  has provided informal guidance on PFS efforts and participated in the  informal working group. The Department of the Treasury (Treasury)  participated in the informal working group. The Departments of Labor  (DOL), Housing and Urban Development (HUD), and Justice (DOJ) and  the Corporation for National and Community Service (CNCS) have  funded or plan to fund specific PFS projects. The Departments of Health  and Human Services (HHS) and Education (Education) have been  exploring opportunities to pursue the PFS concept. We interviewed  officials from each of these agencies about their involvement in PFS. Of  the 8 agencies in the working group, we also interviewed officials and  reviewed budget documents from 6 agencies that typically fund social  programs—CNCS, DOJ, DOL, Education, HHS, and HUD—to identify  illustrative examples of how these agencies have used evidence-based  approaches in their decision making more generally.
We conducted our work from July 2014 to September 2015 in accordance  with generally accepted government auditing standards. Those standards  require that we plan and perform the audit to obtain sufficient, appropriate  evidence to provide a reasonable basis for our findings and conclusions  based on our audit objectives. We believe that the evidence obtained  provides a reasonable basis for our findings and conclusions based on  our audit objectives.
subsections: section_title: Background
subsections: section_title: OMB Guidance Encourages Agencies to Use Evidence-Based Approaches, Including PFS
paragraphs: According to OMB’s 2013 Memo on the  Next Steps in the Evidence and Innovation  Agenda  …evidence includes evaluation results,  performance measures, and other relevant  data analytics and research studies, with a  preference for high-quality experimental and  quasi-experimental studies.
Since 2010, OMB has used its annual budget guidance to broadly  encourage federal agencies to use evidence to consider the effectiveness  of their programs and to institutionalize the use of evidence to foster  innovation rooted in research and rigorous evaluation. In a 2013 memo,  OMB identified areas in which agencies could improve their use of  evidence, including: (1) strengthening evaluation capacity and proposing  new evaluations; (2) developing high-quality, low-cost evaluations and  rapid, iterative experimentation; (3) using innovative outcome-focused  design; and (4) increasing agency capacity to use evidence. In 2015,  OMB required agencies to include an evidence template with their fiscal  year 2017 budget submissions. The template should provide an overview  of evidence-building strategies and identify related priorities. The six  federal agencies in our study that typically fund social programs said they  used evidence to inform a range of program proposals or decisions. See  appendix 2 for illustrative examples of these agencies’ evidence-based  programs.
As part of its agenda to base management and policy decisions on  evidence, the White House and OMB have encouraged agencies to  explore PFS, where appropriate, as one tool to incorporate evidence into  grant making. As of July 2015, four federal agencies have awarded PFS  grants or are developing PFS grant proposals (see table 1).
section_title: PFS Is a Mechanism for Contracting for Social Outcomes
paragraphs: PFS is a program funding mechanism that some U.S. state and local and  foreign governments are using to contract for prevention services to  address entrenched social problems. The first PFS project began in the  United Kingdom in 2010. In a PFS contract, private investors provide  upfront capital that a service provider uses to implement an evidence- based prevention program. An intermediary organization can solicit  funding from the investors, facilitates the flow of capital from the investors  to the service provider, and oversees the provider’s work throughout the  project. If the program is successful, the government entity makes  payments to the intermediary, which in turn may repay the investors their  capital along with interest. These payments are, in theory, based on the  savings that result from decreased use of government services. PFS is  intended to allow government to transfer the risk of paying for a program  that does not achieve desired outcomes from the government to other  participating organizations. If the program is successful, the government  obtains improved outcomes for vulnerable populations. In addition, PFS  can potentially capture cost savings by reducing the need to provide  costly remediation services to the individuals in vulnerable populations.  However, stakeholders said that PFS can also be used to contract for  outcomes that a government deems worthwhile, even if the outcomes do  not generate savings. Because the government makes payments based  on the service provider meeting outcome targets, all parties to the  contract have a vested interest in the success of the service provider.
An individual PFS project comprises a series of contracts and other  vehicles, such as loan and grant agreements, that defines entities’ roles  in the project and identifies the risks they each agree to assume and the  benefits they each stand to receive by virtue of their participation.
Although individual PFS projects may vary in terms of roles and  responsibilities for the parties to the contract, there are five main types of  organizations that commonly participate in a PFS project: government,  intermediaries, investors, service providers, and evaluators (see figure 1).
Government originates a contract for defined outcomes and pays on the  basis of those outcomes being achieved. In the United States, state and  local governments have initiated PFS projects to date, while in the UK  both national and local-level governments have initiated them. Examples  of government entities that have been involved in PFS projects include  state or local programmatic agencies and central management offices,  which often work together collaboratively on a project. As mentioned  above, DOL and CNCS have awarded grants to support state and local  PFS projects.
Investors provide capital to finance a program. They provide capital on  the condition that the government will repay the investment if outcomes  are achieved, usually with a rate of return. In U.S. projects to date, capital  has been provided by a mix of investor types, such as banks,  foundations, and high net worth individuals, whereas in UK projects,  capital has been provided solely by social impact investors.
Intermediaries enter into a government contract in a PFS project and  usually receive a fee for the services they provide. Intermediaries can  take on a variety of roles in a PFS project, such as providing technical  support to governments on the feasibility of a project, identifying the  service provider, raising operating funds from investors, negotiating the  terms of contracts or agreements with governments, service providers,  and investors; disbursing payments to service providers; and overseeing  day-to-day performance management of the project. Examples of  intermediaries that have been involved in PFS projects include nationally  focused nonprofit organizations with public policy and financial expertise  and a locally focused nonprofit organization with experience implementing  programs within local communities.
Service providers enter into contracts with an intermediary or a  government to administer the evidence-based prevention program.  Examples of service providers that have been involved in PFS projects  include nonprofit or charitable organizations with established track  records of implementing programs in specific policy areas, such as job  training or homelessness prevention.
Evaluators measure a program’s outcomes or evaluate a program’s  effect on the target population. The evaluator’s work assures the other  participating organizations that defined outcomes were achieved and are  likely due to the program, rather than to other factors, including chance.  Outcome payments are based on these findings. In some cases, a  validator, independent of the evaluator, is responsible for certifying the  evaluator’s findings. Examples of evaluators that have been involved in  existing PFS projects include a research office within a government  department and an organization with experience evaluating programs  within a specific public policy area, such as early childhood education.
The PFS projects we reviewed generally had three phases: a feasibility  assessment, project design, and program implementation (see figure 2).
section_title: Practices from Federal Experience with Complex Service Acquisitions Can Inform PFS Projects
paragraphs: As described previously, a PFS project is a complex set of contracts  through which multiple organizations work together to deliver social  service programs. PFS projects are similar to complex acquisitions.  Complex acquisitions are defined, in part, by the difficulty in defining  precise terms of exchange for all the characteristics of the product or  service at the time of the contract negotiation. We have previously  identified key factors that government entities should incorporate into  complex acquisitions to achieve improved service acquisition outcomes.  Key literature on complex acquisitions we reviewed reinforces these  factors. Taken together, this understanding can inform the design and  management of PFS contracts. To address the inherent ambiguity in  complex acquisitions, parties can include processes in the contract to  manage decision making and to make ongoing adjustments during  implementation. Specifically, providing clear standards in the following  areas can facilitate PFS contracts:
Valid and well-defined requirements. We have previously reported  that establishing a valid need for a service, and then translating that  need into a service acquisition requirement, is essential for obtaining  the right outcome. Literature on complex contracting underscores the  need for well-defined requirements, but states that it can be  challenging to define all requirements upfront in a complex  acquisition. Undefined requirements leave room for interpretation that,  if not carefully managed, can lead to outcomes that are inconsistent  with expectations.
Properly structured business arrangements. We have previously  reported that once requirements have been defined, it becomes  necessary to develop a business arrangement to meet the  requirements and protect the government’s interests. At a basic level,  properly structuring business arrangements includes defining a clear  scope of expected contractor performance and developing an  objective means of assessing the contractor’s performance. There  should be clearly established relationships among the tasks the  contractor is expected to perform, the contract terms and conditions,  and performance evaluation factors and incentives. Literature on  complex contracting underscores the importance of properly  structured business arrangements, including governance rules, which  should define how the government and contractor will interact to make  decisions during the contract’s implementation.
Proactively managed outcomes. We have previously reported that  managing and assessing post-award performance entails various  activities performed by government officials to ensure that the delivery  of services meets the terms of the contract, including adequate  oversight resources, proper incentives, and a capable workforce for  overseeing contractor activities. Each of these activities requires  metrics and tools to encourage contractors to provide superior  performance and to manage and document acceptable contractor  performance. Literature on complex contracting also underscores  the importance of rewarding successful outcomes, particularly in  cases where it is difficult to define requirements, such as efforts to  acquire services.
section_title: PFS Projects Were Structured in a Variety of Ways and Sought to Achieve Potential Benefits, Such as Improved Social Outcomes and Cost Savings
subsections: section_title: PFS Projects Were Structured to Implement Prevention Programs to Address Entrenched Social Problems
paragraphs: In our selected case illustrations, governments used PFS to address  entrenched social problems. The individuals who face these problems  have complex, multi-faceted needs and are often poorly served by  existing interventions or programs. The organizations participating in the  PFS projects in our study said that these individuals can be better served  through prevention programs focused on defined outcomes. Stakeholders  told us that PFS is just one tool among many for addressing social  problems and that PFS is not a panacea.
Stakeholders said PFS is most appropriate in policy areas in which  populations face complex, entrenched problems. One stakeholder also  noted that PFS may work best on projects that supplement, not supplant,  existing services, such as prevention programs for which existing services  have been insufficient in achieving desired outcomes. Projects in our  study used PFS to implement programs intended to reduce recidivism  among an adolescent jail population, gaps in academic achievement  among preschoolers, homelessness for difficult-to-serve individuals, wait  times for adoption, and to provide preventive health care to address poor  birth outcomes and asthma.
subsections: section_title: Massachusetts: Reducing Recidivism and Increasing Employment among Ex- Offenders
paragraphs: Massachusetts Juvenile Justice PFS  Initiative   The “Roca Model,” which the project is  funding, includes 4 basic elements: outreach  to young men; intensive case management;  life skills, educational, prevocational, and  employment programming; and a connection  to work opportunities. Project stakeholders  believe that the Roca Model will be able to  help young men exiting the criminal justice  system to reduce the average 2.4 years they  spend in prison for new convictions after they  are released.
The Massachusetts Juvenile Justice PFS Initiative, which began in  October 2013, was designed to address high rates of recidivism among  young men. Every year approximately 4,000 high-risk young men age out  of the state’s juvenile justice system or are released from adult  probation. Sixty-four percent of these young men are incarcerated at  least once within 5 years of release. Since its initial design, the project  has been expanded to serve young men connected to the adult  corrections system who are at high risk of future adult incarceration. The  project is being used to fund an intervention provided by Roca, Inc.,  which has been shown to reduce incarceration rates for the young men it  serves by 33 percent.
Massachusetts will make payments on the basis of reduced incarceration,  increased employment, and for each individual program participant who  meets 9 or more times per quarter with a Roca staff member to discuss  job readiness issues. The project’s target impact is a 40 percent reduction  in participants’ days in incarceration compared to a control group during  the 5 years following their enrollment in the program. At that level,  Massachusetts expects to make $21 million in outcome payments and  realize $22 million in gross savings during the approximately 6-year  lifespan of the project. If participants’ days in incarceration are reduced by  55 percent, Massachusetts expects to make the maximum $28 million in  outcome payments and realize $33 million in gross savings. The U.S.  Department of Labor (DOL) awarded Massachusetts an $11.7 million  grant for the project. Up to $10.8 million of this amount is available for  outcome payments associated with the first 2 years of service for up to  535 young men. Massachusetts made its first payments related to the  job readiness metric in May 2015. The state will make its first payments  related to the incarceration and employment metrics in the fall of 2017.  See figure 2 for an example of the roles of the organizations participating  in the project and the size of the project’s investment and potential  savings.
Adolescents released from Rikers Island jail in New York City have a 47  percent chance of returning to jail within 1 year. The New York City  Rikers Island Project, which began in September 2012, was designed to  address this recidivism rate. To do so, Osborne Association and Friends  of Island Academy administered a form of cognitive behavioral therapy  called Moral Reconation Therapy to adolescents at Rikers Island.  Project participants estimated that the city could save $1.7 million in net  long-term costs if participants’ future days in jail are reduced by 11  percent compared to a similar group that did not receive the therapy.  They estimated the city could save $20.5 million if participants’ future  days in jail are reduced by 20 percent. In July 2015, the project’s  evaluator determined that the program did not lead to reductions in  recidivism rates for participants in its 1-year assessment of the project.  The New York City government did not make outcome payments to  investors and the program was discontinued as of August 31, 2015. See  figure 3 for a description of the roles of the organizations participating in  the project and the size of the project’s investment and potential savings.
The New York PFS project, which began in October 2013, was designed  to reduce the high recidivism rate and increase the low employment rate  of high-risk individuals recently released from state prison each year.  According to a report on the project, these individuals spend an average  of 460 days in prison or jail in the 5 years following their release. The  project is being used to fund an intervention by the Center for  Employment Opportunities (CEO). A random control trial (RCT)  evaluation showed CEO’s intervention reduced recidivism for recently  released participants by 16 to 22 percent and days incarcerated for high- risk participants by 30 percent. New York will make payments on the  basis of reduced recidivism, increased employment, and engagement in  transitional jobs for recently released, high-risk individuals. New York  estimates that it can capture savings and benefits ranging from $13  million if recidivism is reduced by 10 percent to $37 million if recidivism is  reduced by 40 percent over a 5-year period. The U.S. Department of  Labor (DOL) awarded New York a $12 million grant for the project. Up to  $11.1 million is available for outcome payments associated with the first 2  years of the project. The first outcome payment would occur in 2017.  See figure 4 for a description of the roles of the organizations participating  in the project and the size of the project’s investment and potential  savings and benefits.
In 2011, a 3-year study conducted by Utah’s Granite School District  showed that between 25 percent and 33 percent of low-income students  in the district would likely be placed in special education without a  preschool intervention. The Utah PFS project was designed to decrease  the use of special education and to address academic achievement gaps  for low-income students who enter kindergarten without having attended  high-quality preschool. The project began in school year 2013-2014, with  outcome payments being supported by Salt Lake County and the United  Way of Salt Lake. Beginning in school year 2014-2015, Utah became the  outcome payor.
The project aims to expand Granite School District’s high-quality  preschool program statewide, which evidence has shown reduces by 95  percent the need for special education among the low-income students  who would otherwise likely be placed in special education programs.  Economically disadvantaged students who are eligible to receive a free or  reduced price lunch are eligible to be enrolled in the program. Of these  students, those with the most extreme negative scores (as reflected by  their scoring two standard deviations below the mean) on a test taken at  the beginning of preschool are tracked from kindergarten through sixth  grade to verify whether they are enrolled in special education. Beginning  in the summer of 2016, Utah will make annual payments to investors that  equal 95 percent of cost savings for each year from kindergarten through  sixth grade that each designated child is not enrolled in special education.  Utah estimates it can save approximately $2,700 per year if a child is not  enrolled in special education. See figure 5 for a description of roles of the  organizations participating in the project and the size of the project’s  investment and potential savings and benefits.
The Peterborough Prison Project, which began in September 2010, was  designed to reduce the recidivism rate among prisoners released from  Peterborough Prison after serving a sentence of less than a year by  improving their access to supportive social services. The project is being  used to fund a range of services for the prisoners, including housing  assistance, drug or alcohol treatment, and mental health support. The UK  Ministry of Justice agreed to make outcome payments on the basis of  reductions in recidivism of 10 percent for each program cohort or group of  prisoners served during a 2-year time frame, or a 7.5 percent reduction  for the three cohorts as a whole. In April 2014, the Ministry of Justice  announced that it would close the Peterborough Project 2 years early  because of a shift to the Transforming Rehabilitation program, which will  provide statutory services similar to those provided in the Peterborough  Project for all short-sentenced prisoners. See figure 6 for a description of  roles of the organizations participating in the project and the size of the  project’s investment and potential savings and benefits.
The London Homelessness Project, which began in November 2012, was  designed to improve outcomes for a cohort of entrenched homeless  individuals who do not respond to traditional models of resettlement and  who are not targeted by other initiatives. The project funds two providers  to deliver services tailored to the specific needs of the individuals within  the cohort. The Greater London Authority will pay the providers for  achieving outcomes including a reduction in the number of individuals  sleeping on the street; an increase in individuals who sustain stable  accommodation and gain employment; a reduction in average use of the  healthcare system for accident or emergency episodes; and reconnecting  individuals to a country outside of the UK if he or she is a foreign national.
Targets for each of these outcomes are based on historical data on  homeless individuals within London. The project does not include an  evaluation for the purposes of determining outcome payments because it  was not possible to create a matched comparison group for the cohort.  This is because other cities in the UK do not have as complete data on  homeless individuals as does London and the cohort is unique within  London. A feasibility study estimated the net present value of providing  services across the five outcome areas to the entire cohort over 5 years  to be approximately £30 million (approximately $49 million). If all of the  project outcomes are achieved, the maximum outcome payments and  administrative costs will total £5 million (approximately $8 million), thus  resulting in substantial public sector savings. See figure 7 for a  description of roles of the project’s participating organizations.
The It’s All About Me (IAAM) adoption project, which began in October  2013, was designed to find adoptive homes for children that otherwise  would not find them. The project funds adoption agencies to recruit  parents to adopt hard-to-place children. The agencies also provide these  parents with support services. The program’s goal is for up to 140 of  these children to be adopted each year. A UK local government pays a  service provider defined amounts at different points in the adoption  process, such as when a service provider places a child with an adoptive  family, and after 1 and 2 years of the child remaining with the adoptive  family. According to the project intermediary, while the maximum cost is  greater than what a local authority usually would pay adoption agencies, it  is also estimated to be about half of the cost of a child remaining in foster  care for 2 years. Evaluations are being conducted on the health and  psychological outcomes for the adopted children; outcomes for families  who adopt the children; and how decision making changes within local  governments as a result of the project. See figure 8 for a description of  the roles of the organizations participating in the project.
In addition to the case illustrations identified above, we examined 3  additional PFS initiatives that have yet to be implemented or were not  implemented.
section_title: South Carolina Maternal Health PFS Initiative
paragraphs: Stakeholders in South Carolina are developing a PFS initiative focused  on controlling costs and improving health and other outcomes of mothers  and newborns in South Carolina’s Medicaid program. The project is  expected to expand the Nurse-Family Partnership program, which,  according to stakeholders, is an evidence-based nurse home visiting  program for first-time, low-income mothers and their children, to rural and  under-served parts of the state. Trained professionals provide services  and support to pregnant women and families with young children,  primarily during visits to families’ homes. According to a feasibility study  prepared for the project, multiple trials and evaluations have  demonstrated NFP’s effectiveness, which includes reduced emergency  room visits and better health outcomes in later years.
section_title: Asthma Impact Model for Fresno (AIM4Fresno)
paragraphs: In Fresno, California, a nonprofit organization is conducting a  demonstration project to determine whether a PFS project is feasible. The  project, which began in March 2013, was designed to improve the health  of low-income children with asthma and reduce the costs that result from  emergency treatments. Fresno, according to the stakeholders, has one of  the highest asthma rates in the United States, and approximately 20  percent of children there have been diagnosed with the disease. The  stakeholders are conducting an evaluation to demonstrate the social and  financial benefits of a comprehensive in-home asthma management  program for children at high risk of asthma emergencies. The program  includes in-home education, environmental assessment, and remediation,  such as mold removal to help manage asthma. Based on the outcomes of  the demonstration project, stakeholders plan to pursue PFS financing to  scale up the program to more children who can benefit.
section_title: Maryland Review of a Potential PFS Initiative
paragraphs: In January 2013, the Maryland Department of Legislative Services (DLS)  published the findings of its review of the feasibility and potential benefits  and risks of a PFS project for financing reentry programs for state  prisoners. The review considered, among other things, whether a PFS  project could generate cost savings for the state. In its review, DLS  assumed that the PFS project would fund a pilot program to serve 1,250  participants over 5 years and reduce recidivism by 10 percent or 20  percent. DLS concluded that even if the pilot program reduced recidivism  by 20 percent, that result would be insufficient to close a wing of a prison  or an entire prison facility, which DLS stated are the largest cost savings  associated with avoided reimprisonment. Therefore, the state would be  unable to realize cost savings through the project. Maryland did not  pursue a PFS project to finance reentry programs after the report was  released.
section_title: Governments, Service Providers, and Investors Pursued PFS for Potential Performance and Financial Benefits
subsections: section_title: Potential Benefits for Governments
paragraphs: Stakeholders in PFS projects stated that governments face significant  challenges in implementing programs that effectively address entrenched  social problems. They also stated that government can benefit from using  PFS as a tool to address these challenges. The benefits of PFS include  the following:  Funding for prevention programs and potential cost savings.  Government stakeholders said constrained budgets in recent years have  made it difficult for governments to fund new prevention programs, even  though they would potentially lead to cost savings resulting from a  reduced demand for remediation services. Because investors provide  capital that a government only has to repay if the specified outcomes are  achieved, a PFS project helps governments implement programs that  they might not otherwise be able to fund. In addition, if successful, a PFS  project reduces the need for remediation services and associated budget  pressures and, therefore, frees up additional resources that government  can reinvest in priority programs. For example, an official from the  Massachusetts Executive Office of Administration and Finance told us  that any savings attributable to the Massachusetts Juvenile Justice PFS  project will likely be reinvested in the same or similar programs. Further,  the recipients of the prevention program are expected to benefit from  improved outcomes, such as better education outcomes resulting from  high-quality preschool.
The ability to implement innovative programs by transferring risk.  PFS is designed so that the government only pays for a program if its  outcomes are achieved, thus shifting the risk of failure to investors.
Stakeholders said that PFS creates incentives to both scale prevention  programs where strong evidence exists and to innovate where it does not  exist. For example, the Massachusetts project expanded a proven  program to reduce recidivism rates. The service provider had extensive  experience working with young men in Massachusetts at risk of  recidivating and had successfully reduced their incarceration rates by 33  percent. The Massachusetts project expanded the service to more than  900 young men in Boston and Springfield, areas where the service  provider had extensive experience. If successful, Massachusetts will pay  for the program and expects to benefit from improved social outcomes  and cost savings in its budget. If the service provider is unsuccessful,  Massachusetts does not pay.
PFS also creates incentives to innovate and apply evidence-based  approaches to new or different populations. In the New York City project  the intervention was administered in a new environment, a jail, with a  service provider with prior experience working in prisons, but without prior  experience with the specific intervention. New York City and the project  intermediary chose Moral Reconation Therapy for the intervention  because of the available evidence base and its flexibility in  implementation that could be adapted to a jail setting. The intermediary  selected a provider to administer the service. The PFS structure allowed  the project’s stakeholders to implement the intervention at Rikers Island  for the first time while providing the city with assurance that investors will  absorb financial loss if the intervention does not reduce participants’  recidivism. When the intervention did not reduce recidivism at the target  rates, New York City did not pay for the program.
A potential benefit of PFS is the opportunity for governments to  implement innovative programs without an established evidence base by  transferring risk to investors. In practice, investors told us they prefer to  back programs that already have a rigorous evidence base because  these programs have a known likelihood for success. In Maryland, the  DLS concluded that PFS may limit the implementation of innovative  programs and government’s opportunity to apply evidence-based  approaches to new populations because investors would be most likely to  back projects with well-established records of success in order to  minimize their risk. Therefore, PFS may exclude new service providers  and program types because of the perceived risk.
One of the organizations involved in the PFS projects in our review did  not think that risks would be transferred from a government to investors.  For example, in its review, the Maryland DLS concluded that in practice  there are significant challenges to transferring risk to investors, such as  identifying investors who are willing to invest in PFS projects and  designing contract mechanisms to fully transfer risk. This was a factor in  the DLS recommendation against using a PFS contract for a prisoner  reentry program. The Maryland DLS stated that it may be difficult to  design contract provisions to provide an enforcement mechanism to  prevent investors from terminating the contract early. In addition, during  the implementation of the project, the costs to administer the intervention  may be greater than the investors’ upfront funding and it is not clear  whether the government or the investors would be responsible for those  additional costs.
Collaboration across government agencies. Government stakeholders  told us that many social services are “siloed” within governments. In other  words, individual agencies, largely on their own, design, budget for, and  implement narrowly targeted social services, such as job training courses  or temporary accommodation for homeless individuals. These services  are usually not integrated with others that could collectively help  vulnerable individuals overcome complicated issues and are potentially  not as effective as a more integrated approach. PFS projects target  outcomes for populations or individuals rather than the output of specific  services, which would be the responsibility of a specific agency. PFS  projects bring together multiple government agencies to work  collaboratively to find solutions for difficult social problems. For example,  the New York project targets improved employment and recidivism  outcomes for program participants and therefore brought together  multiple state agencies to oversee their achievement. Officials from the  New York Department of Labor and the Department of Corrections and  Community Supervision are members of an executive steering committee  that provides strategic direction and represents the project across agency  boundaries. Similarly, a stakeholder in Massachusetts said that PFS can  help break down silos because it can bring together multiple agencies to  work together on an important social problem.
Sustained focus for entrenched social problems. Government  stakeholders told us that governments find it challenging to sustain focus  on difficult social problems over multiple years. Governments generally  operate on annual budget cycles, making it difficult to set resource  allocations over multiple years. In addition, changes in political leadership  may shift policy priorities and support for levels of funding or program  continuation. PFS contracts include upfront agreements about investor  funding levels over multi-year time frames, providing sufficient time to  collect, track, and analyze data, and measure outcomes that take multiple  years to achieve.
A focus on outcomes and building an evidence base. Government  stakeholders told us that governments sometimes are not focused on  their programs’ outcomes because for social services they tend to  contract for inputs instead of outcomes. As a result, governments have  difficulty knowing whether a particular social program is having the  desired effect on its target population. For PFS projects, governments  contract for outcomes, and the contracts generally include a requirement  that the program’s impact be rigorously evaluated, which allows  governments to demonstrate the effect the program has as well as to add  to the evidence base to inform future programs.
section_title: Potential Benefits for Service Providers
paragraphs: Stakeholders told us that PFS offers benefits to service providers,  including the following:  Stable funding for operations. Stakeholders told us that service  providers must often pursue multiple grant and fundraising efforts to fund  their services and enter into contracts in which they are not reimbursed  for the full cost of services provided. PFS provides an opportunity for  stable funding spanning multiple years. Officials from the service  providers in New York and Massachusetts told us that upfront, multiyear  funding for their programs is one of the key reasons why their  organizations were excited about participating in a PFS project. An official  from Roca in Massachusetts said that without the consistent funding  provided under PFS, the organization typically has had to piece funding  together from multiple grants to fund its programs.
Stakeholders also told us that PFS can be preferable for service providers  compared to traditional performance-based contracts. In a traditional  performance-based contract, the service provider is not paid all or a  portion of its fees until it meets defined outcomes. In such a model, the  service provider assumes financial risk that could damage it if it does not  succeed, therefore potentially excluding new or small service providers.  For example, UK government officials told us that in a performance-based  contract, the service provider must cover the initial costs of delivering  services, which is difficult for many nonprofit service providers because  they do not have the capital available to provide services in advance of  being paid. A benefit of PFS is that investors offer service providers full  upfront funding, or cost reimbursement, for service provision while  assuming most or all of the financial risk.
Flexibility in implementing service delivery. Service providers  identified two ways PFS projects can give service providers flexibility  compared to a traditional government contract or grant. In some of our  cases, service providers were using PFS to administer their proven  approaches on a larger scale. For example, officials from the service  provider in New York told us that participation in the PFS project has  allowed the organization to implement its service delivery model, for  which the organization believes it has a solid evidence base, on a larger  scale rather than conform to government-determined service delivery  requirements. PFS can also offer flexibility to service providers in their  day-to-day operations because the government contracts for outcomes,  rather than how the service provider spends the funding on the activities it  undertakes. Service providers said they have more flexibility to tailor their  approaches to meet the needs of the target population.
section_title: Potential Benefits for Investors
paragraphs: Stakeholders told us that PFS also offers benefits to investors, including  the following:  Potential for return on investment. Stakeholders told us that investors  benefit from potential financial returns. Investors conduct due diligence on  projects and invest in projects where they believe there is potential for  return on their investment. Investors and intermediaries noted that the  level of potential returns for PFS projects has so far been below-market,  compared to the level of risk.
Opportunity for social investment. Stakeholders said that certain  investors are interested in making investments with positive social  impacts. Officials from the bank that helped to facilitate investments in the  New York project told us that the individuals who invested in the project  are interested in aligning their investments with their values and creating  positive financial and social impacts. Social investors, such as nonprofit  philanthropic organizations in the United States and UK, said PFS is a  tool to support social programs consistent with their missions. For  example, an official from a philanthropic organization that invested in the  Peterborough Project told us that the organization had previously  provided grants to support services for short-sentenced prisoners and  decided to invest in the project, in part, because it offered a sustainable  solution to reducing recidivism among such prisoners. The organization  reasoned that if the project turned out to be successful and could be  shown to deliver cost savings to government, the UK government would  be incentivized to directly fund services for short-sentenced prisoners in  the future. In addition, social investors told us that because they will be  repaid if a service provider meets outcome measures in a PFS project,  they will have the opportunity to recycle the funds into other projects that  align with their missions.
section_title: Participating Organizations Sought to Manage Risks in the Feasibility Assessment, Design, and Implementation Phases of Complex PFS Contracts
subsections: section_title: Governments, Service Providers, and Investors Recognize that PFS Projects Present Risk
paragraphs: PFS is predicated on the idea that governments shift the risk associated  with developing and implementing a prevention program to investors.  Stakeholders told us that the various parties to PFS contracts face a  range of risks that are inherent to contracts:
Financial risk. The risk that one or more parties to the contract could  lose money by participating in a PFS project. For example, a PFS  project may not generate savings that are greater than its costs, which  can include management fees for an intermediary, evaluation costs,  and investors’ contracted outcome payments. Critics of PFS have  argued that, given the extra costs associated with intermediaries and  investors, using PFS to implement a program is inherently more  expensive than providing the prevention program as a direct  government service. Because interest rates are low for government  borrowing, it could still cost less to borrow funds to pay for the  service. However, some investors and intermediaries said they  believe the complexity and associated costs of implementing PFS  projects may decrease over time if the projects become more  replicable. PFS proponents in general have argued that such direct  government funding of new prevention programs has been difficult to  do in the current budget environment.
At the same time, investors risk losing their investment if the service  provider fails to achieve the desired outcomes. Officials from a bank  that has invested in multiple U.S. projects told us that the organization  carefully considers the evidence of the intervention that the service  provider will deliver in order to determine whether it is willing to take  the financial risk associated with an investment and how that risk  should be priced.
In addition, organizations participating in PFS projects told us that  governments, service providers, and investors have dedicated a  significant amount of time and resources to developing PFS projects,  at times without being compensated for their work. There is a risk that  the parties are unable to agree on a project’s complex details and fail  to launch the project, in which case participants lose the time and  resources that they devoted to developing the project.
Stakeholders identified other risks that could potentially result in  increased financial risk for parties to the PFS contract:
Perverse incentives. The risk that outcome payment structures will  create perverse incentives for stakeholders to take actions at the  expense of the individuals receiving treatment in the project. For  example, creaming is the risk that service providers focus on  individuals that are easiest to serve to achieve the defined outcomes.  As a result of the perverse incentive to focus on those that are easiest  to serve, those that are most in need and most difficult to serve could  receive fewer services or a diminished focus. If creaming occurs, the  government is potentially paying outcome payments when the target  population is not being served.
Termination risk. The risk that one or more parties to the contract  decide to terminate their participation in a PFS project, requiring the  project to end earlier than initially planned. The early termination of a  PFS project presents a risk that the target population may be  disadvantaged or hurt by losing access to the intervention. Officials  from an investor in multiple U.S. projects said that the organization  understands that the early termination of a PFS project has the  potential to negatively affect the programs’ target populations.  Therefore, the organization worked with other parties during contract  negotiations to create rules that would allow for the gradual  termination of unsuccessful projects while taking into account the  needs of current participants.
A variant of termination risk is policy risk, or the risk that a  government initiates a policy change that prevents a PFS project from  operating as initially intended. A policy change could disrupt a service  provider’s program delivery, putting the achievement of outcomes— and the investors’ investment—at risk. For example, the UK Ministry  of Justice terminated the Peterborough Project 2 years early. The UK  government implemented a new initiative called Transforming  Rehabilitation that provides a similar set of services for all short- sentenced prisoners in the UK. Officials from the UK Ministry of  Justice said maintaining the PFS program at Peterborough until 2017  for the third and final cohort as planned was not possible, as the  majority of prisoners within that group would already be receiving 12  months of supervision and rehabilitation as a result of the wider  reforms to probation. According to officials from the intermediary, it  would not have been possible to maintain a control group for the  evaluation of the last cohort in the project. The Ministry of Justice and  other project participants agreed to transition arrangements for the  project that, among other things, established that investors could  receive outcome payments if the project evaluation shows recidivism  across the first two cohorts was reduced by 7.5 percent and that the  Ministry of Justice would pay termination costs. Officials from the  intermediary for the project said that investors are pleased that the  needs of short-sentenced prisoners are being met under Transforming  Rehabilitation. However, because the Peterborough Project ended  early, information and learning about the effectiveness of the program  was not fully realized.
Performance risk. The risk that an intervention fails to achieve  defined outcomes. A government does not make payments if defined  outcomes are not achieved. However, if outcomes are not achieved,  then the government has lost time developing the program and failed  to meet mission-related goals and objectives. Performance risk  includes how the prevention program is operationalized, managed,  and implemented. This risk can be heightened if the intervention is  poorly structured or poorly tailored to the needs of a particular  participant population, or if service delivery is poorly executed, which  could lead to potential harm to program recipients. Investors also risk  losing some or all of their investment. Stakeholders from the  government and an investor in New York City told us that they  considered the implications of potential performance risks in the  project during its conception and development. For example, an  official from the New York City government told us that the parties  involved had extensive discussions, including about program  outcomes and timelines. In addition, officials from an organization that  invested in the project told us that they had to decide how likely,  based on existing evidence, the program was to achieve the  established outcome.
Appropriation risk. From the investors’ perspective, the risk that a  government fails to appropriate funds necessary to make outcome  payments. If a government does not appropriate necessary funds,  investors could lose some or all of their investment, making them less  likely to invest in PFS projects in the future. Investors’ perception of  appropriations risk affects their decision on whether to participate in a  PFS project and on how they view the risk of a project. For example,  officials from a bank that has invested in multiple U.S. projects  identified appropriations risk as one of the primary risks it sees for  investors considering PFS projects. The officials told us that since  investors assume substantial performance risk in the projects, they  must have an assurance that, in the end, the government is going to  make the outcome payments to which it initially committed.  Stakeholders in our UK case illustrations did not view appropriations  risk as a concern in the United Kingdom PFS market.
Reputational risk. The risk that a party’s reputation will be affected  negatively by events that occur in a PFS project. Stakeholders in New  York City told us that they are assuming reputational risk by  participating in the Rikers Island project. For example, a former official  from the New York City government told us that the city took a large  reputational risk by launching the first PFS project in the United  States, and the city is therefore very invested in its success. In  addition, an official from one of the service providers working to  implement the New York City project told us that the organization has  assumed reputational risk by working as the service provider for the  project. If the service provider fails to achieve outcomes for the  project, it could make it challenging to get future contracts.  Furthermore, an official from the organization that is evaluating the  New York City project told us that since there are individuals and  organizations within the criminal justice sector who are skeptical of  PFS or believe that it is a flawed concept, the organization is  potentially risking its reputation within the criminal justice sector by  participating in the project.
section_title: During Project Feasibility Assessments, Governments Reported Seeking to Maximize Cost Savings While Managing Financial and Performance Risk
paragraphs: To assess project feasibility, governments and other stakeholders  identified and evaluated potential projects in an effort to maximize cost  savings and also to manage potential financial and performance risk.  Stakeholders from the selected case studies stated that, in the first phase  of PFS project development, they evaluated the feasibility of using PFS to  address a particular social problem within their jurisdiction. We found that  in practice PFS contracts were complex and governments relied  extensively on outside technical assistance to assess the feasibility of  their projects.
Attention to cost-benefit analysis. Governments estimate the potential  effect the PFS project will have on future budgets to determine if the  project is feasible. Using cost-benefit analysis and cost modeling,  governments evaluate whether the prevention program is likely to  generate cost savings, cost avoidance, or other benefits over the course  of the project that exceed the project’s costs, which include the total  amount of outcome payments and associated administrative costs. Based  on this analysis, governments determine whether a project is feasible. For  example, as noted earlier, the Maryland Department of Legislative  Services determined that the state should not pursue a PFS project  because even relatively large reductions in recidivism among program  participants were unlikely to result in the closure of a wing of a prison  facility, the type of large, fixed-cost savings that it believed would be  needed to cover the program’s costs.
Governments’ cost-benefit analyses may also include indirect costs and  social benefits. For example, New York assessed both the direct cost  savings of reduced prison expenses to the government as well as  quantifiable public sector benefits of savings from reduced crime resulting  from the project. The New York project’s outcome payments are derived  from these estimated public sector savings and benefits. According to a  detailed summary of the project, the state estimated and included an  amount for the cost savings from reduced crime for victims based on a  study that estimated the intangible victim costs. To estimate these costs,  the state used other costs such as medical costs and lost earnings for  crime victims. The state also used a jury-compensation approach that  used the money awarded to victims by juries to estimate the intangible  victim costs of crime. Figure 10 shows the potential savings and  benefits, as well as the outcome payment costs that New York identified  for its PFS project. For example, a 10 percent reduction in recidivism  would result in potential savings and benefits that equal outcome  payment costs. As the rate of reduction in recidivism increases, the  state’s potential savings and benefits (after making outcome payments)  increase.
Identification of entities that capture financial benefits and budget  savings. Stakeholders said that in order to maximize cost savings to the  entity who is sponsoring the PFS project, governments must know which  government budget will capture and benefit from the savings.  Stakeholders said that a barrier to potential PFS projects occurs when  expected savings accrue to a different government entity than the one  that is leading the project. This is referred to as the “wrong pockets  problem.” If the savings resulting from a successful intervention by one  government entity accrue to a different government entity, without  agreement from the other government entity to contribute to outcome  payments in proportion to the savings, the project may not be cost- beneficial to implement. For example, stakeholders in South Carolina  have been developing a PFS project to fund a program designed to  improve maternal health outcomes. Since potential savings generated by  the program would accrue to Medicaid, those savings would be split  between the state and the federal government. Stakeholders told us that  the state would find it challenging to agree to make all of the project’s  potential outcome payments since it would only be capturing  approximately 30 percent of the project’s savings, with the federal  government capturing the rest. A stakeholder in New York City said the  city chose to pursue the Rikers Island project specifically because the city  bears all of its own criminal justice costs and therefore would not have to  involve the state or federal government in developing the project or  paying for the outcomes achieved.
Access to, and capacity for, robust data collection and analytics. To  determine project feasibility, governments must have access to complete  and accurate cost and benefit data and must have the capacity to analyze  it. However, stakeholders said that the availability of government data  on social programs and the capacity to analyze the data is a challenge to  the growth of PFS. Stakeholders said that governments need data  systems in place that allow agencies to collect data and share it between  agencies, as needed. However, in some cases, the necessary data can  be difficult to obtain or share. For example, government stakeholders in  Massachusetts told us that differences in the ways in which jurisdictions  within the state maintained files on individuals on probation made it  difficult to collect and share data on potential program participants at the  beginning of the project’s implementation. Stakeholders said legal  barriers can prevent governments from sharing data across multiple  systems. For example, we previously reported that matching education  and workforce data is challenging in states where collecting a Social  Security number in education data is prohibited by state law or agency  policy.
Stakeholders also expressed concerns that many governments do not  have sufficient data collection and analysis capacity for PFS projects.  Similarly, we have previously reported that it is critical to define the  relevant types of information required for service acquisitions and then  develop the appropriate data systems to collect and provide reliable  data. Governments will need staff with expertise in data analysis and  financial modeling to be able to conduct feasibility studies, identify future  savings, evaluate service delivery, determine outcome payments, and  negotiate PFS contracts. Government officials from both New York and  New York City said that their government’s robust data systems facilitated  their cost-benefit analyses. For example, according to a lessons learned  report from the intermediary in the New York City Rikers Island project,  the city’s strong financial planning structure allowed the city to gain a solid  understanding of the amount of money that it could save based on  changes in the size of its jail population. The city used this information to  establish its willingness to pay for reductions in recidivism attributable to  the intervention being funded by the PFS project. The cost information,  together with historical criminal justice data, allowed the city to identify a  specific reduction in recidivism that would make the PFS project  beneficial for the city. To address challenges with government capacity,  all of the U.S. PFS projects in our study included provisions for technical  assistance from experts affiliated with universities or intermediary  organizations to analyze project data and structure financial models  needed to determine project feasibility and outcome payments.
Sufficient evidence of an intervention’s success. In addition to cost- benefit analysis and access to data, stakeholders told us that during the  feasibility phase it is important to consider the evidence of an  intervention’s past success to help manage the performance risk of the  project. While some stakeholders stressed the importance of sufficient  evidence of an intervention’s success, others noted that there is variation  in the PFS field in the level of evidence and how it relates to the individual  project. For example, evidence supporting an intervention applied to one  population could provide the basis for testing the intervention with a  different population. A PFS project could also be used to implement an  innovative intervention with limited but promising evidence. Participating  organizations—including governments, investors, and service providers— may differ in their tolerance of risk and uncertainty associated with the  evidence of any intervention. Stakeholders from government,  intermediaries, and investors said they also assessed the availability of  service providers who could implement the intervention to determine  whether the project would be feasible. Officials from an investor in  multiple U.S. projects told us that the bank has looked to invest in PFS  projects that feature a service provider that has a track record of success.
Stakeholders said the availability of evidence-based social programs and  service providers with the capacity to implement them is a challenge to  growing the PFS field. Stakeholders expressed concerns that there is not  a sufficient pipeline of nonprofit service providers whose interventions  have been rigorously evaluated. We have previously reported on  challenges nonprofits face related to funding indirect costs, such as  program evaluation. In addition, stakeholders told us that in order to be  successful in an established PFS project, service providers need  experience implementing the evidence-based intervention, the capacity to  implement the intervention on a larger scale, and the ability to collect and  report data for evaluation. The enacting legislation for the Utah PFS  Project includes a grant program to build the capacity of preschool  providers to implement the high-quality preschool model, which is based  on a prior evaluation, so these providers are able to participate in  potential future PFS projects.
Leadership support. Stakeholders said that support from top  government leaders is essential to moving a PFS project forward to the  design and implementation phases of the project. This support can  provide the momentum that is necessary to get a PFS project off the  ground and sustain the project through implementation. State officials in  New York told us that PFS was part of Governor Andrew Cuomo’s policy  agenda during his gubernatorial campaign, as it aligned with his interest  in taxpayer efficiency and a focus on outcomes. In addition, a report on  the project stated that it was conceived when Governor Cuomo outlined a  state priority to use PFS to reduce recidivism and increase employment  among high-risk ex-prisoners. Furthermore, stakeholders in Utah told us  that the long-term prospects of the project were boosted by the Board of  the United Way of Salt Lake and the Mayor of Salt Lake County’s  agreement, during the project’s proof of concept year, to dedicate funds  to support the project’s outcome payments. Stakeholders also said that  leadership from a central government agency can help maximize  collaboration across government agencies. They said that it is important  that a central agency coordinates across multiple departments in a  situation where an intervention in one department will accrue future  savings in a different department or where participation from multiple  departments is needed to implement the project. We have previously  reported that top leadership commitment can improve outcomes and set  the direction, pace, and tone for implementing a significant policy  change. Stakeholders also said it is critical to have a dedicated  government official to manage the development of a PFS project. For  example, the governments in Massachusetts, New York, and Utah had a  full-time position in their budget offices to manage their PFS efforts.
section_title: During Contract Design, Parties Reported Defining Requirements to Manage Risk
paragraphs: Stakeholders said contract negotiations for PFS projects are complex,  lengthy, and resource intensive. Stakeholders from government, service  providers, intermediaries, and investors told us that they required  significant legal and technical capacity to support the development of  these complex contracts because PFS contracts were new and existing  templates were not available to use as a guide. While the same types of  parties were present, the PFS projects in our study had different  contractual arrangements. In some cases, the PFS contract was between  the government and the intermediary, which held additional contracts with  investors and service providers. In other cases, the contract was between  the government, intermediary, and service provider. The intermediary  holds separate agreements with the investors in the project. While the  contractual arrangements varied, stakeholders said that all parties were  involved with negotiating the terms of the contracts.
subsections: section_title: Defining Requirements in Outcome Payment Structures
paragraphs: Parties to the PFS contracts in our study defined the requirements for the  service provider, in part, by using performance targets for specific  outcomes that were rewarded with outcome payments. These  performance targets and associated payouts make up an outcome  payment structure. See table 2 for examples of four different types of  outcome payment structures we observed in the PFS projects we  reviewed. For example, in the Utah High Quality Preschool project, the  outcome payment structure was defined as a set payment for each  student participant who avoided being enrolled in special education each  year from kindergarten through sixth grade. Stakeholders from  governments and service providers said that the outcome payment  structure aligned the incentives of all parties to be invested in the service  provider’s successful delivery of the prevention program. Further, as  suggested in practices for complex service acquisitions, because the  parties to the contract agree in advance to outcome measures and  associated payments, there is less room for interpretation once the  contract is being implemented.
We have previously reported that a balanced set of performance goals  and measures can help organizations address varied aspects of program  performance. In some of the PFS cases in our study, the parties  included a range of performance measures in the outcome payment  structure to more effectively serve the target population. For example, the  New York project included performance measures on both reducing  recidivism and increasing employment for individuals who were recently  released from prison. According to a detailed summary of the project,  formerly incarcerated individuals face numerous challenges, including  barriers to finding a job upon release. Stable employment can be an  important factor in success upon release from prison.
The outcome payment schedules also create financial incentives for  investors and service providers to improve the outcomes for the target  population. Stakeholders told us that multiple outcome measures can be  a tool to incentivize service providers to address various participant  needs. For example, the London Homelessness Project includes outcome  measures to target the different causes of homelessness of individuals in  the cohort. One of the five measures is related to supporting individuals’  progress toward employment, and another outcome measure is related to  moving individuals into stable housing. Outcome payment structures with  higher payouts at the end of the project create incentives for investors  and service providers not to terminate the contract early because they will  receive the majority of their return on investment at the end of the  contract period.
section_title: Challenge of Securing Investors
paragraphs: Stakeholders said that the rates of return offered by PFS projects likely  will not meet market returns available on traditional investments with  similar risk profiles, therefore reducing investor demand. Lower returns  compared to traditional investments could make it more difficult for  governments to attract sufficient capital for PFS projects. Investors  assess the risks of a project to determine whether or not they will invest.  Investors’ valuation of risk is primarily based on several factors related to  managing their financial risk, including evidence of the intervention and  the ability of the service provider to implement it, the outcome payment  structure, and the level of appropriations risk. For example, commercial  investors prefer interventions backed by rigorous evidence and service  providers who have a history of strong performance because it lowers  their financial risk. Governments also manage their financial risk by  determining the outcome payments and rate of return they are willing to  pay to investors. In Utah, the legislation that authorizes the High Quality  Preschool Project included a cap on the return for investors. A Utah  legislator said the cap helps to ensure that the state captures more of the  cost avoidance from reduced use of special education services. The  legislature wanted to ensure they were being good government stewards  because they would not be at the table negotiating PFS transactions. In  other projects, the return on investment was negotiated on a case-by- case basis. For example, in the New York project, there is a cap on the  total outcome payments the state can make over the life of the project. In  the UK, government officials told us that some PFS project outcome  payments and associated rates of return funded by the Department for  Work and Pensions’ (DWP) Innovation Fund were not subject to  negotiation. Instead, DWP developed a rate card, where it defined the  maximum it is willing to pay for specific outcomes (see appendix IV for an  example of a rate card).
In the U.S. projects in our study, parties to the contracts used financing  structures to address challenges in attracting investors, such as a tiered  capital structure or a loan guarantee to secure the upfront capital needed  for the project.
Tiered capital structure. A tiered capital structure, where some investors  take on more risk, can attract bigger pools of capital because it lowers  risk for some investors. In three of our U.S. case illustrations a  combination of commercial and philanthropic investors made up the  capital structure. Stakeholders said that investors’ willingness to accept  risk can vary and that philanthropic investors may be willing to accept  more risk at a lower rate of return than commercial investors. For  example, according to the intermediary in the Massachusetts Juvenile  Justice project, to encourage investment the project capital structure was  split between philanthropic and commercial investors. Commercial  funding partners contributed senior debt, which will be paid out first;  foundations contributed junior debt, which will be paid out following the  senior debt. Philanthropic grants from foundations also served as first- loss capital, which will bear the first loss in the event that the service  provider does not meet the outcome measures. According to the  intermediary in the project, in the first-loss role philanthropic donors can  combine and leverage their grant making with commercial capital.
Loan guarantee. In a different model, a philanthropic investor can  provide a loan guarantee to the primary investor to reduce its risk by  assuming the loan if the service provider does not meet the outcome  targets. For example, in the New York City Rikers Island Project, the city  agreed to a maximum return of $2.1 million which was not high enough  for Goldman Sachs to justify the level of risk it perceived regarding  repayment of its $9.6 million loan. In an effort to balance the risk/reward  trade-off, Bloomberg Philanthropies provided a $7.2 million grant to  MDRC to act as a loan guarantee that would reduce Goldman Sachs’  capital at risk in the event that outcome targets were not achieved. An  investor in the project further explained that although the service provider  had a track record of success, its risk of failure in the NYC Rikers Island  project was higher because it would be implementing a new curriculum in  a difficult environment. The combination of the performance risk and its  status as the first PFS project in the United States were the primary  reasons for the loan guarantee.
section_title: During Project Design, Parties Reported Using PFS Contracts to Establish Governance Structures and Rules for Project Implementation to Manage Risk
paragraphs: Governance rules that define interactions and the authority to make  decisions after the contract has been signed can encourage cooperation  and manage the risk of unexpected termination by any party. For the  projects in our study, the contracts also defined how parties will interact  during the implementation of the project. PFS contracts in our study  included outcome payment structures, oversight structures, governance  rules, and project evaluation mechanisms to manage risk.
Oversight structures. PFS contracts included oversight structures to  manage performance risk during contract implementation. All of the  projects in our study had a governance structure, which defined how  parties would make decisions during implementation of the contract,  including how the project would be managed and overseen. For example,  the New York project had a three-tier governance structure made up of  the following:
An executive steering committee: This committee meets semi- annually to provide strategic direction, monitor compliance with the  agreement, and review reports. Members include the Deputy  Secretary for Civil Rights; the Deputy Secretary for Public Safety; the  Commissioner of Department of Labor (DOL); the Commissioner of  the Department of Corrections and Community Supervision (DOCCS);  CEO of the intermediary; and the Executive Director of the service  provider.
A management committee: The committee meets monthly to monitor  reports and program implementation and provides updates to the  executive steering committee. Members include the Associate  Commissioner of DOCCS; two Deputy Commissioners of DOCCS; the  Deputy Commissioner of DOL; the Executive Director of the service  provider’s regional office; the Director of DOCCS Research; a  representative from the Harvard Kennedy School Social Impact Bond  Technical Assistance Lab; and a director from the intermediary.
Working groups: Working groups meet to discuss operational and  research issues that arise over the course of the project. They are  composed of staff and subject-matter experts from DOCCS; DOCCS  Research; DOL; the service provider; the Harvard Kennedy School  Social Impact Bond Technical Assistance Lab; and the intermediary.
Contract governance rules. Complex contracting practices suggest that  including provisions for how decisions will be made when changes are  needed during service delivery can help address implementation  challenges. To manage various risks and inherent uncertainties, PFS  contracts included governance rules to define how decisions would be  made over the course of the project. PFS contracts included rules to  define decision making after the contract was in place as well as  provisions to address how parties to the contract could be replaced in the  case of underperformance. For example, the New York contract included  governance rules that defined when and how the service provider could  be replaced. The contract specified that any replacement service provider  is subject to approval from the New York Department of Labor and  Department of Corrections and Community Supervision, among others.
PFS contracts also defined the circumstances under which investors  could decide to terminate the project based on the service provider’s  performance. A Massachusetts official said that determining the  provisions that would allow investors to terminate the contract was one of  the most difficult parts of the negotiations. Because the PFS project was  designed for the service provider’s model, Massachusetts wanted to  support the model, which requires a number of years of participation to  achieve outcomes. Massachusetts also wanted to ensure that program  participants would not be left without services as a result of the program’s  early termination. Investors did not want to continue their investment if the  service provider was underperforming. In the contract, the parties agreed  that the lead investor has a right to exit the project at the end of the  second and third years if the service provider’s participant attrition rate  exceeded defined thresholds. The contract also included provisions, such  as the development of a remediation plan, designed to improve the  service provider’s performance if it fell below a defined threshold. These  provisions gave investors the opportunity to reduce their financial risk if  the service provider was underperforming. Allowing for 2 years of  implementation and outcome measurement before investors can exit the  contract also helped mitigate the government’s performance risk because  it allowed enough time for service providers to collect data to measure  outcomes and make adjustments if necessary to improve performance.  Termination provisions also defined what would happen if governments  terminated the contract early. For example, in the NYC Rikers Island  Project, the City of New York could terminate without cause. However, if  the city terminated the contract before the entire evaluation cohort  receives services, it would lose the pay-for-success features of the project  and must reimburse the intermediary for program costs to date.
Stakeholders said that building flexibility into contracts to allow for  negotiations to make operational changes without amending the contract  could help address implementation challenges. Some changes to the  projects required updates to the PFS contract, which were costly and time  intensive because all parties to the contract had to negotiate and agree to  the changes. Stakeholders said that it is important to find a balance  between specifically defining requirements and building sufficient  flexibility into the contract to address challenges during program  implementation. A Massachusetts official said the contract must be  detailed enough to ensure that the program participants are protected and  that risk is properly transferred from the government to other parties.  However, the contract’s terms must also offer the parties a degree of  flexibility to make changes as problems arise, because it is very time  consuming and difficult to amend the contract.
Government budgeting structure. Because appropriations risk was a  key concern of investors, the U.S. cases in our study managed this risk  by establishing different mechanisms to secure funds for outcome  payments:
Massachusetts enacted legislation authorizing the backing of PFS  contract payments with the full faith and credit of the commonwealth  and establishing a Social Innovation Financing Trust Fund into which  annual appropriations may be made.
New York City, if the program had been successful, would have made  outcome payments as it would for any other contractual obligation.  There were two evaluation points that would trigger a possible  payment. The final payment was scheduled to occur 1 year after the  final results, giving the city more time to realize savings.
New York pledged to obtain appropriations to make outcome  payments. The contract includes provisions for an orderly wind down  if the state does not renew the appropriation.
Utah enacted legislation that authorizes a School Readiness  Restricted Account which consists of funds appropriated by the  legislature, federal grant funds, and private donations. Funds in this  account may be used for contracts with independent evaluators,  future outcome payments, and grant awards.
Project evaluation mechanisms. The PFS case studies we examined  used evaluations to manage the governments’ various risks and to ensure  the defined outcomes are achieved. In each case, outcome payments  were contingent upon the evaluator certifying that the agreed-upon  outcomes were met. Four of the evaluations in our case illustrations were  impact evaluations, such as a random control trial (RCT) or a quasi- experimental design, where outcomes for individuals receiving the  intervention were compared to those from a similar control group. Three  of the evaluations used a process to verify that outcomes were being  achieved for the target population. For example, in the Utah project’s  evaluation, the evaluator tests each student participating in the high- quality preschool program at the beginning and end of the school year.  The students who score two standard deviations below the mean on the  pre-test are tracked from kindergarten through sixth grade to verify  whether they are enrolled in special education. The evaluation was  derived from an earlier analysis that showed a decrease in special  education use among students attending the high-quality preschool  program. In the London Homelessness project, service providers submit  evidence for some of the measures to the Greater London Authority,  which is responsible for verifying the attainment of those measures. For  example, for the outcome measure on helping clients into stable housing,  providers submit a form from an individual’s landlord, which the Greater  London Authority verifies and determines whether to make the associated  outcome payment.
Stakeholders said that evaluations can manage potential perverse  incentives, such as creaming. Some projects in our study included a  mechanism to address creaming in the evaluation. For example, the Utah  project on early childhood education addressed the risk of creaming by  only including the children who score two standard deviations below the  mean on the pre-test in the evaluation for outcome payments.
Stakeholders said that evaluations can also help manage perverse  incentives for stakeholders to take actions at the expense of the  individuals receiving treatment in the project. For example, because  outcome payments in the Utah project are made based on students  avoiding special education, there is potentially an incentive to withhold  special education services to increase outcome payments. The Utah  project manages this risk by keeping the identity of the students in the  evaluation cohort confidential, so the teachers who determine special  education needs are not aware of a student’s status in the project. In  another example, a report on the London Homelessness Project stated  that well-defined target populations can reduce the risk of creaming  because the service provider cannot manipulate the population. This  project targets 830 entrenched homeless individuals who had been seen  sleeping rough or who have stayed in a London rough sleeping hostel in  the last 3 months, and who had been rough sleeping at least six times  over the last 2 years.
Stakeholders said that there are challenges to implementing RCT  evaluations for PFS projects. We have previously reported that well- conducted randomized experiments are best suited for assessing  effectiveness when multiple causal influences create uncertainty about  what caused results. Such experiments are often expensive because  they are among the most rigorous forms of evaluation, as well as difficult  or sometimes impossible to carry out. For example, a risk of RCTs is that  the evaluation may be inconclusive if the control group cannot be  maintained and because a large sample size is required to yield  statistically significant results. One stakeholder said that a RCT can  increase the reputational risk to service providers if it is poorly designed  and misses the real outcomes of the program. To manage the risk of  these challenges, stakeholders from several projects identified the need  for an alternative methodology in case the evaluation could not be  implemented due to unanticipated implementation challenges. For  example, in the NYC Rikers Island Project, the initial evaluation plan was  based on a random control trial, but in the jail environment it was not  possible to keep the control and treatment groups separate. Therefore,  project stakeholders designed an alternative methodology, a quasi- experimental design, which provided a sufficient degree of confidence  that the outcomes achieved could be attributed to the program.
Stakeholders also addressed potential ethical concerns associated with  RCTs, such as withholding the service intervention from a population that  could potentially benefit from the services in order to maintain a control  group for an impact evaluation. While there could be ethical concerns  associated with using a control group for a random control trial evaluation  of a service, limited availability of funding often prevents treating the  entire population. Therefore, stakeholders indicated they could possibly  draw from patients on a wait list for the program to serve as a control  group. The Massachusetts project addressed ethical concerns with RCTs  by including a “no denial of service policy” if an individual in the  nonparticipating control group asked to participate in the program. This  design required a significant increase in the evaluation’s sample size to  address potential contamination between the treatment group and the  control group in the sample.
section_title: Ongoing PFS Contracts Show Benefits of Data- Driven Performance Management, but Final Outcomes Are Not Yet Known
paragraphs: During the third phase of a PFS project, service providers implement the  prevention program. To manage performance risk, participating  organizations provide oversight, monitor performance, and evaluate the  extent to which outcomes are achieved, based on the structures and  governance roles established in PFS contracts.
subsections: section_title: Implement Prevention Program Using Data-Driven Performance Management
paragraphs: Actively managing performance to achieve outcomes during contract  implementation is vital to achieving results. While the structures and  membership varied, stakeholders said PFS oversight bodies regularly  reviewed performance data during service delivery. According to  stakeholders, intermediaries and investors can bring performance  management expertise to service providers and provide a rigorous focus  on performance management and accountability. Officials from the  intermediary for the New York City project said they continued to monitor  the implementation of the service delivery using metrics that predict  whether the project will be successful. They said that these metrics and  targets allowed them to troubleshoot project implementation and  intervene early to modify elements of the program that were not working.  For example, there is a community service element to the Moral  Reconation Therapy program that was modified to work in the Rikers  Island jail environment. In another example, officials from the intermediary  for the Peterborough Project said they used data analytics and provided  program management support to help improve service delivery. In 2014  the UK Department of Work and Pensions evaluated its Innovation Fund  PFS projects and found that for its first 10 PFS projects the strong focus  on monitoring and managing project performance built capacity among  service providers and helped to increase performance levels and improve  efficiency of service delivery.
Social service providers often collect and report input and output data in  response to multiple grant requirements, but these data may not always  be useful for monitoring the performance of a PFS project in achieving  outcomes. Service providers said the PFS model may require new  methods to collect and use performance information that shows the  program is being implemented as planned and with fidelity to the model  that was shown to be successful. Some service providers may also need  to invest in building their capacity to collect and report outcome data to  participate in PFS projects. An official from one service provider noted  that her organization invested in data entry and data analyst positions and  has a team that collects, analyzes, and processes data that it submits to  the intermediary.
Projects in our study faced unanticipated challenges in delivering services  and stakeholders said oversight mechanisms helped to identify issues  quickly and facilitated decision making to address problems. For example,  stakeholders in our study reported difficulty in identifying cases for  referrals to the service provider. Maintaining the planned level of referrals  was important because the evaluation and outcome payments were  dependent on the level of participation in the project. An official from the  service provider in the New York project said that approximately 3 to 4  months into program implementation, there was a dip in the number of  potential participants referred to the program. He said the cause was that  the service provider was seeking referrals from parole bureaus where it  had traditionally drawn few participants in order to maintain the project’s  randomized referral and evaluation plan. However, he said this strategy  depressed the overall number of referrals. As a result, they decided to  expand the number of parole bureaus that could serve as a source of  referrals.
Stakeholders said that pilot periods for service delivery before the  evaluation starts could reduce unanticipated challenges during PFS  implementation and reduce risks that service providers will not meet  outcome targets. According to a report on the New York City project, it  may be beneficial to pilot-test a program at full scale and only evaluate it  after it achieves a steady state of operation. On the other hand, pilot  programs require time and money, both of which are likely to be  constrained for PFS projects. The New York City project included a pilot  test conducted at half scale. Stakeholders said that although the  experience was valuable, certain issues came to light only after the  program expanded to full scale. A pilot project can also be used to build  evidence on an intervention to develop a future PFS project. For example,  the Asthma Impact Model for Fresno is implementing a pilot project from  spring 2013 through early 2016. The pilot project includes a random  control trial evaluation on the intervention to demonstrate the outcomes of  a home-based asthma management program. If successful, the results of  the pilot will be used to develop a PFS strategy.
section_title: Evaluate Outcomes and Next Steps
paragraphs: Following program implementation, the parties to a PFS project must  evaluate whether the service provider achieved outcome measures and  determine whether the government must make outcome payments based  on the results. PFS contracts we examined spanned 3 to 10 years and  one project—New York City—had been completed as of the time of our  review. Results for the New York City project were reported in July 2015  and indicated that the program did not reduce recidivism among  adolescents at Rikers Island and therefore did not meet the  predetermined threshold of success: a 10 percent reduction in recidivism.  As a result, New York City did not pay for the program and the  intervention was discontinued as of August 31, 2015. Results of the  evaluation of the initial cohort of the Peterborough Project were reported  in August 2014 and indicated that reoffending rates for participants were  reduced by 8.39 percent compared to a control group. A reduction of at  least 10 percent within the cohort was necessary to trigger an outcome  payment to investors, so no payment was made. However, the project’s  terms state that outcome payments will be made if reoffending rates are  reduced for each individual cohort by 10 percent or across all cohorts by  7.5 percent. Therefore, if the evaluation to be conducted following the  second cohort finds that reoffending rates across both cohorts were  reduced by at least 7.5 percent, the Ministry of Justice will make outcome  payments to investors.
We asked participating organizations what will happen to their programs  at the end of their PFS contracts. Some told us that governments may  consider funding the service delivery directly. In another case, the It’s All  About Me adoption project has been designed so that if local  governments use the service in large numbers over 10 years, the project  will amass sufficient reserves after repaying investors to self-finance the  service in the future. While PFS financial models were designed based on  future government savings, some government stakeholders indicated  they were not planning to reduce future agency budgets if PFS outcomes  are achieved, but will reallocate the savings to address other unmet  needs in the same agency budget.
section_title: The Federal Government Could Play a Role in Funding, Building Capacity for, and Managing Investor Risk in PFS, but a Formal Mechanism to Collaborate and Share Lessons Learned Does Not Exist
subsections: section_title: The Federal Government Could Play Several Roles to Address PFS Challenges at the State and Local Level
paragraphs: While the effectiveness of the PFS model remains to be seen,  stakeholders cited three potential roles the federal government could play  to help address challenges at the state and local levels of government  and further develop the PFS field: providing outcome payments, building  capacity, and providing loan guarantees.
Providing outcome payments. State and federal officials and other  stakeholders said the federal government could provide outcome  payments in PFS projects to address the wrong pockets problem, which  could arise when an intervention results in savings to the federal  government as well as the government commissioning the project. This  issue makes it challenging for state and local governments to have  sufficient funds available to make outcome payments. In our case studies,  some officials said that a key consideration when assessing project  feasibility was whether cost savings to their jurisdiction were sufficient to  justify the cost of implementing the project when other jurisdictions, such  as the federal government, shared in the costs and savings as well. To  date, the federal government has not become a formal party to any PFS  contract in which the federal government would share in the project’s  savings. However, the federal government has provided grants to support  outcome payments for two of the case studies we examined. To assist  New York and Massachusetts with their recidivism PFS projects, DOL  awarded nearly $24 million in grant funding to support outcome payments  for these projects. A stakeholder said that the DOL grant catalyzed the  New York project, allowing them to fund outcome payments for the first  cohort of 1,000 participants. Having support for outcome payments  helped Massachusetts officials increase the number of participants in the  project, thus allowing for a stronger evaluation of the program’s  outcomes.
The UK government also took steps to mitigate challenges related to the  wrong pockets issue. For example, the £20 million (approximately $31  million) Social Outcomes Fund, managed by the Cabinet Office, was  created to address challenges related to public sector spending silos that  occur between agencies in central and local governments trying to  address complex social issues. The Social Outcomes Fund provides  outcome payments for government departments, local authorities, and  other organizations to implement innovative projects to address social  problems where savings could be spread among several government  agencies. This approach addresses the difficulty faced by a single office  attempting to justify the cost of implementing the program. The federal  government could also commission PFS projects and provide outcome  payments for programs where it would capture all of the savings, such as  programs targeting Social Security Disability Insurance.
Similar to state and local governments, as an outcome payor the federal  government could face many of the same potential benefits, risks, and  decisions regarding project feasibility identified by stakeholders in our  study. Playing the role of outcome payor in PFS projects could provide  benefits from potential future savings to the federal government if  outcomes are achieved. For example, a stakeholder said a potential area  where the federal government could play a role as outcome payor is in  the provision of nonmedical asthma management services including in- home education, environmental assessment, and remediation such as  mold removal, which could potentially reduce the number of asthma  attacks and emergency visits by Medicaid recipients. Reducing  emergency room visits paid by the Medicaid program could yield savings  to both the state and federal governments. However, as an outcome  payor, federal agencies would need to manage financial and performance  risks similar to state and local governments. Federal agencies  considering entering a PFS contract as an outcome payor would need to  assess the feasibility of individual projects, including the factors  stakeholders in our study identified, such as evidence on the intervention,  cost-benefit analysis of the project, and the commitment of government  leadership.
Another potential challenge for federal agencies to consider is whether  their existing grant program authority can accommodate outcome  payments. PFS projects can span 3 to 10 years, while grant program  authorities may limit the duration of a grant project. For example, DOJ  officials told us that the reentry grant program authorities under the  Second Chance Act limited the grant performance period to 12 months.  This restriction would have prohibited the use of a PFS project as no  outcomes could be achieved in such a short project period. However, this  time limit was later eliminated for grants funded by the appropriation  made available for DOJ’s PFS efforts for fiscal years 2014 and 2015.  Another issue for agencies to consider is the period of availability of the  funds and what happens to funds obligated for outcome payments if they  are not made because the service provider does not meet the outcome  targets. DOL officials told us that the grant funds obligated for outcome  payments will be deobligated and unavailable for new obligations if  outcome targets are not met in the New York or Massachusetts PFS  projects. In another example, CNCS’s fiscal years 2014 and 2015  appropriations include authority that allows CNCS to reobligate funds to  other authorized projects if outcome targets are not met.
Building capacity for PFS projects. According to stakeholders, the  federal government could make grants for feasibility studies and project  development to build the capacity of state and local governments,  intermediaries, and service providers, and offset administrative costs. In  fiscal year 2014, CNCS awarded $12 million in grants to eight  intermediaries to help governments and nonprofits build their capacity to  implement and structure PFS projects. As of July 2015, the initial eight  intermediaries had made awards to a total of 48 state and local  governments and nonprofit organizations to fund technical assistance on  PFS project development. According to CNCS, the grants address the  limited availability of funding for planning, feasibility studies, and deal  structuring, which has constrained growth in the field. In another example,  DOJ and HUD are jointly pursuing a PFS demonstration project on  permanent supportive housing for the prisoner reentry population. The  project will make grants to intermediary organizations that will work with  state and local governments and service providers to implement PFS  projects and will include an evaluation to determine if this model of  delivering supportive housing can achieve savings by decreasing  government spending on corrections systems, homeless services,  Medicaid, and crisis care services, some of which are federally funded.
The UK central government addressed capacity issues by creating  organizations to manage grant making and technical assistance for  organizations involved in PFS projects. Through its Commissioning Better  Outcomes Fund, the Big Lottery Fund provides up to £6 million  (approximately $9 million) in grants (up to a maximum of £150,000  (approximately $229,000) per project) for technical assistance to assess  the feasibility of PFS projects. It also provides outcome payments to  encourage and sustain government engagement. The Better Outcomes  Fund seeks to increase the capacity of service providers, enhance  understanding of how to organize and implement a successful PFS  project, and increase the knowledge base of local government officials  working with PFS projects in the field. A stakeholder from the Centre for  Social Impact Bonds said these grants have helped improve the technical  and financial capacity of key players in PFS projects.
State and federal officials also said that the federal government is  uniquely positioned to help build capacity by encouraging knowledge  sharing among parties that are implementing PFS projects. For example,  to share knowledge with its grantees, DOL leads a working group for the  state and local government recipients of PFS grant awards in an effort to  increase knowledge and share best practices. DOL officials said that the  monthly meetings have included speakers with PFS expertise, such as  officials from Social Finance, the Peterborough Project, and other federal  agencies. In addition, the White House Office of Social Innovation and  Civic Participation co-hosted, with the Arnold Foundation and the  Nonprofit Finance Fund, a series of PFS Summits in late 2014 and early  2015 to highlight and advance regional activity underway across the  country, build the longer term Pay for Success infrastructure to catalyze  future projects, and facilitate an open, transparent dialogue among public,  private, and philanthropic stakeholders. Officials from one state  government said that knowledge sharing by the federal agencies has  been helpful to the state as it develops and implements PFS projects.
UK government officials also said that knowledge sharing was an  important component of the growth of PFS. As part of the annual  reporting requirements for PFS grants, the Big Lottery Fund requires that  grantees inform officials of plans to communicate information on their  learning and experience as the project progresses. Where the Big Lottery  Fund supports outcome payments, projects are also required to evaluate  their own performance, contribute to an independent program evaluation,  and support dissemination of learning from these evaluations. In the UK,  officials from the Centre for Social Impact Bonds said the Centre has also  addressed knowledge sharing issues by coordinating meetings with other  government organizations, holding workshops, networking, and making  connections between interested parties.
Providing loan guarantees. A challenge to PFS projects that could limit  growth is obtaining capital from investors. Investors said that the federal  government could provide loan guarantees or credit enhancements for  PFS projects to increase the availability of capital by reducing risk for  investors. Stakeholders said that having the backing of the federal  government would provide greater incentives for investment in PFS  projects and help establish the market. For example, Bloomberg  Philanthropies provided a $7.2 million loan guarantee for the New York  City Riker’s Island project to reduce some of the risk to investors of losing  their investment if outcome targets were not achieved. However, some  stakeholders said a government loan guarantee would negate a primary  benefit of PFS to transfer the risk of program implementation away from  the government. Such guarantees provided by the federal government  would potentially leave the federal government liable for paying for a  project that did not achieve agreed-upon outcomes.
section_title: There Is No Formal Mechanism for Federal Agencies to Collaborate on PFS
paragraphs: In our prior work, we stated that collaborative mechanisms, such as  interagency groups or collaboration technology, can be used to develop  policies, implement programs, and share information. The agencies in  our study participated in an interagency PFS working group that met  between 2012 and 2014 to share information on agency efforts related to  PFS. Federal officials said the working group is no longer meeting  because of staff turnover in leadership positions. Officials who  participated in the working group said it was beneficial and furthered their  agencies’ efforts related to PFS. For example, a federal agency official  said the PFS working group helped officials network with their peers and  collaborate on specific efforts. When CNCS developed its fiscal year 2014  PFS grant notice of funding availability, it consulted with other federal  agency officials with PFS expertise. Though officials found these  meetings useful, they were not maintained with a regular schedule and  information resulting from meetings was not documented or broadly  disseminated. In May 2015, OMB officials noted that the White House  and federal agency officials are working to restart interagency meetings  on PFS.
A mechanism for agencies to share knowledge and information on PFS  could give agencies interested in potential PFS projects access to leading  practices and current information on how federal participation is evolving.
In addition, a collaborative mechanism would allow agencies to share  information on more technical aspects of PFS, such as practices on how  to evaluate savings that accrue to federal agencies as the result of  successful PFS efforts. Officials from one federal agency said that  information about the various roles that federal agencies have played in  the PFS field would be informative, given that the approach to supporting  PFS has varied by agency. Having a central mechanism for sharing  information on PFS would also decrease the risk of duplicative efforts in  projects that affect multiple agencies.
section_title: Conclusions
paragraphs: The PFS model is evolving as one among many tools for policymakers to  consider when seeking to provide solutions for some of the nation’s most  vexing social problems. According to stakeholders, PFS allows  governments to shift the focus of social service interventions from costly  remediation to lower cost prevention, while simultaneously mitigating risks  associated with scale-up by funding projects with private investment. For  service providers, the PFS model provides a source of upfront capital for  operations, as well as an opportunity to scale-up promising programs and  refine their capacity for sophisticated data collection and analysis.
As organizations consider pursuing PFS contracts, they will need to think  about how they will manage the risks inherent in PFS contracting. For  example, a government’s costs of implementing a PFS project, which can  include investor returns, management fees, and evaluation costs, can be  high, so the government must decide whether potential benefits outweigh  these costs. Flexibilities built into PFS contracts can help all parties to the  project manage risks by allowing them to make changes as  implementation and evaluation methodologies evolve. In addition, the  parties to a project will need to manage the potential risks that unintended  consequences will negatively affect the population receiving treatment.  Managing these risks requires rigorous design of project implementation  and evaluation methodology to minimize perverse incentives for  organizations to focus treatment on the easiest cases while avoiding the  more challenging ones that could negatively affect outcomes. PFS is  most likely an appropriate tool for prevention programs that address  entrenched social problems. It is important to consider key project  feasibility factors, such as the availability of sufficient evidence on an  intervention’s effectiveness and the capacity of PFS stakeholders to  collect and analyze the data required by these complex projects.
The PFS model is evolving, and early programs are just beginning to  evaluate results. It is too soon to tell if governments will realize the  benefits that induced them to try PFS. For example, a potential benefit of  PFS is that it allows a government to shift the risk of implementing an  innovative program to investors. In practice, investors whose return on  investment is contingent on positive results may prefer projects that are  based on rigorous evidence of success and may avoid innovative  approaches that have not been rigorously tested. If this potential flight to  programs with a strong evidence base turns out to become reality, it may  not make sense for governments to rely on PFS projects. Instead, they  may consider funding these types of programs directly, through traditional  performance-based contracts that incorporate features of PFS projects  that reduce the government’s risk, such as independent evaluation and  governance rules that allow for strong management and oversight. By  undertaking a PFS project to implement a program that is known to be  successful, a government could be taking on extraneous costs for little or  no benefit.
As the PFS field continues to grow and as Congress and federal agencies  consider new investments in these projects, important considerations  remain about the appropriate role for the federal government. Similar to  state and local governments, the federal government faces potential  benefits and challenges stemming from participation in PFS programs. As  federal agencies consider expanding their involvement in PFS, it  becomes increasingly important for officials at all levels of government to  collaborate to share knowledge and experiences.
OMB led an informal working group on PFS for a limited period. Despite  the stated usefulness of this group, notes and findings from the meetings  were not shared among agencies, and the meetings were eventually  discontinued. Having a mechanism for federal collaboration on PFS  projects as the field grows would allow agencies to leverage the  experience of early federal actors in the PFS field and would decrease  the potential for missteps in developing projects due to information gaps  and failure to learn from experience with this evolving tool of government.
section_title: Recommendation for Executive Action
paragraphs: To identify and broadly disseminate information on leading practices and  lessons learned, the Director of OMB should establish a formal means for  federal agencies to collaborate on PFS. This could include creating a  formal working group and providing collaboration technologies, such as  shared databases or web portals.
section_title: Agency Comments and Our Evaluation
paragraphs: We provided a draft of this report to the Director of the Office of  Management and Budget, the Secretaries of the Treasury, Labor,  Housing and Urban Development, Justice, Health and Human Services,  and Education, and the Chief Operating Officer of the Corporation for  National and Community Service.
The Office of Management and Budget (OMB) provided written comments  concurring with our recommendation and is working with agencies to  explore options for continued collaboration on Pay for Success. The  response from OMB is reproduced in appendix V. The Departments of  Labor, Housing and Urban Development, Health and Human Services,  Education, and the Corporation for National and Community Service  provided technical comments that were incorporated into the draft as  appropriate. The Departments of Treasury and Justice had no comments  on the draft report.
We are sending copies of this report to the Director of OMB, the  Secretaries of the Treasury, Labor, Housing and Urban Development,  Justice, Health and Human Services, and Education, and the Chief  Executive Officer of CNCS.
If you or your staff have any questions about this report, please contact  me at (202) 512-6806 or sagerm@gao.gov. Contact points for our Offices  of Congressional Relations and Public Affairs may be found on the last  page of this report. GAO staff who made key contributions to this report  are listed in appendix VI.
section_title: Appendix I: Objectives, Scope, and Methodology
paragraphs: This report responds to a request that we examine Pay for Success (PFS)  to help Congress gain a better understanding of how the mechanism has  been defined in theory and practice and what is known about its  implementation and results to date. We examined (1) how selected PFS  projects have been structured and what potential benefits these projects  can provide; (2) how selected PFS contracts have been structured to  address potential project risks; and (3) the potential roles for the federal  government’s involvement in PFS projects.
To address these objectives, we began by identifying and reviewing  relevant literature on PFS, including work produced by public sector,  nonprofit, academic, and private sector organizations. We also  interviewed individuals with public management expertise who had  examined PFS and published work on or related to PFS. We identified  some of these individuals through our literature review, while others were  referred to us by other experts. These individuals were Rob Duggar of  ReadyNation; Mildred Warner of Cornell University; Leonard Gilroy of the  Reason Foundation; John Roman of the Urban Institute; Jitinder Kohli of  Deloitte Consulting LLP; and Shar Habibi of In the Public Interest. During  these interviews we asked the interviewees, among other things, about  the potential benefits and challenges of PFS; the advantages and  disadvantages that PFS has compared to other program funding  mechanisms; and the potential roles that the federal government could  play in PFS projects.
Based on our literature review and our interviews with individuals  knowledgeable about PFS, we identified jurisdictions in the United States  and abroad that were implementing, planning to implement, or studying  the potential of PFS projects. We then selected seven U.S. and three  United Kingdom (UK) case illustrations to review in depth. We selected  cases from the UK in order to capture the country’s experience as the first  to develop and implement PFS projects as well as because it is the  location with the longest experience with this financing mechanism. We  selected four PFS projects that were being implemented in the United  States at the time we began our engagement in July 2014. Those projects  were the Massachusetts Juvenile Justice PFS Initiative; the New York  City Rikers Island Project; the New York PFS project: Increasing  Employment and Improving Public Safety; and the Utah High Quality  Preschool Program. Because those cases focused heavily on reducing  recidivism and improving early childhood education, we selected two  additional U.S. projects that were in development to reflect an additional  policy area (improving health outcomes). Those projects were the South  Carolina Maternal Health PFS Initiative and the Asthma Impact Model for  Fresno (AIM4Fresno). Finally, we selected one case in which a state  considered but ultimately decided not to move forward with a PFS project.  That case was Maryland’s review of a potential PFS initiative focused on  reducing recidivism. For the UK projects, we selected three projects that  were also in implementation and varied in issue area, structure, and the  participating organization that conceived them. Those cases were the  Peterborough Prison Project; the London Homelessness Project; and the  It’s All About Me Adoption Project, on reducing adoption wait times.
To obtain information about each of these case illustrations, we  conducted site visits to Boston, Massachusetts; New York City, New  York; Albany, New York; Annapolis, Maryland; Salt Lake City, Utah; and  London, United Kingdom. During these visits, we requested and reviewed  documents relevant to the case illustrations, including feasibility studies,  contracts, and evaluation reports. We conducted in-depth interviews with  representatives of organizations participating in the PFS projects in these  locations. For each case, as appropriate, we interviewed representatives  from government entities, intermediaries, service providers, investors,  evaluators, and validators. We conducted some of the interviews by  telephone before or after site visits due to scheduling conflicts. During the  interviews, we asked the interviewees, among other things, about the  potential benefits and risks of PFS; the advantages and disadvantages  that PFS has compared to other program funding mechanisms; the  specific roles their organizations play in PFS projects; the PFS project  contract negotiations, including the establishment of outcome payment  and governance structures; and the potential roles that the U.S. federal  government or UK national government could play in PFS projects.
The interviewees for our case illustrations were representatives from:  Bank of America Merrill Lynch; Bates Wells Braithwaite; Big Lottery Fund;  Big Society Capital; Bloomberg Philanthropies; The Boston Foundation;  Bridges Ventures; Center for Employment Opportunities; Chesapeake  Research Associates; Collective Health; Esmee Fairbairn Foundation;  Goldman Sachs; Greater London Authority; Granite School District  Preschool Services; Harvard University’s Social Impact Bond Technical  Assistance Lab; Imprint Capital; Maryland Department of Legislative  Services; Massachusetts Executive Office of Administration and Finance;  MDRC; New Profit Inc.; New York City Mayor’s Office; New York  Department of Corrections and Community Supervision; New York  Division of the Budget; The Osborne Association; Public Consulting  Group; Roca Inc.; The Robin Hood Foundation; Rockefeller Foundation;  Salt Lake County Mayor’s Office; Sibayltics LLC; Social Finance (UK);  Social Finance (U.S.); St. Giles Trust; Thames Reach; Third Sector  Capital Partners; UK Cabinet Office; UK Department for Communities and  Local Government; UK Ministry of Justice; United Way of Salt Lake; Utah  Governor’s Office of Management and Budget; Utah House of  Representatives; Utah School Readiness Board; Utah State University’s  Early Intervention Research Institute; Vera Institute of Justice; and Voices  for Utah Children.
To assess the PFS contract structure, we reviewed our prior reports and  contracting literature to identify contracting practices from the federal  experiences relevant to PFS contracts, which are complex service  acquisitions. We compared the contracting practices to the PFS contracts  in our study to establish their relevance to the design and implementation  of PFS contracts.
Because the scope of our review was to assess the potential benefits and  risks of PFS and the structure of PFS projects, we did not assess whether  the outcomes established for specific projects and any cost savings that  were identified with their attainment were appropriate, ambitious, or met  other dimensions of quality. In addition, since most of the case  illustrations we reviewed were in early stages of implementation, we did  not request any performance information related to them and did not  assess the likelihood of the projects successfully achieving outcomes.
To obtain information about federal agencies’ involvement in PFS, we  reviewed relevant documentation, including the President’s Budget for  fiscal years 2012 through 2016. Through this review, we identified eight  federal agencies that were involved in an informal working group and  played different roles vis-à-vis PFS. The Office of Management and  Budget, working with the White House Office of Social Innovation and  Civic Participation, has provided informal guidance on PFS efforts, and  participated in the informal working group. The Department of the  Treasury participated in the informal working group. The Departments of  Labor (DOL), Housing and Urban Development (HUD), and Justice  (DOJ), and the Corporation for National and Community Service (CNCS)  have funded or plan to fund PFS projects. The Departments of Health and  Human Services (HHS) and Education (Education) have been exploring  opportunities to pursue the PFS concept. We interviewed officials from  each of these agencies about their involvement in PFS, and asked them,  among other things, about their strategic approaches for using PFS; the  advantages and disadvantages that PFS presents compared to other  financing tools; the amount of funding they allocated to PFS; interagency  collaboration on PFS; and early lessons learned from their involvement in  PFS.
Furthermore, of the eight agencies involved in the informal working group  on PFS, we interviewed officials and reviewed budget documents from six  agencies that typically fund social programs—CNCS, DOJ, DOL,  Education, HHS, and HUD—to identify illustrative examples of how these  agencies have used evidence-based approaches in their decision making  more generally.
We conducted our work from July 2014 to September 2015 in accordance  with generally accepted government auditing standards. Those standards  require that we plan and perform the audit to obtain sufficient, appropriate  evidence to provide a reasonable basis for our findings and conclusions  based on our audit objectives. We believe that the evidence obtained  provides a reasonable basis for our findings and conclusions based on  our audit objectives.
section_title: Appendix II: Illustrative Examples of Evidence-Based Programs from Selected Federal Agencies
paragraphs: To provide examples of how federal agencies are using evidence to  inform decision making, we asked the agencies in our study to provide us  with examples of existing programs or program proposals that were  consistent with Office of Management and Budget (OMB) guidance on  areas in which agencies could improve their use of evidence.
section_title: Appendix III: Overview of PFS Projects in Our Study
subsections: section_title: Investment
paragraphs: $9.6 million loan from  Goldman Sachs. 75 percent  of the loan is guaranteed by  Bloomberg Philanthropies. $16.11 million, including a $8  million loan from Goldman  Sachs and $8.11 million in  funding from various  philanthropic organizations. $13.5 million equity  investment from 40+  investors and foundations.  10 percent of the  investment is guaranteed by  the Rockefeller Foundation. $7 million, including a  $4.6 million loan from  Goldman Sachs and a  $2.4 million subordinate  loan from J.B. Pritzker  Foundation.
section_title: Size of target population Stakeholders •
paragraphs: 5 government offices  (U.S. Department of  Labor, NY Governor’s  Office, the NY Division  of the Budget, NY  Department of Labor,  and NY Department of  Corrections and  Community  Supervision);   Intermediary (Social  Finance)  Service provider  (Center for  Employment  Opportunities)
2 government agencies  (NYC Mayor’s Office and  Department of  Corrections)   Intermediary (MDRC)   2 service providers  (Osborne Association  and Friends of Island  Academy)   1 private-sector  investor (Goldman  Sachs) and 1  philanthropic loan  guarantor (Bloomberg  Philanthropies)   Evaluator (Vera Institute  of Justice)
12 government  agencies; (U.S.  Department of Labor, MA  Executive Office of  Administration and  Finance, MA Office of the  Commissioner of  Probation, and MA  Department of Youth  Services, MA Department  of Corrections; the MA  Parole Board; the Suffolk  County Sheriff’s  Department; the  Middlesex County Sheriff’s  Department; the Hampden  County Sheriff’s  Department; and the  Essex County Sheriff’s  Department.  Also, data  for evaluation and  validation come from the  Executive Office for Labor  and Workforce  Development (wage data)  and the Department of  Criminal Justice  Information Services  (criminal records) investors) and 3  philanthropic  investors (Robin Hood  Foundation,  Rockefeller  Foundation, and Laura  and John Arnold  Foundation)
2 government  offices (Salt Lake  County and the  Utah School  Readiness Board)   Intermediary  (United Way of Salt  Lake)  6 service  providers   1 private-sector  investor (Goldman  Sachs) and 1  philanthropic  investor (J.B.  Pritzker  Foundation)  Evaluator (Utah  State University)
section_title: New York City (NYC) Rikers Island Project
paragraphs: Evaluator (NY  Department of Labor  Research and NY  Department of  Corrections and  Community  Supervision Research)  Validator (Chesapeake  Research Associates)
Intermediary (Third  Sector Capital Partners)   Service provider (Roca)   1 private-sector investor  (Goldman Sachs) and 5  philanthropic investors   Fiscal services provider  (New Profit)   Evaluator (Urban  Institute)   Validator (Public  Consulting Group)
section_title: Number of outcome metrics Evaluation method
section_title: Range of outcome payments First outcome payment due
section_title: Government budget structure
paragraphs: No new budget structure for  outcome payments.
The state requests an annual  appropriation for outcome  payments and backs future  payments with its full faith and  credit.
The state is renewing a 2- year appropriation every  year.
Legislation created a  special account into  which an appropriation  to cover outcome  payments was placed.
section_title: Oversight structure
paragraphs: Intermediary was responsible  for project management,  including making adjustments  to service delivery and  coordinating with and  reporting to other program  partners.
Provided by operating and  oversight committees,  composed of the project’s  stakeholders.
Provided by the executive  steering committee, which  provides strategic guidance,  the management  committee, which provides  operational management,  and working groups, which  provide implementation  support.
Provided by the Utah  School Readiness  Board, which receives  and reviews regular  operational reporting  from the intermediary.
section_title: Appendix IV: Example of United Kingdom Rate Card for Social Impact Bonds
paragraphs: Government officials from the United Kingdom (UK) told us that the rate  of return for Social Impact Bonds funded with the Department for Work  and Pensions’ (DWP) Innovation Fund are not subject to individual  negotiations. The Innovation Fund is a pilot initiative that has  commissioned Social Impact Bonds to help young people who are  disadvantaged participate and succeed in education or training, therefore  improving their employability and reducing their dependency on benefits.  The Innovation Fund is 100 percent outcome based. Investors or  intermediaries work with service providers to bid on Social Impact Bond  projects based on the predetermined outcome payments published in  DWP’s rate card (see figure 11). For example, if a service provider meets  an outcome on an individual’s sustained employment, DWP pays £2,000  ($3,000). DWP makes awards for Social Impact Bond projects through a  competitive bidding process. DWP will pay the investors or intermediaries  solely on the basis of outcomes achieved. The service providers receive  funding from the investors or intermediaries to cover their delivery costs,  so they do not have to take the risk of depending on outcome payments.
section_title: Appendix V: Comments from the Office of Management and Budget
section_title: Appendix VI: GAO Contact and Staff Acknowledgments
subsections: section_title: GAO Contact
section_title: Staff Acknowledgments
paragraphs: In addition to the above contact, Elizabeth Curda (Acting Director) and  Barbara Lancaster supervised this review and the development of the  resulting report. Virginia Chanley, Karin Fangman, Sherrice Kerns, Donna  Miller, and Daniel Webb made significant contributions to this report. In  addition, Joy Booth, Alicia Cackley, Jeffrey DeMarco, Deirdre Duffy, Gale  Harris, Katherine Iritani, Shelby Kain, Adam Miles, Susan Offutt, Erin  Saunders-Rath, Paul Schmidt, and William Woods made key  contributions.