Current barriers to and facilities of undertaking rigorous impact evaluations of Social Impact Bonds (SIBs) - a stakeholder-informed systematic review and thematic synthesis

Background: Social Impact Bonds (SIBs) (an investor backed form of payment-by-results), have attracted considerable political attention and capital since 2010. In the May 2010 General Election, the opposition Conservative Party included SIBs in their manifesto “Big Society, Not Big Government”. They promoted the use of SIBs as a way to achieve value for money and innovation, announcing that they would establish a social investment bank (Big Society Capital) to invest in innovative products such as SIBs (Conservative Party 2010). Since then, 89 SIBs have been launched in 20 countries, with a total investment of over 322 million USD (Social Finance 2017). Despite the political popularity, evaluation practices have differed significantly. In an initial scoping study, the reviewer identified that few SIBs have been subject to detailed evaluation. Many SIBs have not monitored performance against a robust counterfactual and there is a lack of consistency in the processes through which outcome payments are decided.
Objectives: This review aimed to identify the factors that act as barriers to or facilitators of conducting or being involved in rigorous impact evaluations of the efficacy of individual (SIB-funded) programmes. Evaluation of the efficacy of SIB commissioning versus other types of commissioning was outside the scope of this review.
Methods: To identify empirical literature that met the inclusion criteria, the reviewer searched 13 electronic databases, 91 grey literature sources and hand searched 33 academic articles. 92 experts were contacted, to identify additional articles. To enhance the usefulness of this research and minimise biases, this review was informed by an Expert Advisory Panel (10 experts). Prior to establishing the Panel, the project was accepted for ethical approval (CUREC_1A). Included studies needed to be primary research (any research design) or systematic reviews that focus on the design and/or implementation of SIBs and investigate barriers to or facilitators of undertaking or being involved in rigorous impact evaluations of SIBs (as identified by practitioners). Included studies were subject to quality appraisal. To generate new knowledge, a thematic synthesis of qualitative research was conducted. Barriers and facilitators were categorised into descriptive and analytical themes, which were generated inductively.
Results: 5,567 studies were retrieved for title and abstract review. 5,448 studies were excluded; 119 were retrieved for full text review. Of these studies, 25 were included and 94 were excluded. Included studies reported a range of factors which act as barriers to and/or facilitators of undertaking or being involved in rigorous impact evaluations of SIBs. The most frequently reported barriers included: 1) attribution challenges (e.g small sample sizes, no theory of change, overlap of services provided etc); 2) challenges in identifying appropriate outcome metrics; 3) lack of access to required data; 4) lack of financial resources; and 5) differing stakeholder incentives, views and needs. The most frequently reported facilitators included: 1) access to expertise in evaluation, design and data management; 2) access to required data; 3) availability of comparable counterfactual/control group; 4) early and ongoing stakeholder consultation to refine and test outcome metrics; and 5) regular monitoring and performance management.

Keywords: thematic synthesis
Social Impact Bonds
evidence based practice
SIBs
impact evaluation
outcomes based commissioning
systematic review
Development Impact Bonds
DIBs
payment-by- results
impact measurement
Advisor: Humphreys, D