Executive Summary: 

The SIF’s Impact on 

Strengthening 

Organizational Capacity 

A product of the National Assessment of 

the Social Innovation Fund (SIF) 

September 2015 
This report is in the public domain. Authorization to reproduce it in whole or in part is granted. Upon 

request, this material will be made available in alternative formats for people with disabilities. 

Executive Summary: 

The SIF’s Impact on Strengthening Organizational Capacity

September 2015 

Authors 

Xiaodong Zhang, Ph.D. 

Janet Griffith, Ph.D. 

Jing Sun, M.A. 

Lara Malakoff, M.U.R.P. 

Janet Pershing, Ph.D. 

Whitney Marsland, B.A. 

Kurt Peters, Ph.D. 

Elly Field, B.A. 

Prepared For 

Lily Zandniapour, Ph.D. 

Office of Research and Evaluation 

Corporation for National and Community Service 

This document is a product of the National Assessment of the SIF, which seeks to document and capture 

the impact the SIF has on key program stakeholders. Its findings will provide evidence and tell the story 

of the SIF as well as identify lessons learned. The National Assessment is sponsored by the Corporation 

for National and Community Service’s Social Innovation Fund, designed and managed by CNCS Office 

of Research and Evaluation, and conducted by ICF International. 
nationalservice.gov/research 1 

Executive Summary

The Corporation for National and Community Service (CNCS) implements the Social Innovation Fund 

(SIF), an initiative that combines public and private resources to grow the impact of innovative, evidencebased solutions to improve the lives of people in low-income communities throughout the United States. 

Since its authorization in 2009, the SIF has received approximately $50-70 million per year in annual 

appropriations from Congress. These federal dollars are matched by non-Federal funds dollar for dollar 

at both the recipient level and subrecipient level. The SIF leverages federal funds through public-private 

collaborations by awarding grants to highly successful intermediary grantmaking organizations, herein 

referred to as SIF grantees. These organizations, in turn, provide funding to local 

nonprofits that are implementing promising community-based 

programs with evidence of successful outcomes in one of three core 

areas: youth development, economic opportunity, and healthy 

futures. 

The purpose of the SIF national assessment is to facilitate program 

evaluation and management by CNCS, support the greater goal of 

learning from the federal government’s “tiered-evidence”

initiatives, and conduct an independent evaluation of the SIF 

program. The main objective of the national assessment is to look at the impact on the organizational 

capacity, knowledge, and practices of SIF grantees. The assessment focuses on SIF grantees’ adoption of 

evidence-based grantmaking strategies, ability, and willingness to build the evidence base for the service 

models they support, to scale the service models, and to use collaborative approaches to address local 

community needs. The audience for the report includes SIF program managers, CNCS and other federal 

leadership, current and potential SIF grantees, and the broader philanthropic and nonprofit sector. 

The national assessment employs a quasi-experimental design (QED) to evaluate the impact of the SIF on 

organizational change. The findings presented in this report primarily draw on data from surveys of the 

following groups, as well as follow-up interviews with the SIF grantees. 

 SIF grantees compared with themselves over time. Using each SIF grantee as its own comparison,

the evaluation assesses capacity building from a pre-SIF baseline (2009) to the time the survey was

developed in 2014. This comparison allows examination of change by having SIF grantees serve as

their own controls in the analysis, also known as reflexive control. This comparison does not,

however, rule out the possibility that the observed change is part of a more general trend.

 SIF grantees compared with SIF applicants not selected. A comparison of SIF grantees to SIF

applicants who were not selected for grants provides an important counterfactual for the SIF grantee

experience. Those “Non-selected SIF Applicants” included in the comparison submitted applications

that were compliant and scored “satisfactory” or above by CNCS (and its external reviewers) on both

the program review and evaluation review. Comparing the performance of Non-selected SIF

Applicants to that of SIF grantees controls for motivation and helps reduce selection bias.

 SIF grantees compared to a nationally representative sample of grantmaking nonprofits. This 

sample comprises a cross-section of grantmaking nonprofits in the United States that meet basic SIF

eligibility criteria and are similar to the SIF grantees in terms of revenue and grantmaking size.

Selection criteria included the size of the organization (gross revenue of $1M or more) and the

volume of grants made to community organizations (grants totaling $800K or more). Including this

comparison group allows us to control for changes in the broader philanthropic world.
nationalservice.gov/research 2 

Summary of Findings 

The national assessment found promising evidence of improved organizational capacities among the SIF 

grantees. Exhibit ES-1 summarizes the quantitative findings. Between 2009 and 2014, SIF grantees grew in 

organizational capacities and behaviors in 13 out of 14 measures related to selection of grantees, support 

for grantees, evaluation, scaling up, and collaboration. The decrease in one item was largely due to 

change from one grantee.1

To assess whether to attribute these changes to the SIF, the evaluation also examined whether the 

comparison groups of Non-selected SIF Applicants and the National Sample of Grantmaking Nonprofits 

experienced changes in their organizational capacity during this time, and the extent of any change. 

Although all three groups reported important changes, reflecting shifts in the nonprofit world in recent 

years, there were some noticeable differences between the SIF grantees and the comparison groups. Indepth interviews with SIF grantees provided additional insights into the nature of the change they 

experienced. The changes include: 

 Evaluation: Compared to the National Sample of 

Grantmaking Nonprofits, SIF 2010-2012 grantees 

experienced significantly more growth between 

2009 and 2014 in three areas, including: 1) 

conducting rigorous evaluations of the programs; 

2) using evaluation findings to improve programs; 

and 3) using evaluation findings to demonstrate 

and communicate effectiveness of programs 

funded by the organization. The sizes of the 

differences in all three areas were large. 

Compared to Non-selected SIF Applicants, SIF 

2010-2012 grantees experienced significantly greater changes between 2009 and 2014 in the extent to 

which the organization conducted rigorous evaluations of programs funded by the organization and 

the effect size was large. Although the differences in the other two areas were not statistically 

significant due to small sample sizes, the effect sizes were medium and small, respectively. 

Themes that emerged from the follow-up interviews included: 1) the SIF focus on evaluation helped 

grantees plan and implement more—and more formal—evaluation; 2) grantees increased their

evaluation capacity with evaluation staff hires and the use of external evaluation partners; 3) the SIF 

focus on evaluation helped grantees use evidence to improve results; and 4) the assistance helped 

position SIF subgrantees to attract new funders, because they had the skills to conduct rigorous 

evaluation and communicate program effectiveness. 

 

1

 For selecting subgrantees through an open competition, 6 of the 20 grantees reported changes. The average response was largely 

influenced by one grantee who reported “always” using open competition in 2009, but “never” using open competition in 2014. 

“Strengths of the model are in the rigor of the 

evaluations and the partnership and transparency 

of the staff. SIF has proven to have some of the

most rigorous evaluation requirements that any of 

our partners or subgrantees have experienced. 

This led to very strong evaluations. The 

transparency and partnership of the CNCS staff 

was invaluable. CNCS and JBS staff made 

themselves immediately available to talk with our 

staff and the subgrantee evaluation team to help 

determine ways to overcome the obstacles.”

—a SIF grantee
nationalservice.gov/research 3 

Exhibit ES-1. Summary Table for Impact on SIF Grantee Organizational Capacity 

Organizational capacity

Change in 

SIF 

grantees 

from 2009 

to 2014

Difference in 

changes: SIF 2010-

2012 grantees vs 

National Sample of 

Grantmaking 

Nonprofits

Difference in 

changes: SIF 2010-

2012 grantees vs 

Non-selected SIF 

Applicants

Evaluation

A. Conducted rigorous evaluations of programs funded by your 

organization

↑ +Large* +Large*

B. Used evaluation findings to improve programs funded by your 

organization

↑ +Large* +Medium

C. Used evaluation findings to demonstrate and communicate 

effectiveness of programs funded by your organization

↑ +Large* +Small

Support for subgrantees

A. Provided funding to carry out an evaluation or hire an external 

evaluator, as part of the grant or through other means

↑ +Large* No

B. Provided training or technical assistance by your staff 

consultants or other means to conduct rigorous evaluation

↑ +Large* +Small

C. Provided training or technical assistance by your staff 

consultants or other means to support implementation of the 

program

↑ No +Small

Scale up

A. Undertook efforts to scale up existing program(s) – i.e., to 

expand the program(s) within the community or to other 

communities or populations

↑ +Small +Small

B. Selected programs for scale-up based on rigorous evaluation 

that shows them to be effective

↑ +Medium* +Small

Collaboration

A. Participated in funding alliance(s) with other nonprofit sector 

organizations. (For example, co-funding programs through joint 

funding; providing or receiving matching funds; or other 

collaboration) 

↑ +Small No

B. Participated in collaborations with other nonprofit organizations 

to share knowledge

↑ +Small +Small

C. Collaborated with other organizations for purposes of advocacy 

– to advocate for or develop public support for programs or 

approaches to addressing social problems

↑ No No

Selection of subgrantees to fund

A. Used an open competitive process to solicit and review 

applications and to make selection decisions

↓ -Medium* -Small

B. Required applicant organizations to provide evidence of 

intervention effectiveness to be eligible for funding (includes preand post-test outcome data or other evidence based on evaluation 

studies)

↑ -Medium +Small

C. Required applicants to submit a plan for rigorous evaluation of 

intervention to be eligible for funding (that is, quasi-experimental 

designs with a comparison group, experimental designs or other 

similarly rigorous designs)

↑ -Small +Small

↑SIF grantees experienced positive change, ↓SIF grantees experienced negative change 

*statistically significant difference at 0.05 level. A lack of significant difference between SIF grantees and Non-selected SIF Applicants is partly 

attributed to the small sample sizes. 

According to Cohen, Large effect size at 0.8 or more; medium effect size at 0.5-0.79; small effect size at 0.2-0.49; no effect size smaller than 0.2. 

+difference favoring SIF grantee; -difference favoring comparison group 
nationalservice.gov/research 4 

 Support for subgrantees: Compared to the National 

Sample of Grantmaking Nonprofits, SIF 2010-2012

grantees experienced significantly greater changes 

between 2009 and 2014 in the extent to which the 

organization 1) provided funding to subgrantees to 

carry out an evaluation or hire an external evaluator, whether as part of the grant or through other 

means, and 2) provided training or technical assistance to conduct rigorous evaluation. The effect 

sizes in the first two areas were large but negligible in 3) providing TA to support program 

implementation. Compared to Non-selected SIF Applicants, although the change patterns for SIF 

2010-2012 grantees were not statistically significant, we found small differences in favor of SIF 

grantees in the magnitude of the differences regarding provision of TA to evaluation and TA to 

program implementation. 

In addition, follow-up interviews indicated: 1) SIF grantees had increased their investment in 

growing the capacity of their subgrantees in terms of compliance capacity, evaluation capacity, and 

communication capacity; 2) the SIF helped grantees become more strategic in how they support their 

subgrantees; and 3) the support grantees provided gave their subgrantees a solid foundation upon 

which to grow. 

 Scale-up: Compared to the National Sample of Grantmaking Nonprofits, SIF 2010-2012 grantees 

experienced significantly greater changes between 2009 and 2014 in the extent to which they selected 

programs for scale-up based on rigorous evaluation that shows them to be effective, and the effect 

size was medium. Although no statistically significant difference was found in undertaking efforts to 

scale up existing program, we found small differences in favor of SIF grantees in the magnitude of 

the differences. Compared to Non-selected SIF Applicants, although the change patterns for SIF 2010-

2012 grantees were not statistically significant, we found small differences in favor of SIF grantees in 

the magnitude of the differences in both areas 

Themes related to scaling included: 1) the SIF helped grantees develop methods for choosing which 

interventions to scale up and 2) grantees packaged what they learned from the SIF to support scaling.

 Collaboration: No statistically significant differences were found between SIF grantees and the 

comparison groups. However, compared to the National Sample of Grantmaking Nonprofits, we 

found small differences in favor of SIF grantees in the magnitude of the differences regarding 

participating in funding alliances with other nonprofits, 

and participating in collaborations with other nonprofits 

to share knowledge. Compared to Non-selected SIF 

Applicants, we found small differences in favor of SIF 

grantees in the magnitude of the differences in 

collaborations to share knowledge. 

Follow-up interviews revealed: 1) the SIF gave grantees a variety of collaborative learning 

opportunities; 2) the SIF influenced how grantees support peer learning among their subgrantees; 

and 3) the SIF highlighted the value of external partnerships. 

 Selection of subgrantees to fund: Compared to the National Sample of Grantmaking Nonprofits, SIF 

2010-2012 grantees experienced significantly less change between 2009 and 2014 in the extent to 

which the organization used an open competitive process to solicit and review applications and to 

make selection decisions, again largely due to change from one grantee. We also found medium to 

“SIF has been a partner in the work. With the focus 

on innovation, change, and impact, the relationship 

was more dynamic than with other government 

funding relationships.”—a SIF grantee

“The community of funders created through 

monthly calls, issue area groups, and the 

national convenings was extremely strong 

compared to other federal grant programs.”

—a SIF grantee
nationalservice.gov/research 5 

small differences in favor of the National Sample of Grantmaking Nonprofits in all three areas. The 

findings were somewhat inconsistent from the Non-selected SIF Applicants. While the same was true 

in using open competitive process, we found small differences in favor of SIF grantees in effect sizes 

with regard to requiring applicants to provide evidence of effectiveness, and to submit a plan for 

rigorous evaluation in order to be eligible for funding. 

In spite of this, follow-up interviews indicated: 1) the SIF helped grantees build a more systematic 

process for subgrantee selection; 2) the selection process helped grantees identify subgrantees with a 

stronger evidence base; and 3) the SIF’s competitive solicitation process allowed grantees to reach 

subgrantees in new markets and geographic areas. 

The SIF grantees attributed many organizational changes to their participation in the SIF program. In 

particular, they credited technical assistance or support (e.g., coaching, facilitation, tools) received during 

the SIF funding period to help increase capacity or accomplish change. These supports include providing 

guidance on federal compliance, networking opportunities, evaluation plan, general support by SIF 

program office, annual convening, and other outside training and resources. Other factors that 

contributed to the reported change included direction from the organizational leadership and trends in 

the larger grantmaking world.

The SIF grantees observed capacity development among their subgrantees: 1) implementing the 

interventions in their communities; 2) designing and conducting rigorous evaluations of these 

interventions; 3) making use of evaluation findings for program improvement; 4) raising matching funds 

for the intervention; 5) scaling up the intervention; and 6) sharing knowledge and best practices. 

Recommendations 

The SIF grantees regarded rigorous evaluation, collaborating, and engaging 

in peer-learning experience as the main strengths of the SIF model. They 

also valued the transparency and accessibility of the CNCS staff, their 

flexibility, and the requirement for rigorous documentation. While these 

early areas of progress are encouraging, the grantees also offered

recommendations for program improvement. 

 Modifying matching requirement. Although the match requirement 

is set by the SIF statute, many grantees reported that meeting the 

matching requirement was a major challenge. Even for those who had 

early success, by years 2 and 3, several grantees mentioned that it 

became an exercise of moving existing funds around to meet the 

match, but not truly leveraging new dollars.

 Providing even stronger support for complying with federal

regulations. Although the SIF has provided substantial compliance

support, grantees found that federal requirements in the areas of

financial regulations and criminal background checks were particularly

challenging. It may be helpful for CNCS to provide additional support.

 Including a planning year. Although the SIF underwrites evidence-based intervention, a few

grantees recommended including a planning year.
nationalservice.gov/research 6 

 Expanding collaboration. SIF grantees recommended continuing collaboration among SIF grantees 

and funders, and reiterated the importance of securing engagement and feedback from private 

foundations and philanthropies. 

 Improving the application process. SIF grantees and applicants offered additional recommendations 

to improve the SIF application process, such as adjusting the timeframe, including a pre-qualification 

phase, more transparency about implementation requirements, and further streamlining and 

clarifying of application requirements. 

Limitations 

Limitations of this study include the small number of SIF grantees and Non-selected SIF Applicants in the 

sample. The relatively small number of cases renders analyses of the survey data vulnerable to such 

factors as the presence of outliers. We used various strategies to mitigate the issue including the use of 

non-parametric assumption in statistical significance testing, effect size calculations, etc. Another 

limitation is that much of the change was reported retrospectively and depends on self-report. We 

address it by acquiring documentary evidence from SIF grantees to support changes reported in the 

survey. Finally, discussion with CNCS staff and grantees indicates that the experience of the initial 2010 

SIF cohort differed from that of subsequent cohorts, which may affect comparisons and analyses of 

trends. Additionally, the mix of SIF-funded organizations over time has varied, with the result that, 

although this study can provide evidence about the experience of this specific group of organizations, 

because the future mix of organizations and the experience of later SIF cohorts may differ, the potential to 

generalize from this study to the broader population of future SIF grantees is limited. We combined 

qualitative and quantitative methods to obtain more in-depth understanding of survey findings. Overall, 

we tried to carefully present findings so that readers understand the strengths and limitations of the data. 

Next Steps 

Building organizational capacities requires a multi-faceted approach and long-lasting commitment. 

Although this report provides promising evidence that the SIF program moved grantee organizations in 

the right direction, we also found that these changes were often incremental and that time was required 

for changes in one program to permeate the rest of the organization. 

We will continue to examine these questions next year. In addition to relying on surveys to document 

and capture changes in intermediary capacities and experiences over time, and provide broad-brush 

findings, we plan to use other methodologies, such as case studies. Our purpose will be to provide an indepth examination of selected grantees, to understand their internal and external contexts, the factors that 

facilitate and inhibit capacity building, and further uncover best practices and lessons learned. 
Corporation for National and Community Service 

1201 New York Avenue, NW 

Washington, D.C. 20525

TEL: (202) 606-5000 

TTY: (202) 606-3472 

NationalService.gov