The SIF’s Impact on 

Strengthening 

Organizational Capacity

A product of the National Assessment of 

the Social Innovation Fund (SIF)

September 2015
This report is in the public domain. Authorization to reproduce it in whole or in part is granted. Upon 

request, this material will be made available in alternative formats for people with disabilities. 

The SIF’s Impact on Strengthening Organizational Capacity 

September 2015 

Authors 

Xiaodong Zhang, Ph.D. 

Janet Griffith, Ph.D. 

Jing Sun, M.A. 

Lara Malakoff, M.U.R.P. 

Janet Pershing, Ph.D. 

Whitney Marsland, B.A. 

Kurt Peters, Ph.D. 

Elly Field, B.A. 

Prepared For 

Lily Zandniapour, Ph.D. 

Office of Research and Evaluation 

Corporation for National and Community Service 

This document is a product of the National Assessment of the SIF, which seeks to document and capture 

the impact the SIF has on key program stakeholders. Its findings will provide evidence and tell the story 

of the SIF as well as identify lessons learned. The National Assessment is sponsored by the Corporation 

for National and Community Service’s Social Innovation Fund, designed and managed by CNCS Office

of Research and Evaluation, and conducted by ICF International. 
nationalservice.gov/research i 

Table of Contents 

Table of Contents ------------------------------------------------------------------------------------------------------------------ i

Executive Summary --------------------------------------------------------------------------------------------------------------- 1

1. Introduction -------------------------------------------------------------------------------------------------------------- 7

2. SIF National Assessment----------------------------------------------------------------------------------------------- 8

2.1 Evaluation Questions ------------------------------------------------------------------------------------------ 8

2.2 Evaluation Design---------------------------------------------------------------------------------------------- 9

2.3 Data Collection and Analysis------------------------------------------------------------------------------- 10

2.4 Limitations---------------------------------------------------------------------------------------------------- 11

3. Evaluation Findings--------------------------------------------------------------------------------------------------- 12

3.1 Background and Demographics---------------------------------------------------------------------------- 12

3.1.1 Financial Data-------------------------------------------------------------------------------------- 12

3.1.2 Grantmaking Practices---------------------------------------------------------------------------- 13

3.1.3 Federal Funding ----------------------------------------------------------------------------------- 14

3.2 Support Received by SIF Grantees ------------------------------------------------------------------------ 15

3.3 Impact on Organizational Capacity ------------------------------------------------------------------------ 17

3.3.1 Selection of Subgrantees ------------------------------------------------------------------------- 17

3.3.2 Support for Subgrantees -------------------------------------------------------------------------- 20

3.3.3 Use of Data, Evidence, and Evaluation--------------------------------------------------------- 23

3.3.4 Scaling Up Programs with Evidence of Effectiveness---------------------------------------- 29

3.3.5 Collaboration to Address Community Needs-------------------------------------------------- 31

3.4 Reflections on SIF Experience ----------------------------------------------------------------------------- 33

3.4.1 Tiered-Evidence Grant Program----------------------------------------------------------------- 34

3.4.2 Changes across Organizations ------------------------------------------------------------------- 35

3.4.3 Development of SIF Subgrantees’ Capacity --------------------------------------------------- 37

3.4.4 The SIF Model’s Strengths and Benefits, Challenges and Problems ----------------------- 39

3.4.5 Recommendations--------------------------------------------------------------------------------- 40

4. Conclusions and Next Steps ----------------------------------------------------------------------------------------- 40

References ------------------------------------------------------------------------------------------------------------------------- 42

Appendix A. Overview of SIF National Assessment--------------------------------------------------------------------- A-1

Appendix B. Evaluation Methodologies ------------------------------------------------------------------------------------ B-1

Appendix C. Findings from the SIF 2014 cohort ------------------------------------------------------------------------- C-1

Appendix D. SIF Survey Descriptive Data Tables----------------------------------------------------------------------- D-1
nationalservice.gov/research 1 

Executive Summary

The Corporation for National and Community Service (CNCS) implements the Social Innovation Fund 

(SIF), an initiative that combines public and private resources to grow the impact of innovative, evidencebased solutions to improve the lives of people in low-income communities throughout the United States. 

Since its authorization in 2009, the SIF has received approximately $50-70 million per year in annual

appropriations from Congress. These federal dollars are matched by non-Federal funds dollar for dollar 

at both the recipient level and subrecipient level. The SIF leverages federal funds through public-private

collaborations by awarding grants to highly successful intermediary grantmaking organizations, herein 

referred to as SIF grantees. These organizations, in turn, provide funding to local 

nonprofits that are implementing promising community-based 

programs with evidence of successful outcomes in one of three core 

areas: youth development, economic opportunity, and healthy 

futures. 

The purpose of the SIF national assessment is to facilitate program 

evaluation and management by CNCS, support the greater goal of 

learning from the federal government’s “tiered-evidence”

initiatives, and conduct an independent evaluation of the SIF 

program. The main objective of the national assessment is to look at the impact on the organizational 

capacity, knowledge, and practices of SIF grantees. The assessment focuses on SIF grantees’ adoption of 

evidence-based grantmaking strategies, ability, and willingness to build the evidence base for the service 

models they support, to scale the service models, and to use collaborative approaches to address local 

community needs. The audience for the report includes SIF program managers, CNCS and other federal 

leadership, current and potential SIF grantees, and the broader philanthropic and nonprofit sector. 

The national assessment employs a quasi-experimental design (QED) to evaluate the impact of the SIF on 

organizational change. The findings presented in this report primarily draw on data from surveys of the 

following groups, as well as follow-up interviews with the SIF grantees. 

 SIF grantees compared with themselves over time. Using each SIF grantee as its own comparison,

the evaluation assesses capacity building from a pre-SIF baseline (2009) to the time the survey was

developed in 2014. This comparison allows examination of change by having SIF grantees serve as

their own controls in the analysis, also known as reflexive control. This comparison does not,

however, rule out the possibility that the observed change is part of a more general trend.

 SIF grantees compared with SIF applicants not selected. A comparison of SIF grantees to SIF

applicants who were not selected for grants provides an important counterfactual for the SIF grantee

experience. Those “Non-selected SIF Applicants” included in the comparison submitted applications

that were compliant and scored “satisfactory” or above by CNCS (and its external reviewers) on both

the program review and evaluation review. Comparing the performance of Non-selected SIF

Applicants to that of SIF grantees controls for motivation and helps reduce selection bias.

 SIF grantees compared to a nationally representative sample of grantmaking nonprofits. This 

sample comprises a cross-section of grantmaking nonprofits in the United States that meet basic SIF

eligibility criteria and are similar to the SIF grantees in terms of revenue and grantmaking size.

Selection criteria included the size of the organization (gross revenue of $1M or more) and the

volume of grants made to community organizations (grants totaling $800K or more). Including this

comparison group allows us to control for changes in the broader philanthropic world.
nationalservice.gov/research 2 

Summary of Findings 

The national assessment found promising evidence of improved organizational capacities among the SIF 

grantees. Exhibit ES-1 summarizes the quantitative findings. Between 2009 and 2014, SIF grantees grew in 

organizational capacities and behaviors in 13 out of 14 measures related to selection of grantees, support 

for grantees, evaluation, scaling up, and collaboration. The decrease in one item was largely due to 

change from one grantee.1

To assess whether to attribute these changes to the SIF, the evaluation also examined whether the 

comparison groups of Non-selected SIF Applicants and the National Sample of Grantmaking Nonprofits 

experienced changes in their organizational capacity during this time, and the extent of any change. 

Although all three groups reported important changes, reflecting shifts in the nonprofit world in recent 

years, there were some noticeable differences between the SIF grantees and the comparison groups. Indepth interviews with SIF grantees provided additional insights into the nature of the change they 

experienced. The changes include: 

 Evaluation: Compared to the National Sample of

Grantmaking Nonprofits, SIF 2010-2012 grantees

experienced significantly more growth between

2009 and 2014 in three areas, including: 1)

conducting rigorous evaluations of the programs;

2) using evaluation findings to improve programs;

and 3) using evaluation findings to demonstrate 

and communicate effectiveness of programs 

funded by the organization. The sizes of the 

differences in all three areas were large. 

Compared to Non-selected SIF Applicants, SIF 

2010-2012 grantees experienced significantly greater changes between 2009 and 2014 in the extent to 

which the organization conducted rigorous evaluations of programs funded by the organization and 

the effect size was large. Although the differences in the other two areas were not statistically 

significant due to small sample sizes, the effect sizes were medium and small, respectively. 

Themes that emerged from the follow-up interviews included: 1) the SIF focus on evaluation helped 

grantees plan and implement more—and more formal—evaluation; 2) grantees increased their

evaluation capacity with evaluation staff hires and the use of external evaluation partners; 3) the SIF 

focus on evaluation helped grantees use evidence to improve results; and 4) the assistance helped 

position SIF subgrantees to attract new funders, because they had the skills to conduct rigorous 

evaluation and communicate program effectiveness. 

1

 For selecting subgrantees through an open competition, 6 of the 20 grantees reported changes. The average response was largely 

influenced by one grantee who reported “always” using open competition in 2009, but “never” using open competition in 2014. 

“Strengths of the model are in the rigor of the 

evaluations and the partnership and transparency 

of the staff. SIF has proven to have some of the

most rigorous evaluation requirements that any of 

our partners or subgrantees have experienced. 

This led to very strong evaluations. The 

transparency and partnership of the CNCS staff 

was invaluable. CNCS and JBS staff made 

themselves immediately available to talk with our 

staff and the subgrantee evaluation team to help 

determine ways to overcome the obstacles.”

—a SIF grantee
nationalservice.gov/research 3 

Exhibit ES-1. Summary Table for Impact on SIF Grantee Organizational Capacity 

Organizational capacity

Change in 

SIF 

grantees 

from 2009 

to 2014

Difference in 

changes: SIF 2010-

2012 grantees vs 

National Sample of 

Grantmaking 

Nonprofits

Difference in 

changes: SIF 2010-

2012 grantees vs 

Non-selected SIF 

Applicants

Evaluation

A. Conducted rigorous evaluations of programs funded by your 

organization

↑ +Large* +Large*

B. Used evaluation findings to improve programs funded by your 

organization

↑ +Large* +Medium

C. Used evaluation findings to demonstrate and communicate 

effectiveness of programs funded by your organization

↑ +Large* +Small

Support for subgrantees

A. Provided funding to carry out an evaluation or hire an external 

evaluator, as part of the grant or through other means

↑ +Large* No

B. Provided training or technical assistance by your staff 

consultants or other means to conduct rigorous evaluation

↑ +Large* +Small

C. Provided training or technical assistance by your staff 

consultants or other means to support implementation of the 

program

↑ No +Small

Scale up

A. Undertook efforts to scale up existing program(s) – i.e., to 

expand the program(s) within the community or to other 

communities or populations

↑ +Small +Small

B. Selected programs for scale-up based on rigorous evaluation 

that shows them to be effective

↑ +Medium* +Small

Collaboration

A. Participated in funding alliance(s) with other nonprofit sector 

organizations. (For example, co-funding programs through joint 

funding; providing or receiving matching funds; or other 

collaboration) 

↑ +Small No

B. Participated in collaborations with other nonprofit organizations 

to share knowledge

↑ +Small +Small

C. Collaborated with other organizations for purposes of advocacy 

– to advocate for or develop public support for programs or

approaches to addressing social problems

↑ No No

Selection of subgrantees to fund

A. Used an open competitive process to solicit and review 

applications and to make selection decisions

↓ -Medium* -Small

B. Required applicant organizations to provide evidence of 

intervention effectiveness to be eligible for funding (includes preand post-test outcome data or other evidence based on evaluation 

studies)

↑ -Medium +Small

C. Required applicants to submit a plan for rigorous evaluation of 

intervention to be eligible for funding (that is, quasi-experimental 

designs with a comparison group, experimental designs or other 

similarly rigorous designs)

↑ -Small +Small

↑SIF grantees experienced positive change, ↓SIF grantees experienced negative change 

*statistically significant difference at 0.05 level. A lack of significant difference between SIF grantees and Non-selected SIF Applicants is partly

attributed to the small sample sizes.

According to Cohen, Large effect size at 0.8 or more; medium effect size at 0.5-0.79; small effect size at 0.2-0.49; no effect size smaller than 0.2.

+difference favoring SIF grantee; -difference favoring comparison group
nationalservice.gov/research 4 

 Support for subgrantees: Compared to the National

Sample of Grantmaking Nonprofits, SIF 2010-2012

grantees experienced significantly greater changes

between 2009 and 2014 in the extent to which the

organization 1) provided funding to subgrantees to

carry out an evaluation or hire an external evaluator, whether as part of the grant or through other

means, and 2) provided training or technical assistance to conduct rigorous evaluation. The effect

sizes in the first two areas were large but negligible in 3) providing TA to support program

implementation. Compared to Non-selected SIF Applicants, although the change patterns for SIF

2010-2012 grantees were not statistically significant, we found small differences in favor of SIF

grantees in the magnitude of the differences regarding provision of TA to evaluation and TA to

program implementation.

In addition, follow-up interviews indicated: 1) SIF grantees had increased their investment in 

growing the capacity of their subgrantees in terms of compliance capacity, evaluation capacity, and 

communication capacity; 2) the SIF helped grantees become more strategic in how they support their 

subgrantees; and 3) the support grantees provided gave their subgrantees a solid foundation upon 

which to grow. 

 Scale-up: Compared to the National Sample of Grantmaking Nonprofits, SIF 2010-2012 grantees

experienced significantly greater changes between 2009 and 2014 in the extent to which they selected

programs for scale-up based on rigorous evaluation that shows them to be effective, and the effect

size was medium. Although no statistically significant difference was found in undertaking efforts to

scale up existing program, we found small differences in favor of SIF grantees in the magnitude of

the differences. Compared to Non-selected SIF Applicants, although the change patterns for SIF 2010-

2012 grantees were not statistically significant, we found small differences in favor of SIF grantees in

the magnitude of the differences in both areas

Themes related to scaling included: 1) the SIF helped grantees develop methods for choosing which 

interventions to scale up and 2) grantees packaged what they learned from the SIF to support scaling.

 Collaboration: No statistically significant differences were found between SIF grantees and the

comparison groups. However, compared to the National Sample of Grantmaking Nonprofits, we

found small differences in favor of SIF grantees in the magnitude of the differences regarding

participating in funding alliances with other nonprofits, 

and participating in collaborations with other nonprofits 

to share knowledge. Compared to Non-selected SIF 

Applicants, we found small differences in favor of SIF 

grantees in the magnitude of the differences in 

collaborations to share knowledge. 

Follow-up interviews revealed: 1) the SIF gave grantees a variety of collaborative learning 

opportunities; 2) the SIF influenced how grantees support peer learning among their subgrantees; 

and 3) the SIF highlighted the value of external partnerships. 

 Selection of subgrantees to fund: Compared to the National Sample of Grantmaking Nonprofits, SIF

2010-2012 grantees experienced significantly less change between 2009 and 2014 in the extent to

which the organization used an open competitive process to solicit and review applications and to

make selection decisions, again largely due to change from one grantee. We also found medium to

“SIF has been a partner in the work. With the focus 

on innovation, change, and impact, the relationship 

was more dynamic than with other government 

funding relationships.”—a SIF grantee

“The community of funders created through 

monthly calls, issue area groups, and the 

national convenings was extremely strong 

compared to other federal grant programs.”

—a SIF grantee
nationalservice.gov/research 5 

small differences in favor of the National Sample of Grantmaking Nonprofits in all three areas. The 

findings were somewhat inconsistent from the Non-selected SIF Applicants. While the same was true 

in using open competitive process, we found small differences in favor of SIF grantees in effect sizes 

with regard to requiring applicants to provide evidence of effectiveness, and to submit a plan for 

rigorous evaluation in order to be eligible for funding. 

In spite of this, follow-up interviews indicated: 1) the SIF helped grantees build a more systematic 

process for subgrantee selection; 2) the selection process helped grantees identify subgrantees with a 

stronger evidence base; and 3) the SIF’s competitive solicitation process allowed grantees to reach 

subgrantees in new markets and geographic areas. 

The SIF grantees attributed many organizational changes to their participation in the SIF program. In 

particular, they credited technical assistance or support (e.g., coaching, facilitation, tools) received during 

the SIF funding period to help increase capacity or accomplish change. These supports include providing 

guidance on federal compliance, networking opportunities, evaluation plan, general support by SIF 

program office, annual convening, and other outside training and resources. Other factors that 

contributed to the reported change included direction from the organizational leadership and trends in 

the larger grantmaking world.

The SIF grantees observed capacity development among their subgrantees: 1) implementing the 

interventions in their communities; 2) designing and conducting rigorous evaluations of these 

interventions; 3) making use of evaluation findings for program improvement; 4) raising matching funds 

for the intervention; 5) scaling up the intervention; and 6) sharing knowledge and best practices. 

Recommendations 

The SIF grantees regarded rigorous evaluation, collaborating, and engaging 

in peer-learning experience as the main strengths of the SIF model. They 

also valued the transparency and accessibility of the CNCS staff, their 

flexibility, and the requirement for rigorous documentation. While these 

early areas of progress are encouraging, the grantees also offered

recommendations for program improvement. 

 Modifying matching requirement. Although the match requirement is

set by the SIF statute, many grantees reported that meeting the

matching requirement was a major challenge. Even for those who had

early success, by years 2 and 3, several grantees mentioned that it

became an exercise of moving existing funds around to meet the match,

but not truly leveraging new dollars.

 Providing even stronger support for complying with federal

regulations. Although the SIF has provided substantial compliance

support, grantees found that federal requirements in the areas of

financial regulations and criminal background checks were particularly

challenging. It may be helpful for CNCS to provide additional support.

 Including a planning year. Although the SIF underwrites evidence-based intervention, a few

grantees recommended including a planning year.
nationalservice.gov/research 6 

 Expanding collaboration. SIF grantees recommended continuing collaboration among SIF grantees

and funders, and reiterated the importance of securing engagement and feedback from private

foundations and philanthropies.

 Improving the application process. SIF grantees and applicants offered additional recommendations

to improve the SIF application process, such as adjusting the timeframe, including a pre-qualification

phase, more transparency about implementation requirements, and further streamlining and

clarifying of application requirements.

Limitations 

Limitations of this study include the small number of SIF grantees and Non-selected SIF Applicants in the 

sample. The relatively small number of cases renders analyses of the survey data vulnerable to such 

factors as the presence of outliers. We used various strategies to mitigate the issue including the use of 

non-parametric assumption in statistical significance testing, effect size calculations, etc. Another 

limitation is that much of the change was reported retrospectively and depends on self-report. We 

address it by acquiring documentary evidence from SIF grantees to support changes reported in the 

survey. Finally, discussion with CNCS staff and grantees indicates that the experience of the initial 2010 

SIF cohort differed from that of subsequent cohorts, which may affect comparisons and analyses of 

trends. Additionally, the mix of SIF-funded organizations over time has varied, with the result that, 

although this study can provide evidence about the experience of this specific group of organizations, 

because the future mix of organizations and the experience of later SIF cohorts may differ, the potential to 

generalize from this study to the broader population of future SIF grantees is limited. We combined 

qualitative and quantitative methods to obtain more in-depth understanding of survey findings. Overall, 

we tried to carefully present findings so that readers understand the strengths and limitations of the data. 

Next Steps 

Building organizational capacities requires a multi-faceted approach and long-lasting commitment. 

Although this report provides promising evidence that the SIF program moved grantee organizations in 

the right direction, we also found that these changes were often incremental and that time was required 

for changes in one program to permeate the rest of the organization. 

We will continue to examine these questions next year. In addition to relying on surveys to document 

and capture changes in intermediary capacities and experiences over time, and provide broad-brush 

findings, we plan to use other methodologies, such as case studies. Our purpose will be to provide an indepth examination of selected grantees, to understand their internal and external contexts, the factors that 

facilitate and inhibit capacity building, and further uncover best practices and lessons learned. 
nationalservice.gov/research 7 

1. Introduction

The Social Innovation Fund (SIF) is a federal program to leverage public and private resources to grow 

the impact of innovative, evidence-based solutions to improve the lives of people in low-income 

communities throughout the United States. It was authorized by the 2009 Serve America Act and is 

administered by the Corporation for National and Community Service (CNCS). The SIF is part of the 

federal government’s efforts to support “tiered-evidence initiatives” to promote social innovation. 

Since 2010, the SIF has received an annual Congressional appropriation of approximately $50-70 million. 

The program leverages federal funds through public-private collaborations by awarding grants to highly 

successful intermediary grantmaking organizations. Federal dollars are matched up by non-Federal funds 

dollar for dollar at both the recipient level and subrecipient level. In turn, these organizations provide 

funding to local nonprofits that are implementing promising community-based programs with evidence 

of successful outcomes in one of three core areas: youth development, economic opportunity, and healthy 

futures. The SIF is characterized by the unique interplay of six key elements: 

 It relies on intermediary grantmaking institutions to implement the program—they take on the role

of finding, selecting, monitoring, supporting, evaluating, and reporting on the nonprofit

organizations implementing community-based interventions.

 It is a tiered-evidence initiative that requires all funded programs/interventions to demonstrate at

least preliminary evidence of effectiveness, or funding “what works.”

 It requires that all programs or interventions implement a rigorous evaluation that will build on their

level of evidence. 2

 It charges intermediaries with scaling evidence-based programs—increasing impact within their

community or to communities across the country—and as such, grapples with the field-wide

challenge of how best to do so successfully and efficiently.

 It leverages public-private partnerships to effect large-scale community impact in ways that neither a

traditional federal grant nor a philanthropic investment could achieve on its own. This strategy

features a unique leveraged funding model to support nonprofit programs.

 It is committed to improving the effectiveness of nonprofits, funders, and other federal agencies by

capturing and learning about best practices and promoting approaches that will generate the greatest

impact for individuals and communities.

By 2014, the SIF had completed four rounds of grantmaking (in 2010, 2011, 2012, and 2014) and selected a 

total of 26 grantees for 27 grants. These grantees have in turn selected more than 200 nonprofit 

organizations through open and competitive processes to implement promising interventions in youth 

development, economic opportunity, and healthy futures. Exhibit 1 summarizes the number of awards by 

year. 

2

 Although there is no consensus within the evaluation field concerning what constitutes a particular level of evidence, the SIF program divides 

evidence into three categories: preliminary, moderate, and strong (CNCS, n.d.). These divisions are based on how well a particular 

evaluation is able to address concerns about internal and external validity, with evaluations that are more effective generating strong or 

moderate levels and those that are less so generating preliminary levels of evidence. 
nationalservice.gov/research 8 

Exhibit 1. SIF awards by year 

Award year (cohort) Number of Grants to SIF grantees Number of SIF subgrantees funded

2010 11 149

2011 5 48

2012 4 15

2014 7* 40

Total 27 252

Extracted from CNCS website. 

* There were 7 new grants in 2014, but one was to an organization funded in an earlier cohort. As a result, there are 27 SIF grants, but 26

grantee organizations. 

2. SIF National Assessment

The purpose of the SIF national assessment is to facilitate program evaluation and management by 

CNCS, support the greater goal of learning from the federal government’s tiered-evidence initiatives, and 

conduct an independent evaluation of the SIF program. The main objective of the national assessment is 

to look at the impact on the organizational capacity, knowledge, and practices of SIF grantees. To meet 

the need for information about the implementation and effectiveness of the SIF, CNCS contracted with 

ICF International to support CNCS’s Office of Research and Evaluation to implement an independent 

national assessment of the SIF. The assessment focuses on SIF grantees’ adoption of evidence-based 

grantmaking strategies, ability, and willingness to build the evidence base for the service models they 

support, scale the service models, and use collaborative approaches to address local community needs. 

To measure the capacity of the SIF grantees—and to compare them with other grantmaking 

organizations—the emphasis of the evaluation is on changes in organizational behavior in terms of 

policies and practices. The audience for the report includes SIF program managers, CNCS and other 

federal leadership, current and potential SIF grantees, and the larger philanthropic and non-profit sector. 

This chapter provides an overview of the SIF national assessment. Appendix A presents a detailed 

description of the national assessment. Appendix B offers details about the evaluation methodologies, 

including information about research design, sampling, survey instruments, data collection, and analysis. 

2.1 Evaluation Questions 

The SIF national assessment is designed to provide information for CNCS on the implementation and 

effectiveness of the SIF. The key questions for the SIF national assessment are: 

 Does participation in the SIF increase the capacity of organizations (e.g., skills, attitudes, behavior) in

the use of evidence-based grantmaking strategies?

 Does participation in the SIF increase the willingness and ability of organizations to build the

evidence base for high-impact models?

 Does participation in the SIF increase organizational knowledge of how to scale effective program

models?

 Does participation in the SIF facilitate collaborative approaches to addressing local community

needs?

Exhibit 2 presents the SIF logic model developed by the evaluation team to guide the national assessment. 

The logic model was used to help understand the program and its context, as well as to inform evaluation 

design and instrument development. 
nationalservice.gov/research 9 

Exhibit 2. SIF Logic Model 

2.2 Evaluation Design 

The national assessment employs a quasi-experimental design (QED) to evaluate the impact of SIF on 

organizational change. The purpose of this approach is to create a counterfactual to compare SIF grantees 

with what would have happened in the absence of the SIF. The comparisons are: 

 SIF grantees compared with themselves over time. Using each SIF grantee as its own comparison,

the evaluation assesses capacity building from a pre-SIF baseline (2009) to the time the survey was

administered in 2014. This comparison allows examination of change by having SIF grantees serve as

their own controls in the analysis. This comparison does not, however, rule out the possibility that

the observed change is part of a more general trend.

 SIF grantees compared with Non-selected SIF Applicants. A comparison of SIF grantees to Nonselected SIF Applicants who were not selected for grants provides an important counterfactual for the

SIF grantee experience. Those “Non-selected Applicants” included in the comparison submitted

applications that were compliant and scored “satisfactory” or above by CNCS (and its external

reviewers) on both the program review and evaluation review. Examining the performance of Nonselected SIF Applicants relative to SIF grantees controls for motivation and helps reduce selection

bias.

 SIF grantees compared to a nationally representative sample of grantmaking nonprofits. This 

sample comprises a cross-section of grantmaking nonprofits in the United States that meet basic SIF

eligibility criteria and are similar to the SIF grantees in terms of revenue and grantmaking size.

Selection criteria included the size of the organization (gross revenue of $1M or more) and the

volume of grants made to community organizations (grants totaling $800K or more). Including this

comparison group allows us to control for changes in the broader philanthropic world.

Exhibit 3 presents response rates for each respondent group. The surveys achieved 100% response from 

SIF grantees, but lower response rates from Non-selected SIF Applicants and the National Sample of 

Grantmaking Nonprofits. Non-response analysis was conducted to assess the potential bias resulting 
nationalservice.gov/research 10

from these lower response rates. Results from the non-response analysis did not suggest a significant 

potential for non-response bias affecting the survey of Non-selected SIF Applicants or the National 

Sample of Grantmaking Nonprofits. (See details in Appendix B). In addition, we weighted the responses 

for these two groups to improve precision of the estimates. 

Exhibit 3. Response rates for the survey by group 

SIF 2010-

2012

Non-selected SIF 

Applicants 2010-

2012 SIF 2014

Non-selected SIF 

Applicants 2014

National Sample of 

Grantmaking 

Nonprofits

Completed response 20 9 6 9 387*

Response rate 100% 35% 100% 60% 12%

*including 169 non-eligible organizations which do not make grants.

2.3 Data Collection and Analysis 

To provide a well-rounded picture of the organizational changes experienced by the SIF grantees, 

findings presented in this report drew on multiple sources of data: 

 Surveys of SIF grantees and two comparison groups: 1) organizations that applied for the SIF but

were not funded (Non-selected SIF Applicants), and 2) a national comparison sample of United States

grantmaking nonprofit organizations (the National Sample of grantmaking nonprofits).

 Qualitative interviews with 1) SIF grantees, to develop information on evidence to support and

document organizational changes reported in the survey, and 2) a combination of SIF grantees and

other stakeholders (subgrantees, program officers, and others) to develop information briefs on

specific topics of interest to CNCS and the field.

 Data from other sources, including IRS form 990 (financial data submitted by nonprofits).

Building organizational capacities requires a multi-faceted approach and long commitment. According to 

Harsh (2010), capacity building is a recurring cycle of intervention, support, and growth. In developing 

the survey, CNCS and ICF discussed and tested approaches to conceptualizing and measuring 

organizational capacity. Broadly, capacity refers to the potential of an organization to carry out its 

mission effectively. As the evaluation question indicates, this involves an array of elements, including 

financial, human, technological, and other resources; infrastructure to support the work; leadership; 

organizational culture and climate; and engagement and partnership within the organization and 

between the organization and its organizational partners and other stakeholders. 

To measure the capacity of the SIF grantees—and to compare them with other grantmaking 

organizations—CNCS and ICF determined that it was feasible and appropriate to focus on organizational 

behavior. From the perspective of organizational change, behavior is, ultimately, the measure that 

matters. To allow for comparison over a common time frame and to gain insights into factors that 

contributed to change, the survey asked SIF grantees and the comparison organizations about 

organizational behaviors in fourteen different areas in 2009 (before SIF implementation) and in 2014, and 

asked respondents to report factors that contributed to any changes in the extent of these organizational 

behaviors. Cognitive interviews and pretests determined that respondents understood and could report 

on organizational behavior and change in behavior, could make distinctions in the degree of 

organizational behavior using the survey’s 7-point scale, and could report factors that contributed to the 

changes. Because the survey provided an opportunity to ask SIF grantees about their perceptions of 

change among their subgrantees, the survey also asked a question about change in subgrantee capacity. 
nationalservice.gov/research 11

Data analysis focused first on documenting and analyzing change in grantmaking strategies, use of data 

and evaluation, engagement in building an evidence base for programs and scaling programs based on 

evidence of effectiveness, and collaboration to address community needs. 

Comparisons between SIF grantees and other organizations were then used to attribute observed change 

to SIF participation. To test for differences between the SIF grantees and Non-selected SIF Applicants, we 

used non-parametric technique known as Mann-Whitney test to account for 

the small sample sizes of the two groups. For differences between SIF 

grantees and the National Sample of Grantmaking Nonprofits, we 

calculated the confidence interval at 95% around the values for the 

nonprofit sample; then, if the difference between the SIF value and the 

value for the sample is larger than the confidence interval for the sample, 

we can draw the inference that the observed difference is greater than 

would occur by chance (Cochran, 1977; the appropriate test is a one-sample 

t-test). In addition, we used effect size estimates to gauge the practical 

significance of the differences. Statistical significance which measures the 

probability of getting a result, is highly dependent on sample sizes. On the 

other hand, effect sizes measures the magnitude of the difference and is less 

dependent on sample sizes. A small effect can be statistically significant if 

the sample size is large. Cohen (1988) defined effect sizes as "small, d = 0.2," 

"medium, d = 0.5," and "large, d = 0.8".

Next, analyses of open-ended questions from the surveys addressed “how” 

and “why” questions relating to the factors that contribute to the observed change. Finally, interview data 

and documentary evidence provided insights to supplement data from the surveys. 

2.4 Limitations 

The national assessment has both strengths and limitations. In terms of strengths, the inclusion of two 

comparison groups provides an opportunity to address questions about SIF grantees relative to similar, 

highly committed organizations (Non-selected SIF Applicants), and to broader trends in the field of 

nonprofit grantmaking (the National Sample of Grantmaking Nonprofits). In addition, the combination 

of multiple data collection strategies—combining quantitative survey methodology with qualitative 

interview and document review methods—makes it possible to obtain specific and objective information 

about change as well as insights into factors that contribute to reported change. 

Limitations of this study include the small number of SIF grantees and Non-selected SIF Applicants in the 

sample. The relatively small number of cases renders analyses of the survey data vulnerable to such 

factors as the presence of outliers. We used various strategies to mitigate the issue including the use of 

non-parametric assumption in statistical significance testing, effect size calculations, etc. Another 

limitation is that much of the change was reported retrospectively and depends on self-report. We 

address it by acquiring documentary evidence from SIF grantees to support changes reported in the 

survey. Finally, discussion with CNCS staff and grantees indicates that the experience of the initial 2010 

SIF cohort differed from that of subsequent cohorts, which may affect comparisons and analyses of 

trends. Additionally, the mix of SIF-funded organizations over time has varied, with the result that, 

although this study can provide evidence about the experience of this specific group of organizations, 

because the future mix of organizations and the experience of later SIF cohorts may differ, the potential to 

generalize from this study to the broader population of future SIF grantees is limited. We combined 
nationalservice.gov/research 12

qualitative and quantitative methods to obtain more in-depth understanding of survey findings. Overall, 

we tried to carefully present findings so that readers understand the strengths and limitations of the data. 

3. Evaluation Findings

The chapter first describes the background and demographics of the survey respondents, and then 

discusses the support received by SIF grantees. Next, the chapter examines the impact on organizational 

capacities through comparison of SIF grantees with the comparison groups. Finally, it looks at the 

reflections of SIF grantees about their SIF experiences. The evaluation findings consider early SIF grantees 

from 2010-2012 cohorts and their corresponding comparison groups. Because the SIF 2014 grantees were 

only in the first year of SIF funding when the survey was administered, it was too early to expect the 

survey to show any major changes resulting from the SIF grant. Appendix C offers early findings about 

SIF 2014 grantees and 2014 Non-selected SIF Applicants. 

3.1 Background and Demographics 

Demographics about the organizations surveyed help us understand their context and establish a frame 

of reference to interpret the survey results. We specifically looked at financial indicators, grantmaking 

practices, and level of federal funding. 

3.1.1 Financial Data 

Exhibit 4 compares SIF grantee organizations and their comparison groups on three financial indicators 

for 2013: gross revenue, total value of government grants received, and total value of grants made to U.S. 

organizations. These data were obtained from GuideStar based on the nonprofit data submitted to the IRS 

through Form 990 or Form 990-PF.

SIF 2010-2012 grantees were similar to the comparison samples of Non-selected SIF Applicants and the 

National Sample of Grantmaking Nonprofits in terms of gross revenue and government grants received. 

Although similar to Non-selected SIF Applicants in terms of the value of the grants they made to U.S. 

organizations, SIF 2010-2012 grantees gave significantly more to U.S. organizations than did the National 

Sample of Grantmaking Nonprofits. 

Exhibit 4. Financial indicators by respondent group 

Financial indicators

(2013) SIF 2010-2012

Non-selected SIF 

Applicants 2010-2012

National Sample of 

Grantmaking Nonprofits

Gross Revenue n 15 9 147

Mean $76,573,121 $74,927,426 $98,431,599 

Median $44,479,384 $22,234,541 $ 6,833,348 

Total Value of 

Government Grants 

received

n 14 6 121

Mean $5,307,259 $980,788 $3,741,735 

Median $1,411,197 $618,451 $ 660,614 

Total Value of Grants 

made to US 

Organizations

N 15 9 218

Mean $21,573,525

b+ $33,890,743 $12,483,991 

Median $11,462,022 $ 8,278,806 $ 4,111,065 

Statistical significance (p<.05) is noted as: 

 a: significant difference between SIF 2010-2012 and Non-selected SIF Applicants 2010-2012

 b: significant difference between SIF and the National Sample of Grantmaking Nonprofits 

 +: favoring SIF 

 -: favoring the comparison group 

Sample sizes for Exhibit 4 are smaller than those reported in other exhibits due to missing data. 
nationalservice.gov/research 13

3.1.2 Grantmaking Practices 

The survey asked Non-selected SIF Applicants and the National Sample of Grantmaking Nonprofits a set 

of questions concerning their grantmaking practices to ensure that they fit the selection criteria for the 

comparison group. 

SIF grantees were geographically based, issue-based, or both. A great majority of both Non-selected SIF 

Applicants and the National Sample of Grantmaking Nonprofits were geographically based. About half 

these organizations were issue-based, either exclusively or in addition to their geographic focus, meaning 

that their programs focus on specific issue areas but may be located in a variety of geographic locations 

around the U.S. (Exhibit 5). 

Exhibit 5. Grantmaking focus for the comparison groups 

Grantmaking focus

Non-selected 

SIF applicants 

2010-2012 

(n=9)

National Sample 

of Grantmaking 

Nonprofits

(n=217)

Geographically based (make grants to organizations within a community, state, or 

other specified geographic region) 88.9% 80.2%

Issue-based (make grants to organizations whose programs focus on specific issue 

areas, but may be located in a variety of geographic locations around the U.S.) 55.6% 41.9%

Both geographically based and issue-based 44.4% 30.0%

Other 0.0% 16.1%

Exhibit 6 shows that the comparison samples of both Non-selected SIF Applicants and the National 

Sample of Grantmaking Nonprofits were similar in terms of their grantmaking areas, with the majority of 

the organizations awarding grants in the same areas targeted by the SIF: healthy futures, youth 

development, and economic opportunity. A third of the National Sample of Grantmaking Nonprofits 

reported that they make grants, but not in any of the three SIF areas. A greater percent of Non-selected 

SIF Applicants than the National Sample of Grantmaking Nonprofits fund youth development and 

economic opportunity. 

Exhibit 6. Grantmaking areas for the comparison groups 

Grantmaking area

Non-selected 

SIF applicants

2010-2012 

(n=9)

National Sample 

of Grantmaking 

Nonprofits

(n=213)

Community health or healthy futures: Promoting healthy lifestyles and reducing the 

risk factors that can lead to illness. Interventions include disease prevention and lowincome healthcare/outreach.66.7% 67.6%

Youth development (includes early childhood development): Preparing America’s 

youth for success in school, active citizenship, productive work, and healthy and safe 

lives. Interventions include summer or after-school programming, delinquency 

prevention, and youth mentoring/tutoring/volunteers.

100.0% 73.7%

Economic opportunity: Increasing economic opportunities for economically 

disadvantaged individuals. Interventions include job or career training or job 

readiness for adults.

77.8% 55.9%

Makes grants to U.S. nonprofit organizations to carry out programs or interventions, 

but not in any of the areas listed 33.3% 33.8%

Makes grants to U.S. nonprofit organizations, but not to carry out programs or 

interventions 0.0% 7.0%
nationalservice.gov/research 14

Finally, nonprofit respondents in the sample showed limited knowledge of the SIF: 45% of grantmaking 

organizations and 71% of non-grantmaking nonprofits were unaware of the SIF. Only about 10% of 

grantmaking nonprofits and 5% of all nonprofits knew quite a lot or a great deal about the SIF (Exhibit 7). 

Exhibit 7. Level of knowledge about CNCS among nonprofits 

45%

42%

10%

4%

71%

24%

4%

2%

None–not aware of CNCS Social Innovation Fund

Some knowledge–have heard of it and know something 

about it

Know quite a lot about the CNCS Social Innovation Fund

Know a great deal about it–very familiar with the program

Nonprofit (not grant-making) (n=166)

Nonprofit (grant-making) (n=218)

3.1.3 Federal Funding 

The survey asked about other federal funding sources because of indications from CNCS and SIF 

grantees that some grantees found it challenging to comply with federal funding requirements. These 

data provide context for analysis of the SIF experience and capacity development, and suggest 

consequences for subsequent federal funding. 

Exhibit 8 compares the proportion of grantees in each group that received federal funding in 2009 and 

2014. Several interesting findings emerge. First, two-thirds of SIF 2010-2012 grantees had no prior federal 

funding. In addition, among the National Sample of Grantmaking Nonprofits, about two-thirds had 

neither current nor previous federal funding. 
nationalservice.gov/research 15

Exhibit 8. Federal funding experience of SIF grantees and the comparison groups 

The survey also asked SIF grantees whether SIF was the first federal funding their organization had ever 

received. Among the SIF 2010-2012 grantees, half indicated that it was (not shown in exhibits). 

3.2 Support Received by SIF Grantees 

CNCS provides extensive technical 

assistance (TA) and other support to SIF 

grantees. This includes hands-on support 

throughout from CNCS SIF program 

officers; extensive support, TA, and 

review of evaluation plans and products 

by CNCS’s Office of Research and 

Evaluation and their evaluation TA 

contractor; and a variety of efforts to 

engage the grantees as a group (e.g., 

convenings, conference calls, and resource 

sharing). Through these means, CNCS 

provides substantial assistance to 

individual grantees, and also leverages 

their capacity by connecting SIF grantees 

to one another and giving them 

opportunities to share their strengths and 

learn from each other’s strengths. Six key 

elements make up the SIF’s capacity 

building approach (Exhibit 9). The SIF 

builds capacity around evaluation, federal 

grants management and compliance, and 

communication and outreach, and it does this through convenings, the Knowledge Network, and the 

Knowledge Initiative. 

37%

45%

63% 63%

33%

36%

2009 2014

Change: 

8.2%

Change:

3.5%

SIF 2010-2012 

(n=19)

Grant-making 

nonprofits (n=180)

Change: 

0%

Non-selected SIF 

applicants (n=8)

Exhibit 9. Social Innovation Fund (SIF) Capacity Building 

and Knowledge Sharing Program
nationalservice.gov/research 16

The survey asked SIF grantees to reflect on the support from the SIF. Open-ended questions gave them 

opportunities to share feedback about TA or support (e.g., coaching, facilitation, and tools) received 

during the SIF funding period to help increase capacity or make change. Specifically, grantees shared 

feedback on which support was more or less helpful, and whether they would like additional TA or 

support in specific areas. 

Helpful support. Two or more grantees identified each of the following areas as particularly helpful: 

federal compliance, networking, evaluation planning, general support by the SIF program office, the 

annual convening, external contractors, and other external training and resources such as Scaling What 

Works http://www.scalingwhatworks.org, the Knowledge Initiative, and issue area groups. Below are 

excerpts from grantee responses describing how they benefitted:

 We received a lot of support from the SIF evaluation team at CNCS and the evaluation TA contractor with

finalizing the evaluation plans. We also felt the Knowledge Initiative has been really helpful for connecting with

other organizations. The SIF grantee calls with specific topics have helped

us find other intermediaries to connect with and learn from. For instance,

we have coordinated two proposals with another SIF grantee regarding

our work around a shared measurement framework.

 The annual convening is particularly helpful. That face-to-face contact is

very valuable. I think the most helpful resource from SIF was the SIF

program officer—she is outstanding and an incredible resource in terms

of helping navigate requirements of the grant and with planning and

thinking about program objectives and requirements. She was very

helpful in the start-up phases and now in year 3 continues to be helpful

as things change.

Less helpful support. Two or more SIF 2010-2012 grantees reported as 

less helpful: the SIF Evaluation Plan (SEP) review and webinars, evaluation, and the Knowledge 

Network. Other types of support were each cited once by different grantees as less helpful: the annual 

SIF meeting, grantee calls, communications, financial tracking, and compliance coaching. Some of these 

areas coincide with those that other grantees considered to be helpful, showing that these responses 

may vary depending on the individual grantee’s perspectives and experiences. The comments provide 

context or suggestions about ways to improve: 

 SIF grantee calls: The effectiveness of an issue group has varied; some calls and peer exchange are helpful, but

given our different program model, not all topics are relevant.

 SEP webinars: Receive resources/training before grant requirements are released to give grantees and

subgrantees more time to prepare.

 Evaluation. The changes in national evaluation TA provider and expectations of grantees made this difficult.

Evaluation TA is not always helpful. Their approach seems restrictive and they were not always timely.

 Financial tracking: Financial tracking spreadsheets and e-grants are awkward and difficult to report with

accurately.

 Knowledge Network: Make it more user friendly, easier to navigate.
nationalservice.gov/research 17

Areas requiring more support. SIF 2010-2012 grantees identified a wide range of areas in which they 

would have benefited from more support or resources. Areas cited by two or more respondents included 

criminal history check, compliance, implementing evaluation expectations/guidance, and training for 

subgrantees. Other types of support were each suggested once by different grantees: dividing grantees by 

approach rather than issue area, more support for interactions with SIF grant officers, a planning year, 

sustainability, financial reporting tools, TA from the start of the grant, connecting grantees to funders, a 

filing system based on SIF monitoring tools, and bringing subgrantees together to advance philanthropic 

understanding of scaling. The following comments provide additional context: 

 More support around the flexibility or levels of interpretation within SIF would have been helpful in figuring 

out how rigorous evaluation needed to be and what qualified as match or did not. 

 Although none of us were specifically funded to do so, I feel as though the SIF missed an opportunity to bring 

together subgrantees to substantially move forward philanthropic understanding of scaling. There were some 

dynamic conversations that were started at SIF gatherings, and many of the grantees have individually reported 

on their learnings, but, in hindsight, CNCS should have put more effort into bringing these lessons together 

with a focus on informing others. 

3.3 Impact on Organizational Capacity 

The main objective of the national assessment is to look at the SIF’s impact on the organizational capacity 

of grantees, not only with respect to the SIF program, but also organization-wide. This section examines 

organizational capacity in five areas, including selection of subgrantees, support for subgrantees, 

evaluation, scaling up of evidence-based programs, and collaboration. The discussion of each area is 

organized as follows: 

 SIF Grantee Changes. Comparison between the organizational capacity of SIF 2010-2012 grantees in 

2009 and their capacity in 2014 

 Changes for SIF Grantees vs. Comparison Groups. Comparison between SIF 2010-2012 grantees and 

two comparison groups—Non-selected SIF Applicants and the National Sample of Grantmaking 

Nonprofits in terms of any changes in organizational capacity from 2009 to 2014 

 SIF Grantee Observations about Change. SIF grantee observations obtained from open-ended 

questions about organizational capacity 

 SIF Grantee Evidence of Change. Evidence from follow-up interviews with SIF grantees and 

grantee-provided documentation of change 

3.3.1 Selection of Subgrantees 

The survey asked how SIF grantees selected subgrantees in both 2009 and 2014. It also inquired about 

changes the SIF grantees experienced, as intermediary organizations, during this period and, for those 

that reported changes, the major factors that contributed to the changes. 

SIF Grantee Changes. Exhibit 10 shows that about one-third of SIF grantees reported changes in grantee 

selection practices. Between 2009 and 2014, SIF grantees in the 2010-2012 cohort raised the requirement 

that—to be eligible for funding—applicant organizations provide evidence of intervention effectiveness 

(Item B: 5.0 to 5.3) and submit a plan for rigorous evaluation of the intervention (Item C: 4.8 to 5.1). The 

practice of operating an open and competitive process to solicit and review applications and to make 
nationalservice.gov/research 18

selection decisions changed in a negative direction between 2009 and 2014 (Item A: 4.6 to 4.4). The 

reported decrease primarily reflects one grantee’s account.3

Exhibit 10. Changes in how SIF 2010-2012 grantees and comparison groups approached 

subgrantee selection 

Change of rating from 2009 to 2014

(7 point Likert Scale: 1= Not at all to 7= Always)

Percentage of organizations reporting

change over time

A. Used an 

open 

competitive 

process to 

solicit and 

review 

applications 

and to make 

selection 

decisions b

SIF vs Non-selected

Applicants

SIF vs National sample of 

Nonprofits

Effect size (d) -0.3 -0.7

B. Required 

applicant 

organizations 

to provide 

evidence of 

intervention 

effectiveness 

to be eligible 

for funding 

C. Required 

applicants to 

submit a plan 

for rigorous 

evaluation of 

intervention to 

be eligible for 

funding 

SIF vs Non-selected

Applicants

SIF vs National sample of 

Nonprofits

Effect size (d) 0.3 -0.2

SIF vs Non-selected

Applicants

SIF vs National sample of 

Nonprofits

Effect size (d) 0.4 -0.7

4.6 4.4

5 5.3

4.8 5.1

2009 2014

SIF (change = -0.1)

Non-selected SIF Applicants (change = 0.3)

National Sample of Grantmaking Nonprofits (change =0.3)

32%

44%

28%

SIF 2010-

2012 (n=19)

Applicant

(n=9)

Nonprofit

(n=207)

3.5

4

4.8 5.1

3.7

4.5

2009 2014

SIF (change = 0.6)

Non-selected SIF Applicants (change = 0.3)

National Sample of Grantmaking Nonprofits (change =0.8)

42%

33%

44%

SIF 2010-

2012 (n=20)

Applicant

(n=9)

Nonprofit

(n=202)

2

2.3

2.9

3.1

2.4

2.9

2009 2014

SIF (change =0.4)

Non-selected SIF Applicants (change = 0.2)

National Sample of Grantmaking Nonprofits (change =0.5)

26%

22%

25%

SIF 2010-

2012 (n=19)

Applicant

(n=9)

Nonprofit

(n=200)

Statistical significance (p<.05) is noted as: 

 a: significant difference between SIF 2010-2012 grantees and Non-selected SIF Applicants 2010-2012 

 b: significant difference between SIF 2010-2012 grantees and the National Sample of Grantmaking Nonprofits 

 

3

 For selecting subgrantees using open competition, 6 of the 20 grantees reported changes. The average response was largely influenced by 

one grantee who reported always using open competition in 2009 but never using open competition in 2014. 
nationalservice.gov/research 19

Changes for SIF Grantees vs. Comparison Groups. Compared to the National Sample of Grantmaking 

Nonprofits, SIF 2010-2012 grantees experienced significantly less change between 2009 and 2014 on the 

extent to which the organization used an open competitive process to solicit and review applications and 

to make selection decisions, again largely due to change from one grantee. We also found medium to 

small differences in favor of the National Sample of Grantmaking Nonprofits in all three areas. The 

findings were somewhat inconsistent from the Non-selected 

SIF Applicants. While the same was true in using an open 

competitive process, we found small differences in favor of 

SIF grantees in effect sizes with regard to requiring applicants 

to provide evidence of effectiveness, and to submit a plan for 

rigorous evaluation in order to be eligible for funding. 

SIF Grantee Observations about Change. Among SIF 2010-

2012 grantees, a majority attributed changes in their process of 

selecting subgrantees to their participation in the SIF; among 

the National Sample of Grantmaking Nonprofits, a majority 

attributed changes to direction from the organization’s board or other leadership. Both comparison 

groups also mentioned trends in the broader grantmaking world as important factors contributing to 

change. (See Appendix D for details). 

The National Sample of Grantmaking Nonprofits also pointed out that they designed these changes to 

make the grant competition more fair and transparent. One respondent noted that “Donor tolerance for 

funding that does not have measurable results has changed.” Some added that their organizations have 

always required prior evidence in grant applications, but in some cases, prior evidence is not easy to 

come by. For example, one nonprofit respondent observed, “We fund arts experiences; (It is) difficult to 

quantify the impact of arts education experiences in one year.”

In open-ended comments about how their participation in the SIF contributed to their approach to 

selecting subgrantees, SIF 2010-2012 grantees reiterated the themes of implementing application-based 

competitions, emphasizing evidence and evaluation, and increasing use of tools. Below are comments 

that offer context: 

 The SIF RFP process was the first ever for this

organization. We found this process to be helpful and

plan to use it in the future.

 For another program, we now use a competitive

selection process for all of our new, non-SIF sites. We

found the selection process to be a valuable method of

pre-planning and gauging interest and dedication to

the model.

SIF Grantee Evidence of Change. SIF grantees were 

selected through a rigorous, open competition, and 

they select their subgrantees through a similar 

competitive process. For some grantees, conducting a 

competitive selection process was a significant 

change from the way in which they had traditionally 

made funding decisions. Some grantees saw 

Highlight: Focus on Outcomes

One SIF grantee partnered with other funders to 

expand collective focus on outcomes through the 

SIF. A collaboration of funders specified outcomes 

and used a competitive process to select 

subgrantees that have demonstrated these 

outcomes in prior work. Many of these same 

funders then applied this model to another 

collaborative grant program, where they are able to 

identify community programs that clearly 

demonstrate cost savings and sustainability.

Tools and Resources: Assessing Evaluation and 

Financial Management Capacity

One grantee developed a set of tools for assessing 

applicants’ evaluation and financial management capacity. 

Application reviewers used the evaluation assessment tool

to assess the strength of applicants’ current level of 

evidence, based on the soundness of their theory of change, 

their history with using evaluation, and their data collection 

capacity. Reviewers also rated applicants’ capacity to 

conduct an evaluation, based on their budgets, personnel, 

understanding of SIF levels of evidence, and evaluation 

methods. 

The financial management survey required applicants to 

provide copies of documents demonstrating their financial 

capacity, such as IRS forms, financial statements, audits, 

accounting policies, and procedures manuals. It also 

required them to describe the financial management 

qualifications of key staff and discuss their organizational 

policies and procedures for financial management. 
nationalservice.gov/research 20

significant value in a competitive selection process and applied it to other grant programs they manage. 

The follow-up interviews revealed these themes related to the SIF requirement for competitive selection:

 The SIF helped grantees build a more systematic process for subgrantee selection. Although

grantees did not institute many changes, a few did develop and use various combinations of requests

for proposals (RFPs), external and internal review teams, and tools and frameworks for reviewing

funding applications and selecting subgrantees in a systematic way.

 The SIF selection process helped grantees choose subgrantees with strong evidence. SIF grantees

were better able to identify applicants with a strong evidence base and the capacity to build evidence

further. Grantees designed SIF solicitations to clearly articulate the initial level of evidence they

expected of subgrantees.

 The SIF’s competitive solicitation requirement helped grantees reach subgrantees in new markets

and geographic areas. Some grantees reached applicants in markets and geographic areas they

would not have otherwise reached when they shifted from a direct selection process to a competitive

solicitation with the SIF.

3.3.2 Support for Subgrantees 

Grantmaking organizations vary in the extent to which they provide support to subgrantees to develop 

their capacity to accomplish the work. The survey asked respondents about both financial and nonfinancial assistance they provided to help subgrantees implement and evaluate their programs. 

SIF Grantee Changes. Exhibit 11 indicates that about one-third of SIF 2010-2012 grantees reported 

changes in the ways in which they support their subgrantees. On average, SIF grantees in the 2010-2012

cohort expanded the support they gave subgrantees by more often providing funding to carry out an 

evaluation or hire an external evaluator (Item A: 3.0 to 3.9), providing training or 

technical assistance to conduct rigorous evaluation (Item B: 3.6 to 4.3), or 

supporting implementation of the program (Item C: 4.4 to 5.0). 

Changes for SIF Grantees vs. Comparison Groups. Compared to the National 

Sample of Grantmaking Nonprofits, SIF 2010-2012 grantees experienced 

significantly greater changes between 2009 and 2014 in the extent to which the 

organization: 1) provided funding to carry out an evaluation or hire an external 

evaluator, as part of the grant or through other means, and 2) provided training 

or technical assistance to support conducting rigorous evaluation. The effect 

sizes in the latter two areas were large, but negligible in providing TA to support 

program implementation. Compared to Non-selected SIF Applicants, although 

the change patterns for SIF 2010-2012 grantees were not statistically significant, 

we found small differences in favor of SIF grantees in the magnitude of the 

differences regarding provision of TA to evaluation and TA to program 

implementation. 

SIF Grantee Observations about Change. Among organizations that reported changes in how they 

support grantees, the majority of SIF 2010-2012 grantees attributed these changes to their participation in 

the SIF, whereas a majority of the National Sample of Grantmaking Nonprofits attributed these changes 

to direction from the organization’s board/leadership, and to trends in the broader grantmaking world as 

major factors in driving technical assistance to support program implementation. (See Appendix D for 

details.) Another theme identified by SIF grantees in open-ended responses was that their organizational 
nationalservice.gov/research 21

business model had shifted toward a stronger emphasis on grantee support, in terms of either evaluation 

or technical assistance or both. 

Some SIF grantees specifically credited the SIF with changing how they provide TA to subgrantees—not 

only within their SIF programs, but across their broader portfolios. 

 SIF offered both the opportunity and the mandate to increase the level of TA for our subgrantees around

evaluation and program administration. In 2009 we offered small TA grants around program implementation,

and in 2014 we were able to offer larger one-on-one TA awards to our SIF subgrantees, as well as group TA and

training to support data collection and other evaluation activities.

 SIF's requirements around technical assistance and shared learning have impacted our impact work more

generally (not just the body of work that is funded by SIF). More importantly, SIF has been a "lever" that we've

used by demonstrating the government's commitment to this kind of support, we can make a stronger case for

providing it. Also, as part of SIF we joined another grantmaker in making grants, and their promotion of this

message of support beyond the grant has been influential.

Exhibit 11. Changes in support for subgrantees from SIF grantees and comparison groups

Change of rating from 2009 to 2014

(7 point Likert Scale: 1= Not at all to 7= Always)

Percentage of organizations 

reporting change over time

A. Provided 

funding to 

carry out an 

evaluation or 

hire an 

external 

evaluator, as 

part of the 

grant or 

through other 

means b SIF vs Non-selected

Applicants

SIF vs National sample of 

Nonprofits

Effect size (d) -0.1 1.3

B. Provided 

training or 

technical 

assistance by 

your staff 

consultants or 

other means 

to conduct 

rigorous 

evaluation

SIF vs Non-selected

Applicants

SIF vs National sample of 

Nonprofits

Effect size (d) 0.5 0.9

3

3.9

3.2

4.3

2.2

2.6

2009 2014

SIF (change = 0.9)

Non-selected SIF Applicants (change = 1.1)

National Sample of Grantmaking Nonprofits (change =0.4)

40% 44%

26%

SIF 2010-

2012 (n=20)

Applicant

(n=9)

Nonprofit

(n=189)

3.6

4.3

3.7 3.9

2.5

2.9

2009 2014

SIF (change = 0.8)

Non-selected SIF Applicants (change = 0.2)

National Sample of Grantmaking Nonprofits (change =0.4)

35%

44%

28%

SIF 2010-

2012 (n=20)

Applicant

(n=9)

Nonprofit

(n=189)
nationalservice.gov/research 22

Change of rating from 2009 to 2014

(7 point Likert Scale: 1= Not at all to 7= Always)

Percentage of organizations 

reporting change over time

C Provided 

training or 

technical 

assistance by 

your staff 

consultants or 

other means 

to support 

implementatio

n of the 

program SIF vs Non-selected

Applicants

SIF vs National sample of 

Nonprofits

Effect size (d) 0.3 0.1

4.4

5

3.6

3.9

2.9

3.5

2009 2014

SIF (change =0.6)

Non-selected SIF Applicants (change = 0.3)

National Sample of Grantmaking Nonprofits (change =0.6)

35%

33%

36%

SIF 2010-

2012 (n=20)

Applicant

(n=9)

Nonprofit

(n=193)

Statistical significance (p<.05) is noted as: 

 a: significant difference between SIF 2010-2012 grantees and Non-selected SIF Applicants 2010-2012 

 b: significant difference between SIF 2010-2012 grantees and the National Sample of Grantmaking Nonprofits

SIF Grantee Evidence of Change. The SIF 

increased the amount of support that grantees 

were expected to provide to their subgrantees, 

and required them to think carefully about how 

they provided this support. Ultimately, the 

support that subgrantees received through the 

SIF gave them the foundation they needed to 

grow their impact. Follow-up interviews 

uncovered several themes about grantee support:

 SIF grantees increased their investment in growing subgrantees capacity. Grantees invested more

in increasing the capacity of their SIF subgrantees through the SIF, especially capacity related to

compliance, evaluation, and communications.

 The SIF helped grantees become more strategic

about how they support their subgrantees. The

SIF led to a more sophisticated and

standardized approach to TA than many

grantees had used prior to receiving the grant.

Several grantees developed standards for

providing TA across sites.

 The support that SIF grantees provided to their

subgrantees gave them a solid foundation

from which to grow. Grantees helped

subgrantees develop the skills to implement

projects with the level of rigor required by the

SIF. Grantees made efficient use of the TA they

were required to provide by ensuring that it was

useful for the SIF, as well as applicable to efforts well beyond the SIF.

Highlight: Hands-on Support to Subgrantees

The SIF increased the amount of hands-on support and 

oversight that many grantees provided to their 

subgrantee organizations. One grantee developed a 

strategy for providing this hands-on support to its 

subgrantee organizations by implementing frequent 

check-ins with the subgrantees and working closely with 

them throughout the grant. As a result, the grantee 

helped integrate the organizations’ programmatic, 

clinical, finance, compliance, and human resources staff 

to improve program implementation. One subgrantee 

had difficulty developing this level of integration, and 

implemented a corrective action plan to support 

communication across all levels of the organization. 

Tools and Resources: Menu of TA Options

By placing a strong emphasis on approaching TA in a strategic 

way, the SIF helped grantees understand subgrantees’ TA 

needs, plan their TA strategies, and communicate more clearly 

to subgrantees the TA they could provide. One grantee 

developed a resource that summarized the host of technical 

assistance options available to subgrantees so that they could 

better understand the assistance available to them to support 

and sustain the SIF model. 
nationalservice.gov/research 23

3.3.3 Use of Data, Evidence, and Evaluation 

The survey included questions 

about how SIF grantees used 

evaluation in 2009 and 2014 and 

their evaluation resources and 

infrastructure. In addition, 

respondents were asked whether 

their organizations had changed the 

way they use data and evidence, 

and what factors contributed to 

those changes. 

SIF Grantee Changes. Evaluation is 

the area in which SIF grantees 

identified the greatest changes, with 

about two-thirds reporting 

increases in their capacity to 

conduct rigorous program 

evaluation (Item A: 3.1 to 4.5), use 

evaluation findings to improve 

programs (Item B: 3.9 to 4.9), and 

demonstrate and communicate 

program effectiveness (Item C: 3.7 

to 4.9) (Exhibit 12). This outcome 

resonates with the substantial focus that the SIF program places on this area. 

Changes for SIF Grantees vs. Comparison Groups. Compared to the National Sample of Grantmaking 

Nonprofits, SIF 2010-2012 grantees experienced significantly more growth between 2009 and 2014 in all 

three evaluation areas, including 1) conducting rigorous evaluations of the programs; 2) using evaluation 

findings to improve programs; and 3) using evaluation findings to demonstrate and communicate 

effectiveness of programs funded by the organization. The effect size differences in all three areas were 

large. Compared to Non-selected SIF Applicants, SIF 2010-2012 grantees experienced significantly greater 

changes between 2009 and 2014 in the extent to which the organization conducted rigorous evaluations of 

programs funded by the organization, and the effect size was large. Although the differences in the other 

two areas were not statistically significant due to small sample sizes, the effect sizes were medium and 

small respectively. 

Tools and Resources: Subgrantee Performance Dashboard

One grantee developed a performance dashboard to track subgrantee performance on a 

variety of financial, administrative, and programmatic indicators. Organizations are 

flagged, based on performance benchmarks, and red or yellow flags highlight potential 

needs for targeted technical assistance.
nationalservice.gov/research 24

Exhibit 12. Changes in how SIF grantees and comparison groups approached evaluation 

Change of rating from 2009 to 2014

(7 point Likert Scale: 1= Not at all to 7= Always)

Percentage of organizations reporting

change over time

A. Conducted 

rigorous 

evaluations of 

programs 

funded by your 

organization ab

3.9

4.9 5.1 5.4

3.6

4.3

2009 2014

SIF (change = 1.0)

Non-selected SIF Applicants (change = 0.3)

National Sample of Grantmaking Nonprofits (change =0.7)

SIF vs Non-selected

Applicants

SIF vs National sample of 

Nonprofits

Effect size (d) 1.0 1.9

B. Used 

evaluation 

findings to 

improve 

programs 

funded by your 

organization b

SIF vs Non-selected

Applicants

SIF vs National sample of 

Nonprofits

Effect size (d) 0.7 0.8

C. Used 

evaluation 

findings to 

demonstrate 

and 

communicate 

effectiveness 

of programs 

funded by your 

organization b

SIF vs Non-selected

Applicants

SIF vs National sample of 

Nonprofits

Effect size (d) 0.3 0.8

3.1

4.5

3.3 3.6 3.4

4.1

2009 2014

SIF (change = 1.4)

Non-selected SIF Applicants (change = 0.4)

National sample of grantmaking nonprofits (change =0.7)

70%

25%

40%

SIF 2010-

2012 (n=20)

Applicant

(n=8)

Nonprofit

(n=184)

60%

22%

39%

SIF 2010-

2012 (n=20)

Applicant

(n=9)

Nonprofit

(n=183)

3.7

4.9 4.9

5.7

3.8 4.6

2009 2014

SIF (change = 1.2)

Non-selected SIF Applicants (change = 0.8)

National Sample of Grantmaking Nonprofits (change =0.8)

60%

33%

43%

SIF 2010-

2012 (n=20)

Applicant

(n=8)

Nonprofit

(n=181)

Statistical significance (p<.05) is noted as: 

 a: Significant difference between SIF 2010-2012 grantees and Non-selected SIF Applicants 2010-2012

 b: Significant difference between SIF 2010-2012 grantees and the National Sample of Grantmaking Nonprofits 

In addition to change in organizational capacity and practice, the survey asked about changes in 

evaluation staffing and budget. It also examined evaluation capacity through the lens of personnel and 

budget as measures of organizational infrastructure to support growth. 

SIF grantees, Non-selected SIF Applicants, and the National Sample of Grantmaking Nonprofits all 

reported increases between 2009 and 2014 in several indicators of evaluation capacity, including the 
nationalservice.gov/research 25

presence of staff position(s) or groups within the organization dedicated to evaluation; external 

evaluation partners that provide the organization with evaluation services; and line items in the 

organization’s budget dedicated to evaluation. 

SIF 2010-2012 grantees had a significantly higher baseline of evaluation capacity in 2009 than the National 

Sample of Grantmaking Nonprofits, meaning SIF 2010-2012 grantees were more likely in 2009 to have: 1) 

staff position(s) or a group within the organization dedicated to evaluation; 2) external evaluation 

partner(s) that provide evaluation services; and 3) line items in the organization’s budget dedicated to 

evaluation. However, SIF grantees also experienced significantly more growth than did the National 

Sample of Grantmaking Nonprofits in these areas (Exhibit 13).

Exhibit 13. Changes in evaluation infrastructure for SIF grantees and comparison groups 

Did/does your organization 

have the following?

2009 2014

Group n % (yes) n % (yes) Change

Staff position(s) or a group 

within your organization 

dedicated to evaluation 

SIF 2010-2012 20 50.0%b+ 20 70.0% 20.0%b+

Non-selected SIF 

Applicants 2010-2012 8 50.0% 8 75.0% 25.0%

National Sample of 

Grantmaking Nonprofits 187 30.0% 189 38.9% 7.7%

External evaluation 

partner(s)—consultant(s) or 

organization(s)—that provide 

your organization with 

evaluation services

SIF 2010-2012 20 70.0%b+ 20 95.0% 25.0%b+

Non-selected SIF 

Applicants 2010-2012 8 62.5% 8 87.5% 25.0%

National Sample of 

Grantmaking Nonprofits 186 34.9% 189 45.4% 9.6%

Part of the organization's 

budget dedicated to 

evaluation

SIF 2010-2012 20 75.0%b+ 20 95.0% 20.0%b+

Non-selected SIF 

Applicants 2010-2012 8 62.5% 8 87.5% 25.0%

National Sample of 

Grantmaking Nonprofits 187 35.5% 189 47.2% 11.7%

Statistical significance (p<.05) is noted as: 

 a: significant difference between SIF and Non-selected SIF Applicants 

 b: significant difference between SIF and National Sample of Grantmaking Nonprofits 

 +: favoring SIF 

 -: favoring the comparison group 

Exhibit 14 shows that the majority of SIF grantees and Non-selected SIF Applicants reported that their 

evaluation budgets were significantly higher in 2014 than in 2009 in terms both of total dollar amount and 

as a percentage of the organization’s total annual budget. In contrast, the National Sample of 

Grantmaking Nonprofits was more likely to report that their evaluation budgets were about the same in 

2014 as in 2009. 
nationalservice.gov/research 26

13%

17%

30%

16%

33%

25%

63%

50%

35%

7%

5%

2%

5%

National Sample of

Grantmaking

Nonprofits (n=156)

Non-selected SIF

Applicants (n=6)

SIF 2010-2012 (n=20)

Evaluation budget as a % of organization’s total 

annual budget - change from 2009 to 2014

Substantially higher Somewhat higher About the same Somewhat 

lower Substantially lower 

17%

33%

50%

25%

50%

20%

58%

17%

30%

<1%

National Sample of

Grantmaking

Nonprofits (n=160)

Non-selected SIF

Applicants (n=6)

SIF 2010-2012 (n=20)

Total evaluation budget ($) - change from 2009 to 

2014 

Exhibit 14. Reported changes in evaluation budgets of SIF grantees and comparison groups 

SIF grantees significantly exceeded the National Sample of Grantmaking Nonprofits in terms of annual 

evaluation budget as a percent of the organization’s total budget (4.1% higher) in 2009, and experienced a 

greater increase in that budget (1.7%) in 2014 (Exhibit 15). One caveat is that, in some cases, the 

evaluation budget may be included in the program budget rather than a separate line item. In addition, 

the mean values for the National Sample of Grantmaking Nonprofits were skewed by one outliner that 

has a much higher evaluation budget than any others. Across the three groups, no differences were found 

in the number of full-time equivalent (FTE) staff with primary responsibility for evaluation. One possible 

explanations is that SIF grantees tend to hire external evaluators, because internal staff are not equipped 

to conduct the level of rigorous evaluations required. Nevertheless, additional staff may be hired to 

oversee these evaluations. 
nationalservice.gov/research 27

Exhibit 15. Changes in the evaluation budgets of SIF grantees and comparison groups 

Did/does your 

organization have 

the following?

2009 2014

Change 

(mean) Group n Mean Median Min. Max n Mean Median Min. Max

Total annual 

evaluation budget (in 

$) – for in-house 

evaluators or 

external partners

SIF grantees 

2010-2012 10 $860,590 $170,450 $0 $6,300,000 10 $1,789,800 $425,000 $0 $10,100,000 $939,344

Non-selected SIF 

Applicants 2010-

2012

1 $150,000 $150,000 $150,000 $150,000 1 $200,000 $200,000 $200,000 $200,000 $50,000

National Sample of 

Grantmaking 

Nonprofits

66 $568,817 $0 $0 $60,000,000 58 $1,115,810 $0 $0 $63,000,000 $43,195

Annual evaluation 

budget as a % of the 

total organization 

budget

SIF 2010-2012 13 4.1%b+ 1.5% 0.0% 10.0% 13 5.8% 4.0% 0.0% 14.0% 1.7%b+

Non-selected SIF 

Applicants 2010-

2012

0 -- -- -- -- 0 -- -- -- -- --

National Sample of 

Grantmaking 

Nonprofits

76 1.2% 0.0% 0.0% 70.0% 80 2.9% 1% 0% 65.0% 0.8%

Number of full time 

equivalent (FTE) 

staff with primary 

responsibility for 

evaluation

SIF 2010-2012 17 1.0 0.5 0 3 17 1.7 1 0 7 0.7

Non-selected SIF 

Applicants 2010-

2012

3 0.5 0.5 0 1 3 1.3 1 1 2 0.8

National Sample of 

Grantmaking 

Nonprofits

101 1.8 0 0 150 104 3.1 .40 0 200 0.9

Statistical significance (p<.05) is noted as: 

 a: significant difference between SIF and Non-selected SIF Applicants 

 b: significant difference between SIF and National Sample of Grantmaking Nonprofits 

 +: favoring SIF 

 -: favoring the comparison group
nationalservice.gov/research 28

SIF Grantee Observations about 

Change. For those SIF 2010-2012

grantees reporting changes in 

their use of evaluation, the 

majority attributed the change to 

their participation in the SIF. In 

contrast, a majority of the 

National Sample of Grantmaking 

Nonprofits credited direction 

from the organization’s 

board/leadership for the change 

(See details in Appendix D). In 

the open-ended responses, 

respondents noted other factors 

such as increased capacities in 

terms of staff, software, available 

evidence-based programs, and 

framework change, with an emphasis on collective impact evaluation. 

When asked how participation in the SIF had contributed to their organization’s approach to evaluation, 

2010-2012 SIF grantees noted an increase in their communication of findings, appreciation of evaluation, 

and use of resources and tools. 

 SIF's emphasis on rigorous evaluation is giving us an opportunity to demonstrate to our organization as a

whole how evaluation can impact results. It's allowing us to show what it means to fund for impact.

 The SIF helped us look at regional evaluation capacity—both from a grantee and evaluator standpoint. We are

working with other local funders to build capacity. We are much more focused on evaluating at both a program

level and our own community change efforts.

SIF Grantee Evidence of Change. SIF grantees and subgrantees rigorously evaluate their programs to 

strengthen their base of evidence and assess their programs’ effectiveness, efficiency, and impact in the 

communities they serve. SIF grantees planned and implemented more evaluation than they had before 

the SIF, hired evaluation staff to increase their 

evaluation capacity, used evidence to improve 

results, and communicated results to prospective 

funders. Multiple themes emerged from the 

interviews regarding evaluation: 

 The SIF emphasis on evaluation helped its

grantees plan and implement more—and more

formalized—evaluation. To accommodate the

SIF’s strong emphasis on evaluation, grantees

implemented more formalized frameworks to

prepare for and implement evaluations and

became more intentional about how they

related evaluation results to growth.

Tools and Resources: Strategic Scorecard

The SIF served as a pilot for one grantee to be more strategic about measuring 

outcomes. The grantee’s SIF team worked with subgrantees to identify common 

outcomes and indicators on a list of key initiatives. Then, the leadership team and the 

board of directors developed and adopted a new organization-wide scorecard that 

outlined one-year performance metrics and three-year outcomes expected. 

For example, for the economic opportunity key initiative, the 3-year outcome was 

“bridging systems that foster innovative, talent-development approaches, resulting in 

pathways to careers for youth and adults,” and the related 1-year performance metric 

was “increasing cross-sector partners participating in collaborative systems 

development from 18 to 21 to connect individuals to careers and skilled trades.” The 

scorecard provided space to report the projected previous fiscal year’s baseline data; 

previous quarter, current quarter, and year to date data; and the next fiscal year’s goal 

for each key initiative.

The scorecard was unique in that it integrated outcomes measures for all teams, 

including teams not focused on impact (e.g., volunteer management).

Tools and Resources: 

Support for External Evaluator Selection 

Many SIF grantees and subgrantees relied on external evaluators 

to implement rigorous evaluations. One grantee supported its 

subgrantees’ use of external evaluators by providing them with an 

RFP template they could use to select external evaluators and by 

participating in interviews with prospective evaluators. The RFP 

template helped organizations provide prospective evaluators 

with background on the program’s purpose, vision, and mission. It 

also included a sample scope of work that outlined the outcomes 

to be explored and measured, evaluation expectations, and 

potential evaluation challenges. This template helped four 

subgrantees run a competitive selection process to select a new 

external evaluator for its program.
nationalservice.gov/research 29

 Grantees increased their evaluation capacity by hiring evaluation staff and external evaluation

partners. SIF grantees learned quickly that they had to devote a significant amount of capacity to

evaluation, and they engaged staff and external partners to focus specifically on evaluation.

 The SIF emphasis on evaluation helped

grantees use evidence to improve results. The

SIF helped grantees orient their programs toward

using evidence to inform programmatic

improvements in order to achieve a bigger

impact in the communities they serve.

 SIF subgrantees attracted new funders as a

result of their skills in conducting rigorous

evaluation and communicating program

effectiveness. The SIF created a culture of

evaluation, embedding it into funding objectives

and offering funding to subgrantees to conduct

evaluations. Having the capacity to conduct

rigorous evaluations armed grantees and subgrantees with robust data, which enabled them to

communicate more effectively about their program’s impact and value.

3.3.4 Scaling Up Programs with Evidence of Effectiveness 

Increasingly, federal and nonprofit grantmakers are concentrating on scaling up programs that have 

demonstrated evidence of effectiveness. Scale-up may be broadly defined as increasing the impact of a 

program within a community or expanding it to other communities or populations. 

SIF Grantee Changes. About half of the SIF 2010-2012 grantees reported changes in their efforts to scale 

up existing programs (Item A: 4.2 to 5.1), and to select programs for scale-up based on rigorous 

evaluation that demonstrates their effectiveness (Item B: 3.5 to 4.6) (Exhibit 16). 

Changes for SIF Grantees vs. Comparison Groups. Compared to the National Sample of Grantmaking 

Nonprofits, SIF 2010-2012 grantees experienced significantly greater changes between 2009 and 2014 in

the extent to which the organization selected programs for scale-up based on rigorous evaluation that 

demonstrates their effectiveness; the effect size was medium. Although no statistically significant 

difference was found in undertaking efforts to scale up existing programs, we found small distinctions in 

favor of SIF grantees in the magnitude of those differences. Compared to Non-selected SIF Applicants, 

although the change patterns for SIF 2010-2012 grantees were not statistically significant, we found small 

distinctions in favor of SIF grantees in the magnitude of the differences in both areas. 

Highlight: Demonstrating the Benefits of Evaluation

Conducting in-depth program evaluation is a core component 

of the SIF. One grantee is using a randomized controlled trial 

(RCT) to evaluate their programs. Given that subgrantees are 

often focused on using the SIF funding to serve as many 

people as possible with the supportive housing intervention, 

some subgrantees were hesitant to implement an RCT for fear 

of not being able to serve all beneficiaries in need. However, 

the grantee worked closely with the subgrantees to 

understand their concerns and to demonstrate the value of an 

RCT evaluation. If and when the evaluation shows strong 

evidence, subgrantees will see that participants are served in 

the long term, and they can then use these results to scale the 

model to serve more people in need. 
nationalservice.gov/research 30

Exhibit 16. Changes in approach to scaling up programs for SIF grantees and comparison groups 

Change of rating from 2009 to 2014 

(7 point Likert Scale: 1= Not at all to 7= Always) 

Percentage of organizations 

reporting change over time 

A. Undertook 

efforts to scale 

up existing 

program(s)—

i.e., to expand

the program(s) 

within the 

community or to 

other 

communities or 

populations SIF vs Non-selected 

Applicants 

SIF vs National sample of 

Nonprofits 

Effect size (d) 0.4 0.3 

B. Selected 

programs for 

scale-up based 

on rigorous 

evaluation that 

shows them to 

be effective

SIF vs Non-selected 

Applicants 

SIF vs National sample of 

Nonprofits 

Effect size (d) 

4.2

5.1 4.5 5

2.9

4

2009 2014

SIF (change = 0.9)

Non-selected SIF Applicants (change = 0.5)

National Sample of Grantmaking Nonprofits (change =1.0)

53%

38%

53%

SIF 2010-

2012 (n=19)

Applicant

(n=8)

Nonprofit

(n=173)

3.5

4.6

3.4

4

2.4

3.3

2009 2014

SIF (change = 1.1)

Non-selected SIF Applicants (change = 0.6)

National Sample of Grantmaking Nonprofits (change =0.9)

53%

25%

39%

SIF 2010-

2012 (n=19)

Applicant

(n=8)

Nonprofit

(n=170)

0.3 0.6

Statistical significance (p<.05) is noted as: 

 a: significant difference between SIF 2010-2012 grantees and Non-selected SIF Applicants 2010-2012 

 b: significant difference between SIF 2010-2012 grantees and the National Sample of Grantmaking Nonprofits 

SIF Grantee Observations about Change. For those SIF 2010-2012 grantees reporting changes in scaling 

up programs, a majority attributed the changes to the organization’s participation in the SIF, whereas a 

majority of the National Sample of Grantmaking Nonprofits attributed changes to the direction from the 

organization’s board/leadership. SIF 2010-2012 grantees also cited direction from the organization’s 

board/leadership as a major factor (See details in Appendix D). In open-ended questions, respondents 

noted increased capacities in terms of staff, shifts in organizational strategies such as adoption of Resultsbased Accountability, and more funds raised to allow for scaling up. 

Asked how participation in SIF contributed to the organization’s approach to scaling up, SIF 2010-2012

grantees reported a greater understanding of scaling, more available funding, newly created staff 

positions, and evaluation readiness. For example, 

 For some organizations, their investment helps them build their evidence. For the SIF, we continued our

practice of funding organizations that either have been evaluated or were poised for rigorous evaluation within

the SIF 5-year timeframe.
nationalservice.gov/research 31

 SIF is providing us with the evidence needed to scale certain programs. The evaluation findings give the

rationale for scale. We also have a deeper understanding of what it takes to scale programs and how to support

organizations to grow. We would not have had that "forced learning" without SIF.

SIF Grantee Evidence of Change. Themes related to 

scaling are summarized as follows: 

 The SIF helped grantees develop methods for

choosing which interventions to scale up. The

emphasis on evaluation and the TA that grantees

provided to their subgrantees helped them better

understand which programs were likely candidates for

expansion.

 Grantees packaged what they learned from the SIF to

support scaling. The SIF served as a learning lab for

grantees, who took the lessons from the SIF and

applied them within their organizations to support scaling efforts.

3.3.5 Collaboration to Address Community Needs 

By providing SIF grants, CNCS also anticipates wider and deeper collaboration among SIF grantees, with 

other grantmakers, and among subgrantees to address community needs and improve collective impact. 

SIF Grantee Changes. Exhibit 17 shows that about half of the SIF 2010-2012 grantees indicated changes in 

collaboration between 2009 and 2014, reporting considerable changes in participating in funding 

alliance(s) with other nonprofit sector organizations (Item A: 3.6 to 4.3), knowledge sharing (Item B: 4.2 to 

5.2), and collaborating for purposes of advocacy (Item C: 3.5 to 4.1). 

Changes for SIF Grantees vs. Comparison Groups. No statistically significant differences were found 

between SIF grantees and the comparison groups. However, compared to the National Sample of 

Grantmaking Nonprofits, we found small differences in favor of SIF grantees in the magnitude of the 

differences regarding participating in funding alliances with 

other nonprofits, and participating in collaborations with other 

nonprofits to share knowledge. Compared to Non-selected SIF 

Applicants, we found small differences in favor of SIF grantees 

in the magnitude of the differences in collaborations to share 

knowledge. 

SIF Grantee Observations about Change. For those reporting 

changes in collaboration through funding alliances and 

knowledge sharing, a majority of SIF 2010-2012 grantees 

attributed these changes to the organization’s participation in 

SIF and to trends in the broader grantmaking world, whereas a majority of the National Sample of 

Grantmaking Nonprofits attributed these changes to the direction from the organization’s 

board/leadership. SIF grantees and the National Sample of Grantmaking Nonprofits both cited direction 

from the organization’s board/leadership as a major factor behind changes in collaboration for advocacy. 

Trends in the broader grantmaking world also played a key role for many SIF grantees (See details in 

Appendix D). SIF participation was not a major factor here, possibly because federal funding does not 

allow the use of federal funds for lobbying (which may have been conflated with advocacy). 

Tools and Resources: Preparing to Scale

The SIF is focused on growing programs that can make a 

deeper or broader impact on communities. One grantee 

supports scaling by helping its subgrantees describe their 

programs to groups that might adopt them. The organization 

is developing an implementation template in which 

subgrantees record the details of their programs. The 

template calls for the subgrantee to outline the need for the 

program, its vision and mission, its logic model and theory of 

change, and its core program components. The grantee 

encourages subgrantees to use the template to support 

scaling and replication efforts. 
nationalservice.gov/research 32

In open-ended questions, a few respondents also pointed out other factors that contributed to increased 

collaboration, such as the recession and budget cuts, consolidation of services to focus on collective 

impact, and the need for increasing advocacy efforts. 

Regarding the ways that participation in the SIF influenced to their approaches to collaboration, SIF 2010-

2012 grantees emphasized various forms of collaboration among grantees, among funders, and across 

industries. For example, 

 To bolster support of our subgrantees, the organization entered into a strategic collaboration with the evaluator

in our application to become a SIF intermediary. This collaboration extended beyond our SIF work to support

most of our grantees.

 Our previous grantmaking did not necessarily have a match requirement; however, SIF funding does have this

requirement and so we are participating in these efforts with our SIF subgrantees. We've always participated in

industry-specific exchanges; however, now we participate in cross-industry exchanges via the SIF Economic

Opportunity Group.

Exhibit 17. Changes in approach to collaboration by SIF grantees and comparison groups 

Change of rating from 2009 to 2014 

(7 point Likert Scale: 1= Not at all to 7= Always) 

Percentage of organizations 

reporting change over time 

A. Participated 

in funding 

alliance(s) with 

other nonprofit 

sector 

organizations 

SIF vs Non-selected 

Applicants 

SIF vs National sample of 

Nonprofits 

Effect size (d) 0.1 0.2 

B. Participated 

in 

collaborations 

with other 

nonprofit 

organizations to 

share 

knowledge 

SIF vs Non-selected 

Applicants 

SIF vs National sample of 

Nonprofits 

Effect size (d) 0.3 0.4 

3.6

4.3

3.9

4.5

3.1

3.7

2009 2014

SIF (change = 0.8)

Non-selected SIF Applicants (change = 0.6)

National Sample of Grantmaking Nonprofits (change =0.7)

45%

25%

47%

SIF 2010-

2012 (n=20)

Applicant

(n=8)

Nonprofit

(n=172)

4.2

5.3 5.2

5.9

4.1

4.9

2009 2014

SIF (change = 1.0)

Non-selected SIF Applicants (change = 0.6)

National Sample of Grantmaking Nonprofits (change =0.8)

60%

25%

46%

SIF 2010-

2012 (n=20)

Applicant

(n=8)

Nonprofit

(n=172)
nationalservice.gov/research 33

Change of rating from 2009 to 2014 

(7 point Likert Scale: 1= Not at all to 7= Always) 

Percentage of organizations 

reporting change over time 

C. Collaborated 

with other 

organizations 

for purposes of 

advocacy – to 

advocate for or 

develop public 

support for 

programs or 

approaches to 

addressing 

social problems 

SIF vs Non-selected 

Applicants 

SIF vs National sample of 

Nonprofits 

Effect size (d) -0.2 -0.2 

3.5

4.1 4

4.8

3.3

3.9

2009 2014

SIF (change = 0.6)

Non-selected SIF Applicants (change = 0.8)

National Sample of Grantmaking Nonprofits (change =0.7)

40%

38%

41%

SIF 2010-

2012 (n=20)

Applicant

(n=8)

Nonprofit

(n=169)

Statistical significance (p<.05) is noted as: 

 a: significant difference between SIF 2010-2012 grantees and Non-selected SIF Applicants 2010-2012 

 b: significant difference between SIF 2010-2012 grantees and the National Sample of Grantmaking Nonprofits 

SIF Grantee Evidence of Change. The SIF built collaboration and sharing of best practices and lessons 

learned into its model as a way to continually improve the effectiveness of the programs it funds. The SIF 

provided grantees with a variety of opportunities to collaborate and learn from one another, encouraged 

grantees to support knowledge-sharing among their subgrantees, and highlighted the value of 

partnerships with other organizations and potential funders. The following themes emerged from the 

interviews. 

 The SIF gave grantees a variety of collaborative

learning opportunities. The SIF afforded grantees a

variety of opportunities to share resources, knowledge,

and best practices with their peers. Many grantees

used the SIF network to create and engage in various

experience-sharing forums.

 The SIF encouraged grantees to support peer

learning among their subgrantees. The SIF

encouraged grantees to create networks among their

subgrantees around common issues, and used a

variety of ways to do so such as the collaborative

problem solving approach described in the highlight

box.

 The SIF highlighted the value of external

partnerships. The SIF showed grantees the importance

of connecting with partners in the areas in which they

serve, especially for funding purposes. Establishing

relationships with funding partners was essential for SIF grantees to meet the match requirement,

especially for those implementing programs in under-resourced communities.

3.4 Reflections on SIF Experience 

Finally, the survey asked SIF grantees to reflect on their experiences with SIF in areas such as the tieredevidence grant program, major changes, development of subgrantee capacity, strengths and benefits of 

Highlight: Collaborative Problem-Solving

SIF grantees were creative about how they 

brought together peers and partners for idea 

exchange and collective problem solving. One 

grantee hosted a “hackathon,” modeled after 

convenings in which developers collaboratively 

create new software or improve existing software. 

This “hackathon” did not focus on software. 

Instead, it used the “hackathon” collaborative 

problem solving strategy by convening program 

partners to “get inside the youth development” 

system in which they worked to “identify a common 

barrier affecting each organization.” The 

“hackathon” leveraged key staff from multiple 

organizations to address young people’s access to 

public transportation, which can limit their 

participation in the educational and career 

preparation services that are available throughout 

the National Capital Region.
nationalservice.gov/research 34

the SIF model, challenges and problems with the model, and recommendations for improvement. Nonselected SIF Applicants also shared insights about their experiences in applying for the program. These

areas are not relevant to the National Sample of Grantmaking Nonprofits. 

3.4.1 Tiered-Evidence Grant Program 

The survey provided an opportunity to ask grantees for their perspectives on the tiered-evidence 

approach to grantmaking. All SIF 2010-2012 grantees said that tiered-evidence initiatives were at least 

“somewhat effective” in building evidence in a topic area, including 36.8% who said they considered 

these initiatives “very effective” in this regard (Exhibit 18). Grantees who responded “somewhat 

effective” elaborated on such cautions as the need to wait for results or to look at evaluation beyond their 

own program. They also acknowledged that rigorous evaluation may not be essential for every program, 

that a more adaptable approach could be beneficial; and that tiered evidence is more effective with tested 

practices than with pilot programs. They pointed out the possible tension between meeting requirements 

and finding the best method to build evidence, and that it is difficult to implement evaluation and scale 

up simultaneously. The SIF selects programs that have evidence, and does not accept new, but untested 

programs, however promising. Other tiered-evidence programs, such as the U.S. Department of 

Education’s Investing in Innovation (i3), take a different approach and include programs at very early 

stages of development. 

Exhibit 18. How effective do you think tiered-evidence initiatives are in achieving such outcomes as 

building evidence in an area? (n=19) 

Asked about the strengths or contributions of tiered-evidence initiatives, two or more SIF grantees noted 

that these initiatives allow organizations at different levels of maturity to undertake evaluations, and 

inform program improvement/replication. Areas mentioned once by different respondents included 

influencing funders to invest in evidence and effectiveness, rewarding evidence-based intervention, 

encouraging nonprofits to build capacity to reach outcomes, and creating a library of effective, evidencebased interventions other communities can implement. Respondents noted: 

 The contribution of tiered evidence is that it allows for organizations with different levels of maturity and

organizational readiness to seriously undertake evaluation. It is a serious step towards wanting to improve their

programs for the people that they serve. We know every program is not ready for a randomized controlled trial 

Very Effective, 37%

Somewhat 

Effective, 63%

Not Effective, 0%
nationalservice.gov/research 35

(RCT), but organizations can aspire to move their programs in that direction if they are in the habit of being 

accountable and having great performance feedback loops. 

 Tiered-evidence initiatives allowed us to increase our capacity to help subgrantees secure evidence, to

understand how to meet nonprofits where they are in terms of data collection and use, and to understand how to

move them along a continuum in terms of evidence. This is how they will create social change. It also helps to

continue to grow the organization’s knowledge of what works and inform the field about how to get outcomes

for certain populations.

 It has been extremely helpful to have the assistance of the third party evaluation across all of the subgrantee

organizations. Due to this, we will leave the project with solid evidence that will allow us to scale the initiative

to new sites. The evaluation—and particularly the use of RCT—has gained the interest of many new partners.

Grantees also were candid about the problems or limitations of tiered-evidence initiatives. They pointed 

out factors such as limited organizational readiness, cost and time required, disagreement concerning the 

level of evidence standards, difficulty in communicating variation, tendencies to limit innovation, state 

agencies unready to embrace the model, and a tendency to divert focus from what is important to what is 

measurable. For example: 

 Tiered-evidence initiatives have yet to fully show their promise to lead into follow-on funding streams for

programs that show promising evidence. It has also been difficult for some programs to adhere to the short

evaluation timeframes required. Also, tiered-evidence initiatives across government are not yet aligned in their

definition of evidence levels, and the funding allocated is still very small compared to the rest of the federal

budget. Further, state funding tends not to value tiered funding yet.

 Tiered evidence is great in theory, but there are so many variables when dealing with people that sometimes a

more adaptable approach, such as developmental evaluation, is called for. The idea of randomized control trials

as the "gold standard" is a little dated.

 Worst case, they preclude funding for early exploration of truly innovative approaches to some of our most

persistent social challenges.

3.4.2 Changes across Organizations 

Exhibit 19 indicates that 68% of the SIF 2010-2012 grantees reported that their SIF experience had affected 

how they conduct other programs. They cited increased focus on evaluation capacity building, 

compliance, TA provision, competitive selection processes, scaling up programs, collaboration, and 

sustainability. 

 We have increased our focus on evaluation capacity-building in order for our grantees to understand how to use

evidence for program improvement.

 We are taking what we've learned from SIF that has had positive impact in terms of process (such as

competitive bids, an emphasis on accountability), and we are applying those standards to other bodies of work

we fund.

 The SIF experience--the ability to scale up our program, provide intensive national technical assistance,

competitively select subgrantees, and make a commitment to data and outcomes is a goal for all programs at the

organization.
nationalservice.gov/research 36

Exhibit 19. Has your SIF experience affected how you conduct other programs? (n=20) 

* N/A – do not manage any other programs in which the organization funds grantees to carry out projects in communities

The survey asked SIF grantees about major changes that their organizations had experienced during each 

year of the SIF funding as well as the reasons for those changes. Exhibit 20 summarizes the themes 

reported by two or more respondents by implementation year. No consistent theme was identified for 

year 5. 

Exhibit 20. Summaries of changes by year of SIF funding (n=20) 

Changes Frequency Summary of reasons for the change 

Year 1

Increased ability to meet compliance 

requirements 4 The ramp up year was used to set up the new SIF award

Added staff capacity with particular 

expertise 4

Needed to add staff with particular expertise to help grantees meet 

federal/SIF compliance requirements, support SIF work , and 

support evaluation 

Developed grants management/admin 

system 2 The ramp up year was used to set up the new SIF award

Developed partnerships/collaboration 2 For fundraising requirements and nature of the SIF work

Scaling 2

SIF resources (along with private funds) allowed for a movement 

toward scaling effective approaches 

Year 2

Increased ability to meet compliance 

requirements 3 CNCS started to provide more specific instruction and assistance

Implemented the program 2

To get the program running, ensure subgrantees were implementing 

best practices

Focused on evaluation 2 To meet SIF requirement 

Year 3

Worked on sustainability 3 Funding was nearing an end, changed focus to sustainability 

Implementation 2 Continued implementation 

Experienced staff turnover 2 Board decision not to invest in 4th year of match

Year 4

Worked on sustainability 2

Determining the value of the SIF to network partners to identify what 

aspects are sustainable beyond SIF dollars

No, 21%

N/A* , 11%

Yes, 68%
nationalservice.gov/research 37

Finally, the survey asked the sample of Non-selected SIF Applicants about their experience with the 

application process. Fifty percent of the Non-selected SIF Applicants reported that the experience of 

applying for SIF funding had an effect on their organization (not presented in the exhibits). Specifically, 

they mentioned increased attention to evaluation, added clarity and focus, and increased collaboration. 

 The SIF application process forced us to be more strategic in

delivering our message and more collaborative in program

design. It has helped us refine our programming and, we

believe, advance our mission.

 In developing our SIF application we refined our plan for a

statewide funders collaborative and we are now moving

forward with that work. It also helped us engage partners

and define our focus on financial stability.

 We created the Social Innovation Board to raise money for

scaling evidence-based programs, purchased a relational

database outcomes management system to help

collaborations conduct more sophisticated analysis of outcomes, contracted with another organization to help us

create an evaluation and learning system, and just recently hired a full-time evaluation expert who will help

conduct formal evaluations of two key projects involving more than 20 partners. He will also provide training

and technical assistance to partners.

3.4.3 Development of SIF Subgrantees’ Capacity

In order to assess capacity, the survey asked the SIF grantees to rate the capacity of their subgrantees to 

carry out different functions at two points in time: when their SIF funding started, and in 2014. The 

survey also asked SIF grantees to assess the extent to which any changes in subgrantee capacity were 

attributable to their participation in SIF. 

The change in reported subgrantee capacity between the time they received SIF funding and 2014 was 

statistically significant in all seven areas investigated (Exhibit 21). In all but one of these areas (scaling up 

the intervention), a majority of SIF 2010-2012 grantees attributed a substantial amount of the change to 

participation in the SIF (Exhibit 22). 

Exhibit 21. SIF impact on subgrantees as perceived by SIF grantees (n=19) 

To what extent did your subgrantees do this ...?

7 point Likert Scale: 1- Not at all to 7- Always

At the start of 

SIF funding 2014 Change

A. Implement the interventions they are carrying out in their 

communities

3.5 4.4 0.8

B. Design and conduct rigorous evaluations of their interventions 2.3 3.6 1.3

C. Make use of evaluation findings for program improvement 2.6 3.7 1.1

D. Raise matching funds for the intervention 3.2 3.8 0.7

E. Meet federal compliance requirements 2.7 4.2 1.4

F. Scale up the intervention (i.e., increase impact within the 

community or expand to other communities)

2.9 3.8 0.9

G. Share knowledge and best practices 3.1 3.9 0.8
nationalservice.gov/research 38

Exhibit 22. Perceived level of SIF contribution to subgrantee capacity (n=20) 

In open-ended responses, SIF 2010-2012 grantees mentioned four additional benefits to subgrantees: 

increased resources, increased ability to attract future funders, increased ability to manage federal funds, 

and increased technical assistance. 

 All of our subgrantees have increased their capacity in compliance. We provided technical assistance with the

help of a consultant to help subgrantees comply with federal rules and regulations and prepare for A-133

audits. All subgrantees have increased their capacity to prepare for and engage in evaluation. The evaluator

worked with many of our subgrantees to prepare for and execute their evaluations. In addition, we connected

subgrantees with co-investors and helped them increase capacity to raise funds.

 Evaluation was challenging in the early days, but now they are very pleased they have this in place and capacity

is significant higher. Not just the money that improved capacity, but they have improved their data collection

systems, now look at data regularly with their external evaluator (each subgrantee has their own external

evaluator), and receive TA from the external evaluator. The SIF helped increase this capacity, but it was also a

matter of necessity outside SIF.

Asked about other factors contributing to the increased capacity of their subgrantees, SIF grantees 

mentioned the effects of executive leadership/strategic planning, to the impact of coping with a difficult 

funding environment or loss of a contract, enhanced staff capability, organizational changes, and support 

from the collective impact model. 

Grantees also shed light on challenges they faced while increasing subgrantee capacity to implement SIF 

projects, as well as effective approaches to address these challenges. Exhibit 23 summarizes the responses. 

64%

39%

81%

55%

79%

82%

62%

9%

39%

19%

18%

14%

12%

31%

27%

15%

18%

7%

6%

8%

9%

8%

G. Share knowledge and best practices

F. Scale up the intervention (i.e., increase impact within

the community or expand to other communities)

E. Meet federal compliance requirements

D. Raise matching funds for the intervention

C. Make use of evaluation findings for program

improvement

B. Design and conduct rigorous evaluations of their

interventions

A. Implement the interventions they are carrying out in

their communities

A substantial amount Some A little None
nationalservice.gov/research 39

Exhibit 23. Challenges in developing subgrantee capacity to implement SIF projects and effective 

approaches to addressing them 

Challenge in Developing Subgrantee 

Capacity to Implement SIF Projects Effective Approaches

Match restrictions or requirement Reallocate funds; identify other funding sources; devise creative funding streams

Limited resources/staff to support 

capacity building 

Find resources and identify staff expectations for capacity building activities; work 

with external evaluators; work one-on-one with subgrantees on specific elements; 

work with local organizations; stage convenings

Tight evaluation timeline 

Provide interim results so that evaluation can continue after SIF; adding a

planning year would have been helpful

Sustainability beyond SIF Develop sustainability plans; ensure subgrantees establish connections

Staff turnover Provide focused, personal TA; ensure good documentation that builds base 

knowledge in organizations

Federal compliance Provide training and monitoring; develop guidelines

Limited availability of data

Implement tool for collecting data; provide group, in-person capacity building 

training

Top-down nature of the evaluation Help understand the value of co-designing evaluation

Issues with organizational leadership Provide key leadership training

Communicating about evaluation/data 

expectations Create staff position for knowledge sharing

Lack of subgrantee evaluation 

readiness 

Secure support from evaluators

Lack of knowledge about the SIF when 

choosing subgrantees

Secure support from cross-site evaluation team

Scaling Work one-on-one with subgrantees to identify pathways to scale and identify 

funding to test scaling efforts

3.4.4 The SIF Model’s Strengths and Benefits, Challenges and Problems 

In response to open-ended questions, many SIF grantees noted that rigorous evaluation and 

collaborative, engaging peer-learning experiences were major strengths of the SIF model. Other benefits 

included SIF staff transparency and accessibility, flexibility, and the requirement for rigorous 

documentation, which contributed to helping the model work. 

 Strengths of the model are in the rigor of the evaluations and the partnership and transparency of the staff. The

SIF has proven to have some of the most rigorous evaluation requirements that any of our partners or

subgrantees have experienced. This led to very strong evaluations. The transparency and partnership of the

CNCS staff was invaluable. We could call on our program officers and the evaluation staff at any time to help

problem solve. We had a situation where two organizations were experiencing challenges implementing their

approved SEP. CNCS and JBS staff made themselves immediately available to talk with our staff and the

subgrantee evaluation team to help determine ways to overcome the obstacles.

 The SIF has been a partner in the work. With a focus on innovation, change, and impact, the relationship was

more dynamic than some government funding relationships.

 The community of funders created through monthly calls, issue area groups, and the national convenings was

extremely strong compared to other federal grant programs.

According to SIF grantees, the SIF model’s challenges and problems included the matching requirement, 

the rigorous evaluation requirement with limited guidance and funding, federal compliance 

requirements, lack of guidance, and sustainability challenges. 
nationalservice.gov/research 40

 The rigorous evaluations proved challenging, mainly because of the timing of the guidance—it was provided

after subgrantees submitted the first draft of their plans. It appears that CNCS learned from this experience,

and it didn’t occur for subsequent cohorts. The match idea was really great in theory. Our network partners

reported great success at the beginning of the SIF with raising those dollars. It was new and exciting, but by

year 2 and 3, it became an exercise of moving existing funds around to the match, but not really leveraging new

dollars. The role of intermediaries did not garner the acknowledgement and support nationally as originally

expected.

 We consistently hear that raising match funds is a challenge for the subgrantees (and for us as an

intermediary). Meeting all of the federal compliance requirements has also been a challenge.

3.4.5 Recommendations 

The survey also offered an opportunity for SIF 2010-2012 grantees to provide recommendations for 

improving the SIF program. Responses to this question echo some of the themes articulated in the 

challenges discussed above. Recommendations provided by more than one respondent included 

supporting grantees with compliance requirements, learning from intermediary perspectives, 

creating flexibility with the match (which requires a legislative, statutory change), minimizing 

staff turnover at CNCS, and including a planning year. Recommendations suggested 

just once by different respondents included clarity about all 

nuances and requirements, simplifying requirements, 

providing one-on-one TA, restricting small nonprofits from 

applying, extending the evaluation timeline, increasing the 

transparency of the SIF, reducing the burdens placed on SIF 

grantees (including reporting, convening, and 

interviewing), encouraging collaboration among grantees, 

and working with non-traditional funders. Below are 

quotes that illustrate some of these themes. 

 We encourage the SIF to remove restrictions on match funds. Allow time for planning for growth and

evaluation. Play a more active and strategic role with federal and state agencies to serve as the bully pulpit and

connect promising and proven innovation to sustainable funds.

 Listen to intermediaries, understand the nuances of different intermediaries, and understand that they are

experts in subgrantee selection and implementation work. Make sure they understand the nature of being an

intermediary and why they are effective. Ask intermediaries about what intermediaries should know and

understand.

 Allow organizations to have one year of planning and clear expectations around all aspects of the grant. Inform

grantees that capacity building for subgrantees is likely to be a heavy component of the work. Also, reduce the

match requirement so efforts can be focused on capacity and evidence building and support for growth and

sustainability. Lastly, with regard to sustainability, there should be mechanisms in place to make successful

subgrantees aware of federal funding opportunities for which they may qualify as a result of building evidence.

4. Conclusions and Next Steps

Established by the Serve America Act, the SIF program has several stated purposes, including increasing 

the capacity of social entrepreneurs to tackle social issues, seeding experimental initiatives, and providing 

resources to replicate and expand effective programs. The national assessment focuses on the SIF’s impact 
nationalservice.gov/research 41

on organizational capacities. We found evidence of positive changes in almost all target areas. However, 

not all areas evolved in the same way. For example, the greatest changes occurred in how organizations 

perform evaluations, moderate changes took place in support for subgrantees and scaling up, and there 

was less change in collaboration, and selection of subgrantees. Nevertheless, even with the changes, 

many organizations continue to have room to improve their capacity; when asked about how often they 

undertake activities indicative of strong organizational capacities, most of the 2014 SIF grantee responses 

were at or below 5 on a scale of 1-7, with 1 meaning not at all and 7 meaning always. 

Building organizational capacities requires a multi-faceted approach and lengthy commitment. 

According to Harsh (2010), capacity building is a recurring cycle of intervention, support, and growth. 

Although this report has provided promising evidence that the SIF program moved grantee 

organizations in the right direction, these changes were often incremental, and it took time for changes in 

one program to spread to the rest of the organization. 

We will continue to examine these questions next year. In addition to relying on surveys to continue 

documenting and capturing changes in intermediary capacities and experiences over time, and to 

provide broad-brush findings, we plan to use other methodologies and approaches, such as case studies, 

to provide an in-depth examination of selected grantees in order to understand the internal and external 

context, factors facilitating and inhibiting capacity building, and further uncover best practices and 

lessons learned. 
nationalservice.gov/research 42

References 

AAPOR. Response rates: An overview. Retrieved from 

http://www.aapor.org/Response_Rates_An_Overview1.htm. 

Baruch, Y., and Holton, B.C. (2008). Survey response rates and trends in organizational research. Human 

Relations 61:1139-1161. 

Cochran, W.G. (1977). Sampling techniques. New York: John Wiley and Sons.

Cohen, J. (1988). Statistical power analysis for the behavioral sciences (2nd ed.). Hillsdale, NJ: Lawrence 

Earlbaum Associates. 

Corporation for National and Community Service. (n.d.) Social Innovation Fund: Evaluation plan guidance. A 

step-by-step guide to designing a rigorous evaluation. Washington, DC: Corporation for National and 

Community Service. 

Dillman, D. A., Smyth, J. D., & Christian, L. M. (2014). Internet, Phone, Mail, and Mixed-Mode Surveys: The 

Tailored Design Method. New York: John Wiley & Sons.

Groves, R. M., Fowler Jr., F. J., Couper, M. P., Lepkowski, J. M., Singer, E., & Tourangeau, R. 

(2013). Survey methodology. New York: John Wiley & Sons.

Groves, R. M., & Peytcheva, E. (2008). The impact of nonresponse rates on nonresponse bias: A metaanalysis. Public Opinion Quarterly, 72(2), 167-189.

Hager, M.A., Wilson, S., Pollak, T.H., and Rooney, P.M. (2003) Response rates for mail surveys of 

nonprofit organizations: A review and empirical test. Nonprofit and Voluntary Sector Quarterly 32:252-267. 

Harsh, S. (2010). Gaining perspectives on a complex task: A multi-dimensional approach to capacity building. 

Charleston, WV: Appalachia Regional Comprehensive Center at Edvantia. 

LeCompte, M.D., & Schensul, J.J. (1999). Designing and conducting ethnographic research. Walnut Creek, CA: 

AltaMira.

Lester, P. (2015). Social innovation fund: early results are promising. Washington, DC: Social Innovation 

Research Center. 

Saunders, M.N.K. (2012) Web versus mail: The influence of survey distribution mode on employees’ 

response. Field Methods 24:1-18.
nationalservice.gov/research A-1 

Appendix A. Overview of SIF National Assessment 

Social Innovation Fund (SIF) National Assessment Survey: Overview 

January, 2015 

What is the purpose of the survey? Who will use the survey findings? Why is this data collection necessary? 

The survey will collect data needed to assess the implementation and effectiveness of CNCS’s Social 

Innovation Fund (SIF). It will meet the evaluation needs of CNCS management and contribute to the 

evaluation of the Administration’s evidence-based initiatives. There are no data available for this purpose 

from other sources. 

Survey findings will be useful to SIF program managers, CNCS and other federal leadership, current and 

potential SIF grantees, and the larger philanthropic and nonprofit sector. Findings will be reported in 

technical reports to CNCS and information briefs for other stakeholders. 

What research questions does the SIF National Assessment survey address? How does it address them? 

The basic question the SIF National Assessment addresses is: Does participation in SIF increase 

organizations’ capacity (skills, attitudes, behavior) in the areas of evidence-based grantmaking, use of 

rigorous evaluation to build the evidence base and scale up effective program models, and collaborative 

approaches to addressing community needs. The key SIF elements and research questions are shown in 

more detail in Attachment 1, together with the logic model that guides the National Assessment. 

The approach to addressing these research questions is: 

1. Calculate the magnitude of change experienced by SIF grantees and other surveyed organizations

in key capacity areas over the time period from 2009 to 2014 and, for SIF grantees and nonselected applicants, from 2009 to 2014.

2. Test for differences in the magnitude of change between SIF grantees and comparison

organizations to determine whether SIF grantee change can be attributed to SIF participation (test

null hypothesis of no difference).

3. Collect additional descriptive information to increase understanding of SIF program for program

improvement and replication:

a. Describe SIF implementation experience, including program strengths and challenges.

b. Describe how SIF participation and other factors contributed to change in SIF grantees’

capacity and practices.

What groups will be surveyed? When will they be surveyed? How will the survey be administered? 

The three groups that will be surveyed are: 

1. All SIF grantees (n = 26; funded in cohorts between 2010 and 2014)

2. Other organizations that applied for SIF funding and that had applications that were determined

to be compliant and satisfactory in quality, but that did not receive SIF funding (non-selected SIF

applicants; n = 45)

3. Other nonprofit grantmaking organizations that did not apply for SIF funding (a probability

sample will be selected and surveyed, to obtain 400 completed surveys)

Surveys will be administered in 2015 and 2016. The 2015 (Round 1) survey will ask about the 

organization’s capacity in key areas in 2009 (before SIF funding began) and 2014 (after SIF funding was 
nationalservice.gov/research A-2 

provided to SIF grantees) and, for areas where capacity changed, factors that contributed to change. The 

2016 (Round 2) survey will collect data about change between 2014 and 2015 for a subset of the 

organizations surveyed in 2015. 

In each round of the survey, all respondents will be asked a core set of questions about organizational 

capacity and change. These questions will address the key SIF elements and research questions: 

grantmaking strategies; use of rigorous evaluation; organizational infrastructure; scaling up of programs; 

and collaboration. In addition, different subgroups will be asked questions specific to their experience. 

For example, SIF grantees will be asked about support they have received and about change in capacity 

among their subgrantees, as well as reflections on the SIF experience and recommendations. No personal 

or sensitive questions will be asked. Attachment 2 shows the National Assessment questions and the 

survey questions that address each of those questions. 

Slightly different versions of the survey will be required for different respondent subgroups and for 

Rounds 1 and 2. The survey versions and the subgroups to which each will be administered are 

summarized below: 

Group Subgroup/ Cohort

2015 survey administration 

(Round 1)

2016 survey administration 

(Round 2)

Survey 

version Period covered

Survey 

version Period covered

SIF grantees

2010-2012 cohort (N=20) 1 2009-2014 5 2014-2015

2014 cohort (N=7) 2

2009-2014 (pre-SIF) 

and early SIF months 5 2014-2015

Non-selected SIF 

applicants

2010-2012 cohort (N=33) 3 2009-2014 No survey

2014 cohort (N=12) 3 2009-2014 6 2014-2015

National Sample of 

Grantmaking 

Nonprofits

N=400 4 2009-2014 No survey

The survey will be administered online (using SurveyGizmo), with hard copy versions available for 

organizations that do not have email and web access or that prefer hard copies; in addition, to maximize 

responses from the SIF intermediary and non-funded applicant groups, telephone follow-up will be used 

if needed. 

What burden does the survey place on respondents? What will the respondents get in return? 

The total estimated burden for all respondents is 178.2 hours in 2015 and 28.6 hours in 2016, for a total of 

206.8 hours and an annualized burden of 103.4 hours. This estimate is based on a pilot test of the survey. 

The range in estimated time to fill out the survey is from 20 minutes (for the nonprofit comparison group) 

to 40 minutes (for the 2015 round of the survey of 2010-2012 SIF grantees). No other respondent 

recordkeeping or other burden, other than the time to fill out the survey, is involved. 

Respondents will be offered a report on the survey and a benchmarking report that compares their 

organization’s data with aggregate data for other organizations. No payments or gifts will be given for 

participation in the survey. 
nationalservice.gov/research A-3 

Attachment 1: Key SIF Elements, National Assessment Research Questions, and Logic Model 

Key SIF Elements 

The SIF is characterized by the unique interplay of six key elements: 

1. It relies on intermediary grantmaking institutions to implement the program – they take on the 

role of finding, selecting, monitoring, supporting, evaluating and reporting on the nonprofit 

organizations implementing community-based interventions. 

2. It is a tiered-based evidence program that requires all funded programs/interventions to 

demonstrate at least preliminary evidence of effectiveness, or funding “what works.”

3. It requires that all programs or interventions implement a rigorous evaluation that will build on 

their level of evidence. 

4. It charges intermediaries with scaling evidence-based programs – increasing impact within their 

community or to communities across the country – and as such, grapples with a field-wide 

challenge of how best to successfully and efficiently do so. 

5. It leverages public-private partnerships to effect large scale community impact in ways that 

neither a traditional federal grant investment nor a philanthropic grants investment could 

achieve on its own. This includes its unique leveraged funding model to support nonprofit 

programs. 

6. It is committed to improving the effectiveness of nonprofits, funders, and other federal agencies 

by capturing learning and best practices and promoting approaches that will generate the 

greatest impact for individuals and communities. 

Research Questions 

The SIF National Assessment is designed to address five key questions about changes in organizational 

capacity (skills, attitudes, behavior) that flow out of the initiative’s theory of change:

1. Does SIF increase organizational use of evidence-based grantmaking strategies? 

2. Does it increase organizations’ willingness and ability to build the evidence base for high-impact 

models? 

3. Does it increase organizational knowledge of how to scale proven models? 

4. Does it strengthen organizational infrastructure to support the growth of proven models? 

5. Does it facilitate collaborative approaches to addressing local community needs? 

Logic Model 

The exhibit below provides an overview of the logic model that guides the National Assessment: 
nationalservice.gov/research A-4 

Attachment 2: National Assessment Research Questions and Survey Questions 

The exhibit below lists the research questions addressed by the SIF National Assessment and, for each 

research question, shows the corresponding survey questions, provides notes on the groups who will be 

asked to answer the questions, and the survey rounds in which they will be asked. (Because some survey 

versions do not include all the questions and thus have different question numbers, the question numbers 

shown correspond to the most detailed survey, which is “Version 1” and is administered to SIF grantees.)

Research Questions and Survey Items Notes Rd 1 Rd 2

1) Does SIF increase organizational use of evidence-based grantmaking strategies?

Selection of grantees to fund

1A. Use of open competitive process Asked of all respondents. x x

1B. Requirement for evidence of effectiveness Asked of all respondents. x x

1C. Requirement for rigorous evaluation plan Asked of all respondents. x x

23. Major changes experienced in successive years of SIF funding 

(year and reasons)

Asked only of SIF intermediaries. x x

26. Effects of SIF on conduct of other grant programs (yes/no, and 

describe)

Asked only of SIF intermediaries. x x

2) Does it increase organizations’ ability and willingness (skills, attitudes, behaviors) to build the evidence base for 

high-impact models? 

Support provided to grantees funded by organization

2A. Funding for grantee evaluation Asked of all respondents. x x

2B. Technical assistance for grantee evaluation Asked of all respondents. x x

2C. Technical assistance for grantee implementation Asked of all respondents. x x

Organization’s conduct and use of rigorous evaluation

3A. Conduct rigorous evaluation of programs Asked of all respondents. x x

3B. Use evaluation findings to improve programs Asked of all respondents. x x

3C. Use evaluation findings to communicate program effectiveness Asked of all respondents. x x

Support received by SIF intermediaries

9. Support received to increase capacity (source; helpfulness) Asked only of SIF intermediaries. x x

10. Less helpful support; ways to improve Asked only of SIF intermediaries. x x

11. Areas where would have benefited from additional support Asked only of SIF intermediaries. x x

23. Major changes experienced in successive years of SIF funding 

(year and reasons)

Asked only of SIF intermediaries. x x

SIF subgrantee capacity development 

12A. Implement interventions Asked only of SIF intermediaries. x x

12B. Conduct rigorous evaluations Asked only of SIF intermediaries. x x

12C. Use evaluation findings for program improvement Asked only of SIF intermediaries. x x

12D. Raise matching funds Asked only of SIF intermediaries. x x

12E. Meet compliance requirements Asked only of SIF intermediaries. x x

12F. Scale-up intervention Asked only of SIF intermediaries. x x

12G. Share knowledge Asked only of SIF intermediaries. x x

13. SIF contribution to subgrantee capacity development Asked only of SIF intermediaries. x x

14. Other capacity areas Asked only of SIF intermediaries. x x

SIF intermediary capacity development

15. Factors other than SIF that contributed to capacity change (openended)Asked only of SIF intermediaries. x x

16. Challenges to increasing subgrantee capacity, and effective 

approaches (open-ended)

Asked only of SIF intermediaries. x x

23. Major changes experienced in successive years of SIF funding 

(year and reasons)

Asked only of SIF intermediaries. x x

26. Effects of SIF on conduct of other grant programs (yes/no, and 

describe)

Asked only of SIF intermediaries. x x
nationalservice.gov/research A-5 

Research Questions and Survey Items Notes Rd 1 Rd 2

27. SIF elements organization expects to sustain (open-ended) Asked only of SIF intermediaries. x x

SIF and other tiered-evidence programs

20. Contribution to building evidence (amount and description) Asked only of SIF intermediaries. x x

21. Strengths/contributions (open-ended) Asked only of SIF intermediaries. x x

22. Problems/limitations (open-ended) Asked only of SIF intermediaries. x x

3) Does it increase organizational knowledge of how to scale proven models? 

7A. Undertake efforts to scale up existing programs Asked of all respondents. x x

7B. Select programs for scale-up based on rigorous evaluation 

evidence

Asked of all respondents. x x

23. Major changes experienced in successive years of SIF funding 

(year and reasons)

Asked of all respondents. x x

26. Effects of SIF on conduct of other grant programs (yes/no, and 

describe)

Asked only of SIF intermediaries. x x

4) Does it strengthen organizational infrastructure to support growth of proven models?

4. Staff positions; evaluation partner; have evaluation budget in 

2009 and 2014 (yes/no)

Asked of all respondents. x x

5. Evaluation budget (total and % of budget) – compare 2014 with

2009

Asked of all respondents. x x

6. Evaluation budget – total, %, and staff – in 2009 and 2014 Asked of all respondents. x x

23. Major changes experienced in successive years of SIF funding 

(year and reasons)

Asked only of SIF intermediaries. x x

5) Does it facilitate collaborative approaches to addressing local community needs?

8A. Participate in funding alliances with other nonprofits Asked of all respondents. x x

8B. Participate in collaborations to share knowledge Asked of all respondents. x x

8C. Collaborate to develop public support for programs Asked of all respondents. x x

23. Major changes experienced in successive years of SIF funding 

(year and reasons)

Asked only of SIF intermediaries. x x

26. Effects of SIF on conduct of other grant programs (yes/no, and 

describe)

Asked only of SIF intermediaries. x x

Other Information Collected

Organizational demographics

Geographic vs. issue funding focus Asked of other nonprofits (available 

for SIF intermediaries and 

applicants from CNCS sources).

x

Focus of funding (opportunity, youth, community health) Asked of other nonprofits (available 

for SIF intermediaries and 

applicants from CNCS sources).

x

Other information collected from SIF applicants and/or other nonprofits

Awareness of SIF (scale) Asked only of other nonprofits. x

Effect on organization of participation in SIF application process (openended)Asked only of non-funded SIF 

applicants.

x x

Additional reflections on SIF program/experience (asked of SIF intermediaries)

24. What are the strengths/benefits of the SIF model compared with 

other programs you have participated in? Please provide 

examples. (Open-ended)

Asked only of SIF intermediaries. x x

25. What are the challenges/problems of the SIF model compared 

with other programs you have participated in? Please provide 

examples. (Open-ended)

Asked only of SIF intermediaries. x x

28. What advice would you give a federal agency that was 

considering use of an intermediary model similar to SIF? 

(Open-ended)

Asked only of SIF intermediaries. x x
nationalservice.gov/research A-6 

Research Questions and Survey Items Notes Rd 1 Rd 2

29. What recommendations do you have for improving the SIF

program? (Open-ended)

Asked only of SIF intermediaries. x x
nationalservice.gov/research B-1 

Appendix B. Evaluation Methodologies 

This evaluation examines survey data from SIF grantees and from two comparison groups: Non-selected 

SIF applicants, and a National Sample of Grantmaking Nonprofits that did not apply for funding from 

the SIF. The survey data will be collected in two rounds. 

In Round 1 (2015), all SIF grantees and Non-selected SIF Applicants from the 2010-2014 cohorts were 

surveyed, together with a probability sample of other nonprofit grantmaking organizations. The surveys 

asked respondents to report information for two points in time: 2009 and 2014. Requesting information 

about 2009 provided retrospective baseline data reflecting how the organizations perceived their 

situations prior to the award of the first SIF grants. Requesting information about 2014 provided 

information about the recent circumstances of the organizations. This allowed for an analysis of perceived 

change between 2009 and 2014. For areas where change was reported, the survey also collected from 

respondents their perceptions of the reasons for the change. 

In Round 2 (2016), the SIF grantees (all cohorts) and the Non-selected SIF 2014 Applicants will be 

surveyed a second time. Non-selected 2010-2012 SIF Applicants and the National Sample of Grantmaking 

Nonprofits will not be included in the 2016 survey. The National Sample of Grantmaking Nonprofits 

reported their experiences between 2009 and 2014 during Round 1, providing information needed to test 

the study hypotheses related to that sample. For the 2010-2012 Non-selected SIF applicants, key 

information was collected during Round 1. Because it is expected that additional change in 2014-2015 will 

be relatively modest and incremental for both of these comparison groups, the burden that a second 

round of data collection would impose does not appear to be warranted. 

This report is based exclusively on the Round 1 data collection. Because the national assessment will 

cover three different groups (grantees and two comparison groups) across two survey rounds, the 

surveys for each group and each round include core items for all groups, but are also tailored with 

content specific to each group. 

Appendix B provides a detailed discussion of the methodologies involved in evaluating the SIF impact on 

organizational capacities. It begins by describing the research design, sample selection, and statistical 

power. Next, it discusses the survey instruments and the development process. Next, it describes the data 

collection process, the resulting response rates, and non-response analysis. Finally, it presents the data 

analysis approaches. 

B.1. Research Design, Sampling Selection, and Statistical Power 

B.1.1. Research Design 

The SIF national assessment is designed to compare the experience of change in capacity and 

organizational behavior of the SIF grantees, before and after SIF funding, to the experiences of two 

comparison groups also composed of nonprofit populations. The potential respondent universes for 

survey data collection include current SIF grantees (2010, 2011, 2012, and 2014), Non-selected SIF 

Applicants, and the National Sample of Grantmaking Nonprofits. 

 SIF grantees: All of the 26 organizations that received SIF funding (a total of 27 grants) in 2010, 2011,

2012, and 2014 were included in the survey data collection. This universe is defined to include

grantees that are retained in the program over the course of funding as well as those that leave the

program early or that complete their program. Of the 27 funded SIF grants, 20 were funded in 2010-
nationalservice.gov/research B-2 

2012 and 7 were funded in 2014. One of the 26 grantees received two SIF grants and is treated as 

2010-2012 cohort in the survey. 

 Non-selected SIF Applicants: This group includes 45 organizations that submitted SIF applications in

2010, 2011, 2012, or 2014, and whose applications were rated as compliant and of satisfactory quality.

These include 33 organizations that submitted applications in 2010-2012 and an additional 12

organizations that submitted applications in 2014. The final sample is 41 organizations, after

excluding one organization that subsequently went bankrupt and three that received SIF grants in

subsequent years.

 National Sample of Grantmaking Nonprofits: This group includes a probability sample of all

grantmaking nonprofits in the United States. After screening out non-grantmaking organizations (i.e.,

ineligibles), the sample includes 219 complete responses. Details about sampling this group are

described below.

B.1.2. Sampling for the National Sample of Grantmaking Nonprofits 

No sampling plan was required for SIF grantees or Non-selected SIF Applicants, because the survey was 

administered to the entire universe of organizations in these two groups. For other nonprofit 

organizations, a sample was selected to represent this population of organizations, using explicit 

stratification by organizational revenue, and implicit stratification, within each revenue stratum, by 

grants made to U.S. nonprofit organizations. Organizational revenue was divided into seven strata: $1M-

$5M; $5M-$10M; $10M-$20M; $20M-$50M; $50M-$75M; $75M-$100M; $100M+. Within each of these 

revenue strata, organizations were ordered by grants to nonprofits and every mth record was selected, 

such that m = Nh/nh, where Nh is the total number of records in revenue stratum h and nh is the number 

of records to be selected from that stratum. 

Because nonprofit organizations are required to submit IRS Form 990 or Form 990-PF, it is possible to 

identify and select a nationally representative sample of grantmaking nonprofits. The IRS data allow us

to limit the selection by excluding nonprofit organizations that are small in size, and those that do not 

make grants to nonprofit organizations or only make relatively small grants. SIF participation involves a 

size standard: SIF makes grants to grantees of $1 million or more per year for the grant period, who 

commit to making subgrants totaling at least 80% of the SIF grant (i.e., at least $800,000). To define a 

population of grantmaking nonprofits of size comparable to SIF grantees, ICF worked with GuideStar, an 

organization that obtains and manages IRS data for use in research and other applications. We estimated 

that approximately 3,000-4,000 U.S. grantmaking nonprofit organizations meet the size criteria for this 

comparison group (i.e., have revenues of at least $1 million and make grants of at least $800,000 per year). 

The starting point for selection in each revenue stratum is a random number between 1 and m. The 

number of records to be selected from a stratum, nh, is proportional to the size of the stratum in the 

population, such that nh = (Nh/N) ∗ n, where N is the total population size and n is the total number of 

records to be selected. This sample design ensures that the distribution of organizations in the selected 

sample is proportional to the population distribution with respect to revenue and grants to nonprofits. In 

addition, the variance achieved with this sample design is no larger than that for a simple random sample 

and may be smaller, potentially increasing the precision of estimates. 

To achieve an absolute margin of error of at most +/- 5 percentage points on overall estimates from the 

sample data, the target number of completed surveys, nC, for the other grantmaking nonprofit 

organization comparison group was 400. Based on the literature concerning surveys of nonprofits, 
nationalservice.gov/research B-3 

response rates in the range of 30-40% were to be expected (Baruch & Holton, 2008; Hager et al., 2003; 

Saunders, 2012), with sample size calculated using a conservative response rate of 25%. Preliminary 

power analyses indicated that this sample size would provide 0.8 power at α = 0.05 for detecting 

significant effects of size d = 0.2, conventionally considered small effects. Because a lower-than-expected 

response rate was observed during fielding (the final response rate for this group was 13%), the decision 

was made to draw a second sample from the sampling frame using the same stratified design, but limited 

to only those (previously unsampled) records that included an email address on the frame. In total, 3,300 

records were sampled, yielding a total of 262 completed interviews with eligible organizations. 

B.1.3. Statistical Power 

Because the estimates from the SIF grantee data can be treated as population values, the simplest 

approach to comparing means or proportions between the grantees and the two comparison groups (i.e., 

Non-selected SIF Applicants and the National Sample of Grantmaking Nonprofits) are one-sample t-tests 

or z-tests, respectively. 

For the census of Non-selected SIF Applicants, non-response is treated as sampling error, making 

inferential statistics applicable. With 19 cases in the analytic sample for the Non-selected SIF Applicants, 

tests for statistically significant differences between these two groups at α = 0.05 achieve 0.8 power for 

effect sizes of d = 0.68, which would conventionally be considered medium-large effects. 

With 262 cases in the analytic sample for the National Sample of Grantmaking Nonprofits, tests for 

statistically significant differences between these two groups at α = 0.05 achieve 0.8 power for effect sizes 

of d = 0.17, which would conventionally be considered small effects. Thus, the sample sizes available for 

these comparisons provide sufficient power for detecting significant differences even when these 

differences are, in practical terms, very small. 

B.2. Survey Instruments 

B.2.1. Survey Instruments 

Key data for the SIF grantees and the two comparison groups were collected using survey methods. A 

retrospective survey was administered in May-June 2015 for the 2010-2012 cohorts of SIF grantees and 

Non-selected SIF applicants and for the National Sample of Nonprofit Grantmakers. 

The surveys included a combination of closed-ended items (mostly using Likert-type scales) and openended items. All respondents were asked a core set of questions about organizational capacity/behavior 

and change in terms of the use of evidence-based grantmaking strategies; implementation and use of 

rigorous evaluations; scaling up of programs based on evidence of effectiveness; knowledge sharing; 

collaboration to address community needs; and experience with federal funding. Each survey group was 

asked selected questions targeted to their particular situation, in addition to the core questions. These 

tailored questions required four separate versions of the survey for various groups, as follows: 

 Survey 1 (SIF grantees, 2010-2012 cohorts). The SIF grantees were asked the core questions, as well as

questions to collect additional information specific to the SIF experience. The additional questions

asked about the support they have received to carry out the SIF initiative, the capacity of their

subgrantees, and factors that contributed to reported increases in subgrantee capacity, as well as

several questions about the SIF initiative and its contributions and challenges.

 Survey 2 (SIF grantees, 2014 cohort). The 2014 SIF grantee cohort has had only about 6 months of SIF

experience, while the 2010-2012 cohorts had multiple years of SIF experience behind them. For this
nationalservice.gov/research B-4 

reason, the 2014 cohort was asked to answer the core questions with respect to their 2009 to 2014 

experience up to the point when they began receiving SIF funding. In a separate section, the 2014 

cohort survey also asked about support received to carry out the SIF initiative during the early 

months of their SIF experiences and their reflections on SIF based on these months of experience. 

 Survey 3 (Non-selected SIF Applicants). The non-funded SIF applicants were asked the core

questions and a small number of additional questions (e.g., areas of activities they fund) and about

any effects they believe applying for SIF has had on their organization.

 Survey 4 (National Sample of Grantmaking Nonprofits). The other nonprofit organizations were

asked the core questions and a small number of additional questions (e.g., awareness of SIF, their

areas of funding).

Attachment 2 to Appendix A provides a crosswalk of survey items across respondents and data collection 

cycles. The survey question numbers refer to the location of the question in Survey 1 (the version 

administered to 2010-2012 SIF grantees), unless otherwise noted. Some survey items provide information 

related to more than one research question; these are listed under each of the research questions they 

apply to. A number of questions (e.g., ones relating to grantee selection, support, collaboration, and other 

topics) use a Likert scale to record the extent of organizational practices in these areas in 2009, 2014, and 

(for the follow-up surveys) 2015. Similarly, for information collected from SIF grantees about SIF 

subgrantee capacity, a Likert scale is used to assess the level of subgrantee capacity at time of SIF entry 

and the survey. 

B.2.2. Pilot test 

The procedures for data collection and response maximization were tested with a total of nine 

organizations selected to represent the three respondent groups (three per group). In the first phase, 

cognitive interviews were conducted with three respondents (one from each group). In the cognitive 

interviews, respondents were asked to explain their thought process and provide perspectives about the 

survey while they were answering the questions. These interviews provided information that was used to 

evaluate the quality of the response and to help determine whether the question was generating the 

information that was intended. For the second phase, the survey was revised and programmed, then sent 

to six additional respondents (two from each group), with follow-up interviews with three of the six 

respondents (one from each group). 

During the pilot, both survey responses and discussions with respondents indicated that respondents 

were able to provide information about the capacities and practices of their organizations for 2014 and 

2009, and their responses did not suggest a social desirability effect. For example, some respondents 

indicated that there was substantial change in practice between the two points in time, whereas others 

indicated no change, whether the initial level was low or high. The distribution of reported levels and 

change suggested substantial variation among respondents and respondent groups. 

The pilot survey pre-notification and instructions indicated to respondents that they might want to ask 

colleagues to help provide information, and the online survey administration allowed for multiple 

respondents from an organization, if needed. Most pilot respondents said they were able to respond for 

the different topics and for both periods, although a few asked a colleague to help. 

The items that pilot respondents said were most difficult and time-consuming to answer concerned the 

evaluation budget (in dollars and as a percentage of the organization’s overall budget). For respondents 

who could provide the data, the original budget questions were retained. However, a simplified version 
nationalservice.gov/research B-5 

of the question was added that simply asked for a comparison of the magnitude of change in budget 

between the two years, rather than for actual figures. Respondents who had difficulty answering the 

original, more detailed question reviewed the simplified version and said they would be able to answer 

the question framed this way. 

The pilot pre-notification and cover email explained the survey and asked the contact to answer the 

survey themselves or forward it to an appropriate colleague. When specific information was available, 

the survey was addressed to the organization director. If no specific contact information was available, 

the pilot survey was sent to a general contact for the organization. During the interviews, respondents 

were asked to whom the survey should be sent within the organization. Their responses confirmed that, 

whenever possible, it should go to the executive director of the organization or someone in a similarly

senior role, both as a practical matter and to ensure that the organization’s leadership was aware of the 

survey. 

The administration of the programmed version of the pilot survey showed that respondents could access 

the survey link, and most could complete and submit the survey. One respondent thought he had 

completed the survey but was unable to submit it. Another submitted a partially completed copy of the 

survey. Based on this information, the survey administration plan was adjusted in several ways. First, 

each survey was checked as it was returned, and if the survey was only partially completed, there was 

follow up with the respondent. Second, a hard copy was provided for respondents who had problems 

completing the online survey. The respondent was also offered the opportunity to conduct the survey by 

telephone, if the respondent had difficulty with online or hard-copy versions. 

Overall, pilot respondents appeared engaged and interested in the survey issues and questions. In the 

pilot test, respondents were asked if they were interested in receiving a copy of the survey report when 

completed, and were asked to provide contact information if they would like a copy. All said they would 

like to receive a copy, and the contact information they provided will make it possible to send reports to 

them. 

B.3. Data Collection 

Using SurveyGizmo software, the surveys were administered primarily online by emailing participants a 

respondent-specific link to an online survey tool. Online administration lessens the burden on 

participants by allowing them to complete the survey at a time and in a place that is most convenient to 

them. Additionally, respondents have the option of completing the entire questionnaire at once, of 

completing it over several sessions, and of obtaining assistance from colleagues within the organization 

to assist with specific sections (e.g., budget information). Using an online survey format also reduces 

burden by eliminating the time and effort required to return a hard-copy survey by mail. A hard-copy 

version of the survey was made available, but no organizations chose that option. 

B.3.1. Communication and Administration 

For the SIF grantees and Non-selected SIF Applicants, CNCS had contact information that was used to 

send survey communications and information about accessing the survey. For the National Sample of 

Grantmaking Nonprofits comparison group, basic information is available from IRS submissions and 

GuideStar that include employer identification numbers (EIN), addresses, and telephone numbers. 

Contact information (email address) for this sample was identified using multiple sources. For some, 

contact information is available from GuideStar or from the Foundation Directory or similar guides; for 

others, their own websites provide the needed information. In some cases, we telephoned the 
nationalservice.gov/research B-6 

organization to obtain contact information. The survey took several steps to communicate and engage 

with respondents. 

 To the SIF grantees, who are aware of the national assessment, CNCS sent out early communications

through emails to grantees, announcements through the SIF Knowledge Network site, and

communications to such groups as the SIF Evaluation Working Group.

 A pre-notification letter was sent in advance of the survey to all organizations selected for the survey.

The letter provided information about the survey’s purpose, how CNCS will use the data, estimated

time to complete the survey, assurance that participation was voluntary, and a contact person in case

they had questions or wanted more information. Based on the pilot feedback, we sent communication

to the executive directors of the National Sample of Grantmaking Nonprofits, with information about

the survey and a request to provide information about the appropriate person to whom to send the

surveyor, if the organization prefers, to have the executive director receive the survey and direct it to

the appropriate person. After the pre-notification, all organizations that have email were sent the link

to the survey URL.

 We allowed four weeks for organizations to respond before closing the survey (except for the second

nonprofit sample). Survey receipt was closely monitored, including tracking the response rates and

checking to see that the surveys received were complete (e.g., not missing sections) and, as necessary,

following up with reminders to obtain completed surveys or determine reasons for non-completes.

Undeliverables also were monitored, and telephone or emails were used to contact those

organizations.

 Because of the importance of responses from the SIF grantees and the Non-selected SIF Applicant

comparison group, intensive follow-up was used to maximize the response rate for these groups.

These included telephone follow-up and, in two cases, having a trained staff member administer the

survey by telephone and enter the data in the programmed instrument.

 Respondents were sent a “thank you” for their participation. We also sent a survey report and a

benchmarking report that compared their responses with aggregate data from other organizations to

respondents who indicated an interest. SIF grantees also received a request for a 15-30 minute followup interview, in which they were asked to discuss evidence to support their reports of organizational

change in the survey. They also were asked to supply documentation of organizational change.

B.3.2. Response Rates 

Strategies were employed to maximize response rate and minimize issues associated with non-response 

during the survey process: 

 Working with GuideStar, Foundation Directory, and other resources (e.g., organization websites) to

collect accurate and up-to-date contact information for potential survey respondents;

 Using a well-designed, easy-to-use online survey;

 Assuring respondents that their responses would remain confidential;

 Providing respondents with a contact name and telephone number for inquiries;

 Providing respondents with multiple reminders to complete the survey, each time including the

direct link to the web-based survey;
nationalservice.gov/research B-7 

 Providing respondents with an explanation of how their participation would help to inform positive

changes to the field of nonprofit grantmaking, and to social innovation initiatives sponsored by

CNCS’s and other agencies.

 Offering the survey in hard-copy to participants who could not be contacted via email or who

requested a hard-copy survey.

Response rates were computed for all three target populations following Standard Definitions by 

American Association for Public Opinion Research (AAPOR) for web surveys of named persons. B-2 

provides the AAPOR dispositions and final response rates (RR3) for each survey population. 

Exhibit B-2. AAPOR Dispositions and Final Response Rates by Survey Population 

Population

Completed 

Surveys (I)

Ineligible 

(X)

No Response 

(U)

Observed 

Eligibility 

Rate (e)

Total 

Sampled (n)

Response 

Rate (RR3)

SIF grantees 2010-

2012 20 0 0 100% 20 100%

SIF grantees 2014 6 0 0 100% 6 100%

Non-selected SIF 

Applicants (2014) 9 0 6 100% 15 60%

Non-selected SIF 

Applicants (2010-2012) 9 1 (bankrupt) 17 90% 27 37%

National Sample of 

Grantmaking 

Nonprofits

219 168 (nongrantmaking) 2,937 61% 3,324 12%

Final response rates were computed by using the observed eligibility rate e = I/(I + X), where I is the 

number of completed interviews and X is the number of known-ineligible records, to estimate the fraction 

of non-responding organizations (U) that were eligible. Specifically: 

RR3 =

I

I + e(U)

B.3.3. Non-Response Analysis 

Non-response bias occurs when the sample units that respond to a survey, systematically differ with 

regard to survey outcomes compared to sample units that do not respond. Consequently, non-response 

bias in a given survey statistic is a function of both the response rate and the extent to which responding 

and non-responding units differ on that dimension. Although empirical evidence suggests that response 

rates are not correlated with the occurrence of non-response bias (Groves & Peytcheva, 2008), higher 

response rates are generally considered protective against non-response bias, since high response rates 

mitigate the impact of systematic differences when they exist. It is therefore important to conduct an 

analysis of non-response bias to determine the extent to which responding and non-responding units 

differ in ways that could influence survey outcomes. 

Although survey data for non-responding organizations are by definition not available, it is possible to 

evaluate the potential for non-response bias in the current surveys by comparing key characteristics 

known for all sampled organizations (both responding and non-responding). To the extent that these 

characteristics are also associated with survey outcomes, differences in the characteristics between 

respondents and non-respondents raise the potential for non-response bias. 

Using such auxiliary data, non-response bias analyses were conducted for both the National Sample of 

Grant-making Nonprofit organizations (RR3 response rate = 12%) and for the survey of Non-selected SIF 
nationalservice.gov/research B-8 

Applicants (RR3 response rate = 49%). In both cases, the approach was to test for differences between 

responding and non-responding organizations with respect to key characteristics known for both types of 

record. For the sample of other nonprofit organizations, a relatively large number of auxiliary variables 

were available in the GuideStar IRS database from which the original sampling frame was constructed. 

For the Non-selected Applicant sample, auxiliary data were more limited but sufficient for analysis. 

Survey of National Sample of Grantmaking Nonprofits 

Auxiliary data for the other nonprofit organizations comparison group were taken from the GuideStar 

database used to construct the sampling frame. These variables reflect data collected from the 2013 IRS 

Form 990 that each of these organizations is required to file. For this analysis, both respondents 

determined to be eligible (n = 219) and respondents determined to be ineligible (n = 168) were classified as 

responding organizations (for a total of n = 387). The remaining sampled organizations, which did not 

submit completed surveys, were classified as non-respondents (n = 2,938). Variation in the sample sizes 

for responding and non-responding organizations in these analyses is due to missing values in the 

auxiliary variables. 

Exhibit B-3 shows the bivariate comparisons for categorical variables (such as Census region). In these 

cases, the question is whether response rates differ between levels of the categorical variable, and 

differences are tested for statistical significance using the Pearson chi-square test. Exhibit B-4 shows the 

bivariate comparisons for continuous variables (such as organization age). Here, the question is whether 

the mean value of the variable differs between responding and non-responding organizations, with 

differences tested for statistical significance using the independent-samples t-test. All significance tests 

were conducted controlling the Type I error rate at α = .05 

Exhibit B-3. Non-Response bias comparisons for survey of other nonprofit organizations, 

categorical variables 

Variable Level

N 

Sampled

N 

Responding

N NonResponding

Raw 

Response 

Rate

ChiSquare df p

Census 

Region

Midwest 718 96 622 13.40% 3.65 3 .302

Northeast 694 72 622 10.40%

South 1,167 129 1,038 11.10%

West 743 89 654 12.00%

NTEE Code 

(Core 

Categories)

I – Arts, Culture, and Humanities 119 14 105 11.80% 28.96 9 .001

II – Education 463 45 418 9.70%

III – Environment and Animals 117 15 102 12.80%

IV – Health 738 58 680 7.90%

V – Human Services 568 85 483 15.00%

VI – International and Foreign 

Affairs

107 10 97 9.30%

VII – Public, Societal Benefit 1,093 153 940 14.00%

VIII – Region Related 53 5 48 9.40%

IX – Mutual/Membership Benefit 3 0 3 0.00%

X – Unknown, Unclassified 29 0 29 0.00%

Revenue 

Stratum

<$5MM 1,284 142 1,142 11.10% 18.73 2 <.001

$5-20MM 1,082 160 922 14.80%

$20MM+ 951 83 868 8.70%
nationalservice.gov/research B-9 

Exhibit B-4. Non-Response bias comparisons for survey of other non-profit organizations, 

continuous variables 

Variable Response N Mean SD t df p

Organization Age Non-Responding 1,409 35.3 28.6 0.88 1,569 .379

Responding 162 33.3 25.3

Total Revenue Non-Responding 2938 92,088,434 4.37E+08 2.07 3,323 .038

Responding 387 44,954,320 2.56E+08

Receipt Totals Non-Responding 2,938 1.56E+08 1.04E+09 1.76 3,323 .078

Responding 387 61,394,358 3.44E+08

Contributions Non-Responding 2,819 28,350,667 1.12E+08 1.45 3,197 .148

Responding 380 19,741,178 84,904,526

Government Grants Non-Responding 1,059 16,874,360 77,563,158 1.05 1,240 .295

Responding 183 10,514,072 64,545,045

Grants to U.S.

Organizations

Non-Responding 2,935 11,830,001 52,909,774 1.35 3,320 .176

Responding 387 8,166,687 16,476,379

Total Expenses Non-Responding 2,935 82,716,383 4.04E+08 2.01 3,320 .044

Responding 387 40,507,200 2.26E+08

Total Assets (EOY) Non-Responding 2,938 2.21E+08 1.18E+09 1.45 3,323 .148

Responding 387 1.31E+08 8.41E+08

All-Year Employees Non-Responding 2,924 555.7 3,086.9 2.11 3,309 .035

Responding 387 217.4 1,715.1

The bivariate comparisons show that some organizational characteristics are significantly associated with 

whether or not a sampled organization responded to the survey request. Response rates varied 

significantly among National Taxonomy of Exempt Entities (NTEE) categories, with the highest response 

rates observed among Human Services and Public/Societal Benefit organizations (15.00% and 14.00%, 

respectively) and the lowest response rates observed among Health, Unknown/Unclassified, and 

Mutual/Membership Benefit organizations (7.90%, 0%, and 0%, respectively). To the extent that survey 

outcomes differ by NTEE category, this association raises the possibility of some non-response bias, in 

that the responding organizations may not be completely representative of the target population for the 

survey. 

Survey response was also significantly associated with total revenue, total expenses, and number of allyear employees. The pattern was for smaller organizations to respond more frequently than larger 

organizations. However, when breaking total revenue into the three strata used for weighting the survey 

data, a non-linear pattern is observed in which medium-revenue organizations are most likely to respond 

and low- and high-revenue organizations are less likely to respond. To the extent that survey outcomes 

are associated with organization size, these results again raise the possibility of non-response bias. 

However, because the sample was stratified by total revenue, and the weighting included an adjustment 

for non-response within (collapsed, three-level) revenue strata, the weighting should effectively reduce 

any such bias. 

Survey of Non-selected SIF Applicants 

Auxiliary data for the Non-selected SIF Applicant organizations came from multiple sources. For the 

2010, 2011, and 2012 cohorts, SIF application data were available, including the final rating assigned to 

each applicant. Application data were unavailable for the 2014 cohort, however. Data on revenue and 

grants to U.S. organizations came from the same GuideStar database used to construct the sampling 

frame for the survey of other nonprofit organizations, and thus reflects financial records filed with the 

IRS as of 2013. 
nationalservice.gov/research B-10

For this analysis, all respondents were determined to be eligible (n = 19), and the remaining sampled 

organizations, which did not submit completed surveys, were classified as non-respondents (n = 21).4

Variation in the sample sizes for responding and non-responding organizations in these analyses is due 

to missing values in the auxiliary variables. 

The same analytic approach was used to evaluate the potential for non-response bias in the survey of 

Non-selected SIF Applicants and was used for the survey of other nonprofit organizations (described 

above). Exhibit B-5 shows the bivariate comparisons for categorical variables (such as Census region). In 

these cases, the question is whether response rates differ between levels of the categorical variable, and 

differences are tested for statistical significance using the Pearson chi-square test. Exhibit A-6 shows the 

bivariate comparisons for continuous variables (such as revenue). Here, the question is whether the mean 

value of the variable differs between responding and non-responding organizations, with differences 

tested for statistical significance using the independent-samples t-test. All significance tests were 

conducted controlling the Type I error rate at α = .05 

Exhibit B-5. Non-Response Bias Comparisons for Survey of Non-Selected Applicant Organizations, 

Categorical Variables 

Variable Level

N 

Sampled

N 

Responding

N NonResponding

Raw 

Response 

Rate

ChiSquare df p

Cohort 2014 15 9 6 60.00% 2.49 1 .115

2010-2012 26 9 17 34.60%

Census 

Region

Midwest 3 1 2 33.30% 1.41 3 .703

Northeast 2 1 1 50.00%

South 7 2 5 28.60%

West 3 2 1 66.70%

NTEE Code 

(Core 

Categories)

II – Education 5 2 3 40.00% 2.27 4 .686

IV – Health 1 1 0 100.00%

V – Human 

Services

3 1 2 33.30%

VII – Public, 

Societal Benefit

26 12 14 46.20%

X – Unknown, 

Unclassified

1 0 1 0.00%

Revenue 

Stratum

<$5MM 9 4 5 44.40% 2.36 2 .307

$5-20MM 7 5 2 71.40%

$20MM+ 21 8 13 38.10%

Applied More 

than Once?

No 33 13 20 39.40% 1.40 1 .237

Yes 8 5 3 62.50%

Application 

Rank

1 or 2 7 3 4 42.90% 0.12 1 .725

3, 4, or 5 6 2 4 33.30%

4

 The one organization determined not to be eligible due to having entered bankruptcy, and which did not complete a survey, is not included in 

these analyses. 
nationalservice.gov/research B-11

Exhibit B-6. Non-Response Bias Comparisons for Survey of Non-selected Applicant Organizations, 

Continuous Variables 

Variable Response N Mean SD t df p

Organization Age Non-Responding 19 46.05 33.46 0.42 34 .679

Responding 17 41.76 27.42 

Total Revenue Non-Responding 20 44,792,054.00 44,078,279.00 0.03 35 .974

Responding 17 44,207,512.00 64,912,570.00 

Receipt Totals Non-Responding 19 96,699,839.00 139,630,000.00 0.11 34 .914

Responding 17 89,802,711.00 233,350,000.00 

Contributions Non-Responding 18 31,307,778.00 29,048,579.00 -0.52 33 .606

Responding 17 38,907,671.00 54,298,882.00 

Government Grants Non-Responding 11 3,968,798.00 6,592,691.00 -0.66 21 .516

Responding 12 11,326,320.00 36,312,618.00 

Grants to U.S.

Organizations

Non-Responding 18 22,383,509.00 23,706,628.00 -0.38 32 .707

Responding 16 27,146,356.00 47,134,988.00 

Total Expenses Non-Responding 20 39,557,176.00 38,667,169.00 -0.10 35 .919

Responding 17 41,151,817.00 55,426,404.00 

Total Assets (EOY) Non-Responding 20 201,460,000.00 433,650,000.00 0.83 35 .410

Responding 17 99,062,554.00 283,550,000.00 

All-Year Employees Non-Responding 20 151.30 228.10 -0.20 35 .842

Responding 17 172.10 394.00 

Federal Funding 

Requested

Non-Responding 17 2,393,156.00 2,376,380.00 1.41 24 .170

Responding 9 1,253,012.00 420,088.00 

Final Application 

Score

Non-Responding 8 63.31 25.80 0.61 12 .551

Responding 6 52.47 40.50 

Partly due to the small sample sizes available for these analyses, the bivariate comparisons do not show 

any significant associations between the characteristics of Non-selected SIF Applicants and whether or 

not an organization responded to the survey request. Nevertheless, two characteristics showed 

marginally significant associations with response (p < .2). First, the 2014 cohort of Non-selected SIF 

Applicants was more likely to respond (raw response rate = 60.0%) compared to the 2010-2012 cohort 

(raw response rate = 34.6%). Second, the mean dollar amount of federal funds requested in the 

application was lower for responding organizations (M = $1,253,012.00) compared to non-responding 

organizations (M = $2,393,156.00). 

Although the small sample sizes available for this analysis limit the ability to draw reliable conclusions, 

the overall patterns suggest that organization size (in terms of total revenue, receipt totals, contributions, 

grants to U.S. organizations, total expenses, and number of employees) was not systematically related to 

survey response. Although there appeared to be some variation in response rates by Census region, 

NTEE code, number of applications, and application rank, this variation is more likely to be random 

rather than systematic. Finally, as noted, there was a tendency for organizations who applied in 2014 to 

be more likely to respond than earlier applicants, and for the later respondents to have requested less 

federal funding. Overall, this analysis does not suggest a significant potential for non-response bias to 

have affected the survey of Non-selected SIF Applicants. 
nationalservice.gov/research B-12

B.4. Data Analysis 

We employed the mixed-method approach in all phases of the evaluation, including design, data 

collection, and data analysis in order to draw on the strengths of both quantitative and qualitative 

approaches. 

B.4.1. Quantitative Analysis 

The quantitative survey data are primarily Likert scale measures, including measures of the same 

attribute at two time points. The main types of analysis are: 

 Descriptive analyses of changes over time for SIF grantees and comparison groups. These include

means and proportions. Examples include the mean scale value for a group, such as group mean on

the importance of evidence of effectiveness as a basis for funding for SIF grantees; or the proportion

that report different scale values, e.g., the percentage reporting evidence of effectiveness is relied on

to a large or very large extent for funding decisions.

 Group difference analysis between the SIF and comparison groups. For the analyses, we calculated

values such as the mean change score for different groups or the percentage reporting change, and

test differences. The text below describes the specific approach to analyzing group differences.

Data were collected for all SIF grantees, so that this group represents the entire universe of SIF 

intermediary organizations. As such, the measures from the survey of this group are actual population 

values, not estimates. Non-response among the census of Non-selected SIF Applicants (due to 19 of the 41 

censused organizations responding) is treated as random error, statistically equivalent to sampling error, 

and estimates for this group take this variability and the finite population correction into account. The 

National Sample of other Nonprofit Grantmakers was a probability sample drawn from the universe of 

nonprofit grantmaking organizations, and estimates for this group also take into account variability due 

to non-response and sampling error, along with the effects of unequal weighting and the finite 

population correction. 

To test for differences between the SIF grantees and Non-selected SIF Applicants, we used a nonparametric technique known as the Mann-Whitney test to account for the small sample sizes of the two 

groups. For differences between SIF grantees and the National Sample of Grantmaking Nonprofits, we 

calculated the confidence interval at 95% around the values for the nonprofit sample; then, if the 

difference between the SIF value and the value for the sample is larger than the confidence interval for 

the sample, we can draw the inference that the observed difference is greater than would occur by chance 

(Cochran, 1977; the appropriate test is a one-sample t-test.) 

In addition to testing the statistical significance of group differences, we also used effect size estimates to 

gauge the practical significance of the differences. Statistical significance, which measures the probability 

of getting a result, is highly dependent on sample sizes. On the other hand, effect sizes measures the 

magnitude of the difference and is less dependent on sample sizes. A small effect can be statistically 

significant if the sample size is large.

B.4.2. Qualitative Analysis 

To produce informed, actionable findings, we triangulated findings between quantitative and qualitative 

sources to ensure that information from one source, such as stakeholder interviews, was compared with 

other sources, such as document review and survey data. Triangulating results from multiple sources 
nationalservice.gov/research B-13

creates more credible evaluation results and is critical to the validity and reliability of findings 

(LeCompte & Schensul, 1999). 

Qualitative analyses include content analysis of documents, as well as analyses of responses to semistructured interview questions and open-ended survey questions. We reviewed and coded the data to 

identify themes that are common within and across grantees, as well as potential outliers specific to an 

individual intermediary, in an iterative fashion. We developed initial codes for each question. Additional 

codes were developed as they emerged. Comparisons across time and across grantees and other groups 

were conducted to identify patterns of barriers and facilitators to change, and effects of the SIF 

experience. 
nationalservice.gov/research C-1 

Appendix C. Findings from the SIF 2014 cohort 

SIF 2014 grantees were only in the first year of SIF funding when the survey was administered. The 

survey asked about early experiences, since they had a few months in the program at that time, although 

it was not expected that major changes would have yet occurred as a result of the SIF grant. 2014 

responses from SIF grantees and Non-selected SIF Applicants provide useful insights into the application 

experience and SIF start-up, and also can be regarded as baseline data for future data collection and 

analysis. 

C.1. Background and Demographics 

Financial data. SIF 2014 grantees were similar to 2014 Non-selected SIF Applicants in terms of 

government grants received and grants made to U.S. organizations, but had significantly higher gross 

revenue than comparison organizations. 

Compared to data in Exhibit C-1, SIF 2014 grantees were much larger than SIF 2010-2012 grantees on all 3 

financial indicators, suggesting that the 2014 grantees were very different from the earlier cohort. 

Exhibit C-1. Financial indicators by respondent groups 

SIF 2014

Non-selected SIF Applicants

2014

Gross Revenue N 5 9

Mean $ 713,149,014 a+ $ 135,382,776 

Median $ 250,075,637 $ 15,272,282 

Government Grants received N 3 6

Mean $ 25,492,269 $ 21,671,851 

Median $ 1,047,131 $ 280,929 

Grants made to U.S.

Organization 

N 5 8

Mean $ 91,888,326 $ 26,071,513 

Median $ 18,532,535 $ 7,303,501 

Statistical significance (p<.05) is noted as: 

 a: significant difference between SIF and Non-selected SIF Applicants 

 b: significant difference between SIF and the National Sample of Grantmaking Nonprofits 

 +: favoring SIF 

 -: favoring the comparison group 

Federal Funding. Both SIF grantees and Non-selected SIF Applicants in the 2014 cohort increased the 

level of federal funding from 2009 to 2014, as shown in Exhibit C-2. 

Exhibit C-2. Federal funding experience 

2009 2014

Group n % (yes) n % (yes) Change

Received federal government 

funding (other than SIF)

SIF 2014 6 50.0% 6 66.7% 16.7%

Non-selected SIF 

Applicants 2014

9 33.3% 9 55.6% 22.3%

The survey also asked organizations receiving SIF funding whether it was the first federal funding their 

organization had ever received. Among the SIF 2010-2012 grantees, half indicated that the SIF was the 
nationalservice.gov/research C-2 

first time they received federal funding; by contrast, only 16.7% of SIF 2014 grantees reported that they 

had never received federal funding prior to SIF (Exhibit C-3). 

Exhibit C-3. Federal funding receipt by SIF grantees 

Group n % (yes)

Was your SIF funding the first federal government funding your 

organization ever received?

SIF 2010-2012 20 50.0%

SIF 2014 6 16.7%

C.2. Support Received by SIF Grantees

Helpful support. Helpful support for the SIF 2014 grantees included general program support and TA 

(related to fiscal matters and evaluation), new grantee webinars, compliance training, SIF convenings, 

and other information sharing. Specifically, 

 Lots of fantastic support and advice from our Program Officer! Great advice from our “coach” during the

process of defining the subgrantee selection process Connecting with other SIF grantees during our scale-up

phase of the grant and continuing the connection, especially during Year 1.

 The SIF program officer provided extremely helpful support at the beginning of our grant. She connected us

with other organizations implementing collective impact strategies and provided valuable input into our initial

RFP. The evaluation TA contractor has been very helpful in providing assistance on questions related to fiscal

and evaluation matters, respectively. The evaluation contractor has been very open and willing to help answer

questions on any topic for which we needed support. Other SIF intermediaries have been very supportive and

happy to answer any questions, and in two cases, to participate in our evaluator selection process.

Less helpful support. SIF 2014 grantees listed a few areas that could be improved, including 1) offering 

webinars earlier rather than after strategies were already implemented; 2) providing bi-monthly 

conference calls in the form of webinars with presentations and visuals; 3) clarifying points of contact at 

CNCS to avoid confusion; 4) clarifying guidance from the fiscal officer; and 5) offering a media toolkit for 

communication support. 

Areas requiring more support. SIF 2014 grantees said that they could have benefited from more 

interpretation of federal regulation and compliance, clarification of requirements, and engagement of 

external consultants to help meet SIF requirements. One grantee said, “We did not fully understand how to 

apply for reimbursement, and the different reports that we needed to submit. More hands-on upfront support would 

be helpful. It all worked out, but was a little confusing.”

C.3. Impact on Organizational Capacity 

Statistical significance tests did not reveal any differences between SIF 2014 grantees and their Nonselected SIF Applicants in five areas of organizational change, including selection of subgrantees, support 

for subgrantees, evaluation and data use, scale up, and collaboration (Exhibits C-4 through C-11). Survey 

data to be collected in 2016 will allow us to examine the change patterns for the SIF 2014 grantees and in 

comparison with their non-selection counterparts. 
nationalservice.gov/research C-3 

Exhibit C-4. Approaches to selection of subgrantees 

Group

To what extent did your organization do this 

...? 

7 point Likert Scale: 

1- Not at all to 7- Always

2009 2014 Change

A. Used an open competitive process to solicit 

and review applications and to make selection 

decisions

SIF 2014 4.3 (6) 4.8 (6) 0.5

Non-selected SIF 

Applicant 2014

5.6 (9) 5.7 (9) 0.1

B. Required applicant organizations to provide 

evidence of intervention effectiveness to be 

eligible for funding (includes pre- and post-test 

outcome data or other evidence based on 

evaluation studies)

SIF 2014 3.2 (6) 4.0 (6) 0.8

Non-selected SIF 

Applicant 2014

3.3 (9) 4.4 (9) 1.1

C. Required applicants to submit a plan for 

rigorous evaluation of intervention to be eligible 

for funding (that is, quasi-experimental designs 

with a comparison group, experimental designs,

or other similarly rigorous designs)

SIF 2014 1.5 (6) 2.2 (6) 0.7

SIF 2014 1.7 (7) 1.9 (8) 0.3

Statistical significance (p<.05) is noted as: 

 a: significant difference between SIF and Non-selected SIF Applicants 

 b: significant difference between SIF and National Sample of Grantmaking Nonprofits 

 +: favoring SIF 

 -: favoring the comparison group 

Exhibit C-5. Approach to support for subgrantees 

Group

To what extent did your organization do this ...?

7 point Likert Scale: 

1- Not at all to 7- Always

2009 2014 Change

A. Provided funding to carry out an 

evaluation or hire an external 

evaluator, as part of the grant or 

through other means

SIF 2014 2.5 (6) 3.3 (6) 0.8

Non-selected SIF 

Applicant 2014

2.6 (8) 2.6 (8) 0.0

B. Provided training or technical 

assistance by your staff consultants 

or other means to conduct rigorous 

evaluation

SIF 2014 1.3 (6) 2.5 (6) 1.2

Non-selected SIF 

Applicant 2014

3.3 (8) 4.0 (8) 0.8

C. Provided training or technical 

assistance by your staff consultants 

or other means to support 

implementation of the program

SIF 2014 2.8 (6) 4.0 (6) 1.2

Non-selected SIF 

Applicant 2014

4.1 (9) 4.4 (9) 0.3

Statistical significance (p<.05) is noted as: 

 a: significant difference between SIF and Non-selected SIF Applicants 

 b: significant difference between SIF and the National Sample of Grantmaking Nonprofits 

 +: favoring SIF 

 -: favoring the comparison group 
nationalservice.gov/research C-4 

Exhibit C-6. Approach to evaluation 

Group

To what extent did your organization do this ...?

7 point Likert Scale: 1- Not at all to 7- Always

2009 2014 Change

A. Conducted rigorous 

evaluations of programs funded 

by your organization

SIF 2014 2.5 (6) 3.3 (6) 0.8

Non-selected SIF 

Applicant 2014

3.2 (9) 3.4 (9) 0.2

B. Used evaluation findings to 

improve programs funded by your 

organization

SIF 2014 2.3 (6) 3.7 (6) 1.3

Non-selected SIF 

Applicant 2014

4.0 (9) 4.8 (9) 0.8

C. Used evaluation findings to 

demonstrate and communicate 

effectiveness of programs funded 

by your organization

SIF 2014 3.0 (6) 4.3 (6) 1.3

Non-selected SIF 

Applicant 2014

3.3 (9) 4.4 (9) 1.1

Statistical significance (p<.05) is noted as: 

 a: significant difference between SIF and Non-selected SIF Applicants 

 b: significant difference between SIF and the National Sample of Grantmaking Nonprofits 

 +: favoring SIF 

 -: favoring the comparison group

Exhibit C-7. Infrastructure for evaluation 

Did/does your organization 

have the following?

2009 2014

Group n % (yes) n % (yes) Change

A. Staff position(s) or group 

within your organization 

dedicated to evaluation 

SIF 2014 6 0.0% 6 33.3% 33.3%

Non-selected SIF 

Applicant 2014

9 55.6% 9 44.4% -11.2%

B. External evaluation 

partner(s)—consultant(s) or 

organization(s)—that provide 

your organization with evaluation 

services

SIF 2014 6 16.7% 6 100.0% 83.3%

Non-selected SIF 

Applicant 2014

9 66.7% 9 77.8% 11.1%

C. Part of the organization's 

budget dedicated to evaluation

SIF 2014 6 0.0%a- 6 50.0% 50.0%

Non-selected SIF 

Applicant 2014

9 66.7% 9 66.7% 0.0%

Statistical significance (p<.05) is noted as: 

 a: significant difference between SIF and Non-selected SIF Applicants 

 b: significant difference between SIF and the National Sample of Grantmaking Nonprofits 

 +: favoring SIF 

 -: favoring the comparison group 
nationalservice.gov/research C-5 

Exhibit C-8. Perceived changes in evaluation budget 

Was your 2014 

evaluation budget... 

than the evaluation 

budget in 2009? Group n

Substantially 

higher

Somewhat 

higher

About 

the same

Somewhat 

lower

Substantially 

lower

Total evaluation 

budget in dollars

SIF 2014 6 50.0% 0.0% 50.0% 0.0% 0.0%

Non-selected SIF 

Applicant 2014

9 44.4% 0.0% 22.2% 11.1% 22.2%

Evaluation budget as 

a percentage of the 

organization’s total 

budget for the year

SIF 2014 6 50.0% 0.0% 50.0% 0.0% 0.0%

Non-selected SIF 

Applicant 2014

9 11.1% 33.3% 22.2% 11.1% 22.2%

Statistical significance (p<.05) is noted as: 

 a: significant difference between SIF and Non-selected SIF Applicants 

 b: significant difference between SIF and the National Sample of Grantmaking Nonprofits 

 +: favoring SIF 

 -: favoring the comparison group 
nationalservice.gov/research C-6 

Exhibit C-9. Changes in evaluation budget 

Did/does your 

organization have 

the following?

2009 2014

Change 

Group n Mean Median Min. Max n Mean Median Min. Max (mean)

Total annual 

evaluation budget (in 

$) – for in-house 

evaluators or external 

partners

SIF 2014 4 $12,500 $0 $0 $50,000 4 $1,000,000 $1,000,000 $0 $2,000,000 $987,500

Nonselected SIF 

Applicant 

2014

2 $50,000 $50,000 $0 $100,000 1 $0 $0 $0 $0 $-50,000

Annual evaluation 

budget as % of the 

total organization 

budget

SIF 2014 4 0.01% 0% 0.0% 0.02% 3 0.5% 0.5% 0.0% 1.0% 0.5%

Nonselected SIF 

Applicant 

2014

2 2.0% 2.0% 0.0% 4% 2 3.0% 3.0% 0.0% 6.0% 1.0%

Number of full time 

equivalent (FTE) staff 

with primary 

responsibility for 

evaluation

SIF 2014 5 0 0 0 0 5 0.7 0 0 4 0.7

Nonselected SIF 

Applicant 

2014

5 0.3 0 0 1 5 0.8 1 0 2 0.5

Statistical significance (p<.05) is noted as: 

 a: significant difference between SIF and Non-selected SIF Applicants 

 b: significant difference between SIF and the National Sample of Grantmaking Nonprofits 

 +: favoring SIF 

 -: favoring the comparison group 
nationalservice.gov/research C-7 

Exhibit C-10. Approach to scaling up programs 

Group

To what extent did your organization do this ...?

7 point Likert Scale: 1- Not at all to 7- Always

2009 2014 Change

A. Undertook efforts to scale 

up existing program(s) – i.e., 

to expand the program(s) 

within the community or to 

other communities or 

populations

SIF 2014 3.5 (6) 4.7 (6) 1.2

Non-selected SIF 

Applicant 2014

2.9 (9) 4.4 (9) 1.6

B. Selected programs for 

scale-up based on rigorous 

evaluation that shows them 

to be effective

SIF 2014 2.0 (6) 3.3 (6) 1.3

Non-selected SIF 

Applicant 2014

2.3 (9) 3.3 (9) 1.0

Statistical significance (p<.05) is noted as: 

 a: significant difference between SIF and Non-selected SIF Applicants 

 b: significant difference between SIF and the National Sample of Grantmaking Nonprofits 

 +: favoring SIF 

 -: favoring the comparison group 

Exhibit C-11. Approach to collaboration 

Group

To what extent did your organization do this ...?

7 point Likert Scale: 1- Not at all to 7- Always

2009 2014 Change

A. Participated in funding 

alliance(s) with other nonprofit 

sector organizations. (For 

example, co-funding programs 

through joint funding; providing or 

receiving matching funds; or other 

collaboration) 

SIF 2014 2.7 (6) 4.2 (6) 1.5

Non-selected SIF 

Applicant 2014

2.1 (9) 3.9 (9) 1.8

B. Participated in collaborations 

with other nonprofit organizations 

to share knowledge

SIF 2014 3.5 (6) 4.8 (6) 1.3

Non-selected SIF 

Applicant 2014

4.0 (9) 5.2 (9) 1.2

C. Collaborated with other 

organizations for purposes of 

advocacy – to advocate for or 

develop public support for 

programs or approaches to 

addressing social problems

SIF 2014 2.3 (6) 4.8 (6) 2.5

Non-selected SIF

Applicant 2014

3.9 (9) 5.0 (9) 1.1

Statistical significance (p<.05) is noted as: 

 a: significant difference between SIF and Non-selected Applicants 

 b: significant difference between SIF and the National Sample of Grantmaking Nonprofits 

 +: favoring SIF 

 -: favoring the comparison group 
nationalservice.gov/research C-8 

C.4. Reflections on SIF Experience

Although only 6 to 9 months into the first year of SIF grant funding at the time of the survey, several SIF 

2014 grantees reported some changes in terms of implementing policies and procedures, hiring staff, 

evaluation, collaboration, compliance, and scaling. 

 We are working with partners in an enhanced way; we are raising match funding for the initiative in a different

way than we have historically; and we have enhanced our accounting and compliance measures.

 We reviewed our organization's policies and procedures and made several changes to bring them into alignment

with CNCS requirements. We hired one new staff member and reassigned another to manage the SIF grant. We

have adjusted our grant payment process and grants monitoring procedures to accommodate SIF requirements.

We instituted a criminal background check process for SIF-funded staff. We created new tools and manuals,

including terms and conditions, for our SIF grantees. We work more interdependently with other departments

in the foundation, including finance, development, gifts, grants and compliance, and human resources.

SIF 2014 grantees offered additional recommendations for improving the SIF application process, such as 

adjusting the timeframe, including a pre-qualification phase, being transparent about implementation 

requirements, and further clarifying application requirements. 

 Overall the application process was smooth with appropriate levels of guidance. One suggestion is to offer a prequalification phase so that organizations without capacity to implement a SIF grant could be weeded out earlier,

prior to wasting time/resources on the application.

 One of the biggest hurdles for the organization was that what was clear in the implementation was not clear in

the application. In a lot of ways, they didn't understand what they were getting themselves into. They are

pleased with the experience, but there have been a lot of surprises. For example, criminal history check process,

type of rigor of evaluation, lack of flexibility on financial side which affects their ability to work collaboratively.

(Disallowance of in-kind match—now communities have to set up a hierarchy in the way they operate.

Subgrantees can’t sub grant, so they must set up contracts with partners, but none of that counts as match.)

But SIF has imposed a structure, and they are beginning to get through the partnership challenges they would

not have had without the SIF.
nationalservice.gov/research D-1 

Appendix D. SIF Survey Descriptive Data Tables 

Key Findings 

 Organizational capacities: SIF 2010-2012 grantees grew in organizational capacities between 2009 and 2014 in 13 out of 14 measures related to

selection of subgrantees, support for subgrantees, evaluation, scaling up, and collaboration (Tables 1-V).

 Selection of subgrantees to fund: Compared to the National Sample of Grantmaking Nonprofits, SIF 2010-2012 grantees experienced

significantly fewer changes between 2009 and 2014 in the extent to which the organization 1) used an open competitive process to solicit and

review applications and to make selection decisions, and 2) required applicant organizations to provide evidence of intervention effectiveness

to be eligible for funding (Table I).

 Support for subgrantees: Compared to the National Sample of Grantmaking Nonprofits, SIF 2010-2012 grantees experienced significantly

greater changes between 2009 and 2014 in the extent to which the organization 1) provided funding to carry out an evaluation or hire an

external evaluator, as part of the grant or through other means, and 2) provided training or technical assistance by staff consultants or other

means to conduct rigorous evaluation (Table II).

 Evaluation: Compared to Non-selected SIF Applicants 2010-2012 and the National Sample of Grantmaking Nonprofits, SIF 2010-2012 grantees

experienced significantly greater changes between 2009 and 2014 in the extent to which the organization conducted rigorous evaluations of

programs it had funded. Compared to the National Sample of Grantmaking Nonprofits, SIF 2010-2012 grantees experienced significantly

greater changes between 2009 and 2014 in the extent to which the organization used evaluation findings to demonstrate and communicate the

effectiveness of programs it had funded (Table III).

 Development of capacity among SIF subgrantees: SIF 2010-2012 subgrantees experienced positive growth 1) implementing the interventions

they are carrying out in their communities; 2) designing and conducting rigorous evaluations of their interventions; 3) making use of

evaluation findings for program improvement; 4) raising matching funds for the intervention; 5) scaling up the intervention; and 6) sharing

knowledge and best practices (Table VII).

Guide to the Table Shells 

 Main takeaways are summarized at the bottom of each table, except for those tables in sections IX and X which are self-explanatory.

 Tables I through V present both descriptive and inferential statistics on the organizational outcomes for SIF grantees and the two comparison

groups: Non-selected SIF Applicants and the National Sample of Grantmaking Nonprofits. The key information is the comparison between

SIF 2010-12 grantees, Non-selected SIF Applicants 2010-12, and the National Sample of Grantmaking Nonprofits. Any information concerning

SIF 2014 and Non-selected SIF Applicants 2014 (greyed out) is less important because, since SIF 2014 grantees have just begun

implementation, insufficient time has passed for major changes.
nationalservice.gov/research D-2 

 In Tables I through V, the left side contains mean scale values by year and the change. Statistical significance tests were conducted between

groups on 1) change and 2) initial status in 2009. Items showing significant changes were marked with superscripts and highlighted in yellow.

 In Tables I through V, the right side contains descriptives about major factors contributing to the changes, only for those respondents who

reported such changes. A respondent can select multiple responses, and hence 1) the total percentage may exceed 100%; 2) selecting “your

organization’s participation in SIF” as SIF grantees does not affect other response categories; and 3) due to small sample sizes, differences

between SIF 2010-2012 and the Non-selected SIF Applicants were mostly not statistically significant.

 Tables VII and IX reflect SIF grantee information only. The statistics are population values and not estimates.
nationalservice.gov/research D-3 

I. Selection of Subgrantees to Fund 

To what extent did your 

organization do this ...? 5 Major factors that contributed to the change (check all that apply)

Group

7 point Likert Scale: 

1- Not at all to 7- Always

Respondent

s with 

change (n)

Trends in 

the larger 

grantmaking 

world

Your 

organization’s 

participation 

in SIF

Specific 

requirements 

attached to the 

funding your 

organization 

receives

Your organization’s 

board/leadership 

directed 

organization to 

implement change Other

Respondents 

with no 

2009 2014 Change change 6

A. Used an open 

competitive process 

to solicit and review 

applications and to 

make selection 

decisions

SIF 2010-

2012

4.6 

(20)

4.4 

(19)

-0.1b- 6 16.7% 33.3% 33.3% 16.7% 33.3% 13

Non-selected 

SIF 

Applicants 

2010-2012

5.0 

(9)

5.3 

(9)

0.3 4 0.0% -- 50.0% 75.0% 25.0% 5

National 

Sample of 

Grantmaking 

Nonprofits

4.8 

(207)

5.1 

(207)

0.3 53 38.7% -- 24.6% 60.2% 15.9% 150

SIF 2014 4.3 

(6)

4.8 

(6)

0.5 1 100.0% -- 100% 100% 100% 4

Non-selected 

SIF 

Applicants 

2014

5.6 

(9)

5.7 

(9)

0.1 4 25.0% -- 25.0% 75.0% 25.0% 5

5

 Significance (p<.05) were presented as: 

 a: significant difference between SIF and Non-selected SIF Applicants 

 b: significant difference between SIF and the National Sample of Grantmaking Nonprofits 

 +: favoring SIF 

 -: favoring the comparison group 

6

 The change was calculated by averaging the change scores of individual records. 
nationalservice.gov/research D-4 

To what extent did your 

organization do this ...? 5 Major factors that contributed to the change (check all that apply)

Group

7 point Likert Scale: 

1- Not at all to 7- Always

Respondent

s with 

change (n)

Trends in 

the larger 

grantmaking 

world

Your 

organization’s 

participation 

in SIF

Specific 

requirements 

attached to the 

funding your 

organization 

receives

Your organization’s 

board/leadership 

directed 

organization to 

implement change Other

Respondents 

with no 

2009 2014 Change change 6

B. Required 

applicant 

organizations to 

provide evidence of 

intervention 

effectiveness to be 

eligible for funding 

(includes pre- and 

post-test outcome 

data or other 

evidence based on 

evaluation studies)

SIF 2010-

2012

3.5 

(20)

4.0 

(19)

0.6 8 75.0% 62.5% 25.0% 37.5% 12.5% 11

Non-selected 

SIF 

Applicants 

2010-2012

4.8 

(9)

5.1 

(9)

0.3 3 33.3% -- 66.7% 0.0% 0.0% 6

National 

Sample of 

Grantmaking 

Nonprofits

3.7 

(202)

4.5 

(206)

0.8 84 48.9% -- 34.0% 62.6% 9.6% 113

SIF 2014 3.2 

(6)

4.0 

(6)

0.8 3 66.7% -- 66.7% 33.3% 0.0% 3

Non-selected 

SIF 

Applicants 

2014

3.3 

(9)

4.4 

(9)

1.1 7 57.1% -- 28.6% 57.1% 42.9% 2
nationalservice.gov/research D-5 

To what extent did your 

organization do this ...? 5 Major factors that contributed to the change (check all that apply)

Group

7 point Likert Scale: 

1- Not at all to 7- Always

Respondent

s with 

change (n)

Trends in 

the larger 

grantmaking 

world

Your 

organization’s 

participation 

in SIF

Specific 

requirements 

attached to the 

funding your 

organization 

receives

Your organization’s 

board/leadership 

directed 

organization to 

implement change Other

Respondents 

with no 

2009 2014 Change change 6

C. Required 

applicants to submit 

a plan for rigorous 

evaluation of 

intervention to be 

eligible for funding 

(that is, quasiexperimental designs 

with a comparison 

group, experimental 

designs or other 

similarly rigorous 

designs)

SIF 2010-

2012

2.0 

(20)

2.3 

(19)

0.4 5 60.0% 60.0% 20.0% 20.0% 20.0% 14

Non-selected 

SIF 

Applicants 

2010-2012

2.9 

(9)

3.1 

(9)

0.2 2 0.0% -- 100.0% 50.0% 0.0% 7

National 

Sample of 

Grantmaking 

Nonprofits

2.4 

(201)

2.9 

(200)

0.5 46 47.3% -- 35.7% 45.2% 12.6% 150

SIF 2014 1.5 

(6)

2.2 

(6)

0.7 2 50.0% -- 100.0% 50.0% 0.0% 4

Non-selected 

SIF 

Applicants 

2014

1.7 

(7)

1.9 

(8)

0.3 2 100.0% -- 50.0% 50.0% 0.0% 5

Main takeaways: 

 Compared to the National Sample of Grantmaking Nonprofits, SIF 2010-2012 grantees experienced significantly fewer changes between 2009

and 2014 on the extent to which the organization 1) used an open competitive process to solicit and review applications and to make selection

decisions, and 2) required applicant organizations to provide evidence of intervention effectiveness to be eligible for funding.

 For those reporting changes in grantee selection, the majority of SIF 2010-2012 grantees attributed the changes to the organization’s

participation in SIF, whereas a majority of the National Sample of Grantmaking Nonprofits attributed changes to direction from the

organization’s board/leadership. Both groups also mentioned trends within the broader grantmaking world as a major factor.
nationalservice.gov/research D-6 

II. Support for Grantees

To what extent did your 

organization do this ...? Major factors that contributed to the change (check all that apply)

Group

7 point Likert Scale: 

1- Not at all to 7- Always

Respondents 

with change 

(n)

Trends 

in the 

larger 

grantmaking 

world

Your 

organization’s 

participation 

in SIF

Specific 

requirements 

attached to the 

funding your 

organization 

receives

Your 

organization’s 

board/leadership 

directed 

organization to 

implement change Other

Respondents 

with no 

2009 2014 Change change

A. Provided 

funding to

carry out an 

evaluation or 

hire an 

external 

evaluator, as 

part of the 

grant or 

through other 

means

SIF 2010-2012 3.0 

(20) b+

3.9 

(20)

0.9 b+ 8 25.0% 75.0% 25.0% 12.5% 0.0% 12

Non-selected 

SIF Applicants 

2010-2012

3.2 (9) 4.3 

(9)

1.1 4 0.0% -- 75.0% 25.0% 25.0% 5

National 

Sample of 

Grantmaking 

Nonprofits

2.2 

(189)

2.6 

(189)

0.4 41 38.4% -- 37.9% 59.5% 15.1% 140

SIF 2014 2.5 (6) 3.3 

(6)

0.8 2 50.0% -- 50.0% 100.0% 0.0% 4

Non-selected 

SIF Applicants 

2014

2.6 (8) 2.6 

(8)

0.0 0 -- -- -- -- -- 8

B. Provided 

training or 

technical 

assistance by 

your staff 

consultants or 

other means to 

conduct 

rigorous 

evaluation

SIF 2010-2012 3.6 

(20) b+

4.3 

(20)

0.8 b+ 7 28.6% 71.4% 42.9% 0.0% 28.6% 13

Non-selected 

SIF Applicants 

2010-2012

3.7 (9) 3.9 

(9)

0.2 4 0.0% -- 50.0% 25.0% 50.0% 5

National 

Sample of 

Grantmaking 

Nonprofits

2.5 

(189)

2.9 

(192)

0.4 48 51.2% -- 35.0% 51.6% 13.9% 136

SIF 2014 1.3 (6) 2.5 

(6)

1.2 3 66.7% -- 33.3% 66.7% 33.3% 3

Non-selected 

SIF Applicants 

2014

3.3 (8) 4.0 

(8)

0.8 4 50.0% -- 0.0% 25.0% 50.0% 4
nationalservice.gov/research D-7 

To what extent did your 

organization do this ...? Major factors that contributed to the change (check all that apply)

Group

7 point Likert Scale: 

1- Not at all to 7- Always

Respondents 

with change 

(n)

Trends 

in the 

larger 

grantmaking 

world

Your 

organization’s 

participation 

in SIF

Specific 

requirements 

attached to the 

funding your 

organization 

receives

Your 

organization’s 

board/leadership 

directed 

organization to 

implement change Other

Respondents 

with no 

2009 2014 Change change

C. Provided 

training or 

technical 

assistance by 

your staff 

consultants or 

other means to 

support 

implementation 

of the program

SIF 2010-2012 4.4 

(20) b+

5.0 

(20)

0.6 7 42.9% 71.4% 28.6% 14.3% 14.3% 13

Non-selected 

SIF Applicants 

2010-2012

3.6 (9) 3.9 

(9)

0.3 3 66.7% -- 100.0% 0.0% 33.3% 6

National 

Sample of 

Grantmaking 

Nonprofits

2.9 

(193)

3.5 

(193)

0.6 64 38.2% -- 25.5% 57.2% 26.0% 123

SIF 2014 2.8 (6) 4.0 

(6)

1.2 3 33.3% -- 66.7% 66.7% 33.3% 3

Non-selected 

SIF Applicants 

2014

4.1 (9) 4.4 

(9)

0.3 2 50.0% -- 0.0% 50.0% 0.0% 7

Main takeaways: 

 Compared to the National Sample of Grantmaking Nonprofits, SIF 2010-2012 grantees experienced significantly greater changes between 2009

and 2014 in the extent to which the organization 1) provided funding to carry out an evaluation or hire an external evaluator, as part of the

grant or through other means, and 2) provided training or technical assistance by staff consultants or other means to conduct rigorous

evaluation.

 Compared to the National Sample of Grantmaking Nonprofits, SIF 2010-2012 grantees in 2009 were more likely to 1) provide training or

technical assistance by staff consultants or other means to conduct rigorous evaluation, and 2) provide training or technical assistance by staff

consultants or other means to support implementation of the program.

 For those reporting changes in support for grantees, a majority of SIF 2010-2012 grantees attributed the changes to the organization’s

participation in SIF, whereas a majority of the National Sample of Grantmaking Nonprofits attributed changes to direction from the

organization’s board/leadership. Half of the nonprofits also credited trends in the broader grantmaking world as a major factor in TA to

support evaluation.
nationalservice.gov/research D-8 

III. Evaluation

To what extent did your 

organization do this ...?

Major factors that contributed to the change (check all that apply)

Group

7 point Likert Scale: 1-

Not at all to 7- Always

Respondents 

with change 

(n)

Trends in the 

larger 

grantmaking 

world

Your 

organization’s 

participation in 

SIF

Specific 

requirements 

attached to the 

funding your 

organization 

receives

Your 

organization’s 

board/leadership 

directed 

organization to 

implement change Other

Respondent

s with no 

2009 2014 Change change

A. Conducted 

rigorous 

evaluations of 

programs 

funded by your 

organization

SIF 2010-2012 3.1 

(20) 

4.5 

(20)

1.4a+ b+ 14 28.6% 78.6% 14.3% 7.1% 7.1% 6

Non-selected 

SIF Applicants

2010-2012

3.3 

(8)

3.6 

(8)

0.4 2 100.0% -- 100.0% 0.0% 50.0% 6

National 

Sample of 

Grantmaking 

Nonprofits

3.4 

(184)

4.1 

(185)

0.7 62 34.7% -- 22.1% 69.5% 12.6% 109

SIF 2014 2.5 

(6)

3.3 

(6)

0.8 3 66.7% -- 66.7% 66.7% 0.0% 3

Non-selected 

SIF Applicants

2014

3.2 

(9)

3.4 

(9)

0.2 3 66.7% -- 33.3% 33.3% 33.3% 6

B. Used 

evaluation 

findings to 

improve 

programs 

funded by your 

organization

SIF 2010-2012 3.9 

(20)

4.9 

(20)

1.0

b+ 12 33.3% 58.3% 8.3% 33.3% 25.0% 8

Non-selected 

SIF Applicants

2010-2012

5.1 

(9)

5.4 

(9)

0.3 2 50.0% -- 100.0% 0.0% 50% 7

National 

Sample of 

Grantmaking 

Nonprofits

3.6 

(183)

4.3 

(184)

0.7 57 35.8% -- 24.7% 69.9% 12.2% 112

SIF 2014 2.3 

(6)

3.7 

(6)

1.3 3 66.7% -- 66.7% 100.0% 0.0% 3

Non-selected 

SIF Applicants

2014

4.0 

(9)

4.8 

(9)

0.8 5 40.0% -- 40.0% 60.0% 40.0% 4
nationalservice.gov/research D-9 

To what extent did your 

organization do this ...?

Major factors that contributed to the change (check all that apply)

Group

7 point Likert Scale: 1-

Not at all to 7- Always

Respondents 

with change 

(n)

Trends in the 

larger 

grantmaking 

world

Your 

organization’s 

participation in 

SIF

Specific 

requirements 

attached to the 

funding your 

organization 

receives

Your 

organization’s 

board/leadership 

directed 

organization to 

implement change Other

Respondent

s with no 

2009 2014 Change change

C. Used 

evaluation 

findings to 

demonstrate 

and 

communicate 

effectiveness of 

programs 

funded by your 

organization

SIF 2010-2012 3.7 

(20)

4.9 

(20)

1.2 b+ 12 41.7% 66.7% 16.7% 41.7% 16.7% 8

Non-selected 

SIF Applicants

2010-2012

4.9 

(9)

5.7 

(9)

0.8 3 100.0% -- 66.7% 33.3% 33.3% 6

National 

Sample of 

Grantmaking 

Nonprofits

3.8 

(181)

4.6 

(182)

0.8 63 52.2% -- 20.9% 52.2% 15.5% 104

SIF 2014 3.0 

(6)

4.3 

(6)

1.3 3 66.7% -- 66.7% 100.0% 0.0% 3

Non-selected 

SIF Applicants

2014

3.3 

(9)

4.4 

(9)

1.1 6 33.3% -- 16.7% 50.0% 50.0% 3

Main takeaways: 

 Compared to Non-selected SIF Applicants 2010-2012 and the National Sample of Grantmaking Nonprofits, SIF 2010-2012 grantees

experienced significantly greater changes between 2009 and 2014 in the extent to which the organization conducted rigorous evaluations of

programs it had funded. Compared to the National Sample of Grantmaking Nonprofits, SIF 2010-2012 grantees experienced significantly

greater changes between 2009 and 2014 in the extent to which the organization used evaluation findings to demonstrate and communicate the

effectiveness of programs it had funded.

 For those reporting changes in evaluation, the majority of SIF 2010-2012 grantees attributed the changes to the organization’s participation in

SIF, whereas a majority of the National Sample of Grantmaking Nonprofits attributed changes to direction from the organization’s

board/leadership.
nationalservice.gov/research D-10

Did/does your organization have the following?

2009 2014

Group n % (yes) n % (yes) Change

Staff position(s) or group within your organization 

dedicated to evaluation 

SIF 2010-2012 20 50.0%b+ 20 70.0% 20.0%b+

Non-selected SIF 

Applicants 2010-2012

8 50.0% 8 75.0% 25.0%

National Sample of 

Grantmaking Nonprofits

187 30.0% 189 38.9% 7.7%

SIF 2014 6 0.0% 6 33.3% 33.3%

Non-selected SIF 

Applicants 2014

9 55.6% 9 44.4% -11.2%

External evaluation partner(s) -- consultant(s) or 

organization(s) that provide your organization with 

evaluation services

SIF 2010-2012 20 70.0%b+ 20 95.0% 25.0%b+

Non-selected SIF 

Applicants 2010-2012

8 62.5% 8 87.5% 25.0%

National Sample of 

Grantmaking Nonprofits

186 34.9% 189 45.4% 9.6%

SIF 2014 6 16.7% 6 100.0% 83.3%a

Non-selected SIF 

Applicants 2014

9 66.7% 9 77.8% 11.1%

Part of the organization's budget dedicated to evaluation SIF 2010-202 20 75.0%b+ 20 95.0% 20.0%b+

Non-selected SIF 

Applicants 2010-2012

8 62.5% 8 87.5% 25.0%

National Sample of 

Grantmaking Nonprofits

187 35.5% 189 47.2% 11.7%

SIF 2014 6 0.0%a- 6 50.0% 50.0%

Non-selected SIF 

Applicants 2014

9 66.7% 9 66.7% 0.0%

Main takeaways: 

 Compared to the National Sample of Grantmaking Nonprofits, SIF 2010-2012 grantees and Non-selected SIF Applicants 2010-2012 were more

likely to have 1) staff position(s) or groups within the organization dedicated to evaluation; 2) external evaluation partner(s) or organization(s)

that provide evaluation services; and 3) dedicated part of the organization’s budget to evaluation.
nationalservice.gov/research D-11

Was your 2014 evaluation budget... 

than the evaluation budget in 2009? Group n 

Substantially 

higher

Somewhat 

higher

About the 

same

Somewhat 

lower

Substantially 

lower

Total evaluation budget in dollars

SIF 2010-2012 b+ 20 50.0% 20.0% 30.0% 0.0% 0.0%

Non-selected SIF 

Applicants 2010-2012

6 33.3% 50.0% 16.7% 0.0% 0.0%

National Sample of 

Grantmaking Nonprofits 

160 16.7% 24.6% 57.5% 0.4% 0.8%

SIF 2014 6 50.0% 0.0% 50.0% 0.0% 0.0%

Non-selected SIF 

Applicants 2014

9 44.4% 0.0% 22.2% 11.1% 22.2%

Evaluation budget as percentage of 

organization’s total budget for year

SIF 2010-2012 b+ 20 30.0% 25.0% 35.0% 5.0% 5.0%

Non-selected SIF 

Applicants 2010-2012

6 16.7% 33.3% 50.0% 0.0% 0.0%

National Sample of 

Grantmaking Nonprofits 

156 12.9% 16.2% 62.5% 6.8% 1.6%

SIF 2014 6 50.0% 0.0% 50.0% 0.0% 0.0%

Non-selected SIF 

Applicants 2014

9 11.1% 33.3% 22.2% 11.1% 22.2%

Main takeaways: 

 Compared to the National Sample of Grantmaking Nonprofits, 2014 evaluation budgets for SIF 2010-2012 grantees were more likely to be 

higher than in 2009; and the evaluation budget as a percentage of the organization’s total annual budget was also likely to be significantly 

higher. 
nationalservice.gov/research D-12

Did/does your 

organization 

have the 

following?

2009 2014

Change 

Group n Mean Median Min. Max n Mean Median Min. Max (mean)

Total annual 

evaluation 

budget (in $) –

for in-house 

evaluators or 

external 

partners

SIF 2010-2012 10 $860,590 $170,450 $0 $6,300,000 10 $1,789,800 $425,000 $0 $10,100,000 $939,344

Non-selected 

SIF Applicants 

2010-2012

1 $150,000 $150,000 $150,000 $150,000 1 $200,000 $200,000 $200,000 $200,000 $50,000

National Sample 

of Grantmaking 

Nonprofits

66 $568,817 $0 $0 $60,000,000 58 $1,115,810 $0 $0 $63,000,000 $43,195

SIF 2014 4 $12,500 $0 $0 $50,000 4 $1,000,000 $1,000,000 $0 $2,000,000 $987,500

Non-selected 

SIF Applicants 

2014

2 $50,000 $50,000 $0 $100,000 1 $0 $0 $0 $0 $-50,000

Annual 

evaluation 

budget as % of 

the total 

organization 

budget

SIF 2010-2012 13 4.1%b+ 1.5% 0.0% 10.0% 13 5.8% 4.0% 0.0% 14.0% 1.7%b+

Non-selected 

SIF Applicants 

2010-2012

0 -- -- -- -- 0 -- -- -- -- --

National Sample 

of Grantmaking 

Nonprofits

76 1.2% 0.0% 0.0% 70.0% 80 2.9% 1% 0% 65.0% 0.8%

SIF 2014 4 0.01% 0% 0.0% 0.02% 3 0.5% 0.5% 0.0% 1.0% 0.5%

Non-selected 

SIF Applicants 

2014

2 2.0% 2.0% 0.0% 4% 2 3.0% 3.0% 0.0% 6.0% 1.0%

Number of full 

time equivalent 

(FTE) staff with 

primary 

responsibility for 

evaluation

SIF 2010-2012 17 1.0 0.5 0 3 17 1.7 1 0 7 0.7

Non-selected 

SIF Applicants 

2010-2012

3 0.5 0.5 0 1 3 1.3 1 1 2 0.8

National Sample 

of Grantmaking 

Nonprofits

101 1.8 0 0 150 104 3.1 .40 0 200 0.9

SIF 2014 5 0 0 0 0 5 0.7 0 0 4 0.7

Non-selected 

SIF Applicants 

2014

5 0.3 0 0 1 5 0.8 1 0 2 0.5
nationalservice.gov/research D-13

Main takeaways: 

 Compared to Non-selected SIF Applicants 2010-2012 and the National Sample of Grantmaking Nonprofits, SIF 2010-2012 grantees reported 

greater changes in total annual evaluation budgets from 2009 to 2014. 

 Compared to the National Sample of Grantmaking Nonprofits, the changes in annual evaluation budgets, as a percentage of the total 

organization budget, did not differ from SIF 2010-2012 grantees, although the latter had a higher percentage in 2009 (e.g., higher initial status). 

 No differences were observed in the number of FTE staff with primary responsibility for evaluation among three groups. 
nationalservice.gov/research D-14

IV. Scaling Up Evidence-Based Programs: Increasing the Impact of Programs within the Community or in other Communities

To what extent did your 

organization do this ...? Major factors that contributed to the change (check all that apply)

Group

7 point Likert Scale: 1-

Not at all to 7- Always

Respondents 

with change 

(n)

Trends 

in the 

larger 

grantmaking 

world

Your 

organization’s 

participation 

in SIF

Specific 

requirements 

attached to 

the funding 

your 

organization 

receives

Your 

organization’s 

board/leadership 

directed 

organization to 

implement 

change Other

Respondents 

with no 

2009 2014 Change change

A. Undertook efforts to 

scale up existing 

program(s) – i.e., to 

expand the program(s) 

within the community 

or to other 

communities or 

populations

SIF 2010-2012 4.2 

(19)b+

5.1 

(19)

0.9 9 22.2% 66.7% 22.2% 77.8% 22.2% 10

Non-selected SIF 

Applicants 2010-

2012

4.5 (8) 5.0 (8) 0.5 3 33.3% -- 66.7% 66.7% 0.0% 5

National Sample of 

Grantmaking 

Nonprofits

3.1 

(173)

3.9 

(174)

0.8 79 16.0% -- 27.4% 66.5% 16.4% 81

SIF 2014 3.5 (6) 4.7 (6) 1.2 3 66.7% -- 0.0% 66.7% 33.3% 2

Non-selected SIF 

Applicants 2014

2.9 (9) 4.4 (9) 1.6 7 42.9% -- 14.3% 28.6% 42.9% 2

B. Selected programs 

for scale-up based on 

rigorous evaluation that 

showed them to be 

effective

SIF 2010-2012 3.5 (19)

b+

4.6 

(19)

1.1 b+ 9 33.3% 77.8% 11.1% 22.2% 11.1% 10

Non-selected SIF 

Applicants 2010-

2012

3.4 (8) 4.0 (8) 0.6 2 0.0% -- 50.0% 50.0% 0.0% 6

National Sample of 

Grantmaking 

Nonprofits

2.5 

(170)

3.2 

(172)

0.8 53 18.0% -- 32.9% 72.0% 15.7% 104

SIF 2014 2.0 (6) 3.3 (6) 1.3 2 50.0% -- 0.0% 100.0% 0.0% 3

Non-selected SIF 

Applicants 2014

2.3 (9) 3.3 (9) 1.0 5 40.0% -- 20.0% 40.0% 20.0% 4
nationalservice.gov/research D-15

Main takeaways: 

 Compared to the National Sample of Grantmaking Nonprofits, SIF 2010-2012 grantees experienced more growth in selecting programs for 

scale-up based on rigorous evaluation from 2009 to 2014. 

 Compared to the National Sample of Grantmaking Nonprofits, SIF 2010-2012 grantees in 2009 were more likely to 1) undertake efforts to scale 

up existing programs; and 2) select programs for scale-up based on rigorous evaluation that showed them to be effective. 

 For those reporting changes in scaling up programs, a majority of SIF 2010-2012 grantees attributed the changes to the organization’s 

participation in SIF, whereas a majority of the National Sample of Grantmaking Nonprofits attributed the changes to direction from the 

organization’s board/leadership. For understanding efforts to scale up existing programs, most SIF 2010-2012 grantees also cited direction 

from the organization’s board/leadership as a major factor.
nationalservice.gov/research D-16

V. Collaboration to Address Community Needs 

To what extent did your 

organization do this ...?

Major factors that contributed to the change

Group

7 point Likert Scale: 1-

Not at all to 7- Always

Respondents 

with change 

(n)

Trends in the 

larger 

grantmaking 

world

Your 

organization’s 

participation in 

SIF

Specific 

requirements 

attached to 

the funding 

your 

organization 

receives

Your 

organization’s 

board/leadership 

directed 

organization to 

implement 

change Other

Respondents 

with no 

2009 2014 Change change

A. Participated in 

funding alliance(s) 

with other nonprofit 

sector 

organizations. (For 

example, cofunding programs 

through joint 

funding; providing 

or receiving 

matching funds; or 

other 

collaboration) 

SIF 2010-2012 3.6 

(20) b+

4.3 

(20)

0.8 9 33.3% 55.6% 11.1% 33.3% b- 11.1% 11

Non-selected SIF 

Applicants 2010-

2012

3.9 

(8)

4.5 

(8)

0.6 2 100.0% -- 100.0% 100.0% 0.0% 6

National Sample 

of Grantmaking 

Nonprofits

3.1 

(172)

3.7 

(172)

0.7 66 33.1% -- 18.4% 66.7% 15.1% 92

SIF 2014 2.7 

(6)

4.2 

(6)

1.5 4 75.0% -- 0.0% 75.0% 25.0% 2

Non-selected SIF 

Applicants 2014

2.1 

(9)

3.9 

(9)

1.8 6 50.0% -- 0.0% 50.0% 0.0% 3

B. Participated in 

collaborations with 

other nonprofit 

organizations to 

share knowledge

SIF 2010-2012 4.2 

(20)

5.2 

(20)

1.0 12 66.7% 58.3% 8.3% 16.7% 8.3% 8

Non-selected SIF 

Applicants 2010-

2012

5.3 

(8)

5.9 

(8)

0.6 2 50.0% -- 50.0% 50.0% 0.0% 6

National Sample 

of Grantmaking 

Nonprofits

4.1 

(172)

4.9 

(172)

0.8 68 39.9% -- 18.8% 51.1% 19.5% 93

SIF 2014 3.5 

(6)

4.8 

(6)

1.3 5 80.0% -- 0.0% 40.0% 20.0% 1

Non-selected SIF 

Applicants 2014

4.0 

(9)

5.2 

(9)

1.2 5 60.0% -- 0.0% 20.0% 20.0% 4
nationalservice.gov/research D-17

To what extent did your 

organization do this ...?

Major factors that contributed to the change

Group

7 point Likert Scale: 1-

Not at all to 7- Always

Respondents 

with change 

(n)

Trends in the 

larger 

grantmaking 

world

Your 

organization’s 

participation in 

SIF

Specific 

requirements 

attached to 

the funding 

your 

organization 

receives

Your 

organization’s 

board/leadership 

directed 

organization to 

implement 

change Other

Respondents 

with no 

2009 2014 Change change

C. Collaborated 

with other 

organizations for 

purposes of 

advocacy – to 

advocate for or 

develop public 

support for 

programs or 

approaches to 

addressing social 

problems

SIF 2010-2012 3.5 

(20)

4.1 

(20)

0.6 8 50.0% 25.0% 12.5% 62.5% 25.0% 12

Non-selected SIF 

Applicants 2010-

2012

4.0 

(8)

4.8 

(8)

0.8 3 0.0% -- 0.0% 66.7% 0.0% 5

National Sample 

of Grantmaking 

Nonprofits

3.3 

(169)

3.9 

(170)

0.7 56 37.6% -- 14.4% 68.7% 17.5% 99

SIF 2014 2.3 

(6)

4.8 

(6)

2.5 4 25.0% -- 0.0% 75.0% 25.0% 2

Non-selected SIF 

Applicants 2014

3.9 

(9)

5.0 

(9)

1.1 5 40.0% -- 0.0% 40.0% 0.0% 4

Main takeaways: 

 No significant differences were observed in terms of changes in collaboration practices among the three groups.

 Compared to the National Sample of Grantmaking Nonprofits, SIF 2010-2012 grantees in 2009 were more likely to (e.g., had a higher initial

status) 1) participate in funding alliances with other nonprofit sector organizations, and 2) collaborate with other organizations for purposes of

advocacy.

 For those reporting changes in collaboration for funding alliances and knowledge sharing, a majority of SIF 2010-2012 grantees attributed the

changes to the organization’s participation in SIF, whereas a majority of the National Sample of Grantmaking Nonprofits attributed the

changes to direction from the organization’s board/leadership. For most SIF grantees, trends in the broader grantmaking world and direction

from the organization’s board/leadership were also cited as major factors.
nationalservice.gov/research D-18

VI. Support Received by SIF Intermediaries 

Open-ended questions only 

VII. Development of Capacity among Your Organization’s SIF Subgrantees 

To what extent did your subgrantees 

do this ...?

How much of the change in your subgrantees’ capacity 

has been a result of their participation in SIF?

Group

7 point Likert Scale: 1- Not at all to 

7- Always

n

A substantial 

amount Some A little None

Respondents 

with no 

change

At the start of 

SIF funding 2014 Change

A. Implement the interventions 

they are carrying out in their 

communities

SIF 2010-

2012

3.5 (19) 4.4 (19) 0.8 13 61.5% 30.8% 0.0% 7.7% 6

B. Design and conduct rigorous 

evaluations of their interventions

SIF 2010-

2012

2.3 (19) 3.6 (19) 1.3 17 82.4% 11.8% 5.9% 0.0% 2

C. Make use of evaluation findings 

for program improvement

SIF 2010-

2012

2.6 (19) 3.7 (19) 1.1 14 78.6% 14.3% 7.1% 0.0% 5

D. Raise matching funds for the 

intervention

SIF 2010-

2012

3.2 (19) 3.8 (19) 0.7 11 54.5% 18.2% 18.2% 9.1% 8

E. Meet federal compliance 

requirements

SIF 2010-

2012

2.7 (19) 4.2 (19) 1.4 16 81.3% 18.8% 0.0% 0.0% 3

F. Scale up the intervention (i.e., 

increase impact within community, 

or expand to other communities)

SIF 2010-

2012

2.9 (19) 3.8 (19) 0.9 13 38.5% 38.5% 15.4% 7.7% 6

G. Share knowledge and best 

practices

SIF 2010-

2012

3.1 (19) 3.9 (19) 0.8 11 63.6% 9.1% 27.3% 0.0% 8

Main takeaways: 

 SIF 2010-2012 subgrantees experienced positive growth 1) implementing the interventions they are carrying out in their communities; 2) 

designing and conducting rigorous evaluations of their interventions; 3) making use of evaluation findings for program improvement; 4) 

raising matching funds for the intervention; 5) scaling up the intervention; and 6) sharing knowledge and best practices. 

 In almost all these areas, SIF 2010-2012 grantees attributed the changes largely to participation in the SIF. 
nationalservice.gov/research D-19

VIII. Federal Funding 

2009 2014

Group n % (yes) n % (yes) Change

Received federal government funding 

(other than SIF)

SIF 2010-2012 19 36.8% 20 45.0% 5.3%

Non-selected SIF Applicants 2010-2012 8 62.5% 8 62.5% 0.0%

National Sample of Grantmaking 

Nonprofits

181 38.4% 180 41.6% 3.5%

SIF 2014 6 50.0% 6 66.7% 16.7%

Non-selected SIF Applicants 2014 9 33.3% 9 55.6% 22.3%

Group n % (yes)

Was your SIF funding the first federal government funding your organization ever received? SIF 2010-2012 20 50.0%

SIF 2014 6 16.7%

Group n Very Effective Somewhat Effective Not Effective

How effective do you think tiered-evidence initiatives are in 

achieving such outcomes as building evidence in an area?

SIF 2010-2012 19 36.8% 63.2% 0.0%

Main takeaways: 

 Only one-third of the SIF 2010-2012 grantees and the National Sample of Grantmaking Nonprofits received federal government funding in 

2009. In contrast, two-thirds of Non-selected SIF Applicants had received federal funding. In 2014, 8% more SIF grantees received federal 

funding in addition to SIF, whereas the percentage for Non-selected SIF Applicants remained the same and the percentage for the National 

Sample of Grantmaking Nonprofits increased by 4%. 

 For half of the SIF grantees, the SIF funding was the first federal government funding the organization had ever received. 

 One-third of the SIF 2010-2012 grantees thought that tiered-evidence initiatives are very effective in building evidence in an area, whereas 

two-thirds reported that they are somewhat effective. 
nationalservice.gov/research D-20

IX. Reflections on SIF experience 

Group n % (yes) % (no)

% (N/A – do not have any other 

programs in which organization 

funds grantees to carry out 

programs in communities)

Has your SIF experience affected how you conduct those other 

programs?

SIF 2010-2012 19 68.4% 21.1% 10.5%

Group n % (yes) % (no)

Did your organization’s experience in applying for SIF funding have 

an effect on your organization?

Non-selected SIF Applicants 2010-2012 6 50.0% 50.0%

SIF 2014 6 100.0% 0.0%

Non-selected SIF Applicants 2014 9 88.9% 11.1%

X. Background 

Group n % (yes) % (no)

Does your organization make grants to any U.S. nonprofit 

organizations?

Non-selected SIF Applicants 2010-2012 9 100.0% 0.0%

Non-selected SIF Applicants 2014 9 100.0% 0.0%

National Sample of Grantmaking 

Nonprofits

387 56.6% 43.4%

Grantmaking area

Non-selected 

SIF Applicants 

2010-2012 (n=9)

Non-selected 

SIF Applicants 

2014 (n=9)

National Sample 

of Grantmaking 

Nonprofits (n=213)

Community health or healthy futures – Promoting healthy lifestyles and reducing the risk factors that can lead to 

illness (for example, disease prevention, low-income healthcare/outreach) 66.7% 88.9% 67.6%

Youth development (includes early childhood development) – Preparing America’s youth for success in school, 

active citizenship, productive work, and healthy and safe lives (for example, summer or after-school 

programming, delinquency prevention, youth mentoring/tutoring/volunteers)

100.0% 100.0% 73.7%

Economic opportunity – Increasing economic opportunities for economically disadvantaged individuals (for 

example, job or career training or career readiness for adults) 77.8% 66.7% 55.9%

Funds U.S. nonprofit organizations to carry out programs or interventions, but not in any of the areas listed 33.3% 11.1% 33.8%

Funds U.S. nonprofit organizations, but not to carry out programs or interventions 0.0% 11.1% 7.0%
nationalservice.gov/research D-21

Grantmaking location

Non-selected 

SIF Applicants

2010-2012 

(n=9)

Non-selected 

SIF Applicants

2014 (n=9)

National Sample 

of Grantmaking 

Nonprofits (n=217)

Geographically-based (fund organizations within a community, state, or other specified geographic region) 88.9% 77.8% 80.2%

Issue-based (fund organizations whose programs focus on specific issue areas, but may be located in a wide 

variety of geographic locations around the U.S.) 55.6% 77.8% 41.9%

Other 0.0% 22.2% 16.1%

Level of knowledge about CNCS

Grantmaking Nonprofits

(n=218)

Non-grantmaking

Nonprofits (n=166)

None – not aware of CNCS Social Innovation Fund 44.5% 70.5%

Some knowledge – have heard of it and know something about it 41.7% 23.5%

Know quite a lot about the CNCS Social Innovation Fund 10.1% 4.2%

Know a great deal about it – very familiar with the program 3.7% 1.8%
Corporation for National and Community Service 

1201 New York Avenue, NW 

Washington, D.C. 20525

TEL: (202) 606-5000 

TTY: (202) 606-3472 

NationalService.gov