STRENGTHENING ORGANIZATIONAL CAPACITY 

AND PRACTICES FOR HIGH-PERFORMING 

NONPROFIT ORGANIZATIONS: EVIDENCE FROM 

THE NATIONAL ASSESSMENT OF THE SOCIAL 

INNOVATION FUND—A PUBLIC-PRIVATE 

PARTNERSHIP 

XIAODONG ZHANG 

JANET GRIFFITH 

JANET PERSHING 

JING SUN 

LARA MALAKOFF 

WHITNEY MARSLAND 

KURT PETERS 

ELLY FIELD 

ICF 

key words: organizational capacity, Social Innovation Fund, 

national assessment, public private partnership, tiered evidence 

initiative 

ABSTRACT 

The Social Innovation Fund (SIF) is a federal initiative that leverages 

funds through public-private collaborations to support and build the 

evidence base for community programs. The SIF awards grants to 

grantmaking nonprofit organizations that, in turn, select, fund, and 

support local nonprofit organizations to implement community-based 

programs in one of three core areas: youth development, economic 

opportunity, and healthy futures. The federal funds are matched both by 

grantees and by subgrantees, leveraging $3 of funding for each $1 of 

federal funds. The fundamental purpose of the SIF is to strengthen the 

capacity and practices of nonprofit organizations to select and support 

grantees that operate evidence-based programs, implement rigorous 

evaluations, scale up, and collaborate with other organizations. 

Employing a quasi-experimental design, the national 

assessment found improved organizational capacity among SIF 
 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

PAQ FALL 2017 425 

grantees. Between 2009 and 2014, SIF grantees funded in the 2010-

2012 grant cycles reported improvement in 13 out of 14 measures of 

organizational capacity. Compared to applicants for SIF who were not 

selected and to a national sample of grantmaking nonprofits, and 

measured by effect size estimates (Cohen, 1988), SIF had a large 

impact on the evaluation capacity of SIF grantees, a medium impact on 

their support for subgrantees and for scaling, and a small impact on 

subgrantee selection and collaboration. 

INTRODUCTION 

In the past decade, interest has intensified in the practice 

of developing organizational capacity to achieve programmatic 

goals. According to Harsh (2012), capacity building increases 

the ability of an organization to respond effectively to change by 

providing its staff with the skills and tools needed to identify and 

resolve problems over time. Although the nonprofit sector is 

widely viewed as a catalyst for change and a mechanism for 

serving societal needs, it faces substantial capacity challenges. 

De Vita and her colleagues (2001) observed that many small, 

community-based organizations are structurally fragile and many 

larger nonprofits are stretched to their limits. As demands for 

community-based services grow, with the identification of new 

needs and emergence of new service paradigms, nonprofits are 

continually challenged to strengthen their organizational 

capacity. On the other hand, there is a dearth of evidence-based 

information about what works and what does not work in 

building the organizational capacity of nonprofits. This is 

primarily because the philanthropic sector historically has paid 

little attention to capacity building, which was rarely supported 

by funders and of secondary importance to nonprofit managers 

whose priority is to deliver programs and services to people in 

need (Venture Philanthropy Partners, 2001). Nonprofits were 

further stretched during the 2008-12 recession. According to IRS 

data, charities with $50,000 or more fared somewhat worse 

during the recession-driven years of 2008–12 than during the 

previous four years (2004–08), although the difference was more 

modest than some had expected (Brown et.al., 2013). This 

situation is beginning to evolve, however, as more funders 
 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

426 PAQ FALL 2017 

dedicate attention and financial support to developing 

organizational capacity. 

This article contributes to the literature in two important 

ways. First, it provides a case study to test and advance theories 

about developing organization capacity that can be applied to a 

broader context, and presents empirical evidence concerning the 

ways that a high-profile federal program has addressed the twin 

goals of promoting evidence-based innovation while 

strengthening the organizational capacity of high-performing 

nonprofits. Second, the study features several methodological 

elements that enhance the evaluation’s rigor. The evaluation 

employs two comparison groups, measures organizational 

capacity in terms of organizational behaviors, and supplements 

self-report with documentary evidence. In the rest of this article, 

we first present an overview of theoretical perspectives and 

empirical evidence that informed the national assessment. Next, 

we describe the SIF program and the national assessment. 

Finally, we present the findings and conclusions. 

THEORETICAL PERSPECTIVES AND
 

EMPIRICAL EVIDENCE
 

Organizational capacity refers to the wide range of 

capabilities, knowledge, and resources that an organization can 

employ to solve problems and achieve goals (Grantmakers for 

Effective Organizations, n.d.). Harsh (2012) presents a 

“multiple-dimension” approach to capacity building. Her four 

essential dimensions include: type of capacity (human, 

structural, organizational, or material); stage of capacity 

(exploration, emerging, full, or sustainability); level of capacity 

(information, skills, structures, or processes); and desired 

outcome of capacity. She argues that identifying the 

organization’s starting point and capacity goals within these four 

dimensions is essential to designing a successful change 

initiative. She also speaks to the importance of monitoring 

implementation continuously to make mid-course corrections. 

Wandersman and colleagues (2012) provide a logic 

model and roadmap for addressing gaps and improving 

organizational capacity. Much like Harsh’s framework, the first 
 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

PAQ FALL 2017 427 

step involves identifying existing capacity and the desired 

outcomes of capacity building initiatives. This existing capacity 

is then built upon using an iterative process that applies four 

support components—capacity tools, training, technical 

assistance (TA), and quality assurance/quality improvement— 

until goals are achieved. They argue for the integration of all 

four support components in order to overcome the limitations 

and challenges of each individual component. These four 

components also rely upon relationships, both within 

organizations and between organizations and their external 

capacity building providers. This iterative process offers a 

flexible structure and approach that ultimately guides 

organizations to accomplish capacity building outcomes and 

goals. 

Nonprofit organizations face substantial challenges in 

developing organizational capacity. Traditionally, foundations 

make grants based on their assessment of a program’s potential 

efficacy. Although that approach creates an incentive for 

nonprofits to devise innovative programs, it does not necessarily 

encourage their attention to assessing the strengths, goals, and 

needs of the organization itself. Thus, they may lack the 

organizational resources to carry out the programs that they have 

designed and tested (Letts et al., 1997). According to a report 

from the Center for Effective Philanthropy (Buteau & Buchanan, 

2013), providing TA to nurture organizational capacity can often 

mean the difference between making a grant and making an 

impact. Grantees that receive comprehensive assistance from 

their foundation funders report a substantially greater impact on 

their organizations than grantees that receive no assistance. 

Providing this kind of TA requires a substantial commitment of 

time, money, and resources, however. Grantees funded by the 

Social Innovation Fund (SIF) are confronted with additional 

challenges in what is, for many, a new role as the recipient of a 

federal grant. By asking nonprofits to execute a range of 

essential tasks quickly—including fundraising, program 

implementation, and evaluation—the SIF has highlighted core 

strengths as well as areas in which nonprofits would need to 

improve to meet grant requirements (Grantmakers for Effective 

Organizations, 2013). 
 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

428 PAQ FALL 2017 

Empirical research on the effectiveness of capacity 

building efforts varies in scope and methodological rigor. For 

example, using surveys of 125 grassroots organizations that 

participated in capacity building events over five years, Sobeck 

(2008) found a positive relationship between investment in 

capacity building and organizational practices in planning, using 

evaluation strategies, and having grant writing knowledge and 

awareness of opportunities. Many evaluations of capacity 

building tend to rely on self-reports, with limited examples of 

more rigorous designs involving counterfactuals. One such 

example is the impact evaluation of the Compassion Capital 

Fund (CCF) demonstration program. CCF was a federal grant 

that enabled intermediaries to provide support to nonprofits 

designed to improve their organizational capacity, strengthen 

long-term sustainability, and enhance their ability to work 

effectively on target issues. (The level of investment was much 

smaller than the SIF.) The evaluation employed a randomized 

controlled trial (RCT) design and compared 217 treatment 

nonprofits with 168 control nonprofits served by 10 intermediary 

grantees in the 2006 grant cycle. The survey methodology 

featured baseline and 15-month follow-up web-based surveys. 

The evaluation found that the nonprofits that received capacity 

building support experienced significantly higher growth in 

several areas, including organizational development, leadership 

development, and community engagement (Minzner et. al., 

2014). 

Evidence on organizational impact began to emerge 

from some of the early cohorts of the SIF grantees and 

subgrantees. Stakeholder interviews by the Mile High United 

Way suggested that benefits on subgrantee organizations ranged 

from tangible increases in staffing capacity and related structural 

improvements, to improvement in ability to implement 

programming with fidelity across sites, and the new capacity to 

track and monitor programmatic efforts and related child 

outcomes (University of Denver, nd). Similarly, case studies of 

two United Way SIF grantees described how the grants have 

changed the ways organizations approach their work, in 

particular enabling them to adopt or enhance evidence-based 

practices for finding, selecting, and supporting high-performing 
 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

PAQ FALL 2017 429 

nonprofit organizations (Stiefvater & Education Northwest,

2015). On the other hand, participation in the SIF has not been

without challenges. The financial requirements of the program

including match requirements, background checks and related

fiscal documentation burden, as well as the need to keep

programming consistent for research studies can be

overwhelming at times (University of Denver, nd). 

The literature suggests that organizational capacity is

key to an organization’s ability to solve problems and achieve

goals. Nonprofit organizations, which have traditionally focused

on delivering programs, face challenges in developing such

capacity. However, recent trends suggest that more funders are

starting to pay attention to developing organizational capacity

and some of these efforts have begun to show promise in

increasing organizational practices and abilities to design and

implement programs. This article adds to the body of literature

by providing empirical evidence from a nationwide grant

program. 

SOCIAL INNOVATION FUND 

The SIF is a federal program designed to identify and

implement innovative and effective evidence-based solutions to

improve the lives of people in low-income communities

throughout the United States. Authorized by the 2009 Edward

M. Kennedy Serve America Act, it is administered by the

Corporation for National and Community Service (CNCS). The

SIF is one of the federal government’s “tiered-evidence

initiatives,” established to promote social innovation based on

evidence. 

Since its authorization, the SIF has received an annual

Congressional appropriation of approximately $50-70 million.

The program leverages federal funds through public-private

collaborations by awarding grants to larger nonprofit

grantmaking organizations, which serve as intermediaries for the

program. These SIF grantees match the federal funds and in turn

fund local nonprofits (subgrantees), which also provide matching

funds to implement community-based programs in one of three
 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

§

§

§

§

§

§

430 PAQ FALL 2017 

priority areas: youth development, economic opportunity, and 

healthy futures. The SIF’s design integrates six key elements: 

Reliance for mobilization of the network on intermediary 

grantmaking institutions, which find, select, monitor, 

support, evaluate, and report on the nonprofit organizations 

that actually implement the community-based interventions. 

Requirement of a minimum of preliminary evidence of 

effectiveness for all funded programs/interventions, as 

tiered-evidence initiatives. 

Rigorous evaluation that will build upon the 

program/intervention’s level of evidence. 

Scaling evidence-based programs to enable intermediaries to 

increase their impact within communities, locally or across 

the country, bringing them face-to-face with the widely 

shared nonprofit sector challenge of achieving successful 

and efficient scale up. 

A unique funding model that leverages public-private 

partnerships to support nonprofits and effect large-scale 

community impact in ways that neither a traditional federal 

grant nor a philanthropic investment could achieve on its 

own. 

A commitment to improving the effectiveness of nonprofits, 

funders, and federal agencies by capturing and learning 

about best practices and promoting approaches that generate 

the greatest impact for individuals and communities. 

By late 2014, CNCS had completed four rounds of SIF 

grantmaking (in 2010, 2011, 2012, and 2014) and awarded 27 

grants to 26 grantees (one organization received two grants). (No 

grantees were funded in 2013.) These grantees include both large 

nonprofits and foundations such as the Edna McConnell Clark 

Foundation, Jobs for the Future, and Venture Philanthropy 

Partners, and regional intermediaries like United Way Southeast 

Michigan and Mile High United Way. These grantees have in 

turn selected as subgrantees more than 200 nonprofit 

organizations through open, competitive processes. 

Another fundamental purpose of the SIF program is to 

transform the way that nonprofit organizations conduct their 

work by requiring both grantees and subgrantees to build 

organizational capacity in targeted areas of practice. The SIF 
 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

PAQ FALL 2017 431 

selects its grantees through a competitive process that requires 

them to demonstrate both preliminary evidence of their 

programmatic effectiveness and extensive capacity to support 

their subgrantees. To increase support for subgrantees and their 

programs, CNCS has provided extensive TA and other support to 

SIF grantees to foster capacity around evaluation, federal grants 

management and compliance, and communication and outreach. 

CNCS deploys capacity building through its SIF program 

officers, the CNCS Office of Research and Evaluation and its 

evaluation TA contractor, and peer learning opportunities for 

grantees such as convenings, conference calls, and on-line 

resource sharing. 

Lastly, the SIF demands accountability and enforces 

high expectations among grantees by emphasizing the creation of 

an evidence base and the replication and expansion of what 

works. The application of evidence permeates all phases of the 

work from subgrantee selection and program improvement to 

demonstration of results and scale-up. 

THE SIF NATIONAL ASSESSMENT 

In 2013, CNCS funded an independent national 

assessment of the SIF program. Its main objective was to 

examine the impact of the SIF on the organizational capacity and 

practices of its grantees—their entire organizations, not only 

those sections that manage the SIF grant. CNCS’s objectives 

were to increase effective program evaluation and management 

among grantees and further the national goal of learning from the 

federal government’s tiered-evidence initiatives. Between 2014 

and 2015, the national assessment team measured 14 key aspects 

of organizational capacity and practice, comparing the SIF 

grantees with other grantmaking organizations. 

This section describes the evaluation questions that 

guided this research as well as the design, sample, instruments, 

data collection strategy, and analytic approach. 

Evaluation Questions 

To inform CNCS about the SIF program’s 

implementation and impact on the organizational capacity and 
 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

§

§

§

432 PAQ FALL 2017 

practice of grantees, the SIF national assessment addressed four 

key research questions: 

1.	 Does participation in the SIF increase the capacity of 

organizations (for example, skills, attitudes, and 

behaviors) in the use of evidence-based grantmaking 

strategies? 

2.	 Does participation in the SIF increase the willingness 

and ability of organizations to build the evidence base 

for high-impact models? 

3.	 Does participation in the SIF increase organizational 

knowledge of how to scale effective program models? 

4.	 Does participation in the SIF facilitate collaborative 

approaches to addressing local community needs? 

Evaluation Design and Sample 

The national assessment employed a quasi-experimental 

design (QED) to evaluate the SIF’s impact on organizational 

change by comparing the grantees with themselves over time as 

well as creating two counterfactual groups that enabled 

evaluators to compare the experiences of SIF grantees with the 

experiences to be expected in the SIF’s absence. The analysis 

compared SIF grantees with: 

Themselves over time. The evaluation assessed capacity 

building from a pre-SIF baseline (2009) to the time of data 

collection in 2014. This comparison allowed SIF grantees to 

serve as their own controls. This comparison did not rule out 

the possibility that any observed change resulted primarily 

from a general trend or high motivation among grantees. 

“Non-selected SIF applicants.” The evaluation compared 

SIF grantees with applicants that had submitted satisfactory 

SIF applications but did not receive a SIF grant, providing an 

important counterfactual for the SIF grantee experience. 

Comparing the performance of non-selected SIF applicants 

with that of SIF grantees controlled for motivation and 

reduced the potential effects of selection bias. 

A nationally representative sample of grantmaking 

nonprofits. The evaluation compared SIF grantees with a 

sample of U.S. grantmaking nonprofits similar to the SIF 

grantees in terms of revenue and grantmaking size. Including 
 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

PAQ FALL 2017 433 

this comparison group controlled for changes in the broader 

philanthropic/ nonprofit world, such as increasing 

engagement in rigorous evaluation, funding of evidencebased programs, and scaling up of programs with 

demonstrated effectiveness. 

Although the SIF had funded four cohorts at the time of 

data collection, this article examines findings from the first three 

cohorts of 20 grantees (2010, 2011, and 2012) and the 

comparison groups. Because the six 2014 grantees were in their 

first year of SIF funding at the time of data collection, it was not 

realistic to anticipate changes in capacity and practice 

attributable to the SIF. 

Differences in sample sizes for the SIF grantees and the 

two counterfactual groups resulted in variations in statistical 

power for the findings of these comparisons. Although the SIF 

grantee group included only 20 organizations (excluding those 

funded in the 2014 cohort), it represented the entire population 

and a response rate of 100%. With 19 cases in the analytic 

sample for the non-selected SIF applicants, tests for statistically 

significant differences between these two groups at ! = 0.05

achieve 0.8 power for effect sizes of ! = 0.68, considered 

medium-large effects (Cohen, 1988). With 262 cases in the 

analytic sample for the grantmaking nonprofits, tests for 

statistically significant differences between this group and SIF 

grantees at ! = 0.05 achieve 0.8 power for effect sizes of 

! = 0.17, relatively small effects. 

Instruments, Data Collection, and Analysis 

To portray the organizational changes experienced by 

the SIF grantees, the evaluation drew on multiple sources of 

data, including (a) surveys of SIF grantees and the two 

comparison groups, (b) interviews with SIF grantees and other 

stakeholders, and (c) administrative data, including IRS form 

990 (for financial data). 

Instrument development. The national assessment 

examined five domains targeted by the SIF program: use of 

evidence-based grantmaking strategies, implementation and use 

of rigorous evaluations, scaling up of programs based on 

evidence of effectiveness, knowledge sharing, and collaboration 
 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

434 PAQ FALL 2017 

to address community needs. To measure capacity, the study 

drew on respondent reports of changes in organizational 

behavior, because behavior is ultimately the measure that matters 

in considering organizational change. Finally, to allow for 

comparison over a common time frame, the survey asked SIF 

grantees and the comparison organizations about organizational 

behaviors in 2009 (before SIF implementation) and in 2014 (at 

or toward the end of SIF implementation). 

The evaluators applied the retrospective pretest approach 

because the CNCS funded the national assessment several years 

after the grantees began work. Research has shown that 

retrospective pretest is not only simple and convenient but also 

effective in addressing the response shift bias commonly present 

in traditional pretest-posttest methodology, as respondents tend 

to overestimate pretest outcomes (Pratt et al., 2000; Hill & Betz, 

2005; Cantrell, 2010). Cognitive interviews and pretests 

determined that respondents understood and could report on 

changes in organizational behavior, distinguish degrees of 

organizational behavior using the survey’s 7-point scale, and 

report on factors that contributed to the changes. The survey 

instruments featured both closed-ended items (most using Likerttype scales) and open-ended items. Separate survey instruments 

for each group included all core questions, plus additional 

questions targeted to that group’s distinctive experience. 

Data collection. The evaluators sent survey participants 

a respondent-specific email link to an online survey tool. 

Respondents had the option of completing the entire 

questionnaire in one sitting or over several sessions. They were 

invited to involve others in their organization in answering any 

questions. Strategies to increase response rates included: tapping 

multiple sources to compile accurate and up-to-date contact 

information; establishing a user-friendly online survey platform; 

ensuring respondent confidentiality; allowing ample time for 

response; and sending multiple reminders and follow-ups. Once 

a SIF grantee submitted its survey, the evaluators sent a request 

for a 15-30 minute follow-up interview that gave SIF grantees 

the opportunity to discuss evidence and supply documentation 

that would support their observations in the survey about 

organizational change. 
 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

PAQ FALL 2017 435 

The evaluators computed response rates for all three 

target populations (SIF grantees, non-selected applicants, and the 

national sample of grantmaking nonprofits), abiding by the 

American Association for Public Opinion Research (n.d.) 

standard definitions for web surveys of named persons. The 

surveys achieved 100% response from SIF grantees, but lower 

response rates from non-selected SIF applicants (35%) and the 

national sample of grantmaking nonprofits (12%). However, a 

non-response analysis suggested little non-response bias. 

Weighting for the responses from these two groups improved the 

precision of the estimates. 

Data analysis. Data analysis first documented and 

analyzed change in grantmaking strategies reported by SIF 

grantees. Because the estimates from the SIF grantees were 

population values, no statistical significance test was required. 

Comparisons between SIF grantees and the two counterfactual 

groups provided information about whether to attribute observed 

change to SIF participation. The evaluators used different 

analytic approaches to account for differences in sample size 

between the two comparison groups. To account for small 

sample sizes, we used the non-parametric Mann-Whitney test to 

assess differences between the SIF grantees and non-selected SIF 

applicants. To compare the differences between SIF grantees and 

the national sample of grantmaking nonprofits, we calculated the 

confidence interval at 95% around the values for the nonprofit 

sample. 

In addition to the statistical significance test, we 

employed effect size estimates to gauge the practical significance 

of differences in magnitude. Statistical significance, which 

measures the likelihood that a result could occur by chance, is 

highly dependent on sample sizes. On the other hand, effect size, 

which measures the size of difference between groups, is less 

dependent on sample sizes. A small effect can be statistically 

significant if the sample size is large, while a large effect may 

not achieve statistical significance if the sample size is small. 

Cohen (1988) defines effect sizes as "small, d = 0.2," "medium, 

d = 0.5," and "large, d = 0.8." 

Finally, analyses of open-ended questions from the 

surveys, as well as interview data and documentary evidence, 
 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

436 PAQ FALL 2017 

addressed “how” and “why” questions about the factors that 

contributed to the observed change, and provided insights that 

supplemented the survey data. Analysis of qualitative data 

highlighted themes and emergent patterns and used illustrative 

quotes, examples, or vignettes. 

Limitations and Solutions 

The appraisal of organizational change conducted for the 

SIF national assessment had both strengths and limitations. One 

strength was the inclusion of two comparison groups, which 

enabled the evaluation to address questions about SIF grantees 

relative to both similar, highly committed organizations (nonselected SIF applicants) and to broader trends in the field of 

nonprofit grantmaking (the national sample). In addition, 

combining multiple data collection strategies—survey, 

interview, and document review methodologies—allowed the 

collection of specific, objective information about change as well 

as insights into factors that contributed to reported change. 

The limitations of this study include the small number of 

SIF grantees and non-selected SIF applicants in the sample, 

which renders survey data analyses vulnerable to such factors as 

the presence of outliers. Several strategies were used to mitigate 

this issue, including effect size calculations and non-parametric 

assumptions in statistical significance testing. Other limitations 

are the reliance on self-report and retrospective reporting. The 

evaluation addressed these limitations by obtaining evidence 

from SIF grantees to document changes reported in the survey. 

Finally, our discussions with CNCS staff and grantees indicated 

that the experience of the first SIF cohort differed from that of 

subsequent cohorts. They perceived, for example, that the first 

cohort grantees experienced more programmatic “growing 

pains” than subsequent cohorts. These observations may suggest 

that the potential to generalize from this study to the broader 

population of future SIF grantees is limited. This study reports 

evidence about the experience of a specific group of 

organizations, and the future mix of organizations and therefore 

the experiences of later SIF cohorts may differ. 
 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

PAQ FALL 2017 437 

FINDINGS 

This section first describes the organizational 

characteristics of the respondents. Next, it reviews findings 

related to impact on organizational capacity and practices 

through comparison of SIF grantees with the comparison groups. 

Organizational Characteristics 

Data analysis began with a comparison between SIF 

grantees and the two comparison groups on three financial 

indicators (obtained from 2013 federal tax submissions): gross 

revenue, total value of government grants received, and total 

value of grants made to U.S. organizations. The results suggest 

that SIF grantees were similar to both the comparison groups in 

terms of gross revenue and government grants received. 

Although similar to non-selected SIF applicants in terms of the 

value of grants made to U.S. organizations, SIF grantees gave 

significantly more to U.S. organizations than did the national 

sample of grantmaking nonprofits (Table 1). 

The survey asked about prior experience with federal 

funding in part because some SIF grantees, especially those that 

had not previously received federal funds, found it challenging to 

comply with federal funding requirements. These data provide 

context for analysis of the SIF experience and capacity 

development, and suggest considerations for subsequent federal 

funding. The data show that two-thirds of SIF grantees receiving 

federal funding in 2009 and 2014 had no prior federal funding. 

Among the national sample of grantmaking nonprofits, about 

two-thirds had neither current nor previous federal funding 

(Table 2). 
 

 

 

 

 

 

 

 

 

 

438 PAQ FALL 2017 

Table 1 

Financial indicators by respondent group 

National 

Financial 

indicators 

(2013) SIF 

Non-selected 

SIF 

applicants 

sample of 

grantmaking 

nonprofits 

Gross revenue N 15 9 147 

Mean $76,573,121 $74,927,426 $98,431,599 

Median $44,479,384 $22,234,541 $ 6,833,348 

Total value of N 14 6 121 

government 

grants 

received 

Mean $5,307,259 $980,788 $3,741,735 

Median $1,411,197 $618,451 $ 660,614 

Total value of 

grants made to 

U.S. 

organizations 

N 15 9 218 

Mean $21,573,525 

b+ $33,890,743 $12,483,991 

Median $11,462,022 $ 8,278,806 $ 4,111,065 

Statistical significance (p<.05) is noted as: 

a: significant difference between SIF and non-selected SIF applicants 

b: significant difference between SIF and the national sample of grantmaking 

nonprofits 

+: favoring SIF 

-: favoring the comparison group 

Sample sizes for Table 1 are smaller than those reported in other exhibits due to missing 

data. 

Table 2 

Federal funding experience of SIF grantees and the 

comparison groups 

2009 2014 

Change 

Group n % 

(yes) n % (yes) 

SIF 19 36.8% 20 45.0% 5.3% 

Non-selected SIF 8 62.5% 8 62.5% 0.0% 

applicants 

National sample of 

grantmaking 

nonprofits 

181 38.4% 180 41.6% 3.5% 

Impact on Organizational Capacity 

This section examines five aspects of organizational 

capacity targeted by the SIF program: selection of subgrantees, 

support for subgrantees, evaluation, scaling up of evidence-based 
 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

PAQ FALL 2017 439 

programs, and collaboration. The discussion of each aspect is 

organized into three sections: 

1.	 SIF grantee organizational changes (comparison between 

the organizational capacity of SIF grantees in 2009 and 

2014) 

2.	 Changes for SIF grantees vs. comparison groups 

(comparisons between the organizational capacity of SIF 

grantees and two comparison groups in 2009 and 2014) 

3.	 Observations about organizational change (responses 

concerning major factors affecting change and summary 

of open-ended question responses) 

Selection of subgrantees. The survey asked SIF grantees 

about subgrantee selection in both 2009 and 2014. If the 

responses for the two years differed, the survey asked about 

changes that the SIF grantees experienced, as intermediary 

organizations, during this period. 

SIF grantee organizational changes. About one-third 

of SIF grantees reported changes in subgrantee selection 

practices. Between 2009 and 2014, SIF grantees revised their 

funding eligibility guidelines to require applicant organizations 

to provide evidence of intervention effectiveness (Item B: 5.0 to 

5.3) and submit a plan for rigorous evaluation of the intervention 

(Item C: 4.8 to 5.1). However, the practice of operating an open 

and competitive process to solicit, review, and approve or reject 

applications changed in a negative direction between 2009 and 

2014 (Item A: 4.6 to 4.4). (The decrease primarily reflects the 

account of one SIF grantee of the six that responded to this 

question, of using open competition in 2009 but never in 2014 

(Table 3). 
 

To what extent did 

your organization do 

this ...? 

Group 2009 2014 Change 

Effect Respondents 

size with change 

A. Used an open, 

competitive process to 

solicit and review 

applications and to 

make selection 

decisions 

SIF 4.6 4.4 

(20) (19) 

-0.1b- -0.1 6 

Non-selected 

SIF applicants 

5.0 (9) 5.3 (9) 0.3 -0.3 4 

National 

sample of 

grantmaking 

nonprofits 

4.8 5.1 

(207) (207) 

0.3 -0.7 53 

B. Required applicant 

organizations to 

provide evidence of 

intervention 

effectiveness to be 

eligible for funding 

(includes pre- and 

posttest outcome data 

or other evidence based 

on evaluation studies) 

SIF 3.5 4.0 

(20) (19) 

0.6 0.2 8 

Non-selected 

SIF applicants 

4.8 (9) 5.1 (9) 0.3 0.4 3 

National 

sample of 

grantmaking 

nonprofits 

3.7 4.5 

(202) (206) 

0.8 -0.7 84 

C. Required applicants 

to submit a plan for 

rigorous evaluation of 

intervention to be 

eligible for funding 

(that is, quasi -

experimental designs 

with a comparison 

group, experimental 

designs, or other 

similarly rigorous 

designs) 

SIF 2.0 2.3 

(20) (19) 

0.4 0.2 5 

Non-selected 

SIF applicants 

2.9 (9) 3.1 (9) 0.2 0.3 2 

National 

sample of 

grantmaking 

nonprofits 

2.4 2.9 

(201) (200) 

0.5 -0.2 46 

 

 

 

 

 

 

 

 

 

 

 

440 PAQ FALL 2017 

Table 3 

Changes in how SIF grantees and comparison groups 

approach subgrantee selection 

Statistical significance (p<.05) is noted as: 

a: significant difference between SIF and non-selected SIF applicants 

b: significant difference between SIF and the national sample of grantmaking 

nonprofits 

+: favoring SIF 

-: favoring the comparison group 

Sample sizes for Table 3 are smaller than those reported in other exhibits due to missing 

data. 

Changes for SIF grantees vs. comparison groups. 

Compared to the national sample of grantmaking nonprofits, SIF 

grantees experienced significantly less change between 2009 and 

2014 in the extent to which they used an open and competitive 

process to solicit, review, and approve or reject applications, 
 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

PAQ FALL 2017 441 

again largely due to negative change reported by one grantee. 

The results also show medium to small positive differences in 

favor of the national sample of grantmaking nonprofits in all 

three areas. These findings were somewhat inconsistent with the 

comparison to non-selected SIF applicants. The same pattern 

held true for using an open and competitive process, but small 

differences were found in favor of SIF grantees, in effect sizes, 

with regard to requiring applicants to provide evidence of 

effectiveness and submit a rigorous evaluation plan to be eligible 

for funding (Table 3). 

Observations about change. A majority of SIF grantees 

attributed changes in their subgrantee selection process to strict 

selection requirements imposed by their participation in the SIF, 

although they also mentioned other factors that contributed, such 

as trends in the field and direction from the organization’s 

leadership. The other two groups were not asked about SIF 

participation effects, but were asked about other factors. A 

majority of the national sample of grantmaking nonprofits 

attributed changes to guidance from their board or other 

leadership. Both comparison groups also referenced trends in the 

broader grantmaking world as important factors that contributed 

to change. (The tables do not include statistics for these 

findings.) One respondent noted, “Donor tolerance for funding 

that does not have measurable results has changed.” Some added 

that their organizations had always required evidence of 

effectiveness in grant applications, but in some cases, it is 

challenging to identify earlier evidence. For example, one 

nonprofit respondent observed, “We fund arts experiences; [it is] 

difficult to quantify the impact of arts education experiences in 

one year.” 

In open-ended comments about the impact of their 

participation in the SIF on their approach to selecting 

subgrantees, SIF grantees reiterated the themes of implementing 

application-based competitions, emphasizing evidence and 

evaluation, and increasing use of tools. Below are comments that 

offer context. 

The SIF RFP process was the first ever for this 

organization. We found this process to be helpful and 

plan to use it in the future. 
 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

442 PAQ FALL 2017 

For another program, we now use a competitive 

selection process for all of our new, non-SIF sites. We 

found the selection process to be a valuable method of 

pre-planning and gauging interest and dedication to the 

model. 

Support for subgrantees. Grantmaking organizations 

vary in the extent to which they support subgrantees to develop 

their capacity to carry out programs. The survey asked 

respondents about both the financial and non-financial 

assistance they provided to help subgrantees implement and 

evaluate their programs. 

SIF grantee organizational changes. About one-third of 

SIF grantees reported changes in the ways that they support their 

subgrantees. Typically, SIF grantees expanded their support to 

subgrantees by more frequently funding an evaluation or hiring 

an external evaluator (Item A: 3.0 to 3.9), providing training or 

TA to conduct rigorous evaluation (Item B: 3.6 to 4.3), or 

supporting program implementation (Item C: 4.4 to 5.0) (Table 

4). 
 

 

 

 

 

 

 

 

 

 

 

 

 

PAQ FALL 2017 443 

Table 4 

Changes in support for subgrantees from SIF grantees and 

comparison groups 

To what extent did 

your organization 

do this ...? 

Group 2009 2014 Change 

Effect Respondents 

size with change 

A. Provided funding to SIF 

carry out an evaluation 

or hire an external Non-selected 

evaluator, as part of SIF 

the grant or through applicants 

other means National 

sample of 

grantmaking 

nonprofits 

3.0 3.9 

(20) b+ (20) 

0.9 b+ 0.5 8 

3.2 (9) 4.3 

(9) 

1.1 -0.1 4 

2.2 2.6 

(189) (189) 

0.4 1.3 41 

B. Provided training or SIF 

TA by your staff 

consultants or other Non-selected 

means to conduct SIF 

rigorous evaluation applicants 

National 

sample of 

grantmaking 

nonprofits 

3.6 4.3 

(20) b+ (20) 

0.8 b+ 0.4 7 

3.7 (9) 3.9 

(9) 

0.2 0.5 4 

2.5 2.9 

(189) (192) 

0.4 0.9 48 

C. Provided training or SIF 

TA by your staff 

consultants or other Non-selected 

means to support SIF 

implementation of the applicants 

program National 

sample of 

grantmaking 

nonprofits 

4.4 5.0 

(20) b+ (20) 

0.6 0.3 7 

3.6 (9) 3.9 

(9) 

0.3 0.3 3 

2.9 3.5 

(193) (193) 

0.6 0.1 64 

1 Statistical significance (p<.05) is noted as: 

a: significant difference between SIF and non-selected SIF applicants 

b: significant difference between SIF and the national sample of grantmaking 

nonprofits 

+: favoring SIF 

-: favoring the comparison group 

Changes for SIF grantees vs. comparison groups. 

Compared to the national sample of grantmaking nonprofits, SIF 

grantees experienced significantly greater changes between 2009 

and 2014 in the extent to which the organization: (a) provided 

funding to carry out an evaluation or hire an external evaluator 

and (b) provided training or TA to support rigorous evaluation. 

The effect sizes in the latter two areas were large, but negligible 

for TA to support program implementation. Although the change 
 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

444 PAQ FALL 2017 

patterns for SIF grantees were not statistically significant, we 

found small differences in favor of SIF grantees, compared to 

non-selected SIF applicants, in the magnitude of TA provided to 

support evaluation and program implementation (Table 4). 

Observations about change. Among organizations that 

reported changes in their TA support to grantees for program 

implementation, the majority of SIF grantees attributed the 

changes to their participation in the SIF, but the majority of the 

national sample of grantmaking nonprofits attributed such 

changes to direction from their board or other leadership and to 

trends in the broader grantmaking world. Another topic that SIF 

grantees identified in open-ended responses was a shift in their 

organizational business model toward a greater emphasis on 

grantee support, either for evaluation or TA or both. Some SIF 

grantees specifically credited the SIF with changing how they 

provide TA to subgrantees—not only within their SIF programs, 

but across their broader portfolios. 

SIF offered both the opportunity and the mandate to 

increase the level of TA for our subgrantees around 

evaluation and program administration. In 2009 we 

offered small TA grants around program implementation, 

and in 2014 we were able to offer larger one-on-one TA 

awards to our SIF subgrantees, as well as group TA and 

training to support data collection and other evaluation 

activities. 

SIF's requirements around TA and shared learning have 

impacted our impact work more generally, not just the 

body of work that is funded by SIF. More importantly, 

SIF has been a lever that we've used. By demonstrating 

the government's commitment to this kind of support, we 

can make a stronger case for providing it. 

Use of data, evidence, and evaluation. The survey 

asked about ways that SIF grantees used evaluation in 2009 and 

2014, their evaluation resources, and infrastructure. In addition, 

we asked respondents whether their organizations had changed 

their use of data and evidence, and what factors contributed to 

those changes. 

SIF grantee organization changes. SIF grantees 

identified the greatest changes in their use of evaluation, with 
 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

PAQ FALL 2017 445 

about two-thirds reporting increases in their capacity to conduct 

rigorous program evaluation (Item A: 3.1 to 4.5) and to interpret 

evaluation findings to improve programs (Item B: 3.9 to 4.9) and 

demonstrate and communicate program effectiveness (Item C: 

3.7 to 4.9) (Table 5). This outcome is consistent with the SIF 

program’s substantial focus on evaluation. 

Changes for SIF grantees vs. comparison groups. 

Compared to the national sample of grantmaking nonprofits, SIF 

grantees experienced significantly more growth between 2009 

and 2014 in all three evaluation areas, including (a) conducting 

rigorous program evaluations; (b) using evaluation findings to 

improve programs; and (c) using evaluation findings to 

demonstrate and communicate the effectiveness of programs 

funded by the organization. The effect size differences in all 

three areas were large. 

Compared to non-selected SIF applicants, SIF grantees 

experienced significantly greater changes between 2009 and 

2014 in the extent to which they conducted rigorous evaluations 

of programs funded by the organization, and the effect size was 

large. Although the differences in the other two evaluation areas 

were not statistically significant, the effect sizes were medium 

and small, respectively (Table 5). 
PAQ FALL 2017

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

446 

Table 5 

Changes in how SIF grantees and comparison groups
 

approach evaluation
 To what extent did 

your organization do 

this ...? 

Group 2009 2014 Change 

Effect 

size 

Respondents 

with change 

Conducted rigorous 

evaluations of programs 

funded by your 

organization 

SIF 3.1 4.5 

(20) (20)
 

1.4a+ b+ 0.7 14
 

Non-selected SIF 

applicants 

3.3 3.6 

(8) (8)
 

0.4 0.9 2
 

National sample 

of grantmaking 

nonprofits 

3.4 4.1 

(184) (185) 

0.7 1.9 62 

B. Used evaluation 

findings to improve 

programs funded by 

your organization 

SIF 3.9 4.9 

(20) (20)
 

1.0 b+ 0.5 12
 

Non-selected SIF 

applicants 

5.1 5.4 

(9) (9)
 

0.3 0.7 2
 

National sample 

of grantmaking 

nonprofits 

3.6 4.3 

(183) (184) 

0.7 0.8 57 

C. Used evaluation 

findings to demonstrate 

and communicate 

effectiveness of 

programs funded by 

your organization 

SIF 3.7 4.9 

(20) (20)
 

1.2 b+ 0.7 12
 

Non-selected SIF 

applicants 

4.9 5.7 

(9) (9)
 

0.8 0.3 3
 

National sample 

of grantmaking 

nonprofits 

3.8 4.6 

(181) (182) 

0.8 0.8 63 

1 Statistical significance (p<.05) is noted as: 

a: significant difference between SIF and non-selected SIF applicants 

b: significant difference between SIF and the national sample of grantmaking 

nonprofits
 

+: favoring SIF
 

-: favoring the comparison group
 

Given the SIF’s intense emphasis on evaluation, the 

survey also asked about changes in evaluation staffing and 

budget. SIF grantees, non-selected SIF applicants, and the 

national sample of grantmaking nonprofits all reported increases 

between 2009 and 2014 in several indicators of evaluation 

capacity: (a) staff position(s) or groups within the organization 

dedicated to evaluation; (b) external evaluation partners that 

provide the organization with evaluation services; and (c) line 

items in the organization’s budget dedicated to evaluation. SIF 

grantees had a significantly higher baseline for evaluation 

capacity in 2009 than did the national sample of grantmaking 

nonprofits: SIF grantees were more likely to have (a) staff 

position dedicated to evaluation; (b) an external evaluation 
 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

PAQ FALL 2017 447 

partner; and (c) dedicated line items. SIF grantees also 

experienced significantly more growth in these areas than did the 

national sample of grantmaking nonprofits(Table 6). 

The perception data in Table 6 are further substantiated 

by organizations’ financial data (Table 7). The majority of SIF 

grantees and non-selected SIF applicants reported significantly 

higher evaluation budgets in 2014 than in 2009, both in total 

dollar amount and percentage of the organization’s annual 

budget. In contrast, the national sample of grantmaking 

nonprofits was more likely to report that their evaluation budgets 

were about the same in 2014 as in 2009. 

Table 6 

Changes in the evaluation infrastructure of SIF grantees and 

comparison groups 

Did/does your 2009 2014 

organization have 

the following? 

Group n % (yes) n % (yes) Change 

Staff position(s) or a SIF 20 50.0% b+ 20 70.0% 20.0% b+ 

group within your 

organization that is 

Non-selected SIF 

applicants 8 50.0% 8 75.0% 25.0% 

dedicated to 

evaluation 

National sample 

of grantmaking 187 30.0% 189 38.9% 7.7% 

nonprofits 

External evaluation SIF 20 70.0% b+ 20 95.0% 25.0% b+ 

partner(s)— 

consultant(s) or 

Non-selected SIF 

applicants 8 62.5% 8 87.5% 25.0% 

organization(s)—that 

provide your 

organization with 

evaluation services 

National sample 

of grantmaking 

nonprofits 186 34.9% 189 45.4% 9.6% 

Part of the SIF 20 75.0% b+ 20 95.0% 20.0% b+ 

organization's budget 

is dedicated to 

Non-selected SIF 

applicants 8 62.5% 8 87.5% 25.0% 

evaluation National sample 

of grantmaking 187 35.5% 189 47.2% 11.7% 

nonprofits 

Statistical significance (p<.05) is noted as: 

a: significant difference between SIF and non-selected SIF applicants 

b: significant difference between SIF and the national sample of 

grantmaking nonprofits 

+: favoring SIF 

-: favoring the comparison group 
 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

448 PAQ FALL 2017 

Table 7 

Reported Changes in the evaluation budgets of SIF grantees 

and comparison groups 

Was your 

2014 

evaluation 

budget... 

than the 

evaluation 

budget in 

2009? Group n 

Substantially

higher 

 Somewhat

higher 

About

the 

same 

Somewhat 

lower 

Substantially 

lower 

Total 

evaluation 

budget in 

dollars 

SIF b+ 20 50.0% 20.0% 30.0% 0.0% 0.0% 

Nonselected SIF 

applicants 

6 33.3% 50.0% 16.7% 0.0% 0.0% 

National 

sample of 

grantmaking 

nonprofits 

160 16.7% 24.6% 57.5% 0.4% 0.8% 

Evaluation 

budget as a 

percentage of 

organization’s

total annual 

budget 

SIF b+ 20 30.0% 25.0% 35.0% 5.0% 5.0% 

Nonselected SIF 

applicants 

6 16.7% 33.3% 50.0% 0.0% 0.0% 

National 

sample of 

grantmaking 

nonprofits 

156 12.9% 16.2% 62.5% 6.8% 1.6% 

Statistical significance (p<.05) is noted as: 

a: significant difference between SIF and non-selected SIF applicants 

b: significant difference between SIF and the national sample of grantmaking 

nonprofits 

+: favoring SIF 

-: favoring the comparison group 

SIF grantees significantly exceeded the national sample 

of grantmaking nonprofits in terms of annual evaluation budget 

as a percent of the organization’s total budget (4.1% higher) in 

2009, and in terms of increase in that budget (1.7%) in 2014 

(Table 8). One caveat is that, in some cases, the evaluation 

budget may be part of the program budget rather than a separate 

line item. In addition, the mean values for the national sample of 

grantmaking nonprofits were skewed by one outlier with a much 

higher evaluation budget than any others. Across the three 

groups, we found no differences over time in the number of fulltime equivalent (FTE) staff with primary responsibility for 

evaluation. One explanation is that SIF grantees tend to hire 

external evaluators, because internal staff are not equipped to 
 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

PAQ FALL 2017 449 

conduct the rigorous evaluations required. Nevertheless, 

additional staff may be hired to oversee these evaluations. 

Table 8 

Changes in the evaluation budgets of SIF grantees and 

comparison groups 

Did/does your 2009 2014 Change 

organization 

have the 

following? 

Group n Mean Median n Mean Median 

Total annual 

evaluation 

budget (in $) – 

for in-house 

evaluators or 

external 

partners 

SIF 10 $860,590 $170,450 10 $1,789,800 $425,000 $939,344 

Non-selected 1 $150,000 $150,000 1 $200,000 $200,000 $50,000 SIF applicants 

National 

sample of 66 $568,817 $0 58 $1,115,810 $0 $43,195 

grantmaking 

nonprofits 

Annual SIF 13 4.1% b+ 1.5% 13 5.8% 4.0% 1.7% b+ 

evaluation 

budget as a 

Non-selected 

SIF applicants 0 -- -- 0 -- -- --

percentage of National 

the 

organization’s 

total budget 

sample of 

grantmaking 

nonprofits 

76 1.2% 0.0% 80 2.9% 1% 0.8% 

Number of SIF 17 1.0 0.5 17 1.7 1 0.7 

FTE staff with 

primary 

Non-selected 

SIF applicants 3 0.5 0.5 3 1.3 1 0.8 

responsibility National 

for evaluation sample of 

grantmaking 

101 1.8 0 104 3.1 .40 0.9 

nonprofits 

Statistical significance (p<.05) is noted as: 

a: significant difference between SIF and non-selected SIF applicants 

b: significant difference between SIF and the national sample of grantmaking 

nonprofits 

+: favoring SIF
 

-: favoring the comparison group
 

Observations about change. A majority of those SIF 

grantees reporting changes in their use of evaluation attributed 

the change to their participation in the SIF. In contrast, a 

majority of the national sample of grantmaking nonprofits 

credited direction from the organization’s board or other 

leadership for the change. In the open-ended responses, SIF 

grantees noted other factors such as increased capacity in terms 

of staff, software, evidence-based programs, and framework, 

with an emphasis on collective impact evaluation. When asked 
 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

450 PAQ FALL 2017 

how participation in the SIF had contributed to their 

organization’s approach to evaluation, SIF grantees noted an 

increase in their communication of findings, appreciation of 

evaluation, and use of resources and tools. 

SIF's emphasis on rigorous evaluation is giving us an 

opportunity to demonstrate to our organization as a 

whole how evaluation can impact results. It's allowing 

us to show what it means to fund for impact. 

The SIF helped us look at regional evaluation capacity— 

both from a grantee and evaluator standpoint. We are 

working with other local funders to build capacity. We 

are much more focused on evaluating at both a program 

level and our own community change efforts. 

Scaling up programs with evidence of effectiveness. 

Increasingly, federal and nonprofit grantmakers are seeking to 

scale up programs that have demonstrated evidence of 

effectiveness. Scale-up may be broadly defined as increasing the 

impact of a program within a community or expanding it to other 

communities or populations. 

SIF grantee organizational changes. About half the SIF 

grantees reported changes in their efforts to scale up existing 

programs (Item A: 4.2 to 5.1) and select programs for scale-up 

based on rigorous evaluation that demonstrates their 

effectiveness (Item B: 3.5 to 4.6) (Table 9). 
 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

PAQ FALL 2017 451 

Table 9 

Changes in approach to scaling up programs for SIF grantees 

and comparison groups 

To what extent 

did your 

organization do 

this ...? 

Group 2009 2014 

Effect Respondents 

Change size with change 

A. Undertook efforts 

to scale up existing 

program(s) – that is., 

to expand the 

program(s) within 

the community or to 

other communities or 

populations 

SIF 4.2 

(19)b+ 

5.1 

(19) 

0.9 0.5 9 

Non-selected SIF 

applicants 

4.5 (8) 5.0 (8) 0.5 0.4 3 

National sample 

of grantmaking 

nonprofits 

3.1 

(173) 

3.9 

(174) 

0.8 0.3 79 

B. Selected programs SIF 

for scale-up based on 

rigorous evaluation Non-selected SIF 

that showed them to applicants 

be effective National sample 

of grantmaking 

nonprofits 

3.5 (19) b+ 4.6 

(19) 

1.1 b+ 0.5 9 

3.4 (8) 4.0 (8) 0.6 0.3 2 

2.5 

(170) 

3.2 

(172) 

0.8 0.6 53 

1 Statistical significance (p<.05) is noted as: 

a: significant difference between SIF and non-selected SIF applicants 

b: significant difference between SIF and the national sample of grantmaking 

nonprofits
 

+: favoring SIF
 

-: favoring the comparison group
 

Changes for SIF grantees vs. comparison groups. 

Compared to the national sample of grantmaking nonprofits, SIF 

grantees experienced significantly greater changes between 2009 

and 2014 in the extent to which they selected programs for scaleup based on rigorous evaluation that demonstrates effectiveness; 

the effect size was medium. Although we found no statistically 

significant differences in scale up among the three groups, we 

found small distinctions in favor of SIF grantees in the 

magnitude of those differences. Although the change patterns for 

SIF grantees were not statistically significant, compared to nonselected SIF applicants, we detected small differences in favor of 

SIF grantees in the magnitude of differences in both capacities. 

Observations about change. For those SIF grantees 

reporting changes in scaling up programs, a majority attributed 

them primarily to the organization’s participation in the SIF, 

whereas a majority of the national sample of grantmaking 
 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

452 PAQ FALL 2017 

nonprofits attributed changes to direction from the organization’s 

board or other leadership. (SIF grantees also cited direction from 

their board/leadership as a major factor.) In response to openended questions, SIF grantees noted increased capacity in terms 

of staffing, shifts in organizational strategies such as adoption of 

“results-based accountability,” and more funds raised for scaling 

up. Asked how participation in SIF contributed to their approach 

to scaling up, SIF grantees reported a greater understanding of 

scaling, more available funding, newly created staff positions, 

and evaluation readiness: 

SIF is providing us with the evidence needed to scale 

certain programs. We also have a deeper understanding 

of what it takes to scale programs and how to support 

organizations to grow. We would not have had that 

"forced learning" without SIF. 

We are taking what we’ve learned from SIF that has had 

positive impact in terms of process (such as competitive 

bids, an emphasis on accountability), and we are 

applying those standards to other bodies of work we 

fund. 

Collaboration to address community needs. By 

awarding SIF grants, CNCS also anticipates prompting wider 

and deeper collaboration by SIF grantees—with subgrantees, 

other SIF grantees, and grantmakers—to address community 

needs and improve collective impact. The requirement for 

raising and sustaining match funding requires collaboration 

with other funders, but CNCS is interested in fostering broader 

community among funders. 

SIF grantee organizational changes. About half the SIF 

grantees indicated changes in collaboration between 2009 and 

2014 and reported considerable increases in their participation in 

funding alliance(s) with other nonprofit sector organizations 

(Item A: 3.6 to 4.3), knowledge sharing (Item B: 4.2 to 5.2), and 

collaboration for purposes of advocacy (Item C: 3.5 to 4.1) 

(Table 10). 
 

 

 

 

 

 

 

 

 

 

 

PAQ FALL 2017 453 

Table 10 

Changes in approach to collaboration by SIF grantees and
 

comparison groups
 To what extent did 

your organization do 

this ...? 

Group 2009 2014 Change 

Effect 

size 

Respondents 

with change 

A. Participated in 

funding alliance(s) 

with other nonprofit 

sector organizations. 

(For example, coSIF 3.6 (20) 4.3 (20) b+ 

0.8 0.4 9 

Non-selected 

SIF applicants 

3.9 (8) 4.5 (8) 0.6 0.1 2 

National 3.1 3.7 0.7 0.2 66 

funding programs 

through joint funding; 

providing or receiving 

matching funds; or 

sample of 

grantmaking 

nonprofits 

(172) (172) 

other collaboration) 

B. Participated in 

collaborations with 

other nonprofit 

organizations to share 

SIF 4.2 (20) 5.2 (20) 1.0 0.7 12 

Non-selected 

SIF applicants 

5.3 (8) 5.9 (8) 0.6 0.3 2 

National 4.1 4.9 0.8 0.4 68 

knowledge sample of (172) (172) 

grantmaking 

nonprofits
 

C. Collaborated with 

other organizations for 

purposes of 

advocacy—to 

SIF 3.5 (20) 4.1 (20) 0.6 0.3 8 

Non-selected 

SIF applicants 

4.0 (8) 4.8 (8) 0.8 -0.2 3 

National 3.3 3.9 0.7 -0.2 56 

advocate for or 

develop public support 

for programs or 

approaches to 

sample of 

grantmaking 

nonprofits 

(169) (170) 

addressing social 

problems 1 Statistical significance (p<.05) is noted as: 

a: significant difference between SIF and non-selected SIF 

applicants 

b: significant difference between SIF and the national sample of 

grantmaking nonprofits 

+: favoring SIF 

-: favoring the comparison group 

Changes for SIF grantees vs. comparison groups. No 

statistically significant differences emerged between SIF 

grantees and the comparison groups. However, compared to the 

national sample of grantmaking nonprofits, we found small 

differences in favor of SIF grantees in the magnitude of their 

participation in funding alliances with other nonprofits and 
 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

454 PAQ FALL 2017 

collaborations with other nonprofits to share knowledge. 

Compared to non-selected SIF applicants, we found small 

differences in favor of SIF grantees in the magnitude of 

collaborations to share knowledge (Table 10). 

Observations about change. A majority of SIF grantees 

that reported changes in collaboration through funding alliances 

and knowledge sharing attributed these changes to their 

participation in SIF and to trends in the broader grantmaking 

world. A majority of the national sample of grantmaking 

nonprofits, on the other hand, attributed such changes to 

direction from their board or other leadership. 

SIF grantees and the national sample of grantmaking 

nonprofits both cited direction from the organization’s board or 

other leadership as a major factor behind increased collaboration 

for advocacy. (SIF participation was not a major factor, because 

grantees cannot legally use federal funds for lobbying.) This is 

one illustration of the impact on SIF grantees of trends in the 

broader grantmaking world. In response to open-ended 

questions, a few SIF grantees noted other factors that contributed 

to increased collaboration, such as the recession and budget cuts, 

consolidation of services to augment collective impact, and the 

need for intensified advocacy. 

Regarding the ways that participation in the SIF 

influenced their approaches to collaboration, SIF grantees 

emphasized various forms of collaboration among grantees, 

funders, and industries. For example, 

To bolster support of our subgrantees, the organization 

entered into a strategic collaboration with the evaluator 

in our application to become a SIF intermediary. This 

collaboration extended beyond our SIF work to support 

most of our grantees. 

Our previous grantmaking did not necessarily have a 

match requirement. However, SIF funding does have this 

requirement, and so we are participating in these efforts 

with our SIF subgrantees. We've always participated in 

industry-specific exchanges; however, now we 

participate in cross-industry exchanges via the SIF 

Economic Opportunity Group. 
 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

PAQ FALL 2017 455 

CONCLUSIONS 

The SIF program has sought to increase the capacity of 

social entrepreneurs to tackle social issues, seed experimental 

initiatives, and provide resources to replicate and expand 

effective programs. The national assessment described in this 

article sought to identify the SIF’s influence on grantee capacity 

to meet these goals. The evaluation found evidence of positive 

changes in almost all target capacities. Table 11 shows that 

between 2009 and 2014, SIF grantees reported improvement in 

13 of 14 measures of organizational capacity. However, not all 

areas of capacity evolved in the same way. 

Compared to non-selected SIF applicants and the 

national sample of grantmaking nonprofits, as measured by 

effect size estimates, SIF had a large impact on evaluation 

capacity. Followup interviews suggested that the SIF focus on 

evaluation helped grantees plan and implement more—and more 

formal—evaluation; grantees increased their evaluation capacity 

with evaluation staff hires and the use of external evaluation 

partners; the SIF focus on evaluation helped grantees use 

evidence to improve results; and the assistance helped position 

SIF subgrantees to attract new funders, because they had the 

skills to conduct rigorous evaluation and communicate program 

effectiveness. 

Data suggested a medium impact of the SIF on grantees’ 

support for subgrantees Followup interviews further indicated 

that SIF grantees had increased their investment in growing the 

capacity of their subgrantees in terms of compliance capacity, 

evaluation capacity, and communication capacity; the SIF helped 

grantees become more strategic in how they support their 

subgrantees; and grantee support gave subgrantees a solid 

foundation upon which to grow. 

A medium effect was also observed in the area of scaling 

up by SIF grantees. Themes related to scaling noted that the SIF 

helped grantees develop methods for choosing which 

interventions to scale up; and grantees packaged what they 

learned from the SIF to support scaling among the subgrantees. 

While the SIF appears to have had a small impact on 

subgrantee collaboration, the followup interviews still showed 
 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

456 PAQ FALL 2017 

that the SIF gave grantees a variety of collaborative learning 

opportunities; the SIF influenced how grantees support peer 

learning among their subgrantees; and the SIF augmented the 

extent to which grantees engaged in external partnerships. 

Although a small effect was observed for grantees’ 

practices regarding subgrantee selection, followup interviews 

indicated some promising trends including that the SIF helped 

grantees build a more systematic process for subgrantee 

selection; the selection process helped grantees identify 

subgrantees with a stronger evidence base; and the SIF’s 

competitive solicitation process allowed grantees to reach 

subgrantees in new markets and geographic areas. 

The SIF grantees regarded rigorous evaluation and 

engaging in peer-learning experience as major strengths of the 

SIF model. They also valued the transparency and accessibility 

of the CNCS staff and their flexibility. Even with these changes, 

many organizations continue to have room to improve their 

capacity. When asked how often they undertake activities 

indicative of strong organizational capacity in the survey, most 

SIF grantees scored themselves at or below 5 on a scale of 1-7, 

with 1 meaning “not at all” and 7 “meaning always.” 

While the evidence of progress is encouraging, the 

grantees offered recommendations for improving the SIF 

program. Many grantees reported that meeting the match 

requirement was a major challenge. Even among those who had 

early success, by years 3 to 5, several grantees mentioned 

undertaking an exercise to move existing funds around to meet 

the match, but not truly leveraging new dollars. In addition, 

although many of the federal requirements were not SIF-specific, 

grantees considered the federal compliance in financial 

regulations and criminal background checks as particularly 

burdensome. SIF grantees and applicants offered additional 

recommendations to improve the SIF application process, such 

as adjusting the timeframe, including a pre-qualification phase, 

more transparency about implementation requirements, and 

further streamlining and clarifying of application requirements. 

Finally, SIF grantees recommended continuing collaboration 

among SIF grantees and funders, and reiterated the importance 
 

PAQ FALL 2017 457 

of securing engagement and feedback from private foundations 

and philanthropies. 

Table 11 

Summary table for impact on SIF grantee organizational 

capacity 

Organizational capacity 

Change 

in SIF 

grantees 

from 

2009 to 

2014 

Difference in 

changes: SIF 

2010-2012 

grantees vs. 

national 

sample of 

grantmaking 

nonprofits 

Difference in 

changes: SIF 

2010-2012 

grantees vs. 

non-selected 

SIF 

applicants 

Evaluation 

A. Conducted rigorous evaluations of programs 

funded by your organization 

↑ +Large* +Large* 

B. Used evaluation findings to improve 

programs funded by your organization 

↑ +Large* +Medium 

C. Used evaluation findings to demonstrate and 

communicate effectiveness of programs funded 

by your organization 

↑ +Large* +Small 

Support for subgrantees 

A. Provided funding to carry out an evaluation 

or hire an external evaluator, as part of the grant 

or through other means 

↑ +Large* No 

B. Provided training or TA by your staff 

consultants or other means to conduct rigorous 

evaluation 

↑ +Large* +Small 

C. Provided training or TA by your staff 

consultants or other means to support 

implementation of the program 

↑ No +Small 

Scale up 

A. Undertook efforts to scale up existing 

program(s)—that is, to expand the program(s) 

within the community or to other communities 

or populations 

↑ +Small +Small 

B. Selected programs for scale -up based on 

rigorous evaluation that shows them to be 

effective 

↑ +Medium* +Small 
 

Table 11, continued 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

458 PAQ FALL 2017 

Difference in 

Organizational capacity 

Change 

in SIF 

grantees 

from 

2009 to 

2014 

changes: SIF 

2010-2012 

grantees vs. 

national 

sample of 

grantmaking 

nonprofits 

Difference in 

changes: SIF 

2010-2012 

grantees vs. 

non-selected 

SIF 

applicants 

Collaboration 

A. Participated in funding alliance(s) with other 

nonprofit sector organizations (for example, cofunding programs through joint funding; 

providing or receiving matching funds; or other 

collaboration) 

↑ +Small No 

B. Participated in collaborations with other 

nonprofit organizations to share knowledge 

↑ +Small +Small 

C. Collaborated with other organizations for 

purposes of advocacy—to advocate for or 

develop public support for programs or 

approaches to addressing social problems 

↑ No No 

Selection of subgrantees to fund 

A. Used an open, competitive process to solicit 

and review applications and to make selection 

decisions 

↓ -Medium* -Small 

B. Required applicant organizations to provide 

evidence of intervention effectiveness to be 

↑ -Medium +Small 

eligible for funding (includes pre- and posttest 

outcome data or other evidence based on 

evaluation studies) 

C. Required applicants to submit a plan for 

rigorous evaluation of intervention to be eligible 

for funding (that is, quasi-experimental designs 

with a comparison group, experimental designs, 

or other similarly rigorous designs) 

↑ -Small +Small 

↑SIF grantees experienced positive change 

↓SIF grantees experienced negative change 

*statistically significant difference at 0.05 level. A lack of significant difference 

between SIF grantees and non-selected SIF applicants is partly attributed to the 

small sample sizes. 

According to Cohen, large effect size at 0.8 or more; medium effect size at 0.5-0.79; 

small effect size at 0.2-0.49; no effect size smaller than 0.2. 

+difference favoring SIF grantee 

-difference favoring comparison group 

Building organizational capacity requires a multi-faceted 

approach and lengthy commitment through a recurring cycle of 

intervention, support, and growth (Harsh, 2012). The SIF 

national assessment has provided promising evidence that the 

SIF moved grantee organizations in the direction of increased 
 

 

 

 

 

 

 

 

PAQ FALL 2017 459 

capacity to make and support investments in program 

improvement and expansion. It also reminds funders and 

nonprofits that organizational changes require patience. As 

several respondents pointed out, almost everything about 

developing capacity takes more time and presents more 

complexity than one might anticipate. Nevertheless, building 

capacity in nonprofits has the potential to strengthen not only 

their individual organizational competency, but that of the 

community of nonprofit organizations as well. 
 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

460 PAQ FALL 2017 

REFERENCES 

American Association for Public Opinion Research. (n.d.). 

Response rates: An overview. Retrieved from 

https://www.aapor.org/Education-Resources/ForResearchers/Poll-Survey-FAQ/Response-Rates-AnOverview.aspx. 

Brown, M.S., McKeever, B., Dietz, N., Koulish, J., and Pollak, 

T. (2013). The impact of the great recession on the 

number of charities. National Center for Charitable 

Statistics. Washington, DC: Urban Institute. 

Buteau, E., & Buchanan, P. (2013). Working well with grantees: 

A guide for foundation program staff. Cambridge, MA: 

The Center for Effective Philanthropy. 

Cantrell,	 P., (2010). Traditional vs. retrospective pretests for 

measuring science teaching efficacy beliefs in preservice 

teachers. School Science and Mathematics, 103(4), 177-

185. 

Cohen, J. (1988). Statistical power analysis for the behavioral 

sciences. Hillsdale, NJ: Lawrence Erlbaum Associates. 

Corporation for National and Community Service. (n.d.). Social 

Innovation Fund: Evaluation plan guidance. A step-bystep guide to designing a rigorous evaluation. 

Washington, DC: Corporation for National and 

Community Service. 

De Vita, C. J., Fleming, C., & Twombly E. C. (2001). Building 

nonprofit capacity: A framework for addressing the 

problem. In C. J. De Vita & C. Fleming (Eds.). Building 

capacity in nonprofit organizations. Washington DC: 

The Urban Institute. 

Grantmakers for Effective Organizations. (2013). Investing for 

impact: Building the capacity and infrastructure of highperforming nonprofits. Washington, DC: Grantmakers for 

Effective Organizations. 

Grantmakers	 for Effective Organizations. (n.d.). What is 

nonprofit capacity and why does it matter? Washington, 

DC: Grantmakers for Effective Organizations. 
 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

PAQ FALL 2017 461 

Harsh, S. (2012). Taking successful change initiatives beyond 

capacity: A multiple-dimension approach to capacity 

building. Fairfax, VA: ICF International. 

Hill, L.G., & Betz, D.L. (2005). Revisiting the retrospective 

pretest. American Journal of Evaluation, 26, 501-517. 

Letts, C.W., Ryan, W., & Grossman, A. (1997). Virtuous capital: 

What foundations can learn from venture capitalists. 

Harvard Business Review (March-April), 1-8. 

Minzner, A., Klerman, J. A., Markovitz, C. E., & Fink, B. 

(2014). The impact of capacity-building programs on 

nonprofits: A random assignment evaluation. Nonprofit 

& Voluntary Sector Quarterly, 43(3), 547-569. 

doi:10.1177/0899764013491013. 

Pratt, C. C., McGuigan, W. M., Katzev, A. R. (2000). Measuring 

program outcomes: Using retrospective pretest 

methodology. American Journal of Evaluation, 21(3), 

341-349. 

Sobeck, J. L. (2008). How cost-effective is capacity building in 

grassroots organizations? Administration in Social Work, 

32(2), 49-68. doi:10.1300/J147v32n0204. 

Stiefvater, E., & Education Northwest (2015). Lessons and 

Stories from the Social Innovation Fund: From 

Transactional and Transformational—United ways 

Partner for Impact. Washington, DC: Corporation for 

National and Community Service, Social Innovation 

Fund. 

University of Denver (n.d.) Mile High United Way: Impact of the 

Social Innovation Fund on Participating Organizations. 

Denver, CO: University of Denver, Graduate School of 

Social Work Butler Institute for Families. 

Venture Philanthropy Partners. (2001). Effective capacity 

building in nonprofit organizations. Washington, D.C.: 

Venture Philanthropy Partners. 

Wandersman, A., Chien, V. H., & Katz, J. (2012). Toward an 

evidence-based system for innovation support for 

implementing innovations with quality: Tools, training, 

technical assistance, and quality assurance/quality 

improvement. American Journal of Community 

Psychology, 50.