Special Issue: Evaluation Technical Assistance

Evaluation Review 

1-24 

Lessons From the a The Author(s) 2020 

Article reuse guidelines: 

Social Innovation sagepub.com/journals-permissions 

DOI: 10.1177/0193841X20976241 

journals.sagepub.com/home/erx 

Fund: A Tested

Evaluation Technical

Assistance Approach

to Strengthening

Evaluation Practice

and Building a Body of

Evidence

Lily Zandniapour1 and Mary Hyde1

Abstract

The Social Innovation Fund (SIF), a program of the Corporation for 

National and Community Service that received funding from 2010 to 

2016, is one of a set of tiered evidence initiatives that was designed and 

implemented at the federal level during President Obama’s administration. 

The key objectives of the initiative were to (1) invest in promising interventions that address social and community challenges and grow their 

1 Research and Evaluation, Corporation for National and Community Service, Washington, 

DC, USA 

Corresponding Author:

Mary Hyde, Director of Research and Evaluation and/or Lily Zandniapour, Research and 

Evaluation Manager; Corporation for National and Community Service, 250 E Street SW, 

Washington DC 20525, USA. 

Email: mhyde@cns.gov 
2 Evaluation Review XX(X)

impact and (2) invest in evaluation and capacity building in order to support 

the development and use of rigorous evidence to measure the effectiveness 

of each funded intervention (i.e., to “move the evidence needle”) and 

inform decision making. The SIF proved successful in strengthening and 

sustaining the capacity of its implementing partners to conduct rigorous 

evaluations when put through a robust impact evaluation of its own at the 

national level. It has also spurred high-quality local evaluations that are 

building knowledge and a body of evidence across the supported program 

models to inform practice. The SIF’s evaluation technical assistance program was critical to its success, and as such, its design and approach holds 

interesting lessons for the larger field. This article discusses the structure 

and key features of the SIF as a grant making model, its evaluation requirements, and embedded approach and process for evaluation capacity building 

and the delivery of technical assistance, the tools and resources that it 

generated to support its goals, the evidence supporting its success, and 

how those lessons can inform other organizations and initiatives. 

Keywords

rigorous evaluation, evaluation technical assistance, capacity building, 

evidence-based grant making, Social Innovation Fund, tiered evidence 

Introduction

During the last 2 decades, there has been a growing interest in and demand

for evidence-based solutions to persistent social problems (Haskins &

Margolis, 2014; Nussle & Orszag, 2014). In practice, this has meant an

increased emphasis and reliance on evidence produced through evaluations

designed to assess causal relationships between social programs and

improvements in the negative conditions and outcomes that often affect

vulnerable populations in low-income communities across the country

(Maynard et al., 2016). Within the context of the federal government, tiered

evidence initiatives have been an important strategy for incentivizing the

use of causal evidence in the design and implementation of social programs

(Government Accountability Office [GAO], 2016; Haskins & Baron, 2011;

Haskins & Margolis, 2014; Lester, 2015). More specifically, these initiatives use evidence to inform grant making, guiding decisions about which

applicants receive funding and the dollar value of grant awards. Evidence is

also used to assess implementation of social programs, document their
Zandniapour and Hyde 3 

impacts, and inform future funding priorities (GAO, 2016). In addition, all

of these federal initiatives have some form of embedded evaluation technical assistance (ETA) component in order to make them operational. As

such, they offer interesting case examples in terms of how they not only

build the evidence base for social programs but also strengthen evaluation

capacity and practice by providing technical assistance.

The Corporation for National and Community

Service’s (CNCS) Social Innovation Fund (SIF)

The SIF, administered by the CNCS, is one of the federal tiered evidence

initiatives that offers an interesting case example of ETA. The SIF received

government funding from 2010 to 2016. Annual Congressional appropriation for the initiative was US$50 million, except for 2014, when the program received a US$70 million appropriation. According to CNCS program

and administrative data, SIF generated over US$1 billion in investments for

evidence-based community solutions in 46 states and Washington, DC.

Private and local funders multiplied the US$352 million direct federal

investment in awards, providing more than US$707 million in additional

funding to expand evidence-based programming. These investments provided support to 553 nonprofit organizations in close to 2,000 service

locations across the country and provided social programming to more than

857,000 individuals and their families.

The SIF was committed to investing in innovative and promising solutions that had demonstrated some evidence of effectiveness through prior

evaluation studies. For the SIF, “social innovation” meant “new ways to

solve old problems that are faster, cost-effective, data-driven, and lead to

better results for the public good.” SIF made grant awards to experienced

intermediary organizations (“grantees”) with strong skills and track records

of success in selecting, validating, and growing or scaling high-performing

programs; using evaluation findings for making programmatic decisions;

and providing training and technical assistance to their past grantees.

Funded organizations possessed the capacity to manage a federal grant and

engage with third-party evaluators. Organizations were funded to implement social programs designed to improve education, economic, and health

outcomes for vulnerable communities. Funded organizations were required

to conduct third-party evaluations of the sponsored social programs with the

goal of advancing the body of evidence.
4 Evaluation Review XX(X)

The SIF had two distinct grant programs: the SIF Classic Grant Program

and the SIF Pay for Success Grant Program.2 The focus of this article is on

the SIF Classic Grant Program that implemented six rounds of grant making

starting in 2010 and awarded a total of 39 grants. These SIF grantees were

selected through an open, competitive process. The SIF grants ranged from

US$1 million to US$10 million per year, for up to 5 years. Table 1 summarizes the number of awards by year.

Table 2 lists all of the SIF grantees, their projects, and number of awards.

Although CNCS made its last set of SIF grant awards in fiscal year 2016,

SIF grantees and their subaward recipients (“subgrantees”) will continue to

administer their programs until their federal funding is exhausted.

The SIF ETA Program

Given the high evaluation and evidence bar of the initiative, SIF grantees

and subgrantees were required to commit significant time and resources to

ensure formal evaluations of each program model that received funding

from the SIF. The agency quickly realized the need to provide resources

if the initiative was to fully meet its inherent evidence building goals and

developed a comprehensive ETA program (see Figure 1 for the theory of

change).

By statute, the SIF was to reserve not more than 5% of the funds appropriated for a fiscal year to support, directly or through contract with an

independent entity, research and evaluation activities to evaluate the eligible entities and community organizations receiving grants under the initiative. The SIF ETA program was supported by third parties3 with a value of

between US$4 million and US$5 million for services provided to an approximately US$350 million funding stream over a 7-year period of funding

(representing around 1.3% of the initiative’s appropriation funds and 1.6%

of the federal grant funds based on 80% grant disbursement ratio). This only

reflects the external costs of the ETA program and does not include the

agency’s costs, which primarily included CNCS staff time. Management and

technical direction of the work at the agency level required at least 1.5–2 fulltime equivalent research and evaluation staff and a significant amount of time

on the part of the SIF program officers and leadership.

Although participation in the array of ETA activities was not mandatory

for SIF grantees, subgrantees, and their evaluation partners, SIF evaluation

requirements and expected milestones were requirements of the program,

and as such, SIF stakeholders engaged in those activities in order to meet

program requirements and expectations. The array of activities that
Zandniapour and Hyde 5 

Table 1. SIF Classic Grants by Award Year. 

Award Year Number of Grants to SIF Grantees 

2010 11 

2011 5 

2012 4 

2014 7 

2015 8 

2016 4 

Total 39 

Note. No new awards were made in 2013. SIF 1⁄4 Social Innovation Fund. 

comprised the ETA program and the process for their delivery to SIF

grantees and their partners are described in the next section.

SIF ETA Activities and Delivery Process

The ETA program activities were tied to the evaluation planning, implementation, monitoring, and reporting expectations and requirements of the SIF.

The evaluation planning process took place during the first year of the grant.

Once an evaluation plan was approved, the evaluation moved to the second

phase: implementation and reporting. The time frame for evaluation plan

implementation and reporting was based on the length of the study and varied

for different interventions. On average, the studies were expected to take

around 3.5 years to complete. During this period, grantees would submit

interim evaluation reports, which were reviewed and compiled by CNCS.

The evaluation reports were used to meet CNCS’s information needs and

provide feedback to grantees and their partners. The technical guidance and

standardized feedback provided to the grantees, theirsubgrantees, and respective evaluation partners were all intended to strengthen the capacity and

deliverables for the program at each phase of the evaluation. Figure 2 provides a visual overview of this process (Zandniapour & Deterding, 2017).

CNCS developed a comprehensive ETA program organized around a

flexible coaching model that included individual consultation and customized resource development. This framework of teaching and learning

strategies (see Figure 3) is consistent with the empirical and theoretical

evaluation capacity building (ECB) literature (e.g., Baizerman et al.,

2002a; Labin et al., 2012) and the evaluation capacity building approach

used by CNCS for other programs.
 





 





 







 

 



 

 

 







6 Evaluation Review XX(X)

Table 2. SIF Classic Grantees, Their Projects, and Number of Awards. 

Grantee, Award Year(s), Number Grantee, Award Year(s), and Project Number 

and Project Name(s) of Awards Name(s) of Awards 

AARP Foundation (2014, 2015) 

Back to Work 50þ: Women’s 

Economic Stability Initiative 

Scaling Reading Success 

AIDS United (2010) 

Access to Care Initiative (A2C) 

Annie E. Casey Foundation (2015) 

Learn and Earn to Achieve Potential 

(LEAP) 

Boston Foundation (2014) 

Boston Coaching for Completion 

Initiative 

Capital Area United Way (2012) 

Capital Area United Way Social 

Innovation Fund 

Capital Impact Partners (2011) 

Stewardship Capacity Fund 

Corporation for Supportive Housing 

(2011) 

Reducing Risky Behaviors Through 

Supportive Housing 

The Edna McConnell Clark Foundation 

(2010) 

The Edna McConnell Clark 

Foundation Social Innovation Fund 

Foundation for a Healthy Kentucky 

(2010) 

Kentucky Health Futures Initiative 

Friends of the Children (FOTC) (2016) 

Friends of the Children 

Greater Twin Cities United Way, in 

Partnership with Generation Next 

(2012) 

STRIVE Alliance 

GreenLight Fund (2012) 

GreenLight Fund Social Innovation 

Fund 

Jobs for the Future, Inc. (2010, 2014) 

National Fund for Workforce 

Solutions 

Opportunity Works 

2 Mile High United Way (2011) 1 

Early Literacy Social Innovation 

Fund 

1 Missouri Foundation for Health (2010) 1 

Strategic Innovation in Missouri 

1 Nebraska Children and Families 1 

Foundation (2015) 

Connected Youth Initiative 

1 New Profit Inc. (2010) 1 

Pathways Fund 

1 The North Carolina Partnership for 1 

Children, Inc. (2016) 

Shape NC: Healthy Starts for 

Young Children Initiative 

1 REDF (2010, 2015) 2 

REDF Social Innovation Fund 

Scaling the Impact of Social 

Enterprise 

1 Share Our Strength (2014) 1 

No Kid Hungry 

1 Silicon Valley Community Foundation 1 

(2014) 

The Big Lift 

1 United Way for Southeastern Michigan 2 

2011, 2016) 

Greater Detroit Early Childhood 

Innovation Fund 

Bib to Backpack 

1 United Way of Central Indiana (2016) 1 

Great Places 2020 

1 United Way of Greater Cincinnati 1 

(2010) 

Greater Cincinnati Social 

Innovation Fund 

1 United Way of Greenville County 1 

(2014) 

OnTrack Greenville 

2 United Way of Lane County (2015) 1 

Kids in Transition to Schools (KITS) 

(continued)
 















Zandniapour and Hyde 7 

Table 2. (continued) 

Grantee, Award Year(s), Number Grantee, Award Year(s), and Project Number 

and Project Name(s) of Awards Name(s) of Awards 

The John A. Hartford Foundation (2012) 

IMPACT 

Local Initiatives Support Corporation 

(2010, 2015) 

Expanding Financial Opportunity 

Centers 

Bridges to Career Opportunity 

Program 

Mayor’s Fund to Advance New York 

City (2010, 2015) 

Economic Opportunity Social 

Innovation Fund 

Connections to Care 

Methodist Healthcare Ministries of 

South Texas, Inc. (2014) 

S ́ı Texas: Social Innovation for a 

Healthy South Texas 

1 U.S. Soccer Foundation (2011) 1 

Soccer for Success 

2 Venture Philanthropy Partners (2010) 1 

youthCONNECT Social Innovation 

Fund 

2 Youthprise (2015) 1 

Opportunity Reboot 

1 Total Number of Awards 39 

Note. SIF 1⁄4 Social Innovation Fund; CNCS 1⁄4 Corporation for National and Community 

Service. 

Evaluation training and technical assistance were provided on emerging

“hot topics” of importance to the SIF evaluation work (e.g., one example was

on Propensity Score Matching, a strategy that many ofthe SIF evaluations were

using to develop their comparison group and was deemed important to cover).

CNCS also supported quarterly meetings of an Evaluation Workgroup, peer

learning, and evaluation-focused sessions at convenings. Access to resources

and expert advice was provided to grantees on the SIF Knowledge Network, the

program’s virtual platform for the grantee community of practice.

The following case example (Table 3) illustrates how grantees were

supported in their efforts to strengthen their evaluation capacity and

advance the evidence base of their programs.

Evidence of SIF ETA Effectiveness

The key question about the SIF ETA program is whether or not it worked.

Empirical evidence of the SIF ETA’s effectiveness is found in three sources

as follows: (1) findings from the national assessment of the SIF which

provides rigorous evidence that the program was successful in building

the evaluation capacity and practices of participating organizations, (2) the

caliber of the evaluation designs implemented by SIF grantees, and (3) the
Figure 1. Social Innovation Fund theory of change. 

8 
Zandniapour and Hyde 9 

credibility of the positive and causal findings obtained through successful

implementation of these evaluation designs.

Results From the SIF National Assessment

The SIF national assessment was designed to facilitate program evaluation

and management by CNCS and support the greater goal of learning from the

federal government’s tiered evidence initiatives.4 A rigorous quasiexperimental impact study was conducted at the funding stream level to

assess the effectiveness of the program in bringing about positive capacity

change within the participating organization, particularly the intermediary

organizations who were the most immediate sphere of influence for the SIF

(Zhang et al., 2017; Zhang & Barrero, 2016; Zhang et al., 2015; Zhang &

Sun, 2016; Zandniapour et al., 2017). The study aimed to measure change in

a number of areas but inclusive of evaluation capacity, a key focus of this

program, and relied on two comparison groups (a nonselected but eligible

sample of SIF applicants and a nationally representative sample of similar

nonprofit organizations across the country) to capture program impact. At

the core of this study were the following questions: “Is SIF a transformative

program in terms of improving organizational practices and policies of its

grantee organizations?” and “What can be learned from the implementation

and results of this program?”

Findings from this evaluation showed that the program has been successful in meeting its goals and has indeed facilitated organizational change

(Zhang et al., 2015). The study found evidence of improved organizational

capacities among the SIF grantees in 13 of 14 areas tied to the key features

of the SIF. Of note is that impact findings were most pronounced and

statistically significant in the domains of evaluation and evidence-driven

decision making: (1) conducting rigorous evaluations of the programs, (2)

using evaluation findings to improve programs, and (3) using evaluation

findings to demonstrate and communicate effectiveness of programs funded

by the organization.

The study found a combination of program features that were key to

SIF’s impact and success in improving the capacity of partner organizations. These included financial investment and resourcing of evaluation;

accountability (clear requirements and intensive coaching and ongoing

monitoring of progress); and training, technical assistance, and other hands

on support from CNCS researchers and their ETA provider, and support and

engagement of the SIF program staff. The program also produced a community of professionals supporting one another in carrying out evaluation.
Figure 2. Social Innovation Fund evaluation process. 

10 
 

























Zandniapour and Hyde 11 

CNCS’s ECB Approach: Crosswalk with Teaching and Learning Strategies

from the Literature*

What the literature

says:

What we do:

Appreciate inquiry 

Coaching/mentoring 

Community of practice 

Internship 

Involvement in 

an evaluation process 

Meetings 

Technical assistance 

Technology 

Training 

Written materials 

Coaching/mentoring 

1:1 customized and flexible coaching/mentoring for 

grantees, subgrantees and their evaluation partners 

Community of practice: Social Innovation Fund 

Knowledge Network 

Involvement in an evaluation process 

Evaluations conducted per requirements 

Feedback on evaluation briefs including explanation 

of evaluation use 

Meetings 

SIF Convenings 

Conference calls 

Evaluation Workgroup meetings 

Technical assistance 

Ad hoc for grantees 

1:1 coaching with grantees, subgrantees and their 

evaluation partners 

Conference calls/webinars on “hot topics” 

Technology 

Evidence Exchange 

Evaluation Resources webpage 

Community of practice platform 

Recorded trainings, webinars (audio/video) and 

modules 

Training 

Training curricula 

Webinars 

Self-paced modules 

Written materials 

Standardized feedback forms for evaluation plan 

reviews and interim and final report reviews 

Tools and resources including guidance documents, 

checklists, papers, and topical resources 

Presentation materials and curricula 

Figure 3. Corporation for National and Community Service’s Evaluation Capacity 

Building (ECB) framework and ECB literature crosswalk. 

Peer support among the SIF grantees and support from SIF grantees to the

organizations they fund were important to successfully implementing the

SIF program.
Table 3. Case Example of SIF Evaluation Technical Assistance. Case Description Evaluation Challenges Technical Assistance Provided Methodist Healthcare Ministries of South Texas, Inc.

(MHM) had eight separate subgrantees. Thesesubgrantees all had slightly different interventions

and target populations but were generallyimplementing an integrated health careintervention for a low-income, predominantly

Hispanic population in the Rio Grande Valley. Thegrantee came to the SIF program with the goal of

also doing a meta-analysis of all subgrantees, in

addition to individual subgrantee-level evaluations.

Results:The MHM completed all subgrantee evaluations 

(four conducted experimental design impact evaluations and four used quasi-experimentaldesign impact studies. MHM meta-evaluationof Social Innovation for a Healthy South Texas (S ́ı Texas) was completed successfully. 

Each subgrantee had its own interventionand distinct eligibility criteria, research questions, and study designs The evaluation proposals of the subgranteesvaried in quality and subgrantees proposed to use different measures and methods 

CNCS worked with MHM to develop individual subgrantee evaluation proposals and strengthen thestudy design and analytic strategies of each subgranteeCNCS developed a common framework across subgrantees so that all subgrantees were using similar measures and asking similar research questionsCNCS developed the meta-analysis and strategies for how they could conduct an overarching evaluation based on the new subgrantee study designs 

(continued)

12 
Table 3. (continued) 

Case Description Evaluation Challenges TecThe meta-evaluation study included a portfoliowide impact study using a quasi-experimental

design across the eight subgrantees with pooled individual-level data. The study examined the effectiveness of enhanced Integrated Behavioral Health (IBH) on improving patient health outcomes compared to participants receiving the subgrantee’sstandard of care or similar services. The study produced a range of findings. One of the positive findings was that after 12 months of care, S ́ı Texas participants showed 

significantly lower depressive symptoms and blood sugar levels than did participants who received each subgrantee’s standard of care. 

Note. SIF 1⁄4 Social Innovation Fund; CNCS 1⁄4 Corporation for National and Community Service. 

13 
14 Evaluation Review XX(X)

Strengthening Evaluation Designs and Building Evidence

The SIF grantees were not mandated to use a specific evaluation design to

build evidence for their social programs. SIF grantees and subgrantees

adopted a variety of study designs when evaluating their programs and

impacts. The type of study pursued depended on the program’s size, service

approach, the stage of existing evidence and implementation, allocated

budget, and other considerations. Decisions around the design were made

at the SIF grantee or local level with CNCS consultation and feedback.

SIF’s portfolio of local evaluations consists of about 135 studies that are

in different stages of development, implementation, and completion and

vary in terms of design, scope, and outcomes.

Most of these use a combination of designs including process and cost

studies to build knowledge about the workings of these promising interventions and improve implementation and include a comparison group to assess

effectiveness. A subset examines fidelity questions or questions regarding

feasibility of conducting future experimental or quasi-experimental studies

(Spera et al., 2015). Among SIF-sponsored evaluations, 30% (40 studies)

are implementing an experimental design, 53% (72 studies) employ a quasiexperimental design, and 17% (23 studies) have nonexperimental designs.

As has been stated previously, the SIF grantees and their subgrantees

were expected to advance the evidence base of social programming and

potentially increasing the number of interventions with moderate or strong

evidence of effectiveness. The evidence tiers used by the SIF are described

as follows.

Evidence Tiers

As outlined in the Social Innovation Fund Notices of Funding Availabilities, the tiers of evidence are defined as follows:

Strong evidence means evidence from previous studies on the program, the

designs of which can support causal conclusions (i.e., studies with high

internal validity), and that, in total, include enough of the range of participants and settings to support scaling up to the state, regional, or national

level (i.e., studies with high external validity). The following are examples

of strong evidence: (1) More than one well-designed and well-implemented

experimental study or well-designed and well-implemented quasiexperimental study that supports the effectiveness of the practice, strategy,

or program or (2) one large, well-designed, and well-implemented
Zandniapour and Hyde 15 

randomized controlled multisite trial that supports the effectiveness of the

practice, strategy, or program.

Moderate evidence means evidence from previousstudies on the program, the

designs ofwhich can support causal conclusions(i.e.,studies with high internal

validity) but have limited generalizability (i.e., moderate external validity).

This can also include studies for which the reverse is true—studies that only

support moderate causal conclusions but have broad general applicability. The following examples would constitute moderate evidence: (1) At

least one well-designed and well-implemented experimental or quasiexperimental study supporting the effectiveness of the practice, strategy,

or program, with small sample sizes or other conditions of implementation or analysis that limit generalizability; (2) at least one well-designed

and well-implemented experimental or quasi-experimental study that

does not demonstrate equivalence between the intervention and comparison groups at program entry, but that has no other major flaws related to

internal validity; or (3) correlational research with strong statistical controls for selection bias and for discerning the influence of internal factors.

Preliminary evidence means evidence that is based on a reasonable hypothesis

supported by credible research findings. Thus, research that has yielded promising results for either the program model or a similar program model will

meet CNCS criteria. Examples of research that meet the standards include (1)

outcome studies that track participants through a program and measure participants’ responses at the end of the program and (2) third-party pre- and

posttest research that determines whether participants have improved on an

intended outcome.

In terms of the level of evidence targeted by these evaluation studies,

15% (20 evaluations) targeted a strong level of evidence for the program,

while 56% (76 evaluations) targeted moderate level of evidence and 29%

(39 evaluations) aimed to achieve a preliminary level of evidence. Among

studies that have been completed and published to date (106 evaluations),

10% (11 studies) have achieved a strong level of evidence, 29% (31 studies)

achieved a moderate level of evidence, and the remaining 60% (64 studies)

produced a preliminary level of evidence based on the review of the study

design, implementation, and results.

Of the evaluations that were completed, 53% (56 studies) achieved the

level of evidence they were targeting, 5% (five evaluations) achieved a

higher level of evidence (three aimed for moderate and achieved a strong

evidence level and two aimed for preliminary and achieved a moderate

level of evidence), and 42% (45 evaluations) achieved a lower than targeted
16 Evaluation Review XX(X)

evidence level in the end.5 Among the latter group, five were aiming for a

strong level of evidence and attained a moderate evidence level, 34 targeted

moderate and landed in the preliminary tier and six aimed at strong and

ended in preliminary level of evidence. There are various explanations for

why the program evaluations did not achieve the intended evidence results

and a great many have to do with what transpires in the process of implementation (in terms of data collection such as issues with sample attrition,

access to administrative data, statistical power, baseline equivalence of

treatment and counterfactual group, and others) that affect the internal and

external validity of studies and in some cases it may be due to findings and

results. The SIF National Assessment’s meta-synthesis of findings based on

a subset of evaluation reports across the SIF evaluations provided the following insights: “Evaluation-related lessons concerned adequate funding

for data collection, sizable administrative datasets, tracking longitudinal

outcomes, and others, such as statistical power, detail about the control

condition, and selection of inexperienced sites” (Zhang, 2015, p. 2).

Taken together, these success indicators provide evidence that the SIF

ETA program was indeed effective in building evaluation capacity, as

intended, strengthening evaluation practices among funded organizations and

advancing the body of evidence for social programs designed to improve

educational, economic, and health outcomes in underresourced communities.

Discussion

The process of operationalizing the SIF ETA program was a rich and generative experience that produced many findings, resources, and lessons that

could be used to strengthen evaluation capacity and practices and increase

the potential for improving the evidence base for social programs. This

discussion is organized in two sections: (1) challenges identified and agency

responses to these challenges and (2) lessons and takeaways to inform and

strengthen future efforts and initiatives.

Early Challenges and Opportunities

CNCS’s operationalization of the ETA program and its requirements experienced a number of early challenges. These challenges required adjustments and in many ways informed the development of the ETA program.

These challenges and responses are summarized in Table 4.

Tiered evidence initiatives, including SIF, were the first of their kind.

While there is a literature on evaluation capacity building and a handful of
 

 





 

Zandniapour and Hyde 17 

Table 4. Challenges Encountered and Corresponding Responses. 

Challenges Responses 

Inadequate and uneven preparation among 

funded grantees for meeting all SIF 

evaluation requirements (e.g., accurately 

assessing existing body of evidence, finding 

the right evaluation partner, budgeting for 

high-quality evaluations, assessing 

evaluation readiness among subgrantees, 

implementing high-quality evaluations) 

Significance: The agency quickly realized 

the need to provide resources if the 

initiative was to fully meet the inherent 

evidence-building goals of the initiative 

Inadequate early communication 

of expectations with grantees and their 

partners about evaluation requirements 

Significance: Given that SIF grantees and 

their subgrantees had developed standing 

contracts with their research partners on 

their own, when issues were discovered 

with their proposed designs and 

processes, it was difficult and complicated 

to fix those issues and ensure that they 

align with CNCS expectations 

Failure to foresee or account for inevitable 

lags in communications, given the number 

of various parties involved in the initiative 

Significance: Time lines established for 

evaluation plan development, refinement, 

review and approval were often unrealistic 

and in some cases created delays in 

implementation of the funded project 

CNCS developed a comprehensive 

evaluation technical assistance program 

organized around a flexible coaching 

model that included individual 

consultation and customized resource 

development 

Clarified goals and expectations for 

grantees and their partners 

Clarified roles and responsibilities of 

CNCS/SIF evaluation team versus the SIF 

program office and the evaluation 

technical assistance provider to address 

issues of monitoring, compliance 

enforcement, communication, and the like 

Developed standardizing review forms to 

ensure a systematic and objective, and 

transparent approach to the review 

processes 

Revised turnaround times for reviews of 

evaluation plans and final reports 

Note. SIF 1⁄4 Social Innovation Fund; CNCS 1⁄4 Corporation for National and Community 

Service. 

case examples that offer some practice guidance (e.g., Baizerman et al.,

2002b; Cheng & King, 2017; Cousins & Bourgeois, 2014; Hilton &

Libretto, 2017), there was very limited experience within federal government with designing and operationalizing a comprehensive technical assistance program like the one developed for the SIF. Early challenges noted

above are only a handful of examples, as there were many more details and

issues that required careful considerations and response. The ETA program
18 Evaluation Review XX(X)

developed in response to these early challenges may be of broad use by

practitioners, grant makers, and evaluators alike.

Lessons Learned From the SIF ETA Program

Leadership prioritization. Staff participation in ETA was facilitated by

engaged and supportive organizational and program leadership. SIF was

very intentional on this score and reinforced buy-in by engaging intermediaries’ executive directors throughout the grant period and communicating

about the shared objectives of the initiative to ensure there would be no lapse

in prioritization on the part of the initiative’s primary stakeholders—SIF’s

grantees. It is also worth noting that the SIF grantees represented highcapacity organizations, and this general capacity paired with leadership

support facilitated a readiness for the evaluation demands of this initiative.

Trusting relationships. Planning and conducting evaluations in the real world

poses many challenges and issues. Being able to openly discuss concerns

and issues as they come up and having confidence in that all parties are

vested in the successful resolution of those challenges are key for effective

capacity building and assistance. Having a mindset that encourages holding

back rather than sharing is a detriment to open and effective communication.

Timing matters. ETA is most effective when it is delivered “in time” to meet

the needs of the organizations where they are and when there is absorption

capacity. Due to the technical nature of many of the concepts and issues that

are covered in ETA, when these types of information and requirements are

communicated generally, they are difficult to process and absorb by the lay

person. Communicating relevant content at the right time, when there is

attention and focus on them by the recipient, enables them to connect to it

and take it in. Moreover, effective ETA requires flexibility to meet the

needs of the organizations and their partners, not overburdening them and

ensuring that they are able to engage.

Customized coaching models. Different people have different learning styles.

In the absence of direct knowledge of the learning style of the individuals, it

is important to use different formats and delivery mechanisms to respond to

diverse learning styles and facilitate take-up. Diversity of readiness and

specific needs of the organizations within their particular contexts requires

a model that is tailored or customized for the individual recipients and
Zandniapour and Hyde 19 

appropriate to the status of the existing evidence base and level of program

maturity.

Dedicated resources. ETA is most effective and meaningful when it is offered

in addition to financial and nonfinancial resources for evaluation to the

grantee. This makes for a holistic package of support and acknowledges

the value and benefits of evaluation capacity while recognizing that there

are costs associated with evaluation and participation in capacity building.

SIF recognized the need for both financial and nonfinancial resources. In

addition to allocating financial resources for ETA, many resources were

developed to assist grantees along the way. Table 5 shows some of the

resources and tools developed for the SIF grantees and their partners for

each phase of the evaluation process.

Progress measurement and a strong feedback loop are important for continuous

program improvement and achieving the intended goals of ETA. Identifying

indicators and milestones for the ETA effort and capturing challenges,

successes, and lessons learned helped promote learning, improvement, and

increased ETA effectiveness. CNCS in consultation with the ETA provider

developed a set of dashboard indicators and metrics for tracking progress.

Among indicators and milestones used were total number of TA hours, as

well as average number of hours of TA by grantee, the status of evaluation

plans and reports as well as status of feedback and progress on those plans

and reports, the types of study designs, evaluation budgets by types of

study designs, and incoming, targeted and attained levels of evidence for

the interventions. The dashboard report also included ETA topics by

phase, common challenges faced by grantees or their subgrantees, best

practices or lessons learned that could inform more targeted support.

Progress on the ETA activities was reported and tracked on a weekly and

quarterly basis.

The SIF and more specifically its ETA program’s design and implementation provides a successful framework for supporting the planning and

implementation of rigorous local evaluations while building the infrastructure and capacities of its partner organizations to conduct such studies and

use the generated evidence for decision making purposes. An independent

national impact evaluation has verified the program’s success in building

evaluation capacity. Important information about the 135 interventions that

are addressing pressing challenges in local communities has been produced

and their evidence base strengthened. SIF has increased the number of

interventions with moderate and strong levels of evidence of addressing
 

20 Evaluation Review XX(X)

Table 5. Sample ETA Resources by Topic and Evaluation Phase. 

ETA Delivery Phase Sample ETA Topics ETA Resource 

Evaluation Planning Evaluation plans SIF Evaluation Plan (SEP)

Establishing a baseline Guidancea

assessment of a program’s The SIF Rubricb 

evidence base Evaluator Screening Tipsc

Screening and selecting a Evaluation Budgeting 

third-party evaluator Guidanced 

Evaluation designs and Impact Evaluability

budgets Assessment Tool,e 

Impact evaluation readiness Working with

Institutional review boards institutional review

boardsf

Self-paced Modules 

(tutorials)g 

Evaluation Using administrative data Secondary/ 

Implementation Data collection Administrative Data 

& Monitoring Trouble-shooting design Use and Reporting, 

implementation challenges Implementation 

Reporting 

Outcomes/Impact 

Reporting Checklists 

Evaluation Evaluation reports SIF Reporting Guidance

Reporting documenth 

Note. ETA 1⁄4 evaluation technical assistance. a

Social Innovation Fund Evaluation Plan Guidance can be accessed at https://www.nationalser 

vice.gov/impact-our-nation/evidence-exchange/social-innovation-fund-evaluation-plan-gui 

dance b

The Social Innovation Fund rubric for assessing levels of evidence can be accessed at 

http://www.nationalservice.gov/sites/default/files/resource/SIF-Rubric-Assessing-levels-of-evi 

dence.12.19.14.pdf c

Evaluator Screening Tips can be accessed at https://www.nationalservice. 

gov/sites/default/files/resource/Evaluator-Screening-Tips.pdf d

Evaluation Budgeting Guidance 

can be accessed at https://www.nationalservice.gov/impact-our-nation/evidence-exchange/SIFBudgeting-Rigorous-Evaluation e

The Impact Evaluability Assessment Tool can be accessed at 

http://www.nationalservice.gov/sites/default/files/resource/FR_SIFImpactEvaluabilityAssess 

mentTool_Final_2016.pdf f

Working with Institutional Review Boards can be accessed at 

https://www.nationalservice.gov/sites/default/files/resource/SIF_IRB_Tips_7_2014.pdf g

The 

self-paced modules can be accessed at http://sep-tutorials.org/. h

Social Innovation Fund Evaluation Reporting Guidance can be accessed at https://www.nationalservice.gov/impact-ournation/evidence-exchange/social-innovation-fund-evaluation-reporting-guidance 

important social problems. The SIF ETA program provides another case

example that advances the science and practice of evaluation capacity

building.
Zandniapour and Hyde 21 

Declaration of Conflicting Interests

The author(s) declared no potential conflicts of interest with respect to the research,

authorship, and/or publication of this article.

Funding

The author(s) received no financial support for the research, authorship, and/or

publication of this article.

Notes

1. The Social Innovation Fund required funded intermediary organizations to match

every federal dollar one-to-one and also required the same one-to-one match of

the nonprofit and public organizations they funded as subgrantees.

2. In 2014, 2015, and 2016, Corporation for National and Community Service was

authorized to use up to 20% of the annual Congressional appropriations of Social

Innovation Fund (SIF) funds to test Pay for Success (PFS) approaches. PFS is an

innovative model for tying the funding for an intervention to achievement of its

outcomes. The goal of PFS is to leverage philanthropic and private sector investments to deliver better outcomes, enabling government or other payers (such as

school districts or hospitals) to pay only when outcomes are achieved—that is, to

pay only for what worked. Following the launch of the SIF PFS program, the

original SIF Grant Program is referred to as SIF “Classic” in order to distinguish

it from the PFS awards.

3. Corporation for National and Community Service (CNCS) procured the professional services of research and evaluation consulting firm(s) for undertaking the

evaluation technical assistance for the Social Innovation Fund. The evaluation

technical assistance providers that CNCS worked with included Abt Associates

and JBS International. Abt Associates worked with CNCS in the early phase of

the SIF and JBS International served as the evaluation technical assistance contractor from 2011 through 2016.

4. The Social Innovation Fund Classic National Assessment products suite can be

accessed at https://nationalservice.gov/impact-our-nation/evidence-exchange/

SIF-Classic-National-Assessment

5. The completed reports are posted on the Corporation for National and Community Service Evidence Exchange, which can be accessed at https://www.national

service.gov/impact-our-nation/evidence-exchange

References

Baizerman, M., Compton, D. W., & Stockdill, S. H. (2002a). The art, craft, and

science of evaluation capacity building. New Directionsfor Evaluation, 93, 1–120.
22 Evaluation Review XX(X)

Baizerman, M., Compton, D. W., & Stockdill, S. H. (2002b). Summary and analysis

of the case studies: Themes across the cases. New Directions for Evaluation, 93,

101–108.

Cheng, S., & King, J. A. (2017). Exploring organizational evaluation capacity and

evaluation capacity building: A Delphi study of Taiwanese elementary and

junior high schools. American Journal of Evaluation, 38, 521–539.

Corporation for National and Community Service. (n.d.). Social Innovation Fund

evaluation plan guidance: A step-by-step guide to designing a rigorous evaluation. https://www.nationalservice.gov/impact-our-nation/evidence-exchange/

social-innovation-fund-evaluation-plan-guidance

Corporation for National and Community Service, Office of Research and

Evaluation. (2014). Impact evaluability assessment tool (lead author, Lily

Zandniapour). http://www.nationalservice.gov/sites/default/files/resource/FR_

SIFImpactEvaluabilityAssessmentTool_Final_2016.pdf

Cousins, J. B., & Bourgeois, I. (2014). Cross-case analysis and implications for

research, theory, and practice. New Directions for Evaluation, 141, 101–119.

Government Accountability Office. (2016). Tiered evidence grants: Opportunities

exist to share lessons from early implementation and inform future federal

efforts. http://www.gao.gov/assets/680/679917.pdf

Haskins, R., & Baron, J. (2011). Building the connection between policy and evidence: The Obama evidence-based initiatives. Brookings Institution Press.

https://www.brookings.edu/wp-content/uploads/2016/06/0907_evidence_

based_policy_haskins.pdf

Haskins, R., & Margolis, G. (2014). Show me the evidence. Brookings Institution

Press.

Hilton, L., & Libretto, S. (2017). Evaluation capacity building in the context of

military psychological health: Utilizing Preskill and Boyle’s multidisciplinary

model. American Journal of Evaluation, 38, 393–404.

Labin, S., Duffy, J., Meyers, D. C., Wandersman, A., & Lesesne, C. A. (2012). A

research synthesis of the evaluation capacity building literature. American

Journal of Evaluation, 33, 307–338.

Lester, P. (2015). Social Innovation Fund: Early results are promising. The Social

Innovation Research Center. http://www.socialinnovationcenter.org/wp-content/

uploads/2015/07/Social_Innovation_Fund-2015-06-30.pdf

Maynard, R., Goldstein, N., & Nightingale, D. S. (2016). Program and policy

evaluation in practice: Highlights from the federal perspective. In L. R. Peck

(Ed.), Social experiments in practice: The what, why, when, where, and how of

experimental design & analysis. https://doi.org/10.1002/ev.20209

Nussle, J., & Orszag, P. (Eds.). (2014). Moneyball for government. Disruption Books.
Zandniapour and Hyde 23 

Spera, C., DiTommaso, A., Hyde, M., & Zandniapour, L. (2015). The Social

Innovation Fund: Pioneering an evidence-based investment model.

Corporation for National Community Service. https://www.nationalservice.

gov/impact-our-nation/evidence-exchange/SIF-Pioneering-Investment-Model

Zandniapour, L., & Deterding, N. M. (2017). Lessons from the Social Innovation

Fund: Supporting evaluation to assess program effectiveness and build a body of

research evidence. American Journal of Evaluation. http://journals.sagepub.

com/doi/pdf/10.1177/1098214017734305

Zandniapour, L., Griffith, J. D., Zhang, X., Sun, J., & Pershing, J. (2017).

Strengthening organizational practice: Evidence from the Social Innovation

Fund national assessment. The Journal of Nonprofit Education and

Leadership, 7(4). http://js.sagamorepub.com/jnel/article/view/8642

Zandniapour, L., & Vicinanza, N. (2013). Budgeting for rigorous evaluation:

Insights from the Social Innovation Fund. Corporation for National

Community Service. http://www.nationalservice.gov/impact-our-nation/evi

dence-exchange/Budgeting_Rigorous_Evaluation

Zhang, X. (2015). Meta-synthesis of evaluations across the SIF: Interim report

(Prepared for the Corporation for National and Community Service, Office of

Research and Evaluation). ICF International. https://nationalservice.gov/sites/

default/files/evidenceexchange/Interim_FR_SIFNA_SynthesisReport_Final.pdf

Zhang, X., & Barrero, J. (2016). Issue Brief #5: Designing and implementing rigorous evaluation for the Social Innovation Fund (Prepared for the Corporation

for National and Community Service, Office of Research and Evaluation). ICF

International.

Zhang, X., Griffith, J., Pershing, J., Sun, J., Malakoff, L., Marshland, W., Peters, K.,

& Field, E. (2017). Strengthening organizational capacity and practices for highperforming nonprofit organizations: Evidence from the national assessment of

the Social Innovation Fund-A public-private partnership. Public Administration

Quarterly, 41(3).

Zhang, X., Griffith, J., Sun, J., Malakoff, L., Pershing, J., Marshland, W., Peters, K.,

& Field, E. (2015). The SIF’s impact on strengthening organizational capacity

(Prepared for the Corporation for National and Community Service, Office of

Research and Evaluation). ICF International.

Zhang, X., & Sun, J. (2016). Meta-analysis of evaluations across the Social

Innovation Fund program: Final report (Prepared for the Corporation for

National and Community Service, Office of Research and Evaluation). ICF

International.
24 Evaluation Review XX(X)

Author Biographies

Lily Zandniapour, PhD Research and Evaluation Manager, is an economist with

more than 20 years of experience in the applied research and evaluation field across

the private, public, and nonprofit sectors. She supports CNCS’ mission, programs,

and policy priorities through the design, implementation and oversight of research,

evaluation, and capacity building activities, as well as knowledge management and

communications.

Mary Hyde, PhD Director of Research and Evaluation, is a community psychologist with more than 20 years of research, evaluation, technical assistance, and

training experience in social and human services. She is responsible for advancing

the evidence base for national service programs, sponsoring scholarship on volunteering and other forms of civic engagement, as well as fostering a culture of

evaluation within the agency and the field.