2016 CLASSIC GRANT COMPETITION – EVIDENCE AND EVALUATION OVERVIEWOVERVIEW WEBINAR TRANSCRIPT

nationalservice.gov/SIF

2016 Classic Grant Competition – Evidence and Evaluation

Overview Webinar Transcript

This document is a transcript of a webinar covering the 2016 Social Innovation Fund Classic Notice of Federal

Funding Availability (NOFA). It has been very lightly edited for print, and is presented in a conversational,

rather than a formal tone. For official guidance, please see the NOFA.

Welcome to the 2016 SIF Classic Grant

Competition “Evidence and 

Evaluation” Overview. Thanks for your 

interest in the Social Innovation Fund. 

My name is Lois Nembhard and I am

the Deputy Director of the Social

Innovation Fund. 

The purpose of this webinar is to

provide you with an overview of the 

SIF’s evidence and evaluation

expectations and requirements. 

Hopefully you have listened to or 

participated in the SIF Overview and

NOFA Overview webinars, because on 

today’s webinar, we will assume that

you understand how the SIF works and

will not be providing background 

information. 

Our agenda today includes: Introduction of Presenters; Overview of role of Evidence and Evaluation in SIF;

Description of evaluation Requirements; the Grantee Perspective; and questions and answers. 

Your phones have been muted to reduce background noise. You can submit questions at any time in the Q&A

section of the webinar on your screen, as well as by pressing *1 whenever prompted by the operator.
2016 GRANT COMPETITION – SIF OVERVIEW WEBINAR TRANSCRIPT SOCIAL INNOVATION FUND - CLASSIC

nationalservice.gov/SIF 2

Our presenter today is Andrea Robles, 

Research Analyst in the CNCS Office

of Research and Evaluation. Sarah

Gallagher, Director of Strategic

Initiatives from the Corporation for

Supportive Housing, was going provide 

the grantee perspective. She had a 

family emergency. I will be reading her 

remarks. 

I will now turn you over to Andrea.

Hi everyone and thank you for joining 

us today. I am going to begin with the 

role of evidence and evaluation in the 

SIF. But before we begin I just want to 

say a few things about where SIF fits in 

terms of program development. 

The nonprofit marketplace offers many

sources of funding for new ideas to be 

explored and tested. These new ideas

are at the earliest stages of idea

development and innovation. As you

can see from this graphic, SIF is not 

intended for these start-ups but for

models that have at least some existing

evidence of results. In other words, SIF

is intended for innovations that already

have some research of evaluation that

have shown positive results, and are

ready for more substantial evaluation. 

By a more substantial evaluation we

mean implementing an intervention that can be evaluated using a Quasi-Experimental Design (also known as a 

QED) or a Random Control Trial (RCT).
2016 GRANT COMPETITION – SIF OVERVIEW WEBINAR TRANSCRIPT SOCIAL INNOVATION FUND - CLASSIC

nationalservice.gov/SIF 3

Additionally, SIF is intended for innovations that are poised for expansion to more people or communities; in

other words, they are ready to be scaled up. 

As you have heard while listening to the other webinar, it is important to note that the SIF competition is

intended for grantmaking institutions that we refer to as intermediaries. It is the intermediary that will

implement and evaluate the promising innovation itself; it is the intermediary that will select subgrantees (such

as nonprofits) to implement and evaluate promising models and innovations. 

We will discuss the evidence tiers in a 

few minutes but just so you know, as of 

July 2015, the majority or 72 percent,

of SIF grantees started with

interventions assessed at preliminary. 

As you see, with investments in

rigorous evaluation designs, the 

majority or 83% are on the way

towards reaching a strong or moderate

level of evidence.

I should note however, that in 2010, our 

first funding year, interventions were

not required to reach moderate or

strong levels of evidence, thus you see

that there are some interventions that will reach only a preliminary level of evidence. And this is no longer 

allowed under the SIF. We will talk about this more in a few minutes. 

As you know by now from listening to

the SIF NOFA webinars and reading

the NOFA or other information on the 

SIF website, reliance on valid evidence

is a fundamental tenet of the Social

Innovation Fund. The SIF employs

evidence and evaluation in two primary

ways:

First, SIF examines intermediaries’ 

experience and capacity to use evidence 

to assess the effectiveness of their

programs and interventions, and to

drive impact. SIF also examines

whether intermediaries are proposing or

using program models with at least “preliminary” evidence of results. I will expand on this further in just a few

slides.
2016 GRANT COMPETITION – SIF OVERVIEW WEBINAR TRANSCRIPT SOCIAL INNOVATION FUND - CLASSIC

nationalservice.gov/SIF 4

Second, SIF also aims to grow the body

of evidence that exists regarding

interventions that work and can

demonstrate positive impacts.

Intermediaries commit to increase their

evidence base to achieve “moderate” or

“strong” levels of evidence through

rigorous evaluation for each

intervention or program model. So,

let’s unpack the evidence tiers and 

framework a little further.

So in SIF, what does it mean to

increase an evidence base of an

intervention? This schematic of what

we call the “Evidence Framework” and

it helps to visualize an increase in an

evidence base for an intervention

within the SIF timeframe. The evidence

framework is also sometimes referred

to as the “evidence continuum.” Given

that our end goal is to grow the body of

evidence about program models and

interventions that actually work, it is

critical that we appropriately categorize

each intervention or program model

within the continuum at their point of entry in the SIF program. Where an intervention or program model lands

in this classification is what we consider the starting point or the incoming level of evidence.

The starting point or as we refer to incoming level of evidence for a program model may be preliminary,

moderate or strong, depending on the existing body of evidence behind the intervention, which I will explain in

more detail in the next slide. Then, during the SIF subgrant period of three to four years, the grantees’ goal is to

construct and implement rigorous evaluation designs that will increase the body of evidence behind each

intervention or program model and move them along this continuum.
2016 GRANT COMPETITION – SIF OVERVIEW WEBINAR TRANSCRIPT SOCIAL INNOVATION FUND - CLASSIC

nationalservice.gov/SIF 5

What is a body of evidence? This

includes the types of studies that have 

been already conducted on the 

intervention or program model, the 

methodologies used to conduct those 

prior studies, and the results of the 

studied interventions. 

So how does a grantee increase the 

body of evidence? As I stated in the 

preceding slide, the grantees construct 

and implement rigorous evaluation

designs by which we mean high quality,

independent and unbiased evaluations 

that are consistent with the principles of 

scientific research.

So just to give you an example, to obtain SIF funding, an intervention must, at a minimum, be assessed as

having Preliminary level of evidence which means demonstrating that the program model or intervention has

“yielded promising results for either the program or a similar program.” Specifically, the program must have at

least some outcome information from a pre- and post-test without a comparison group, or post-test comparison 

between program and comparison groups, to just to name a few.

Thus advancement on the continuum for this type of intervention that shows positive results from an outcome 

evaluation using a pre- and post-test approach, may mean conducting an impact evaluation using a quasiexperimental or experimental design study that can get them to achieve a moderate level of evidence.

If for example, a program currently has positive results on a single site randomized controlled trial, advancing

the evidence may entail conducting a random control trial across multiple sites.

And I want to reemphasize, it is the expectation of SIF that each program model or intervention you fund in

your portfolio will achieve moderate or strong evidence as defined in this continuum by the end of its three to

four year grant period. 

Now that we have reviewed the 

fundamental roles that evidence and

evaluation play in the Social Innovation 

Fund, I would like to discuss the most 

essential evaluation requirements for

grantees once they have been selected

into the SIF program.
2016 GRANT COMPETITION – SIF OVERVIEW WEBINAR TRANSCRIPT SOCIAL INNOVATION FUND - CLASSIC

nationalservice.gov/SIF 6

If you are selected and awarded SIF

funds, there are two fundamental

responsibilities that you will have in

terms of evaluation:

First, if you are not proposing your own

intervention but funding subgrantees’

proposed interventions, the subgrantees

you select will also need to demonstrate

that they are proposing models or

interventions with at least preliminary

levels of evidence.

Second, within the SIF timeframe, each

intermediary will need to execute a

series of systematic evaluation activities in collaboration with CNCS, your subgrantees, and your evaluation

experts.

If your subgrantees will propose their

own interventions, CNCS will work

with you in a consultative manner to

establish the incoming level of

evidence for the proposed intervention

of the short-listed subgrant candidates

since they also must enter the program

with at least preliminary levels of

evidence.

At this point, many of you are asking,

how do you assess interventions’

incoming levels of evidence? In a

previous slide, I gave you a few

examples of how interventions or

programs may be assessed, but I also

want to share with you a tool, that we 

also call a rubric, that we use and have

shared with our grantees to determine

incoming levels of evidence of their

applicants’ proposed interventions.

I know the rubric’s print in this

PowerPoint slide is small and hard to

read, but the rubric, along with

instructions on how to use it, can be found on the SIF NOFA website under “Supplemental Evaluation
2016 GRANT COMPETITION – SIF OVERVIEW WEBINAR TRANSCRIPT SOCIAL INNOVATION FUND - CLASSIC

nationalservice.gov/SIF 7

Resources.” Although the process of determining the incoming level of evidence for an intervention requires

thoughtful assessment that can be complicated and often requires technical knowledge of research and

evaluation design and methods, this rubric provides a framework for assessing the existing body of evidence

behind an intervention based on past research and evaluation studies conducted on the program.

I don’t want to spend too much time on

how one determines an intervention’s

level of evidence, but I do want to share

this wonderful decision tree based on

the SIF rubric that one of our 2014

grantees, Methodist Healthcare

Ministries of South Texas, created to

simplify some of the main steps to

determine how a study may qualify.

They were kind enough to share this

graphic in the hope that it may be useful

to some of you as you are looking at the

rubric.

Assuming that the study shows positive

results on relevant outcomes, the next question, and that’s the study that the subgrantee would present to you,

the next question to ask is: what is the subject of the study?

Is the study your proposed intervention or a different organization’s intervention?

Is the proposed intervention:

• Identical to the studied intervention? (In other words, “replicated with fidelity.”)

• Similar to the studied intervention?

• Or a combination of the studied intervention and other interventions?

Lastly, what type of research design did the study use? Again, we direct you to the rubric for more detail.

Besides the evaluation requirement that

an intervention enters the program with

at least a preliminary level of evidence,

the second main responsibility for the

grantees, is that within the SIF subgrant

period of three to four years, each

intermediary will need to execute a

series of systematic evaluation activities

in collaboration with CNCS, your

subgrantees, and your evaluation

experts.

This slide presents a snapshot of the

SIF Evaluation Program Process in

order to better ground you in SIF’s

evaluation and evidence activities.
2016 GRANT COMPETITION – SIF OVERVIEW WEBINAR TRANSCRIPT SOCIAL INNOVATION FUND - CLASSIC

nationalservice.gov/SIF 8

Once intermediaries are awarded a SIF grant, they receive further orientation on the initiative’s evaluation

program. 

Next each intermediary develops a portfolio evaluation strategy that provides a framework for CNCS and the 

SIF intermediary to enter a discussion on how the intermediary will approach the evaluation of its interventions 

or program models. Depending on the models and intended outcomes, this evaluation strategy determines

whether sponsored interventions will be evaluated using multiple evaluation plans, or whether one evaluation

plan will be used across a range of subgrants, or possibly both. 

The next step in the planning process possibly is the development of SIF evaluation plans for all funded 

interventions that will build on and increase its evidence base. These plans go through a rigorous vetting process 

and must be approved by CNCS before they are implemented. It typically takes a little more than t a year

following the grant awards for intermediaries to have approved evaluation plans in place.

Once these plans are approved, the implementation and reporting phase begins. Intermediaries are expected to

closely monitor all evaluation activities tied to their portfolio of funded interventions and report progress and

results of those evaluations to CNCS.

Intermediaries share results of the conducted evaluations in their portfolio by submitting interim and final

evaluation reports to CNCS and ultimately make final reports available to the public in order to increase the 

base of knowledge about your funded interventions. 

Throughout the process, CNCS and its evaluation technical assistance provider will assist the intermediaries and

their partner organizations in their evaluation efforts, offer an array of evaluation capacity building services

including individualized feedback, advice, coaching, and other supportive services. 

Also, during this time intermediaries are expected to provide evaluation capacity building and technical

assistance to their subgrantees. 

I’m going to hand this over in just a 

minute but I want to emphasize three

particular points for those of you that

will go on to complete and submit an

application to the program.

First, CNCS expects that intermediaries

contract with independent evaluation

experts and/or require their subgrantees 

to contract with such experts. As you

have heard in this presentation, this

grant will require time, energy, and

technical expertise. Please carefully

weigh the capacity and skillset on your

own staff and consider how you might

partner with external advisors and experts who can support your Social Innovation Fund efforts.

Second, CNCS recognizes that rigorous evaluations are expensive and that nonprofits often are unfamiliar with

these costs. Given the central role that evaluation plays in the success of the Social Innovation Fund, please be 

advised that intermediaries and their subgrantees are expected to allocate appropriate resources to cover the 

many activities we have reviewed here today. 
2016 GRANT COMPETITION – SIF OVERVIEW WEBINAR TRANSCRIPT SOCIAL INNOVATION FUND - CLASSIC

nationalservice.gov/SIF 9

Finally, I really want to emphasize that CNCS is committed to supporting intermediaries in their efforts to meet

their goals and increase the evidence of program effectiveness within their portfolios. CNCS sees this process of 

ongoing evaluation and knowledge building that can improve grantee and subgrantee programs as a key aspect

of the Social Innovation Fund. 

I’ll now pause for questions. 

Please submit them via the Q&A

section on your screen or press *1 for

the operator. 

[Questions asked during the webinar 

were added to the Frequently Asked 

Questions page available at 

http://www.nationalservice.gov/buildyour-capacity/faqs] 

Now that I have covered the basics

around the role of evidence and

evaluation I will cover the remarks that 

Sarah Gallagher has prepared.

When we started on the SIF journey, 

CSH was looking to develop housing 

solutions for the cohort of homeless 

individuals with chronic health 

conditions who are super utilizers of 

crisis health services.

Our solution to this complex problem 

which we are testing through the SIF 

subgrantees in 4 distinct communities 

brings together the best of what we 

know works in ending homelessness with some of the most innovative solutions for improving health and 

lowering health care costs. We are bringing together supportive housing, using a housing first approach that 

helps people move directly into affordable housing and then offers voluntary services to support housing 

stability, coupled with data driven targeting to identify and engage super-utilizers. Additionally, the model is 

underscored with the added component of care coordination, patient navigation, and direct linkages to primary 

and behavioral health care.

Through SIF, CSH saw a rare opportunity to broaden the conversation around the integration of health and 

housing and build the business case to a scale our efforts through new financing mechanisms available via 

Medicaid.

The CSH SIF initiative is a five-year demonstration program where we will house 549 individuals in this 

integrated model.
2016 GRANT COMPETITION – SIF OVERVIEW WEBINAR TRANSCRIPT SOCIAL INNOVATION FUND - CLASSIC

nationalservice.gov/SIF 10

But our true goal in implementing SIF is to develop a model that we can replicate across the country... to create 

a blueprint for linking mainstream housing and health resources and scale the model.

This is what makes the evaluation component of SIF so crucial. Through analysis of administrative data, along 

with qualitative analysis, the evaluation will hopefully not only demonstrate positive impacts on individual lives 

but will also engage new systems, document the service delivery model and provide the lessons learned and 

policy recommendations needed to bring SIF to scale.

Given CSH’s desire to have one single evaluation that captured the impact of the model as well as its 

distinctions across all four sites, we decided to design what is called a “UniSEP” Unified Subgrantee Evaluation 

Plan.

While utilizing the UniSEP model allowed us to support the sub-grantees in a more proactive way and take 

some of the evaluation responsibilities off of them, it brought to light some key considerations that CSH needed 

to address around:

• Relate to the selection of Subgrantees;

• The internal management of the SIF evaluation; and

• Technical support needed by the SIF grantees.

When thinking about your capacity in each of these areas there are some questions that you might want to ask of 

your organization and your potential sub-grantees

• Who on your staff has extensive knowledge of evaluation research designs and implementation?

• To what extent can you support the development of evaluation plans for other organizations or

• Potential subgrantees?

• Have you conducted an RFP process before?

Based on your responses, you may need to think about centralizing some of the evaluation responsibilities 

within your organization and building the capacity of your organization in the areas you have identified.

At CSH, we dedicated a person to over-see the evaluation. Additionally each site has a TA liaison who works 

with them on-the ground to provide TA around implementation, evaluation and sustainability efforts.

Also, as part of your selection process, CSH recommends assessing the ability of each site to access the 

administrative other data you are prioritizing in your evaluation design as well as the evaluation readiness of 

potential subgrantees. This will allow you to gain an understanding of the organization’s evaluation capacity.

While CSH has always prioritized evaluation in its own initiatives, the SIF allowed CSH to heavily invest in 

evaluation for the first time and to local supportive housing providers to build their capacity to collect data and 

imbed evaluation in their work. For many of our subgrantees, this was the first time they participated in a RCT 

design evaluation and this was an opportunity to support and build their capacity in this area.

CSH took two important steps that have been crucial to its journey as a SIF intermediary: we contracted with an 

interdisciplinary team at NYU and we created an evaluation team to support the technical assistance with each 

of the subgrantees.

Through SIF, each subgrantee received direct support in implementing the evaluation design, including RCT, in 

accessing and collecting administrative and client level data and in using early findings to support both

implantation and scaling efforts.

Internally, CSH meets monthly with all staff working on the SIF to share updates, problem solve and coordinate 

across all aspects of the effort and we meet biweekly with the NYU team to ensure successful roll out of the 
2016 GRANT COMPETITION – SIF OVERVIEW WEBINAR TRANSCRIPT SOCIAL INNOVATION FUND - CLASSIC

nationalservice.gov/SIF 11

evaluation. In this call, we provide status updates from both CSH and NYU perspective, talk about data 

collection and quality assurance related to data and talk through the analysis and timeline. We also field sitespecific requests, coordinate on presentations and ensure that we have real time interim findings that we can 

communicate out.

Each subgrantee had support from both the CSH technical assistance team as well as direct support from the 

NYU evaluation team through regular webinars, one on one calls and even an annual site visit.

Our evaluation partner is also a key participant in our annual subgrantee convening.

This direct connection between the evaluator and the sub-grantees has been extremely valuable in building 

subgrantee investment in the evaluation.

Just to briefly summarize, I know we’ve 

gone through a lot quickly:

First, SIF funds promising interventions

or program models with at least

preliminary evidence that can advance to

moderate if not strong; it is not intended

for start-ups.

Second, we are funding experienced

intermediaries that have the

organizational capacity to adequately

support and implement rigorous

evaluations. And you heard some of that 

from Lois’ remarks.

And third, the budget needs to be adequate to fund an intervention and a rigorous evaluation and, if appropriate,

there needs to be sufficient budget for subgrantee support.

There’s more information on this in the NOFA; I think pages 16 to 17 have more in terms of the evaluation 

capacity. If you have any questions, I’m happy to answer them.

[Questions asked during the webinar were added to the Frequently Asked Questions page available at 

http://www.nationalservice.gov/build-your-capacity/faqs]

So one of the things we have now that 

we didn’t have before is the Evidence 

Exchange where you can see some final 

reports that have come about through 

some of the grantees who have finished. 

So I would definitely say to look at that 

and see what that looks like. And again, 

these kinds of resources, such as the 

rubric, see what it means to have an 

incoming level of evidence, as well as 

the budget paper that spells out, again as 

Lois said, the early years but you can 

still see what people have, on average, 

spent. As well as the NOFA in terms of 
2016 GRANT COMPETITION – SIF OVERVIEW WEBINAR TRANSCRIPT SOCIAL INNOVATION FUND - CLASSIC

nationalservice.gov/SIF 12

what would be expected because these are rigorous evaluations, and again you don’t want to come in with an 

intervention that hasn’t been evaluated before. 

We have some key documents and resources listed on the PowerPoint. 

So in particular we wanted to draw your attention to the evaluation resources: the rubric that Andrea covered in 

her presentation; the evaluation planning process guide; and then the evaluation budgeting guide which I think 

you’ll find helpful. And then we also have the SIF Evaluation Plan Guidance, please note however that’s a very 

lengthy document that will be your key resource if you are approved for funding. If you wanted to flip through 

it, it gives you what will be expected of you. 

And then we’ll end up with the how to 

reach us.

SIFApplication@cns.gov

Leave a message at 202.606.3223 

Or you can sign up on our website for 

information and updates on SIF. 

Thank you everybody. We appreciate 

your attendance and as Lois said, don’t 

hesitate to contact us any of those 

different ways if you have any 

questions.