Version 12.19.14, Office of Research and Evaluation, CNCS

Social Innovation Fund’s Rubric

for Identifying Subgrantee Applicants’ Incoming Level of Evidence

This rubric is intended to be a tool to assist Social Innovation Fund’s (SIF) intermediaries in 

assessing subgrantee applicants’ incoming level of evidence (i.e., Preliminary, Moderate, Strong) 

of their proposed interventions. The process of determining the incoming and final level of 

evidence1for an intervention requires thoughtful assessment that can be complicated and often 

requires technical knowledge of research and evaluation design and methods. The rubric 

introduced in this document provides a framework for assessing the existing body of evidence 

behind an intervention based on past research and evaluation studies conducted on the program. 

If you have any questions on how to use this rubric or how you are assessing proposed 

interventions, please feel free to reach out to us (i.e., staff from the Office of Research and 

Evaluation or SIF POs) and we will work with you in consultation with the JBS to answer them.

I. ATTAINING A LEVEL OF EVIDENCE

As described in the SIF NOFA, all SIF funded interventions require at least a Preliminary level 

of evidence upon entering the SIF. This level of evidence will be demonstrated by studies 

conducted prior to applying for SIF funding. 

 To attain the Preliminary level of evidence required for SIF funding, an intervention 

must, at a minimum, have a study that has “yielded promising results for either the 

program or a similar program.” Specifically, the program must have at least some 

outcome information such as pre- and post-tests without a comparison group, or post-test 

comparison between program and comparison groups.

 To attain a Moderate level of evidence, an intervention needs to have evidence “from 

studies whose designs can support causal conclusions (i.e., studies with high internal 

validity2), but have limited generalizability (i.e., moderate external validity3), or studies 

with high external validity, but moderate internal validity.”4 Studies with high internal 

validity will likely use Quasi-experimental Designs (QED) (such as a matched 

comparison group or a comparative interrupted time series design) or Randomized 

 

1CNCS/SIF and JBS team conduct an assessment of the incoming final level of evidence of all interventions 

funded under the SIF grant and classify the interventions according to that level of evidence. The program ensures

that the evaluation planned and implemented under SIF advances that intervention’s base of evidence and targets 

moderate or strong level of evidence.

2

Internal validity for a study is the extent to which the observed difference in the average group outcomes 

(usually program participants versus control or comparison group members) can be causally attributed to the 

intervention or program.

3External validity for a study is the extent to which evaluation results are applicable to groups other than 

those in the research.

4Moderate internal validity could come from a study having a comparison group formed without statistical 

matching techniques, statistical matching techniques that resulted in lower than desirable pre-test group equivalence 

or an interrupted time series design without a comparison group.
Version 12.19.14, Office of Research and Evaluation, CNCS

Controlled Trials (RCT) also known as Experimental Designs. At least one study with 

high internal or external validity is typically needed to attain a Moderate level of 

evidence.

 To attain a Strong level of evidence, an intervention should have designs that “can 

support causal conclusions (i.e., studies with high internal validity), and studies that in 

total include enough of the range of participants and settings to support scaling up to the 

state, regional, or national level (i.e., studies with high external validity).” Interventions 

that enter the SIF with a Strong level of evidence would have conducted either one large, 

multisite RCT or QED study or several smaller RCT or QED studies either in different 

locations or with different populations.

II. ASSESSING INCOMING LEVEL OF EVIDENCE

Although there are several factors to consider when assessing a subgrantee’s incoming level of 

evidence, this rubric focuses on two important sets of factors: 1) the similarity of the proposed 

SIF intervention to the previous studied intervention(s) in terms of where and how they were 

implemented (see Income Evidence Review Rubric); and, 2) type of study or studies conducted

which yielded positive results.

1.) Similarity of the proposed SIF interventions to previously studied interventions: Identify 

how the previously studied interventions relate to the intervention being proposed by the 

SIF applicant. Specifically, assess how closely the interventions studied match the 

proposed intervention in terms of two dimensions: 

a) Was the intervention implemented by the applying organization or a different 

one?; and,

b) How closely matched is the studied intervention to the proposed intervention?

Was the studied intervention identical or very similar to the proposed intervention 

in terms of content, delivery or target population, or was it substantially modified, 

adapted, or combined with other interventions?

2) Type of study conducted which yielded positive results: Identify which types of

research or evaluation designs have been used in prior studies which yielded positive 

results of the proposed intervention.

 What types of studies showed positive results rather than null or negative, results 

for the outcomes targeted by the applicant program?

o For example, as you read through a subgrantee’s application material, do 

they cite positive results from studies that have used designs such as a preand post-tests with a single group? Do they cite studies that have used a 

study with a matched comparison group? Do they report on a randomized 

controlled trial? 
Version 12.19.14, Office of Research and Evaluation, CNCS

III. ISSUES TO CONSIDER

Additional issues to consider when assessing incoming evidence:

1) Adapting an intervention or combining multiple evidence-based interventions may lower 

incoming level of evidence for the “new” intervention.

2) Even with the same study design (e.g., a single site RCT), a proposed intervention using 

evidence from studies of a similar intervention may have a lower incoming level of 

evidence than an intervention using studies from the identical intervention.

3) Unless an intervention is being intentionally replicated with fidelity, studies for the same 

intervention conducted by a different program or organization may also offer lower levels 

of evidence than studies conducted by the proposing subgrantee.

4) A study or studies (i.e., single group pre-post test) conducted in a different organizational 

context than the one being proposed does not likely have sufficient evidence to be 

considered preliminary under SIF standards. This is due to the fact that the preliminary 

evaluations such as single group pre-post tests do not have sufficient internal validity to 

show that the program “causes” the outcome. For studies that only offer pre-post testing, 

it is possible that something in the program context other than the intervention (e.g. how 

they select their participants) may be causing the changes seen by that program.

IV. USING THE RUBRIC

To use the Incoming Evidence Review Rubric on the next page and find an intervention’s 

incoming level of evidence:

1. Review each study the subgrantee offers as incoming evidence for their proposed 

intervention. Identify the studies that generally show positive, rather than null or 

negative, results for the outcomes targeted by the applicant’s program.

2. Determine the connection of the subgrantee’s proposed intervention to the studied 

intervention and use the labels in the top row “Similarity to Proposed Intervention” to 

select the column that best represents how that study relates to the proposed intervention.

3. Put a check in the box(es) of the column selected in step 2 above, where it intersects with 

the row for the design type used in the study from the choices in the left hand column

“Study Design”.

4. Review each study and check each relevant box until the reviewers have exhausted the 

studies and can clearly see where it all lands in terms of level of evidence.

5. After following this procedure for each study cited by the grantee, the highest-ranked 

checked box (e.g. Preliminary, Moderate, Strong) is the incoming level of evidence for 

the subgrantee’s proposed intervention.
Version 12.19.14, Office of Research and Evaluation, CNCS

Incoming Evidence Review Rubric

Applicant/Subgrantee___________________________________ Highest Ranked Checked Box:______________________________

*(To be designated Strong+ and to be exempted from the requirement to attain moderate evidence with their SIF evaluation, the program would need an extensive, multi-site history of 

RCT’s/QED’s with the population in question.)

Similarity to Proposed Intervention:

How matched is the studied intervention to the proposed Intervention 

by organization and similarity to intervention? Was it done by:

A different 

organization 

doing a similar,

but not identical 

intervention?

A different organization 

doing an identical 

intervention? (The 

proposed intervention will 

be replicated with fidelity)

The same organization 

doing a combination of 

interventions that 

include the one 

studied?

The same organization 

doing an intervention that 

is similar, but not 

identical to the studied 

intervention?

The same 

organization, doing 

exactly the same 

intervention?

What type of design was used for studies where the results on relevant 

outcomes were positive?

None or none known Not yet preliminary

(__)

Not yet preliminary

(__)

Not yet preliminary

(__)

Not yet preliminary

(__)

Not yet preliminary

(__)

Implementation only Not yet preliminary

(__)

Not yet preliminary

(__)

Not yet preliminary

(__)

Not yet preliminary

(__)

Not yet preliminary

(__)

Pre-post testing Not yet preliminary

(__)

Not yet preliminary

(__)

Not yet preliminary

(__)

Not yet preliminary/ 

Preliminary (Depending on 

the extent of similarity)

(__)

Preliminary

(__)

Pre-post or post only with nonmatched comparison group, or 

interrupted time series with no 

comparison group

Not yet preliminary

(__)

Not yet preliminary

(__)

Preliminary

(__)

Preliminary

(__)

Preliminary

(__)

Single site, well designed and 

implemented QED or RCT 

Preliminary

(__)

Preliminary

(__)

Preliminary

(__)

Preliminary

(__)

Moderate

(__)

Two or three well designed and 

well implemented single site RCTs

or QEDs

Preliminary

(__)

Moderate

(__)

Preliminary

(__)

Preliminary/Moderate 

(Depending on extent of 

modification)

(__)

Moderate

(__)

National/large scale multi-site well 

designed and well implemented 

QED or RCT, or multiple (three or 

more) well designed and well 

implemented QEDs or RCTs in 

different locations

Preliminary

(__)

Strong*

(__)

Preliminary

(__)

Preliminary/Moderate 

(Depending on extent of 

modification)

(__)

Strong*

(__)