U.S. flagAn official website of the United States government
        Here’s how you know
    Goverment symbol**Official websites use .gov**
            A **.gov** website belongs to an official government organization in the United States.
        Lock symbol**Secure .gov websites use HTTPS**
            A lock (
              A locked padlock
                 ) or **https://** means you’ve safely connected to the .gov website. Share sensitive
              information only on official, secure websites.
 Skip to main content JoinDonate
    About
    About
          About
        
About menu block one
  

        
        What We Do
                Our Impact
                Agency FAQs
About menu block two
  

        
        Our Team
                Careers
About menu block three
  

        
        Agency Overview
                            Budget and Performance Plans
                Disability and Accessibility
                Federal Register Notices
                Open Government
                Plain Writing
                Privacy Policy
                Strategic Plan

  

  
Serve
    Serve
          Serve
        
Serve menu block one
  

        
        AmeriCorps
                            AmeriCorps NCCC
                AmeriCorps State and National
                AmeriCorps VISTA
                AmeriCorps Seniors
                            AmeriCorps Seniors Foster Grandparent Program
                AmeriCorps Seniors RSVP
                AmeriCorps Seniors Senior Companion Program
Serve menu block two
  

        
        Volunteer
                            9/11 Day
                MLK Day
Serve menu block three
  

        
        Find Your Fit
                Serve FAQs

  

  
Partner
    Partner
          Partner
        
Partner menu block one
  

        
        How it Works
                            AmeriCorps NCCC
                AmeriCorps Seniors Foster Grandparent
                AmeriCorps Seniors RSVP
                AmeriCorps Seniors Senior Companion
                AmeriCorps State and National
                AmeriCorps VISTA
                Research and Evaluation
                Volunteer Generation Fund
Partner menu block two
  

        
        Partnerships
                Communities and Initiatives
                Funding Opportunities
Partner menu block three
  

        
        Partner FAQs

  

  
Members & Volunteers
    Members & Volunteers
          Members & Volunteers
        
Members & Volunteers menu block one
  

        
        Your Benefits
                Segal AmeriCorps Education Award
                Education Award FAQs
Members & Volunteers menu block two
  

        
        Alumni
                Members in VISTA
Members & Volunteers menu block three
  

        
        Member and Volunteer FAQs
                Safety and Security

  

  
Grantees & Sponsors
    Grantees & Sponsors
          Grantees & Sponsors
        
Grantee & Sponsors menu block one
  

        
        State Service Commissions
                State and National Direct Grantees
                State Subgrantees
                VISTA Sponsors
                Research and Evaluation Grantees
                Volunteer Generation Fund and Days of Service Grantees
Grantee & Sponsors menu block two
  

        
        RSVP Grantees
                Foster Grandparent Program Grantees
                Senior Companion Program Grantees
                Senior Demonstration Program Grantees
                NCCC Project Sponsors
Grantee & Sponsors menu block three
  

        
        Manage Your Grant
                Grantee and Sponsor FAQs

  

  
Newsroom
    Newsroom
          Newsroom
        
Newsroom menu block one
  

        
        News
                Blog
Newsroom menu block two
  

        
        Stories
                Events
Newsroom menu block three
  

        
        In the News
                Communication Resources



  
Contact
    Contact
    JoinDonate
search Breadcrumb
    Grantees and Sponsors
                                                      Evidence Readiness Resources
 Scaling Checklists: Assessing Your Level of Evidence and Readiness (SCALER)
 Submitted by 
kwilkerson
 on 
Tue, 12/28/2021 - 17:21Current
            Overview
Step 1: Is there evidence your intervention is effective?
Overview Overview
	Identifying Evidence of Effectiveness Is there evidence your intervention is effective?
	Results Results
SCALER Step 1: Overview
The first step in the SCALER is to answer the question “Is there evidence that my intervention is effective?” The answer to this question is important because it determines what your organization needs to do to prepare to scale an intervention. If the intervention does not have evidence that it is effective, your organization should build that evidence before scaling. If the intervention has evidence that it is effective, your organization can then determine its readiness to scale the intervention.
Intervention evidence components
	Research
	rigor
	Intervention
	effectiveness
	Intervention
	relevance
Research rigor

Rigor refers to having a well-designed and well-implemented impact study that clearly separates the effects of an intervention from other factors that may have affected the outcomes. This also means impartial researchers have designed and conducted the study to ensure independence of the study findings.
Intervention effectiveness

An intervention is shown to be effective when rigorous research demonstrates favorable impacts on the targeted outcomes for people receiving the intervention services.
Intervention relevance

Relevance refers to whether the research findings are generalizable to the context in which your intervention will be scaled.
Start
FAQ
Who should complete this step?We recommend designating one person or role to be the SCALER owner, such as the intervention’s program or data manager. Because personnel at all levels are involved in intervention scaling, the SCALER owner will want to involve others including:

Executive directors
	Program managers or frontline supervisors
	Measurement, learning, and evaluation personnel (data managers or internal evaluation specialists)What information do I need?Completing Step 1 requires gathering knowledge about any research studies that have been conducted on your intervention. You can draw on rich information from external sources, such as the research clearinghouses listed on the SCALER Resources page, to understand if existing research shows that an intervention you want to scale has already been proven effective.How should I complete Step 1?Organizations can take one of several approaches:

Involved personnel could complete the checklist on their own, discuss responses, and come to agreement on scores at a meeting organized by the SCALER owner.
	Several personnel could fill out the checklist together during a meeting.
	The SCALER owner could gather the necessary information and complete the checklist.In taking any of these approaches, the SCALER owner should set up a timeline for completing the tool.How can I use the Step 1 results?The printable Step 1 Results report summarizes whether evidence of effectiveness exists for your intervention. With this summary, you can:

Share the results with key stakeholders, organizational leaders, and funders.
	Use the results to identify limitations in the research that has been conducted on your intervention.
	Periodically search for research on your intervention and update Step 1 if you find studies that potentially demonstrate your intervention’s effectiveness. If you cannot find such studies, use Step 2 to help build such evidence.
FAQ
Why is evidence of intervention effectiveness important?Because intervention scaling aims to improve lives for larger numbers of people and communities, organizations should only seek to scale interventions that they expect will maintain or surpass their beneficial impacts for participants after scaling occurs. Rigorously conducted evaluations can establish causal impacts—meaning that well-designed and implementation impact evaluations can show that the intervention, and not other factors, led to improvements in participants’ outcomes.Where can I find information about whether evidence of effectiveness exists for your intervention?Research clearinghouses provide information to help organizations identify research on the effectiveness of a wide variety of interventions. Although the quality of evaluation research varies greatly, the research uses common principles that define the level of rigor that a study demonstrates to show that an intervention caused beneficial outcomes for participants. The SCALER Resources lists various research clearinghouses that you can visit to see if evidence of effectiveness already exists for your intervention.
Use this checklist to assess whether your intervention has evidence of effectiveness. Under “Research rigor,” “Intervention effectiveness,” and “Intervention relevance,” select the items that reflect the nature of the research that has been conducted on your intervention. Should one or more of the items in the checklist remain unchecked, it means your organization does not yet have evidence of your intervention’s effectiveness, and such research evidence must be built.
Research rigor?
  A third party conducted the research
          This means an independent evaluator who is external to your organization or the intervention and who has no vested interest in the evaluation findings conducted the research.
The research included a counterfactual
          This means the research design included a comparison group, which shows what participants’ outcomes would have been had they not received the intervention.
The research had no confounding factors
          This means the research design allowed the evaluators to distinguish between changes in outcomes that were due to the intervention and not to other factors.
The research had low attrition, no reassignment, and baseline equivalence with the final sample
          This means few people in the treatment or comparison group left the study (low attrition), no people in the comparison group switched to the treatment group or vice versa (no reassignment), and the treatment and comparison groups used in the analysis did not differ on key characteristics at the start of the study, other than one group received access to intervention services and the other did not (baseline equivalence with the final sample).Intervention effectiveness?
  The research showed evidence of effectiveness
          This means the amount (such as number of studies) and rigor of the research on your intervention is adequate for you to consider the intervention effective.Intervention relevance?
  The research is appropriate to the context or environment in which scaling is planned
          For example, the locations or populations that were used to establish evidence for the intervention’s effectiveness are similar to or relevant to the plans you have for scaling your intervention.Back

Next
SCALER Step 1: Results
summary_results_output
            
    
    
    


  
  
  
  


  


  

Evidence of your intervention’s effectiveness that aligns to your plans for scaling does not yet exist. Your organization will need to build such evidence.

Before your organization invests its resources to scale an intervention, the intervention should first have demonstrated effectiveness. Such evidence should be based on rigorously conducted research that demonstrates that the intervention itself contributed to changes in participant outcomes. The evidence must also be relevant to the plans your organization has for scaling the intervention so your organization can expect the intervention will maintain its effectiveness when at scale. Because you were unable to identify any existing evidence that meets these standards, you will need to complete Step 2 of the SCALER to help build such evidence.

Research Rigor does not exist
Intervention Effectivenessdoes not exist
Intervention Relevancedoes not exist

What do your results mean?
Return to edit

Research Rigor
**A third party did not conduct the research**If internal personnel conducted an evaluation on the intervention you are seeking to scale, consider having a third-party evaluator conduct an evaluation as well. Using a third-party evaluator with no vested interest in the intervention to conduct the research study provides credibility and objectivity to the evaluation findings.

**Research did not include a counterfactual**Because the research design for the intervention did not include a comparison group as a counterfactual, it is difficult to conclude what participants’ outcomes would have been had they not received the intervention. If you are seeking to build evidence for the intervention in the future, work with a third-party evaluator to use a randomized controlled trial or a matched comparison group in your research design to create a counterfactual.

**Research had confounding factors**If confounding factors were present in the evaluation of the intervention, it is difficult to conclude that any changes in participants’ outcomes were due to the intervention and not to other factors. If you seek to build evidence for the intervention in the future, view the SCALER Resources for more information on common confounding factors so you and your third-party evaluator can design a study that will avoid them.

**Research did not have low attrition, reassignment was used, and baseline equivalence did not exist with the final sample**The presence of these factors shows that the treatment and comparison groups used in the final sample were similar to each other before the treatment group participants received the intervention. If differences between the two groups exist, then such differences, and not the intervention itself, can affect study outcomes. If you seek to build evidence for the intervention in the future using a third-party evaluator, view the SCALER Resources for more information on these factors to understand how a study should be designed and implemented to isolate the impacts of your intervention.intervention effectiveness

**The research did not show evidence of effectiveness**If research has not shown the intervention to be effective, then it is not yet ready to scale. Review the conducted research to identify possible explanations for why it did not show evidence of the intervention’s effectiveness—for example, the evaluation might not have contained a large enough sample size to demonstrate impacts or the intervention was not implemented as intended—and consider how to improve upon these factors when seeking to build your own evidence in the future.intervention relevance

**The research is not appropriate to the context or environment in which scaling is planned**This means the intervention might not be as effective when used with populations or in locations that are different from what was included in research that showed the intervention’s effectiveness. If this occurs, you may need to build evidence for the intervention using populations or locations that ultimately align with your plans to scale the intervention.
step1_result_guidance
        You have completed Step 1 of the SCALER. Based on your results, rigorous evidence that shows your intervention’s effectiveness and is relevant to your plans to scale the intervention does not yet exist. Proceed to Step 2 of the SCALER, which aims to help your organization build evidence for your intervention.

Back

Print Step 1 Results

continue to Step 2

Need to finish later?

Update or view your checklists in the future with your unique link:

Copy my unique Step 1 link to the clipboard

Email my unique Step 1 link.

Copy my unique Step 1 link to the clipboard


return home view SCALER resources
 Back to main content Sign up to hear about our latest news, events, and opportunities.
  Sign up for our newsletter
    Footer about
  

        
        About
                            What We Do
                Our Impact
                Our Team
                Careers
                Agency OverviewFooter serve
  

        
        Serve
                            AmeriCorps
                AmeriCorps Seniors
                Volunteer
                Find Your FitFooter partner
  

        
        Partner
                            How it Works
                Funding OpportunitiesFooter Members & Volunteers
  

        
        Members & Volunteers
                            Your Benefits
                Segal AmeriCorps Education Award
                Alumni
                Safety and SecurityFooter grantees & sponsors
  

        
        Grantees & Sponsors
                            eGrants
                History Checks
                LitmosFooter newsroom
  

        
        Newsroom
                            News
                Blog
                Stories
                Events
                In the News
                Communication Resources
      Bringing Americans together to serve communities

Follow us
  
  1-800-942-2677
            
    help@americorps.gov
    USA.gov / Office of Inspector General / FOIA and Privacy Act / Federal Register notices / No FEAR Act/ Employee Whistleblower Rights / Contractor and Grantee Whistleblower Rights / Office of Special Counsel / STOCK Act / Fraud Alert

Accessibility / Privacy policy / Site map / Donate / Join

**AmeriCorps.gov**
An official website of AmeriCorps. Produced and published at taxpayer expense. 
 You are leaving AmeriCorps.gov
 The website you’re going to is not part of the AmeriCorps domain and may not be under AmeriCorps’ control. Its privacy and security practices and policies may differ from AmeriCorps’. AmeriCorps is not responsible for the link, nor does it endorse the content of the third-party website. Click on the continue button to proceed to the external website; otherwise click cancel to stay on AmeriCorps.gov.
 Cancel Continue