U.S. flagAn official website of the United States government
        Here’s how you know
    Goverment symbol**Official websites use .gov**
            A **.gov** website belongs to an official government organization in the United States.
        Lock symbol**Secure .gov websites use HTTPS**
            A lock (
              A locked padlock
                 ) or **https://** means you’ve safely connected to the .gov website. Share sensitive
              information only on official, secure websites.
 Skip to main content JoinDonate
    About
    About
          About
        
About menu block one
  

        
        What We Do
                Our Impact
                Agency FAQs
About menu block two
  

        
        Our Team
                Careers
About menu block three
  

        
        Agency Overview
                            Budget and Performance Plans
                Disability and Accessibility
                Federal Register Notices
                Open Government
                Plain Writing
                Privacy Policy
                Strategic Plan

  

  
Serve
    Serve
          Serve
        
Serve menu block one
  

        
        AmeriCorps
                            AmeriCorps NCCC
                AmeriCorps State and National
                AmeriCorps VISTA
                AmeriCorps Seniors
                            AmeriCorps Seniors Foster Grandparent Program
                AmeriCorps Seniors RSVP
                AmeriCorps Seniors Senior Companion Program
Serve menu block two
  

        
        Volunteer
                            9/11 Day
                MLK Day
Serve menu block three
  

        
        Find Your Fit
                Serve FAQs

  

  
Partner
    Partner
          Partner
        
Partner menu block one
  

        
        How it Works
                            AmeriCorps NCCC
                AmeriCorps Seniors Foster Grandparent
                AmeriCorps Seniors RSVP
                AmeriCorps Seniors Senior Companion
                AmeriCorps State and National
                AmeriCorps VISTA
                Research and Evaluation
                Volunteer Generation Fund
Partner menu block two
  

        
        Partnerships
                Communities and Initiatives
                Funding Opportunities
Partner menu block three
  

        
        Partner FAQs

  

  
Members & Volunteers
    Members & Volunteers
          Members & Volunteers
        
Members & Volunteers menu block one
  

        
        Your Benefits
                Segal AmeriCorps Education Award
                Education Award FAQs
Members & Volunteers menu block two
  

        
        Alumni
                Members in VISTA
Members & Volunteers menu block three
  

        
        Member and Volunteer FAQs
                Safety and Security

  

  
Grantees & Sponsors
    Grantees & Sponsors
          Grantees & Sponsors
        
Grantee & Sponsors menu block one
  

        
        State Service Commissions
                State and National Direct Grantees
                State Subgrantees
                VISTA Sponsors
                Research and Evaluation Grantees
                Volunteer Generation Fund and Days of Service Grantees
Grantee & Sponsors menu block two
  

        
        RSVP Grantees
                Foster Grandparent Program Grantees
                Senior Companion Program Grantees
                Senior Demonstration Program Grantees
                NCCC Project Sponsors
Grantee & Sponsors menu block three
  

        
        Manage Your Grant
                Grantee and Sponsor FAQs

  

  
Newsroom
    Newsroom
          Newsroom
        
Newsroom menu block one
  

        
        News
                Blog
Newsroom menu block two
  

        
        Stories
                Events
Newsroom menu block three
  

        
        In the News
                Communication Resources



  
Contact
    Contact
    JoinDonate
search Breadcrumb
    Grantees and Sponsors
                                                      Evidence Readiness Resources
 Scaling Checklists: Assessing Your Level of Evidence and Readiness (SCALER)
 Submitted by 
kwilkerson
 on 
Tue, 12/28/2021 - 17:24Current
            Wizard Page
Step 2: Building evidence of effectiveness
Overview Overview
	Building Evidence of Effectiveness Building Evidence of Effectiveness
	Results Results
SCALER Step 2: Overview
If an intervention does not have evidence that it is effective, your organization can draw on Step 2 of the SCALER to build that evidence before scaling. This step consists of a process—grounded in standards of rigorous research and evaluation—that you can use to gain knowledge about an intervention’s implementation and impact.

This process begins with clearly defining the intervention and the path it takes to affect outcomes. Once the intervention is implemented consistently and has suggestive evidence that participants’ outcomes improve after receiving the intervention, the process moves to designing and implementing an impact evaluation. This requires executing an evaluation that meets standards for research rigor and demonstrates the intervention’s effectiveness. If the evaluation shows evidence of effectiveness, the intervention is ready to scale. If it does not, you will tweak the intervention in ways to increase its likelihood of being effective, and the evidence-building process begins again.
Building evidence components
	Define
	intervention
	Lay evaluation
	groundwork
	Prepare for
	evaluation
	Conduct evaluation and
	assess effectiveness
Define intervention

For an organization to implement an intervention consistently and reliably produce beneficial outcomes for participants, the intervention must be (1) clearly defined, (2) implemented as part of the organization’s regular routine, and (3) and implemented according to the defined program model.
	Additionally, any modifications the organization makes to the intervention model should be informed by research.
Lay evaluation groundwork

Building research evidence and using it to improve an intervention requires a culture of measurement, learning, and evaluation. In such a culture, an organization is willing to apply what it learns to improve its intervention’s performance.
	Stakeholders should agree on the need for evidence, the outcomes and measures to include in an evaluation, and the overall goals of an evaluation, along with generally supporting the evaluation.
Prepare for evaluation

Working with third-party evaluators to conduct a rigorous evaluation of an intervention’s effectiveness requires adequate financial resources and personnel.
	Adequate resources should cover costs associated with implementing the evaluation and organizational personnel to support the evaluation’s implementation.
Conduct evaluation and assess effectiveness

Once an organization has prepared for its evaluation, it can work with the third-party evaluators to conduct the study.
	Ideally, the study should be rigorous enough to allow the organization to assess whether its intervention has evidence of effectiveness (as discussed in Step 1). The study should also be relevant to the context in which the organization plans to scale the intervention.
Start
FAQ
Who should complete this step?We recommend designating one person or role to be the SCALER owner, such as the intervention’s program or data manager. Because personnel at all levels are involved in intervention scaling, the SCALER owner will want to involve others including:

Executive directors
	Program managers or frontline supervisors
	Measurement, learning, and evaluation personnel (data managers or internal evaluation specialists)
	Administrative personnel (human resources, development, or purchasing personnel)What information do I need?Completing Step 2 (building evidence) requires enough knowledge and understanding of the following aspects of the intervention and your organization:

How the intervention was defined and implemented at the organization, which includes the intervention’s core activities and the resources needed to carry out these activities
	The organization’s support for conducting an evaluation of the intervention
	The availability of organizational resources and personnel to support an evaluation
	Familiarity with how research should be conducted in a manner that meets standards for rigorHow should I complete Step 2?Organizations can take one of several approaches:

Involved personnel could complete the checklist on their own, discuss responses, and come to agreement on scores at a meeting organized by the SCALER owner.
	Several personnel could fill out the checklist together during a meeting.
	The SCALER owner could gather the necessary information and complete the checklist.How can I use the Step 2 results?The printable Step 2 Results report summarizes whether evidence of effectiveness exists for your intervention. With this summary, you can:

Share the results with key stakeholders, organizational leaders, and funders.
	Use the results to identify areas that need further attention, resources, or support as you work on building evidence for your intervention and prioritize next steps.
	To gauge your progress, complete Step 2 multiple times as you prepare and continue your efforts to build evidence for your intervention’s effectiveness.
FAQ
How can a theory of change (TOC) help my organization?A well-articulated TOC establishes the inputs, activities, outputs, and outcomes needed for the intervention to succeed. Identifying these components in a TOC provides a clear sense of what is needed to produce outputs and what needs to be measured to determine that the intervention improves participant outcomes.Why is building stakeholder agreement on the need for evidence important?It is critical that all stakeholders, including organizational leaders and personnel involved in implementing the intervention, support research and evaluation efforts before they are designed and begin. This helps ensure everyone involved in the evaluation will contribute accurate and reliable information about the intervention. If stakeholders are not willing and able to participate in an evaluation, the evaluation will likely not be feasible or as useful as possible, even if it is conducted.What costs and resources are associated with preparing for evaluation?Planning for and allocating sufficient resources to evaluation is important. Not allocating sufficient funding to evaluation activities could lead to less rigorous or poorly implemented research that does not meet the standards of rigor for showing an intervention is effective. Visit the SCALER Resources for more information on the evaluation-related costs to consider and how costs can vary depending on the scope of the evaluation.Who should conduct the evaluation?A third-party evaluator should be engaged to conduct a rigorous evaluation of intervention effectiveness. A third-party evaluator should be objective and not have a vested interest in whether the intervention produces beneficial impacts for participants. When the evaluation is being implemented, organizational personnel and evaluators should work together to make sure the research tasks are carried out as planned.Why do some evaluations show no impacts?Three reasons might explain why evaluation efforts do not show evidence of effectiveness:
Method failure occurs when a study’s design or implementation was flawed (for example, the sample size was too small to show statistically significant impacts). If this happens, your organization should conduct a new study that addresses the flaw (for example, by increasing the sample size).
	Implementation failure occurs when an intervention is not implemented as intended (that is, without fidelity to the model). If this happens, your organization should take steps to improve implementation fidelity before conducting a new study.
	Theory failure occurs when the intervention is not actually linked to the desired outcome. If this causal link does not exist, revisit the intervention or key components.
Use this checklist as you seek to build evidence of effectiveness for your intervention. Under “Define intervention,” “Lay evaluation groundwork,” “Prepare for evaluation,” and “Conduct evaluation and assess effectiveness,” select the items that reflect the status of your efforts in building evidence.

Ideally, you would make sure all items in a section are complete before moving to the next section. Ultimately, should the intervention be shown to be effective, you would get ready to scale the intervention (see Step 3 of the SCALER). Should the evidence not indicate effectiveness, the organization should continue to build evidence.
Define intervention?
  Defined theory of change (TOC) for the intervention
          This means there is a clear description of the human, financial, and physical resources (inputs) and the intervention’s core components (activities) that are necessary to change participants’ outcomes.
Intervention mature and stable
          This means the organization implements the inputs and activities identified in the TOC as part of a regular organizational routine, and personnel implementing the intervention are proficient and skilled and have managerial and administrative support.
Intervention implemented as described in TOC (with fidelity)
          This means the intervention is delivered according to the defined program model.
Intervention, including any modifications, informed by research
          This means modifications made to an intervention in the hopes of improving participants’ outcomes are based on other existing research and evidence.Lay evaluation groundwork?
  Agreement among organization, partners, and funders on need for evidence
          This means, for example, stakeholders understand the value of using a TOC to improve the design and implementation of an intervention and, subsequently, generating estimates of the intervention’s impact.
Agreement among organization, partners, and funders on outcomes and measures
          This means there is a consensus among stakeholders on the outcomes the intervention is trying to influence and the goals for the extent to which outcomes should be improved.
Agreement among organization, partners, and funders on evaluation goals
          This means the stakeholders agree on the intervention’s TOC, implementation goals for the intervention (outputs and outcomes), research questions for the evaluation, and what constitutes evidence of effectiveness.
Organization, partners, and funders support evaluation
          This means that stakeholders, including organizational leaders and personnel involved in implementing the intervention, support the evaluation efforts before they begin.Prepare for evaluation?
  Financial resources available
          This means there are resources to cover costs associated with implementing the evaluation such as contracting with third-party evaluators, purchasing and preparing systems to record data that will be analyzed for the evaluation, paying fees for other experts in research design and implementation, and accounting for personnel time in collecting data for the evaluation.
Personnel available to work with third-party evaluators
          This means, for example, that organization personnel help third-party evaluators understand the inputs and activities identified in the intervention’s TOC, work collaboratively with the evaluators during the evaluation’s implementation, and provide insights to support the evaluation’s data analysis.Conduct evaluation and assess effectiveness?
  Research conducted by third-party evaluators in a manner that meets standards for rigor
          This means having a well-designed and well-implemented impact study conducted by an independent evaluator (see Step 1 for standards to identify effective interventions).
Research showed beneficial results for participants
          This means the intervention is shown to be effective for improving participant outcomes.
Context or environment in which the research was conducted is generalizable to the organization’s plans for scaling its intervention
          This means the evidence for the intervention’s effectiveness is applicable to the locations where or populations for which your organization plans to scale its intervention.Back

Next
SCALER Step 2: Results
summary_results_output
            
    
    
    
    
 
Your organization should continue its efforts to build evidence of your intervention’s effectiveness that also aligns to your plans for scaling.

Before your organization invests resources to scale an intervention, the intervention should have demonstrated evidence of effectiveness. To build such evidence, the intervention’s model must be defined and stable, and informed by research, and your organization must implement the intervention with fidelity to that model. Once that condition has been met, your organization can then lay the groundwork and prepare for conducting an evaluation of your intervention. After laying the groundwork and preparing for evaluation, your organization can find and work with a third-party evaluator to carry out a rigorous research study that demonstrates the intervention’s effectiveness and aligns to your plans for scaling the intervention. Use the guidance in the results below to identify the areas that need to be addressed to build evidence. View the SCALER Resources for more information that can help your evidence-building efforts and return to Step 2 of the SCALER to document any progress you make.

Define interventioncontinue making progress
Lay evaluation groundworkcontinue making progress
Prepare for evaluationcontinue making progress
Conduct evaluation and assess effectivenesscontinue making progressDefine InterventionLay evaluation groundworkPrepare for evaluationConduct evaluation and assess effectiveness

Define
	Intervention
	Lay evaluation
	groundwork
	Prepare for
	evaluation
	Conduct evaluation and
	assess effectiveness
define_intervention
        
  
  
  
  
What do your results mean?
Return to edit

Define intervention
**The intervention does not have a defined theory of change (TOC)**Begin working with others at your organization to develop a defined TOC for your intervention. A TOC defines an intervention by demonstrating how the intervention’s resources and activities are structured with the intent of changing participant knowledge, attitudes, behaviors, or conditions in ways that improve outcomes. Developing a TOC with this level of detail generally begins with thinking about the desired outcomes that the intervention will affect and then identifying all the assumptions about how the intervention might do so. Articulating the assumptions for how the outcomes will be affected is critical for the intervention’s success because it helps establish the inputs, activities, outputs, and outcomes needed for the intervention to succeed. View the SCALER Resources for more information on how to develop a TOC for your intervention.

**The intervention is not yet mature and stable**The inputs and activities identified in the TOC must be implemented and supported as part of a regular organizational routine. Conducting an evaluation before an intervention is mature and stable might not provide an accurate assessment of the intervention’s potential to improve participant outcomes. If your intervention is having difficulties with achieving maturity and stability, consider using formative and outcome evaluations to gather the information needed to refine the intervention. During a formative evaluation, organizations typically seek to determine if an intervention is feasible before fully implementing it. For an outcome evaluation, organizations monitor whether the intervention is achieving the desired outcomes and might decide that additional or different inputs are needed, or that the intervention’s activities need to be tweaked.

**The intervention is not yet implemented with fidelity as described in the TOC**Implementing an intervention with fidelity as described in the TOC is important because it can affect how well the intervention can succeed in improving participant outcomes. Your organization should investigate the underlying issues with implementation fidelity, such as aspects of service delivery adhering to the intervention model as intended.

**The intervention, including any modifications, is not informed by research**Your organization may seek to adapt an intervention in the hopes of improving upon participant outcomes. It might also make adaptations due to external contextual factors—for instance, funding might become available to engage a new population, and an organization can seek to adapt its intervention to apply for and use that funding. Ideally, these adaptations would be informed by other evaluation studies to illustrate how the adapted intervention can produce the desired outcomes for participants.
summary_tab_lay_groundwork
        
  
  
  
  
What do your results mean?
Return to edit

Lay evaluation groundwork
**Your organization, partners, and funders do not yet agree on the need for evidence**Building research evidence and using it to improve an intervention requires a culture of measurement, learning, and evaluation. In such a culture, your stakeholders would have a common understanding about the value of and need for research and evaluation in decision making. In practice, this means that your stakeholders understand the value of using a TOC to improve the design and implementation of your intervention and, subsequently, generating estimates of the intervention’s impact. To help build such a culture and agreement on the need for evidence, your organization and stakeholders should have discussions on the value of evaluation and research and how they can contribute to strategic decision making for your intervention.

**Your organization, partners, and funders do not yet agree on outcomes and measures**It is important to build consensus among key stakeholders on the outcomes your intervention is trying to influence and the goals for the extent to which outcomes should be improved. Your organization should first understand how your stakeholders define the effectiveness that the intervention must display, which will help determine the outcomes and measures the evaluation should include.

**Your organization, partners, and funders do not yet agree on evaluation goals**Establishing evaluation goals will guide how the evaluation should be designed and carried out to help ensure it produces the relevant and meaningful information your organization and stakeholders need. Seek out the evaluation interests and needs that your various stakeholders may have, and identify commonalities among your stakeholders and areas that should be prioritized. Fostering relationships and collaboration among your stakeholders can also help them understand each other’s needs, which may help in building consensus around the evaluation goals.

**Your organization, partners, and funders do not yet support the evaluation**Stakeholder support is needed to carry out a successful evaluation because stakeholders often play an important role in how well the intervention or the evaluation is implemented. For example, your partners may play a key role in delivering certain intervention activities or serve as referral sources of clients who will participate in the intervention and evaluation. You might also need to seek approval from your funders to dedicate resources to conducting the evaluation. Building support from your stakeholders can include demonstrating they can benefit from the evaluation, as building evidence for the intervention can increase the intervention’s future funding opportunities and sustainability. Also seek to understand whether your stakeholders are experiencing any barriers that might inhibit their support for the evaluation.
summary_tab_prepare_evaluation
        
  
  
What do your results mean?
Return to edit

Prepare for evaluation
**Financial resources are not yet available**Research and evaluation require financial resources, such as infrastructure expenses for systems to record data the evaluation will analyze, fees for experts in evaluation design and implementation, and personnel time to collect data for the evaluation. Not allocating sufficient funding to evaluation activities could lead to less rigorous or poorly implemented research that does not meet the standards of rigor for showing an intervention is effective. As you seek funding for a rigorous study, be aware of any evaluation-related funding requirements that might need to be addressed or incorporated into the goals your organization has for assessing its intervention. For example, a funder might require organizations to administer periodic assessments to participants as a way of monitoring growth.

**Personnel are not yet available to work with evaluators**Personnel implementing the intervention—or managing those who are—play an important role in supporting an evaluation. They can provide third-party evaluators with information about the intervention, including important details on the background and context for the intervention’s implementation. These personnel can also provide important insights on evaluation findings. If these personnel are not able to participate in the evaluation, the evaluation will likely not be feasible or as useful as possible, even if it is conducted. In preparation for the evaluation, work with your organizational leadership to ensure the necessary personnel have carved out sufficient time for evaluation activities.
summary_tab_conduct_evaluation
        
  
  
  
What do your results mean?
Return to edit

Conduct evaluation and assess effectiveness
**The research was conducted in a manner that does not meet standards for rigor**Work with your third-party evaluator to understand why the evaluation was unable to establish evidence of effectiveness due to lack of rigor. Research method failure occurs when the evaluation design or its implementation was flawed. For example, the evaluation might not have contained a large enough sample to demonstrate statistical significance or issues may have arisen when the evaluation was carried out—such as random assignment was not properly conducted—that inhibited the evaluators from adequately assessing the intervention’s effectiveness. In such cases, another evaluation that addresses these issues might be needed.

**The research did not show beneficial results for participants**Some evaluations are unable to establish evidence of effectiveness due to implementation failure or theory failure. Implementation failure occurs when the intervention was not implemented with fidelity to the intended model, so the evaluation cannot capture its true impact. In such cases, intervention implementation would need to be improved to ensure fidelity. Theory failure occurs when the intervention is not actually linked to the desired outcome. In some cases, this causal link might not exist, and you will need to revisit the intervention or key components. This can also occur when organizations implement adaptations to serve a new target population of participants or to deliver services in a new type of setting. Work with your organization’s leadership and personnel to identify the factors, processes, or assumptions that could have contributed to the intervention not producing beneficial participant results and seek to resolve them before engaging in another evaluation.

**The context or environment in which the research was conducted was not generalizable to the organization’s plans for scaling its intervention**This means the intervention might not be as effective when used with populations or in locations that are different from what was included in research that showed the intervention’s effectiveness. Continue your evidence-building efforts for your intervention by conducting another evaluation that includes populations or locations that better align with your plans to scale the intervention.
step2_result_guidance
        You have completed Step 2 of the SCALER. Based on your results, your organization needs to continue its efforts to build evidence of effectiveness for your intervention. Ideally, you would return to Step 2 to update your results as you make progress in this area and until such evidence is built before proceeding to Step 3 of the SCALER (to determine your intervention’s and organization’s readiness to scale).

Back

Print Results

Home

Need to finish later?

Update or view your checklists in the future with your unique link:

Copy my unique Step 2 link to the clipboard

Email my custom link to edit, view, or complete my saved SCALER checklists.

return home view SCALER resources
 Back to main content Sign up to hear about our latest news, events, and opportunities.
  Sign up for our newsletter
    Footer about
  

        
        About
                            What We Do
                Our Impact
                Our Team
                Careers
                Agency OverviewFooter serve
  

        
        Serve
                            AmeriCorps
                AmeriCorps Seniors
                Volunteer
                Find Your FitFooter partner
  

        
        Partner
                            How it Works
                Funding OpportunitiesFooter Members & Volunteers
  

        
        Members & Volunteers
                            Your Benefits
                Segal AmeriCorps Education Award
                Alumni
                Safety and SecurityFooter grantees & sponsors
  

        
        Grantees & Sponsors
                            eGrants
                History Checks
                LitmosFooter newsroom
  

        
        Newsroom
                            News
                Blog
                Stories
                Events
                In the News
                Communication Resources
      Bringing Americans together to serve communities

Follow us
  
  1-800-942-2677
            
    help@americorps.gov
    USA.gov / Office of Inspector General / FOIA and Privacy Act / Federal Register notices / No FEAR Act/ Employee Whistleblower Rights / Contractor and Grantee Whistleblower Rights / Office of Special Counsel / STOCK Act / Fraud Alert

Accessibility / Privacy policy / Site map / Donate / Join

**AmeriCorps.gov**
An official website of AmeriCorps. Produced and published at taxpayer expense. 
 You are leaving AmeriCorps.gov
 The website you’re going to is not part of the AmeriCorps domain and may not be under AmeriCorps’ control. Its privacy and security practices and policies may differ from AmeriCorps’. AmeriCorps is not responsible for the link, nor does it endorse the content of the third-party website. Click on the continue button to proceed to the external website; otherwise click cancel to stay on AmeriCorps.gov.
 Cancel Continue