Welcome!

Using Evidence for Scaling Community-Based Interventions That Work 

Webinar

We’ll get started in just 

a couple of minutes.

Please be sure to 

access the Audio by 

clicking the speaker 

icon above.
CNCS Webinar Series on Scaling

•

•

This webinar is also part of The Evidence and 

Evaluation Community of Practice Workshop 

Series, organized by Federal agencies in 

partnership with the OMB Evidence Team. 

To learn more please reach out to Erica Zielewski

at ezielewski@omb.eop.gov

2
Using Evidence for Scaling CommunityBased Interventions That Work

Mary Hyde, Anthony Nerino, and Lily Zandniapour, Office of Research and 

Evaluation mhyde@cns.gov, anerino@cns.gov, lzandniapour@cns.gov
CNCS Webinar Series on Scaling

Today’s Webinar (#3): Perspectives from the Field/Grantee 

Experiences

Previous webinars:

Webinar #1: June 13, 2018, 2:00 - 3:00 p.m. ET

•

•

Presentation Title: Using Evidence for Scaling Community-Based 

Interventions That Work

Presenters: Nan Maxwell, Ph.D., Senior Researcher, and Scott 

Richman, Ph.D., Survey Researcher, Mathematica Policy Research

Webinar #2: November 14, from 2:00 - 3:30 PM ET

•

•

•

Presentation Title: The Power of National Service: Improving 

Children’s Literacy Outcomes 

Presenters: Carrie Markovitz, Ph.D. and Marc W. Hernandez, Ph.D., 

University of Chicago; Sadie O’Connor, Reading & Math, Inc.

https://www.nationalservice.gov/impact-our-nation/researchevaluation/webinars 4
Why Scaling?

•

•

•

•

•

Bring solutions that work into communities in 

need across the nation

Challenge for funders and practitioners - no 

formula or “correct” pathway for doing this

Need deeper understanding of what it takes to 

successfully scale interventions that work

Requires insight, resources, experience, and 

readiness at many levels

Experiential learning is important to this effort

5
Why this Panel?

•

•

•

•

Breadth of perspective

Range of experience in building evidence and 

scaling

Bring reflections on context and decision making 

process

Wealth of knowledge from successes and 

challenges

6
Questions for Presenters

•

•

•

What did you do to get evidence and what drove 

your decision to scale?

What was the critical preparatory work and how 

is your organization supporting scale-up?

What were key challenges and what have you 

learned about effective scaling?

7
Perspectives from the Field/Grantee Experiences

Panelists:

•

•

•

•

•

LaVal Brewer, Former Executive Director, Playworks

Lara Dreier, Director of Federal Partnerships, College 

Possible

Sadie O'Connor, Managing Director, Reading & Math, Inc.

Adam Maurer, HR Consultant, Citizen Schools

Dean Elson, Chief Knowledge Officer, Reading Partners

8
Playworks: What did you do to get evidence 

and what drove your decision to scale?

• Playworks:

–

–

–

–

–

Randomized controlled study back in 2013

Continuous annual survey and data analysis from our 

current partners

Grew our evaluations team to enable deeper dives into 

parts of the data

Recognized that our Theory of Change was scalable 

outside our current footprint

Desire to fully achieve our vision

9
What was the critical preparatory work and how is 

your organization supporting scale-up?

• Playworks:

–

–

–

–

Work with our state commission on expansion of our AC 

footprint

Study of the geographic region we desired to grow in

Discovering the partners who have like minded growth 

plans

Personal visits to the regions / locations

10
What were key challenges and what have you 

learned about effective scaling?

• Playworks:

–

–

–

Plan (determine outcomes),test small, evaluate test 

against outcomes, pilot small expansion

Distance from our core office and leadership staffing 

causes a strain on the team

Small differences in the way partner schools engage 

with CBO’s requires leaders to adapt/modify the pilot, 

not the outcomes

11
Making college admission and success 

possible for students from low-income 

backgrounds
Rationale for Scaling

+

+

+

Internal data

–

–

–

–

Persistence and graduation rates

External data

College enrollment and 

graduation rates by income 

bracket

Workforce data

Financial sustainability

Decreased reliance on grants
Preparation for Scaling

+

+

Listening 

sessions with 

institutions

Building the 

business case 

for our 

program
Challenges in Scaling

+

+

Grant compliance & 

training

Program fidelity
What is Reading Corps?

Reading Corps is a tutoring program that places 

AmeriCorps members into preschools and elementary 

schools to help students become proficient readers by 

the end of 3rd grade.

Tutors provide one-on-one, small group or classroom 

wide support to students identified as at-risk. 

Evidence-based interventions, data-based decision 

making and onsite coaching are pillars of the model. 

In 16 year history, scaled from 24 AmeriCorps members 

to more than 1500 in 600+ locations in Minnesota, and is 

now replicated in 12 states.

16
What did you do to get evidence? What drove 

your decision to scale?

Establish Promise

Internal evaluation of 

alignment between 

design and application.

Key Questions

Is the program model 

stable? 

Is it promising? 

+ 

+ 

Process evaluations

Descriptive analyses 

of pre-post outcomes

Refine and 

Evaluate

External evaluation of 

causal impact and 

iterative improvement

Key Questions

Is Reading Corps 

effective? 

Is the infrastructure 

strong?

+ 

+ 

 Special education 

cost analysis

NORC Evaluation 

(MN)

Scale and Improve

External evaluation of 

causal impact of the 

program at scale. 

Key Questions

Is Reading Corps effective 

across implementation 

environments?

+ NORC Multi-State 

Evaluation

Continuous 

Improvement

Ongoing internal 

research projects to 

improve programming

Key Questions

Reading Corps works—

now, how can the program 

become more effective and 

efficient?

+ Multiple internally 

lead projects
18

What was critical preparatory work, and how 

is your organization supporting scale-up?

1. Essential elements of the model were clear

2. Early results prove model to be effective

3. Finding people who influence resources

4. Constant refinement of operational systems
19

What were key challenges and what have you 

learned about effective scaling?

1. Statewide infrastructure

2. System for student data management

3. Raising money
OUR MISSION

Citizen Schools seeks to ensure 

that every adolescent has the 

skills, mindsets and networks to 

inspire them as lifelong learners 

and position them to thrive in 

the 21st century. 
OUR VISION

By 2025, one million 

adolescents will have access to 

high quality, hands-on learning 

experiences with experts that 

engage and prepare them to 

navigate future pathways.
CITIZEN SCHOOLS OVERVIEW

OUR PROGRAM

We are re-imagining the way students learn in the classroom, after school and within

our communities to ensure equal opportunity for all.

EXPANDING THE CLASSROOM: EXPANDING THE DAY: EXPANDING THE COMMUNITY:

Catalyst Expanded Learning Time US2020

Empowering science 

teachers, in partnership 

with volunteers, to 

deliver high-quality, 

project-based STEM 

learning experiences 

during the school day.

Partnering with schools 

to provide academic 

support, high school and 

college pathway 

programming, and our 

signature 

apprenticeships.

Connecting STEM 

mentors with nonprofit 

orgs and schools to offer 

hands-on STEM 

education and makercentered learning 

opportunities for 

students.
CITIZEN SCHOOLS OVERVIEW

OUR THEORY 

OF IMPACT

The Megaphone:

TRANSFORM EDUCATION

DIRECT SERVICE:

Innovate & 

prove 

affordable, 

effective models

SYSTEMIC IMPACT:

Students have the 

skills, networks and 

mindsets needed to 

thrive in the 21st 

century economy
OUR FY19 FOOTPRINT

CITIZEN SCHOOLS OVERVIEW

MASSACH

Boston

Somerville

NEW YORK

New York

Buffalo

NORTH CAROLINA

Charlotte

Durham

SOUTH CAROLINA

Allendale

TEXAS

Dallas

PENNSYLVANIA

Philadelphia

Pittsburgh

WASHINGTON, D

ILLINOIS

Chicago

INDIANA

Indianapolis

OHIO

Cincinnati

IDAHO CALIFORNIA

Oakland

Redwood City

San Jose

Silicon Valley

Ventura County

Downey

OREGON

The Dalles

LOUISIANA

Baton Rouge

OKLAHOMA

Tulsa

Catalyst

Expanded Learning Time

US2020

NEW 

MEXICO

Socorro

GEORGIA

Atlanta

MICHIGAN

Detroit USET

C

KANSAS

Wichita

TS
CNCS Scaling Webinar

Dean Elson, Chief Knowledge Officer
Our Vision

▪ We envision a future where all children in the U.S. 

have the reading skills necessary to reach their full 

potential.

Our Mission

▪ Our mission is to help children become lifelong 

readers by empowering communities to provide 

individualized instruction with measurable results. 

Our Program

▪ Our evidence-based tutoring program is a publicschool literacy intervention powered by 

AmeriCorps members and community volunteers 

who twice weekly serve K-4 students who struggle 

with reading.
Evidence Building & Decision to Scale Up

| 27

Early Evidence of 

Success

• Internal data for several 

years showed strong 

results around student 

improvement on 

literacy skills and 

reducing grade-level 

reading gaps

• 2009-12: Matched 

comparison study with 

design and oversight 

from Stanford 

University

Social Innovation Fund 

(SIF): MDRC Evaluation

• SIF / EMCF provided 

funding, an opportunity 

to request matching 

funds from donors, and 

oversight for an 

independent, highly 

rigorous evaluation

• MDRC conducted a 

randomized control trial 

(RCT) from 2012 to 2015, 

including planning, 

implementation and 

impact data collection, 

and cost study

SIF #2: APA 

Evaluation

• Second SIF grant 

supported scaling to 

Colorado and further 

exploring 

implementation and 

impact

• 5-year study in CO 

revealed additional 

program insights and 

yielded more positive 

impact data

Decision to Scale

• Early evidence of 

success and student 

impact

• Valued asset to school 

and district partners; 

high satisfaction and 

long waiting lists

• Alignment among 

national and regional 

board members and 

leadership team to meet 

the size of the challenge
Critical Preparatory Work / How the Org Supports Scaling Up

| 28



















Clear program model / theory of change / logic model

Alignment among leadership team, board & staff

Key functions have strong leaders and functional 

expertise

Expansion “checklist” and clear timeline

Achieving clarity about where to scale & why

Opportunities for sustainable fundraising and school 

and district partnerships

Understanding donor interest and alignment to 

mission, strategy, and goals

Align annually on priorities, debate the “must have” 

investments vs. “nice to have” projects

Data for learning and to tell an impact story
Key Challenges and Lessons Learned About Effective Scaling

| 29















Develop effective measurement and data 

systems

Document success

Balance scaling with core program 

improvement

Encourage innovation and expand sources of 

creativity

Secure multi-year funding commitments

Create and sustain operational efficiencies

Prepare the talent bench
Closing Remarks: Dr. Mary Hyde
Question & Answer

•

•

•

You can ask a question at any time by typing it in 

the Q&A box below

We will answer questions in the order they come 

in

For reports, please check out the Evidence 

Exchange: https://nationalservice.gov/impact-ournation/evidence-exchange

31
Thank you!

•

•

•

Thank you for attending 

today’s webinar

A recording will be available 

online within the next few 

weeks

To help us improve future 

webinars, please take a few 

minutes to fill out this survey

32