KEYSTONE SMILES 

AMERICORPS 

EVALUATION REPORT

 

Wolfrom Consulting

September 29, 2020 
1 | Page

Table of Contents

Introduction ............................................................................................................................................... 2-3 

Program Theory, Logic Model, and Outcomes of Interest ........................................................................ 3-5 

Logic Model................................................................................................................................................ 6-7 

Research Question Addressed in the Study.................................................................................................. 8 

Study Components................................................................................................................................... 8-11

Data Collection and Analysis....................................................................................................................... 11

Quantitative Data .................................................................................................................................. 12-17

Quantitative Data Analysis.......................................................................................................................... 18

Qualitative Data ..................................................................................................................................... 18-25

Qualitative Data Analysis ........................................................................................................................... 26

Conclusions............................................................................................................................................ 26-28
2 | Page

Keystone SMILES AmeriCorps Evaluation

Introduction 

This evaluation outlines an impact evaluation for Keystone SMILES AmeriCorps, an AmeriCorps 

State and National (ACSN) program that exists to serve unmet, critically important needs as 

identified and underserved by committed and engaged host site stakeholders. Through the

evaluation process, Keystone SMILES ascertained the effects of the targeted academic 

assistance activities provided by its members in the schools that they serve. 

This evaluation is a continuation of the impact evaluation that was required by the Corporation 

for National and Community Service (CNCS) for a select group of AmeriCorps state competitive 

grantees during the 2014 – 2017 grant cycle. Keystone SMILES was a part of the select group 

and received technical assistance from the National Opinion Research Center (NORC) at the 

University of Chicago during the first year of the grant cycle. The NORC technical assistance 

team provided guidance on the evaluation requirements:

• Large grantees, including Keystone SMILES, must conduct an independent evaluation to 

measure the impact of their programs. 

• In order to demonstrate the impact of a program, a comparison group must be used.

• Data collection should be conducted at two time points for both beneficiaries as well as 

an identified comparison group of individuals that did not participate in the program.

• Grantees are not expected to evaluate all of their service activities but are required to 

evaluate at least one of their major service activities.

• A minimum of one year of program activities should be evaluated.

• For programs that are implemented in multiple sites, not all sites need to be evaluated. 

However, a representative sample of sites should be included.

The technical assistance team met several times with Keystone SMILES and the evaluator

throughout the first year (2014-2015) of the grant cycle. An evaluation plan was designed and 

accepted by the technical assistance team. The plan was then shared with the State 

Commission and CNCS staff. 

A final evaluation report was submitted to the State Commission and CNCS in December 2017. 

Although qualitative data from host site interviews revealed themes indicating schools depend 

on the Keystone SMILES AmeriCorps program to provide a quality educational experience for all 
3 | Page

students, the limited quantitative data available at the time, was not conclusive that receiving 

services from Keystone SMILES AmeriCorps results in higher test scores on the fourth grade 

PSSA mathematics assessment. However, only one year of data was studied. Consequently, it 

was recommended that the review of the data continue for a minimum of three years to 

determine what effect, if any, the Keystone SMILES AmeriCorps program on the achievement of 

students. The impact study was continued and this report details the findings. 

Program Background and Problem Definition: Together with its host site stakeholders, Keystone 

SMILES AmeriCorps recruits, trains, develops, and engages AmeriCorps members to provide 

academic support with host site school districts and alternative education and mentoring with 

community organizations; community strengthening service to host sites; and member and 

community training. During this program cycle, they provided structured programs to offer 

academic and social/behavior tutoring support within 60 school buildings and a wide array of 

service-learning support services across 12 community-based host sites across 12 counties in

western and northwestern Pennsylvania. The stated mission of Keystone SMILES is “to 

empower and strengthen people of all ages with a focus on children and youth, to enhance the 

quality of their lives through learning and service.”

To address the academic and social needs of struggling rural learners, Keystone SMILES 

AmeriCorps has created, implemented, and maintained a National Service network of partners 

and programs that provide expanded learning opportunities for students. As school districts 

and community agencies, those institutions that stand most ready to directly impact student's 

learning, face diminishing budgets and increasing performance targets, expanded learning 

opportunities become the most effective way to provide much needed interventions. 

 

Program Theory, Logic Model, and Outcomes of Interest

Keystone SMILES AmeriCorps’ expanded learning opportunities include before, during, and 

after school programs and summer learning initiatives that broaden and extend learning for 

youngsters. Expanded learning systems help students be more connected to school, build selfconfidence, and connect with caring adults, so every student can thrive. Students not only learn 

how to do better in school, they find new ways to ignite their passions and to engage in 

learning and life. During the program cycle, Keystone SMILES sought to provide two types of 

expanded learning activities to 20,100 (6,700 per year) rural children and youth. The two types 

of expanded learning activities included one-on-one and small group tutoring, other classroom 

support, summer learning; and service learning projects focused on rural communities, persons 

with disabilities, Opportunity Youth, community revitalization, and veterans and military 

families. Not every student served participated in the same types of interventions; instead, 

members, utilizing the expertise of school leaders, molded services to meet student needs. 
4 | Page

Members employed one-on-one and small group instruction, including Response to 

Intervention (RTI) Model, Credit Recovery, Title I remediation and support, standardized test 

preparation, and progress monitoring to accelerate acquisition of knowledge and skills for 

students. The scope of member service was intentionally wide and varied, allowing school 

leaders to identify and address local needs through various assessments and educator expertise 

so that member engagement both supplemented and complemented, not replaced, existing 

programming.

One-on-one and small group instruction are methods of academic intervention designed to 

provide early, systematic assistance to students who are having difficulty learning. One-on-one 

and/or small group instruction gives students individualized attention focused on improving 

academic deficiencies in specific skills, concepts, and/or subject areas. The Response to 

Intervention (RTI) Model seeks to prevent academic failure through early intervention, frequent 

progress measurement, and increasingly intensive research-based instructional interventions 

for students who continue to have difficulty. Likewise, Title I remediation and support are

provided in schools that have identified 35% of their students as being disadvantaged or at 

risk. Schools providing Title I services must provide identified students with additional 

educational opportunities designed to help them meet academic requirements. Credit recovery 

programs aim to help schools graduate more students by giving students who have fallen 

behind the chance to "recover" credits. Finally, progress monitoring is a scientifically based

practice that is used to assess students' academic performance and evaluate the effectiveness 

of instruction.

The target population for one-on-one and small group tutoring, other classroom support, 

and/or summer learning was at-risk students, in grades PK-12, who were identified as 

economically disadvantaged, having special and/or exceptional needs, and/or who did not meet 

proficiency benchmarks on curriculum based assessments and/or PSSA or Keystone Exams. The

duration and dosage of before, during, and after school programs was 3-5 times per week for 

30-90 minutes a session, for 32 weeks. The dosage and duration of summer programs was 3-5 

days per week for 3-8 hours per day, for 1-8 weeks.

 

The other expanded learning activity of the Program Model that members engaged in centered 

around service learning projects focused on engaging and serving, rural communities, persons 

with disabilities, Opportunity Youth, and/or veterans and military families. Like academic 

assistance, service learning improves academic engagement. By engaging in service that 

teaches real-life skills, students discover a meaning and context to their studies that is 
5 | Page

unattainable through traditional textbooks. This realization motivates struggling learners and 

teaches the value of academic pursuits.

The duration and dosage of Keystone SMILES AmeriCorps members' projects was conducted 

during a concentrated block of time over several weeks or months for a total of no less than 30 

hours. The target population was students, in grades PK-12, receiving other education-related 

interventions. 

For the past 16 years, Keystone SMILES AmeriCorps members have been using pre and post 

surveys, administered twice a year, to measure changes in student engagement. The rubric 

assessments used for the surveys are based on the Summerbridge School Success Rubric 

provided by nationalserviceresources.com. In the last several years, pre and post survey data 

indicates that there has been an increase in student engagement for those students receiving 

services from Keystone SMILES AmeriCorps members, thus providing evidence that the shortterm outcome of the program is being met. 

The purpose of this evaluation study was to determine if the medium-term outcome of 

increased academic performance of students receiving education related interventions

provided by Keystone SMILES AmeriCorps members is also being met. Specifically, this was 

achieved by the analysis of standardized test scores, along with the inclusion of the qualitative

interview data. 
6 | Page

Logic Model: 

Problem Inputs Activities Outputs Short-Term 

Outcomes 

Mid-Term 

Outcomes 

Long-Term 

Outcomes 

The community 

problem that the 

program activities 

(interventions) are 

designed to 

address.

Resources that are 

necessary to 

deliver the 

program activities 

(interventions), 

including the 

number of 

locations/ sites 

and number/type 

of AmeriCorps 

members.

The core activities 

that define the 

intervention or 

program model that 

members will 

implement or 

deliver, including 

duration, dosage 

and target 

population.

Direct products 

from program 

activities.

Changes in 

knowledge, skills, 

attitudes and 

opinions. These 

outcomes, if 

applicable to the 

program design, 

will almost always 

be measurable 

during the grant 

year.

Changes in 

behavior or action. 

Depending on 

program design, 

these outcomes 

may or may not be 

measurable during 

the grant year.

Changes in 

condition or status 

in life. Depending 

on program design, 

these outcomes 

may or may not be 

measurable during 

the grant year.

Some programs, 

such as 

environmental or 

capacity- building 

programs, may 

measure changes 

in condition over a 

period as short as

one year.

54% of children 

living in the rural, 

Northwestern PA 

counties served by 

Keystone SMILES 

live below 200% of 

the poverty level, 

impacting their 

academic 

performance and 

stifling their 

dreams. In School 

District 

Performance 

Profiles released in 

October 2016, the 

school districts 

served by the 

program averaged 

a 69% Building 

Level Academic 

Score. Despite the 

academic needs 

demonstrated by

155 AmeriCorps

members (70 Full 

Time, 12, Reduced 

Full Time, 12 Half 

Time, 54 Quarter

Time, and 7

Minimum Time); 60 

school buildings 

and 12 satellite 

sites;6 Core Staff 

with 70 years 

combined National 

Service 

Experience; 10 

Supervisory Staff 

with 102 years 

combined National 

Service 

Experience; 

Episodic and 

Ongoing 

Volunteers; Board 

of Directors; 

Member Training

Core Activities 

Members will 

provide expanded 

learning activities 

such as 

1.) one- on- one 

and small group 

tutoring,

other classroom 

support, and 

summer learning.

Duration/Dosage: 

Before, During and 

After School 

programs - 3- 5 times 

per week for 30- 90 

minutes a session, for 

32 weeks Summer 

Programs: 3- 5 days 

per week for 3-8 

hours per day, for 1-8 

weeks

6700 students will 

be served in 

CNCS- supported 

K- 12 education 

program annually.

4020 students will 

demonstrate 

increased

academic 

performance in 

math or literacy 

(one grade level). 

Outcomes will be 

measured by Pre 

and Post 

standardized test 

scores.

1340 students will 

demonstrate 

increased 

academic 

engagement 

through improved 

attitude, leadership 

and selfconfidence, and 

increased 

motivation and 

organization.

Increased 

academic 

engagement and 

performance will 

result in improved 

graduation rates in 

rural school 

districts.

Increased access to 

post- secondary 

opportunities 

resulting in 

improved life- time 

earning potential. 

Over time, these 

changes will 

empower students 

and their families

to escape the 

cycle of poverty in 

rural western 

Pennsylvania. 

As rural citizens 

escape the poverty 

cycle, rural 

communities can 

reclaim resources 

to revitalize 

industry and 

stimulate

economic growth. 
7 | Page

Problem Inputs Activities Outputs Short-Term 

Outcomes 

Mid-Term 

Outcomes 

Long-Term 

Outcomes 

these sub- par 

scores, a study 

conducted by the 

Center for Rural PA 

found that 91% of 

rural 

superintendents 

cited academic 

support programs 

as an essential 

element of meeting 

student needs in 

their districts, only 

18% of all eligible 

students received 

academic support. 

Keystone SMILES 

AmeriCorps strives 

to stand in this 

gap, providing

rural schools and

students with 

expanded learning 

activities to 

address the 

academic needs 

created by the 

poverty they live

in.

supported by 

Clarion University

Target Population - 

At risk students, in 

grades PK- 12, who 

have been 

identified as 

economically 

disadvantaged, 

having special or 

exceptional needs, 

and/ or who have 

not met proficiency 

benchmarks in 

curriculum based 

assessments

and/ or on PSSA 

or Keystone 

Exams.

2) service- learning 

projects focused 

on:

Rural communities,

Persons with 

disabilities, 

Opportunity Youth, 

Community 

Revitalization and 

Veterans and 

military families.

Duration/Dosage: 

Projects will be 

conducted during 

a concentrated 

block of time over 

several weeks or 

months.

Target Population 

Students, in grades 

PK-12 receiving 

education-related 

interventions 

Outcomes will be 

measured by a 

program 

developed rubric. 

Rubrics have been 

used by the 

program for 16 

years, so

historical and 

longitudinal data 

will be compiled 

and analyzed.
8 | Page

Research Question Addressed in the Study

Do students in schools who receive targeted services from Keystone SMILES AmeriCorps 

members achieve better mathematics scores on the Pennsylvania System of School Assessment 

(PSSA) standardized test than students in schools with similar academic and socioeconomic 

backgrounds who do not receive services? 

Study Components

Evaluation Design and Rationale

The design chosen for this study was quasi-experimental (QED). (Due to the nature of schools, it 

is not feasible to conduct a randomized control trial (RCT).) The study employed school-level 

matched comparison groups by percent proficiency. An analytic treatment group was identified 

and evaluated against an analytic comparison group.

Keystone SMILES AmeriCorps serves rural school districts in northwestern Pennsylvania that 

range in size between approximately 750 and 2,500 in total student population. Due to the 

relatively small student population size of the school districts that Keystone SMILES serves, 

student-level comparison groups could not be used as the sample size would not be large 

enough to enable accurate and/or reliable statistical judgments. Additionally, most comparison 

schools (those not participating in the Keystone SMILES AmeriCorps Program) would not 

provide student data as they would have no incentive to so. However, comparison school-level 

data is available and readily accessible. 

Analytic Sampling Method

The target population for analysis was fourth grade students receiving academic assistance

from Keystone SMILES AmeriCorps members. Focused academic assistance to the students was 

one-on-one and/or in small group settings. The comparison data that was used is the 

Pennsylvania System of School Assessment (PSSA) standardized test data. 

The analysis group was identified through Keystone SMILES host site application and 

verification processes, June 2015 – September 2019. Initial data was gathered and analyzed 

during the 2015-2016 school year as part of the previous program cycle’s evaluation. The same 

methodology continued to evaluate this program cycle (2016-2017 through 2018-2019). The 

evaluator worked with Keystone SMILES AmeriCorps during the host site recruitment process to 

make sure host sites were aware of the study. Throughout the 2015-2016 school year, district 

and school level achievement and demographic data available on the Pennsylvania Department 
9 | Page

of Education (PDE) and the Pennsylvania Value Added Assessment System (PVAAS) websites 

was utilized to determine school-level match comparison groups by performance quartiles. 

Primary Data Source

In Pennsylvania, school districts are locally controlled. There is no statewide curriculum that all 

districts are mandated to follow. However, there are academic standards that all schools must 

align their curricula to and student proficiency on those standards is assessed each spring using 

the PSSA. Other than the PSSA, there is no other common system of assessment (formative, 

benchmark, diagnostic, etc.) that is consistent across the Commonwealth of Pennsylvania; 

hence, the summative PSSA summative achievement data was used for this evaluation. The 

strength of this study design is that it will be using standardized PSSA test data. The PSSA test is 

a standards-based, criterion-referenced assessment used to measure a student’s attainment of 

the academic standards while also determining the degree to which school programs enable 

students to attain proficiency of the standards. Every Pennsylvania student in grades 3 through 

8 is assessed in mathematics. PSSA data is disaggregated by grade level, content area, and 

demographics and is publicly available. 

Additionally, the English Language Arts (ELA) and Mathematics PSSA are administered to all 

students in grades 3-8. Historically, based on need, Keystone SMILES AmeriCorps members 

have focused the majority of their academic assistance efforts on students in grades K-5. Due to 

the lack of available reliable data for grades K-2, not wanting to use first year (grade 3) PSSA 

testing data, and historical drops in grade 5 PSSA Mathematics data, it was determined that 

fourth grade mathematics data would be analyzed for purposes of this evaluation. It should be 

noted that ELA data was not considered for analysis in this evaluation because soon before this 

evaluation began, the Pennsylvania Department of Educations (PDE) had recently transitioned 

from assessing ELA via two separate tests – one reading and one writing – to a combined 

reading and writing ELA assessment for all grades 3-8. 

The outcome measure for this evaluation study is directly aligned to the data source 

measurement. Grade 4 analysis groups received education-related interventions throughout 

the 2015-2016 (previous program cycle), 2016-2017, 2017-2018, and 2018-2019 school years. 

Academic assistance occurred in before, during, and after school programs, 3-5 times per week 

for 30-90 minutes each session, for a minimum of 32 weeks. Specific learning activities were 

determined through various performance assessment sources and classroom observations and 

were aligned to the Pennsylvania Core Standards and Assessment Anchors.

The evaluator analyzed the 2015 PSSA results to determine comparison match school groups 

based on relatively similar academic achievement and demographics. 
10 | Page

Supplemental Data Sources 

The evaluator conducted periodic interviews with host site supervisors to gather qualitative 

information about the perceived impact of the academic assistance provided by the 

AmeriCorps members. The aforementioned secondary data source was only collected for the 

intervention treatment group. 

Study Limitations

The primary limitation of this design is that it only measured one post-intervention data set, the 

PSSA for one grade level in one content area (mathematics). There is no other standardized 

assessment that all host site schools use to measure mathematics achievement, hence no other 

measurable data sets are available for collection and analysis. The PSSA is the only common 

assessment for all schools across Pennsylvania. Another limitation of the study is the use of 

school-level matched comparison groups, potentially diluting the data set. As noted earlier, 

student-level comparison groups could not be used as the sample size was not large enough to 

enable accurate and/or reliable statistical judgments. Additionally, it was highly unlikely that 

student level data would be made available by comparison districts (those districts not 

participating in the Keystone SMILES AmeriCorps Program). 

Finally, the original sample size included 17 treatment schools. A total of 12 schools 

participated in the Keystone SMILES AmeriCorps program for all four school years of this study. 

While some schools exited the program, others entered.

Target Population/Analytics Sample

Keystone SMILES AmeriCorps members serve students in the following rural Pennsylvania 

counties: Allegheny, Clarion, Crawford, Elk, Erie, Forest, Jefferson, Lawrence, McKean, Mercer, 

and Venango. Collectively, they represent nearly one quarter of the state's area at 8,205 square 

miles. Nearly one in four children (23%) residing in these counties lives below the poverty level 

(USDA,ERS, 2014). An additional 31% live in families classified as "working poor," a group that 

includes children in homes where parents or guardians are employed or looking for work but 

whose total household income is below 200% of the poverty level (Kids Count Data, 2009-

2011). Taken together, these statistics represent an astonishing 54% of children in the counties 

served by Keystone SMILES AmeriCorps.

The nature of the Keystone SMILES AmeriCorps Program is to be flexible and to address needs 

in the grade levels and content areas that are most vulnerable in the rural areas that it serves. A 

review of host sites indicated that there were a high percentage of fourth grade students 

receiving services from Keystone SMILES AmeriCorps members. The trend continued for the

program cycle and an adequate number of fourth grade students received services to yield an 
11 | Page

acceptable sample size. A power analysis was used to determine an adequate sample size for 

both the analysis and comparison groups. 

Data Collection and Analysis 

Treatment schools are those that had an AmeriCorps member serving in them during the 2014-

2015, 2015-2016, 2016-2017, 2017-2018, and 2018-2019 school years. A formula was created 

to determine comparison schools based on total number of students, the number of historically 

underperforming students, and percent proficiency of each group. Once the 2016 PSSA data 

was released in September 2016, the evaluator began to conduct a statistical analysis of the 

quantitative data (PSSA) using regression analysis. The evaluator also conducted host site 

interviews and provides qualitative data to supplement the statistical analysis of the 

quantitative (PSSA) data. 
12 | Page

Quantitative Data 

Treatment School 1 

Edinboro Elementary (General McLane SD) 2015 2016 2017 2018 2019

Yr. to Yr. 

Avg. 

Growth 

Avg. 

Growth 

2015-2019

Total Number of Students 76 80 79 70 90

Percent Proficient 57.9% 56.3% 60.8% 62.8% 62.3% 2% 8%

Historically Underperforming Students 26 22 31 26 44

Percent Proficient 23.1% 40.95% 35.5% 46.1% 54.5% 28% 136%

Comparison School 1 

Gwyn-Nor Elementary (North Penn SD) 2015 2016 2017 2018 2019

Yr. to Yr. 

Avg. 

Growth 

Avg. 

Growth 

2015-2019

Total Number of Students 80 94 90 80 62

Percent Proficient 55.1% 43.6% 50% 52.6% 43.5% -5% -21%

Historically Underperforming Students 25 33 33 30 27

Percent Proficient 28% 21.3% 33.3% 13.3% 29.6% 24% 6%

Treatment School 2 

Elk Valley Elementary (Girard SD) 2015 2016 2017 2018 2019

Yr. to Yr. 

Avg. 

Growth 

Avg. 

Growth 

2015-2019

Total Number of Students 124 113 118 120 115

Percent Proficient 58.9% 64.6% 55.9% 71.7% 70.4% 6% 20%

Historically Underperforming Students 69 68 72 83 68

Percent Proficient 50.7% 54.4% 44.4% 62.7% 63.2% 8% 25%

Comparison School 2 

Francis Sheckler Elementary (Catasauqua Area SD) 2015 2016 2017 2018 2019

Yr. to Yr. 

Avg. 

Growth 

Avg. 

Growth 

2015-2019

Total Number of Students 117 124 117 115 84

Percent Proficient 59.9% 56.5% 47.8% 41.7% 54.7% -1% -9%

Historically Underperforming Students 66 84 83 73 52

Percent Proficient 45.5% 48.8% 42.2% 27.4% 50% 10% 10%
13 | Page

Treatment School 3 

Fairview Elementary School (Fairview SD) 2015 2016 2017 2018 2019

Yr. to Yr. 

Avg. 

Growth 

Avg. 

Growth 

2015-2019

Total Number of Students 115 119 128 134 129

Percent Proficient 64.3% 71.5% 75% 66.4% 62.8% 0% -2%

Historically Underperforming Students 31 35 39 40 39

Percent Proficient 45.1% 45.7% 43.6% 42.5% 38.5% 0% -15%

Comparison School 3 

Bethel Springs Elementary (Garnet Valley SD) 2015 2016 2017 2018 2019

Yr. to Yr. 

Avg. 

Growth 

Avg. 

Growth 

2015-2019

Total Number of Students 113 100 131 94 83

Percent Proficient 65.5% 73% 72.5% 72.3% 73.5% 3% 12%

Historically Underperforming Students 31 27 26 20 25

Percent Proficient 41.9% 48.1% 42.3% 65% 48% 8% 15%

Treatment School 4

Hillview Intermediate Center (Grove City SD) 2015 2016 2017 2018 2019

Yr. to Yr. 

Avg. 

Growth 

Avg. 

Growth 

2015-2019

Total Number of Students 139 146 140 138 141

Percent Proficient 59.8% 65.1% 70% 70.3% 71.6% 5% 20%

Historically Underperforming Students 56 63 57 62 58

Percent Proficient 39.3% 42.9% 49.2% 56.5% 56.9% 10% 45%

Comparison School 4

Central Columbia Elementary (Central Columbia SD) 2015 2016 2017 2018 2019

Yr. to Yr. 

Avg. 

Growth 

Avg. 

Growth 

2015-2019

Total Number of Students 139 141 138 132 135

Percent Proficient 57.5% 65.3% 74.6% 68.1% 64.4% 3% 12%

Historically Underperforming Students 55 56 57 52 63

Percent Proficient 32.7% 44.% 56.2% 50% 44.5% 10% 36%
14 | Page

Treatment School 5

Iroquois Elementary (Iroquois SD) 2015 2016 2017 2018 2019

Yr. to Yr. 

Avg. 

Growth 

Avg. 

Growth 

2015-2019

Total Number of Students 102 85 98 94 83

Percent Proficient 37.2% 44.7% 55.1% 47.9% 47% 7% 26%

Historically Underperforming Students 70 56 64 73 55

Percent Proficient 24.3% 37.5% 48.4% 38.4% 32.7% 12% 35%

Comparison School 5

Aloysius Fitzpatrick Elementary (Philadelphia SD) 2015 2016 2017 2018 2019

Yr. to Yr. 

Avg. 

Growth 

Avg. 

Growth 

2015-2019

Total Number of Students 96 87 67 96 83

Percent Proficient 36.4% 26.4% 20.9% 22.9% 31.3% -1% -14%

Historically Underperforming Students 73 65 67 94 61

Percent Proficient 28.7% 26.2% 20.9% 23.4% 18.1% -10% -37%

Treatment School 6

Kane Elementary (Kane SD) 2015 2016 2017 2018 2019

Yr. to Yr. 

Avg. 

Growth 

Avg. 

Growth 

2015-2019

Total Number of Students 90 90 83 71 68

Percent Proficient 35.5% 43.3% 42.1% 28.2% 41.2% 8% 16%

Historically Underperforming Students 47 54 55 45 41

Percent Proficient 19.1% 33.3% 36.4% 22.2% 36.6% 27% 92%

Comparison School 6

Canton Area Elementary (Canton Area SD) 2015 2016 2017 2018 2019

Yr. to Yr. 

Avg. 

Growth 

Avg. 

Growth 

2015-2019

Total Number of Students 87 73 73 58 64

Percent Proficient 39.1% 47.9% 43.8% 41.4% 36% -1% -8%

Historically Underperforming Students 50 48 38 40 36

Percent Proficient 24% 41.6% 31.6% 27.5% 22.2% 4% -8%
15 | Page

Treatment School 7

McKean Elementary (General McLane SD) 2015 2016 2017 2018 2019

Yr. to Yr. 

Avg. 

Growth 

Avg. 

Growth 

2015-2019

Total Number of Students 83 54 73 72 69

Percent Proficient 61.5% 68.5% 56.2% 65.3% 71% 5% 15%

Historically Underperforming Students 28 27 36 37 31

Percent Proficient 50% 59.2% 47.2% 45.9% 64.5% 9% 29%

Comparison School 7

Media Elementary (Rose Tree SD) 2015 2016 2017 2018 2019

Yr. to Yr. 

Avg. 

Growth 

Avg. 

Growth 

2015-2019

Total Number of Students 85 87 82 70 74

Percent Proficient 62.3% 64.3% 74.4% 62.8% 79.7% 8% 28%

Historically Underperforming Students 27 25 17 26 22

Percent Proficient 48.1% 48% 35.2% 53.8% 54.5% 7% 13%

Treatment School 8

North Clarion Elementary (North Clarion SD) 2015 2016 2017 2018 2019

Yr. to Yr. 

Avg. 

Growth 

Avg. 

Growth 

2015-2019

Total Number of Students 32 45 42 40 43

Percent Proficient 43.8% 15.6% 54.8% 30% 41.9% 45% -4%

Historically Underperforming Students 18 22 21 15 19

Percent Proficient 27.8% 9.1% 42.8% 6.7% 31.6% 148% 14%

Comparison School 8

Wharton Elementary (Uniontown Area SD) 2015 2016 2017 2018 2019

Yr. to Yr. 

Avg. 

Growth 

Avg. 

Growth 

2015-2019

Total Number of Students 35 25 31 28 30

Percent Proficient 45.7% 36% 38.7% 64.3% 40% 4% -12%

Historically Underperforming Students 22 15 17 16 19

Percent Proficient 31.8% 33.4% 23.6% 50% 42.1% 18% 32%
16 | Page

Treatment School 9

North East Intermediate School (North East SD) 2015 2016 2017 2018 2019

Yr. to Yr. 

Avg. 

Growth 

Avg. 

Growth 

2015-2019

Total Number of Students 111 132 126 131 130

Percent Proficient 54.9% 58.3% 61.1% 51.9% 60% 3% 9%

Historically Underperforming Students 61 68 66 77 80

Percent Proficient 42.6% 47% 53.1% 41.6% 53.8% 8% 26%

Comparison School 9

Jersey Shore Elementary (Jersey Shore SD) 2015 2016 2017 2018 2019

Yr. to Yr. 

Avg. 

Growth 

Avg. 

Growth 

2015-2019

Total Number of Students 108 100 109 90 92

Percent Proficient 55.5% 56% 50.4% 52.2% 40.2% -7% -28%

Historically Underperforming Students 58 54 66 40 60

Percent Proficient 41.4% 44.4% 32.7% 32.5% 35% -3% -15%

Treatment School 10

Northwestern Elementary (Northwestern SD) 2015 2016 2017 2018 2019

Yr. to Yr. 

Avg. 

Growth 

Avg. 

Growth 

2015-2019

Total Number of Students 57 77 70 77 65

Percent Proficient 73.7% 68.9% 61.5% 59.8% 72.3% 0% -2%

Historically Underperforming Students 38 50 49 49 43

Percent Proficient 65.7% 62% 55.1% 53.1% 60.4% -2% -8%

Comparison School 10

Kratzer Elementary (Parkland SD) 2015 2016 2017 2018 2019

Yr. to Yr. 

Avg. 

Growth 

Avg. 

Growth 

2015-2019

Total Number of Students 59 66 71 75 71

Percent Proficient 74.6% 77.3% 74.7% 74.7% 70.4% -1% -6%

Historically Underperforming Students 38 32 28 31 44

Percent Proficient 63.1% 62.5% 50% 54.9% 63.6% 1% 1%
17 | Page

Treatment School 11

Punxsutawney Elem School (Punxsutawney SD) 2015 2016 2017 2018 2019

Yr. to Yr. 

Avg. 

Growth 

Avg. 

Growth 

2015-2019

Total Number of Students 136 143 152 156 139

Percent Proficient 55.1% 53.9% 38.8% 34.6% 46.1% -2% -16%

Historically Underperforming Students 92 87 99 104 95

Percent Proficient 46.7% 41.3% 31.3% 29.8% 41.1% -1% -12%

Comparison School 11

Grandview Intermediate School (Derry Area SD) 2015 2016 2017 2018 2019

Yr. to Yr. 

Avg. 

Growth 

Avg. 

Growth 

2015-2019

Total Number of Students 133 147 157 144 144

Percent Proficient 54.9% 55.1% 54.2% 56.3% 70.9% 7% 29%

Historically Underperforming Students 82 84 78 92 74

Percent Proficient 46.3% 44.1% 46.2% 50% 56.7% 5% 22%

Treatment School 12

Reynolds Elementary (Reynolds SD) 2015 2016 2017 2018 2019

Yr. to Yr. 

Avg. 

Growth 

Avg. 

Growth 

2015-2019

Total Number of Students 87 71 77 54 76

Percent Proficient 43.4% 39.5% 48.1% 46.3% 44.85 1% 3%

Historically Underperforming Students 49 46 46 51 41

Percent Proficient 36.8% 23.9% 34.8% 43.2% 34.1% 3% -7%

Comparison School 12

Williams Valley Elementary (Williams Valley SD) 2015 2016 2017 2018 2019

Yr. to Yr. 

Avg. 

Growth 

Avg. 

Growth 

2015-2019

Total Number of Students 92 70 81 89 71

Percent Proficient 40.2% 28.5% 35.8% 28% 31% -4% -23% 

Historically Underperforming Students 47 38 47 53 41

Percent Proficient 36.2% 10.5% 17% 22.7% 29.3% 13% -19% 
18 | Page

Quantitative Data Analysis

Overall, a total of eight of the twelve (67%) treatment schools scored better than the 

comparison schools on at least three of the four measurements: 

• Year to Year Average Growth for Total Number of Students

• Year to Year Average Growth for Historically Underperforming Students

• Average Growth 2015-2019 for Total Number of Students

• Average Growth 2015-2019 for Historically Underperforming Students 

Those schools were: Edinboro, Elk Valley, Hillview, Iroquois, Kane, North Clarion, North East, 

and Reynolds. Three schools scored worse: Fairview, McKean, and Punxsutawney. 

Northwestern and its comparison school scored relatively about the same. 

Edinboro, Iroquois, Kane, and North East scored higher than their comparison schools in all four 

areas. Hillview scored higher in three of the four categories and the same for Year to Year 

Average Growth for Historically Underperforming Students. Elk Valley and Reynolds scored 

higher than their comparison schools in all areas except for Year to Year Average Growth for 

Historically Underperforming Students; whereas North Clarion scored higher than their 

comparison schools in all areas except for Average Growth 2015-2019 for Historically 

Underperforming Students. 

Northwestern scored higher than its comparison school in Year to Year Average Growth for 

Total Number of Students and Average Growth 2015-2019 for Total Number of Student but 

lower in the other two categories. McKean scored lower in all categories except Year to Year 

Average Growth for Historically Underperforming Students. Both Fairview and Punxsutawney 

scored lower than their comparison in all for areas. 

Qualitative Data

Interviews were conducted with 10 of the 12 treatment host sites. All schools were invited to 

participate in the interview process. Present during the interview was the host site coordinator 

and/or the site supervisor. 

Interview Questions:

Question #1: During the past three school years, did an AmeriCorps member serve in least one 

of your elementary schools that serves 4th graders?
19 | Page

Question #2: Was the focused academic assistance provided by the AmeriCorps member(s) 

one-on-one, small group, or both?

Question #3: How were students that were served by an AmeriCorps member identified? 

Question #4: How often were identified students provided focused education-related 

interventions by an AmeriCorps member?

Question #5: Describe the AmeriCorps member(s) service learning project(s)?

Question #6: Did organization and completion of assignments increase, decrease, or remain the 

same for students who received services provided by an AmeriCorps member?

Question #7: Did motivation and interest in school increase, decrease, or remain the same for 

students who received services provided by an AmeriCorps member?

Question #8: Did students who received services provided by an AmeriCorps member have an 

improved attitude towards school?

Question #9: Did academic performance increase, decrease, or remain the same for students 

who received services provided by an AmeriCorps member?

Question #10: Overall, how would you rate the effectiveness of improving academic 

performance in regarding to the services provided by the AmeriCorps member? Scale: 1 

(failure) to 10 (success)

Question #11: How would you rate the overall success of the AmeriCorps program in your 

school? Scale: 1 (failure) to 10 (success)

Question #12: What were the most important benefits/outcomes of the program?

Question #13: Any factors that contributed to the success or failure of the program?

Question #14: Any other comments?
20 | Page

Treatment School: Edinboro Elementary (General McLane School District) 

Question #1 Yes

Question #2 Both

Question #3 DIBELS and MAP assessments

Question #4 20-30 minutes four times per week

Question #5 Troop care packages 

Question #6 Increased

Question #7 Increased

Question #8 Yes

Question #9 Increased

Question #10 8 

Question #11 8 

Question #12 Able to provide additional interventions to more students

Question #13 Success and Challenge: recruiting the “right” person

Question #14 Recently there has been a lot of questions around the signing off on timesheets

Treatment School: Elk Valley (Girard School District)

Question #1 Yes

Question #2 Both

Question #3 DIBELs Next and Accessing the Code

Question #4 60 mins/day, three or four days a week 

Question #5 School beautification; Targeted STEM activities; Diversity lessons

Question #6 Increased

Question #7 Increased

Question #8 Improved

Question #9 Increased

Question #10 10

Question #11 8 

Question #12 Meet the academic, social, and emotional needs of more students; Improved 

relationships with families

Question #13 Challenge: recruiting the right people with the necessary skills 

Question #14 The commitment of members shows true dedication to service

Treatment School: Fairview Elementary School (Fairview School District) 

Question #1 Yes

Question #2 Both
21 | Page

Treatment School: Hillview Intermediate School – 2017 information. Multiple attempts were made to 

update answers. 

Question #1 Yes

Question #2 Both

Question #3 AIMSweb and Progress Monitoring

Question #4 1⁄2 hr/each day

Question #5 Informal

Question #6 Couldn’t recall specifics 

Question #7 Yes

Question #8 Yes – particularly in grades 6-12

Question #9 Yes

Question #10 Yes

Question #11 10

Question #12 9 

Question #13 Provide 1:1 support to most needy students; Relationship building

Question #14 Quality members who are dedicated to service; Lack of communication from SMILES 

– change in staff

Question #15 Very successful program – blessed and appreciative 

Question #3 DIBELS and MAP assessments

Question #4 20-30 min/daily 

Question #5 Collected blankets for NICU; collected personal care items for VA Group; and 

provided breakfasts for veterans

Question #6 Increased

Question #7 Increased

Question #8 Yes

Question #9 Increased

Question #10 10

Question #11 10

Question #12 Ability to focus on building relationships and provide academic support for 

underserved population

Question #13 Challenge: limitations on what members can do & when they can provide services 

(no extracurricular – which would lead to better/more relationships)

Question #14 Service learning is now a part of school district culture. It’s now in the district’s 

mission statement. Good to model service for students, staff, and community. 
22 | Page

Treatment School: Iroquois Elementary (Iroquois School District)

Treatment School: Kane Elementary (Kane School District)

Question #1 Yes

Question #2 Both

Question #3 MTSS – Multi-Tiered Systems of Support (meet monthly) and Universal Screenings 

[Tier 1, 2, 3]

Question #4 Reading Buddies (10-15 students) about 45 minutes/daily; 90 min reading daily in 

classrooms; Push in math support for 2 hrs/day; Daily assistance with intervention 

time

Question #5 Book Swap – used book swap with entire school; Bingo – books for prizes

Question #6 Increased

Question #7 Increased

Question #8 Yes

Question #9 Increased

Question #10 10

Question #11 9.5

Question #12 Provide services that couldn’t otherwise support; Individualized support to 

vulnerable population; Improved test scores and grade level success; Practical and 

meaningful experience for member. 

Question #13 Challenge: recruitment of members

Question #14 Working with SMILES has been positive. Very organized and good team to partner 

with.

Question #1 Yes

Question #2 Both

Question #3 aimswebPlus; Classroom Diagnostic Tool; A2i; and Fountas and Pinnell

Question #4 30 min/daily

Question #5 STEM Night; Enrichment Summer Camp; Penny War Fundraiser; Positive Behavior 

Video Project; Coordination of Grade Level Library

Question #6 Increased

Question #7 Increased

Question #8 Yes

Question #9 Increased

Question #10 8 

Question #11 8 

Question #12 Support of students, faculty, and community; Connections to community and school; 

Relationship building 

Question #13 Challenge: Limitations of the types of services the members can engage in

Question #14 Members were huge help with food distribution during COVID
23 | Page

Treatment School: McKean Elementary (General McLane School District

Question #1 Yes

Question #2 Both

Question #3 DIBELS and MAP assessments

Question #4 20-30 minutes four times per week

Question #5 Transition Books for 4th to 5th Grade and Preschool to Kindergarten

Question #6 Increased

Question #7 Increased

Question #8 Yes

Question #9 Increased 

Question #10 8 

Question #11 8 

Question #12 Provide support to students have learning needs

Question #13 Success and Challenge: recruiting the “right” person

Question #14 Would like to have more consistency of timing of application each year

Treatment School: North Clarion Elementary (North Clarion School District)

Question #1 Yes

Question #2 Small Group

Question #3 aimswebPlus

Question #4 30-40 min/daily

Question #5 Community Garden & Trail; Goodwill Clothing Drive

Question #6 Increased

Question #7 Increased

Question #8 Yes

Question #9 Increased

Question #10 8 

Question #11 7 

Question #12 Provide academic support to needy students

Question #13 Success: when you have the “right” person and education award for members

Challenge: recruitment

Question #14 It’s always good to have more adults helping students 
24 | Page

Treatment School: North East Intermediate School (North East School District) 

Question #1 Yes 

Question #2 Both

Question #3 Teacher referral; PSSA scores; MAP assessments; Study Island

Question #4 3 sessions/week for 60 minutes each session

Question #5 Recycling Program; Kindness Rocks Program; Spring into Action Clean-Up

Question #6 Increased

Question #7 Increased

Question #8 Yes

Question #9 Increased

Question #10 10

Question #11 9 

Question #12 Increase in student achievement; Relationship building with students & families to 

improve school engagement; Support to staff

Question #13 Success: quality of member; Challenge: recruitment

Question #14 Really appreciate the program and what we’re able to offer our families. 

Treatment School: Northwestern Elementary (Northwestern School District)

Question #1 Yes

Question #2 Both

Question #3 Classroom Diagnostic Tests 

Question #4 20-25 minutes/daily 

Question #5 Presentation on Proper Disposal of American Flag 

Question #6 Same

Question #7 Same

Question #8 N/A

Question #9 Same

Question #10 3 

Question #11 5 

Question #12 Success: good experience for members; hiring former members

Challenge: community and staff perception of program; recruitment 

Question #13 Increased adult contact with students

Question #14 Change in district leadership affected success of program 
25 | Page

Treatment School: Punxsutawney Elementary School (Punxsutawney School District) 

Question #1 Yes

Question #2 Both

Question #3 Via IST Program – PSSA scores, Classroom Diagnostic Tool, STAR, Study Island

Question #4 60 mins/daily

Question #5 Beautification of Courtyard; Google Classroom Resources, Cafeteria Mural, Recess 

Equipment 

Question #6 Increased

Question #7 Increased

Question #8 Improved

Question #9 Increased

Question #10 10

Question #11 10

Question #12 Support for teachers and extra support for students

Question #13 Challenge: recruitment – right person with appropriate skill set

Question #14 Have had some of the best members over the past couple of years

Treatment School: Reynolds Elementary (Reynolds School District)

Question #1 Yes

Question #2 Mainly small group

Question #3 DIBELS, Classroom Diagnostic Tests; local assessments

Question #4 30 min/day

Question #5 Toiletries care packages for overseas service members; Food Drive for Super Bowl

Question #6 Same

Question #7 Increased

Question #8 Yes

Question #9 50% Increased

50% Same

Question #10 6 

Question #11 8.5

Question #12 Many current teachers are former members; Ability to build positive relationships 

with specific students; Help to build student confidence; Male member was positive 

role model for many students

Question #13 Success: quality of candidates

Challenges: needy population; Lack of follow-through of families; Increased mental 

health issues of students; Population of available candidates; Background and skills 

of members.

Question #14 No additional comments
26 | Page

Qualitative Data Analysis

Some general themes that emerged from the qualitative analysis:

• Most schools provided both one-on-one and small group focused academic assistance 

to students.

• A variety of assessments were used to identify students to receive AmeriCorps services.

• The dosage varied by host site.

• All members engaged in service learning projects that benefitted the school community 

and/or the community at large. 

• Organization, completion of assignments, motivation and interest in school, and 

academic performance increased or improved as a result of AmeriCorps services.

• On a scale of 1 (failure) to 10 (success), eight out of twelve schools rated the services of 

AmeriCorps members an 8 or higher relative to the effectiveness of improving academic 

performance 

• On a scale of 1 (failure) to 10 (success), eight out of twelve schools rated the overall 

Keystone SMILES AmeriCorps program an 8 or higher. 

• Support to teachers, support to students, increased academic performance, and 

relationship building were benefits of the AmeriCorps program.

• Finding the right member is a key to a successful AmeriCorps program.

• Availability of qualified personnel hinders the ability to get the right person to serve as a 

member. 

• Host sites (schools) couldn’t provide the educational experience they do without the 

AmeriCorps program.

Conclusions 

Eight of the twelve (67%) treatment schools involved in the evaluation scored better than the 

comparison schools on at least three of the four measurements:

• Year to Year Average Growth for Total Number of Students

• Year to Year Average Growth for Historically Underperforming Students

• Average Growth 2015-2019 for Total Number of Students

• Average Growth 2015-2019 for Historically Underperforming Students 

Based on the quantitative data, there is evidence that receiving services from Keystone SMILES 

AmeriCorps results in higher test scores on the 4th grade PSSA mathematics assessment. Four 

years of PSSA data was analyzed for this evaluation. It is recommended that the review of the 
27 | Page

data continues to determine if similar findings persist. Additionally, now that the revised English 

Language Arts (ELA) PSSA assessment has been in existence for several years, it is 

recommended that 4th grade ELA also be included in the evaluation of the Keystone SMILES 

AmeriCorps program since many members assist students with the acquisition of literacy skills. 

It was noted several times during the host site interviews, that recruitment of qualified 

candidates is a major challenge for schools. School administrators cited that in northwestern 

Pennsylvania there are fewer and fewer students majoring in education and social services, 

thus limiting the number of qualified candidates for available teaching positions within their 

schools, and even less to serve as AmeriCorps members. Recruitment of members seems to be 

a very real risk to the success of the Keystone SMILES AmeriCorps program. 

As noted in the Study Limitations section, when the evaluation began over five years ago, the 

original sample size included 17 treatment schools. A total of 12 schools participated in the 

Keystone SMILES AmeriCorps program for all four school years of this program cycle study. 

While some schools exited the program, others entered. A component should be included in 

future evaluations to include some data from those schools that participated in the program for 

less than the entire evaluation cycle. 

It should again be noted that the scope of this evaluation was limited to one subject area 

(mathematics) in one grade level (4th grade). It is advised that in the future a more 

comprehensive evaluation model should be considered in an effort to assess all aspects of the 

Keystone SMILES AmeriCorps program. 

During the interview process with school personnel, the following statements were recorded: 

• “Because of the AmeriCorps Program and the support it provides, we have fewer 

students needing intensive interventions.”

• “AmeriCorps enables us to have the majority of our students working at or above grade 

level.”

• “With AmeriCorps, we’re able to have extra sets of hands and eyes helping our students, 

as well as extra hearts showing them love.”

• “It’s a great opportunity for young people to gain experience and serve their 

community.”

• “AmeriCorps is a quality program that allows us to provide more support to our 

families.”

• “We don’t want to think of a day when we don’t have an AmeriCorps Program in our 

district. We really don’t know how we’d be able to offer the same support to our 

students.” 

• “AmeriCorps is a cost-effective way to have more adults helping students in our school.”
28 | Page

The themes that emerged from the host site interviews indicate that schools heavily depend on 

the Keystone SMILES AmeriCorps program to provide a quality educational experience for all 

students, especially those students needing extra support. Without the services provided by 

AmeriCorps members in schools, the students most in need of interventions, academic 

assistance, and additional support would simply not receive the extra assistance they need to 

be successful.