West Virginia University Research Corp

Energy Express 

Evaluation 2019 

AmeriCorps State

Written by

Ben Edmonds, Ph.D.

Emeritus Faculty

Indiana University

Submitted to

Andrea Price

Director, Energy Express

October 16, 2019
1 

TABLE OF CONTENTS

 Page

Introduction 2 

Energy Express: The Program and its Goals 3 

Data Collection: Instruments and Procedures 5 

Results 8 

Discussion and Recommendations 19 

References 24 

Tables

Table 1 Demographic Characteristics of All Children Enrolled in Energy Express 

2019 

9 

Table 2 Demographic Characteristics of the Children in the Sample and Control 

Group

10 

Table 3 Percent and Number of Children who Increased or Maintained Reading 

Achievement Scores

11 

Table 4 Results of the Wilcoxon Signed Ranks Test in Z-Scores on the WoodcockJohnson. 12 

Table 5 Comparison of Groups Conducted Using the Mann-Whitney U test on 

Gain Scores for W 

14 

Table 6 Demographic Characteristics of the AmeriCorps Volunteers 15 

Table 7 Percentage of Mentors who Maintained or Increased Scores in a Domain 17 

Table 8 Percentage of Community Coordinators who Maintained or Increased 

Scores in a Domain

19 

Appendices

Appendix A AmeriCorps Mentor 2019 Service Surveys 25 

Appendix B AmeriCorps Community Coordinator 2019 Service Surveys 28 
2 

Introduction

Energy Express is a West Virginia University Extension program that provides services 

to children living in rural West Virginia in either a five- or six-week format during the summer 

break. In 2019, children in 79 low-income rural communities across 38 West Virginia counties 

benefited from this program. Energy Express receives funding from three main sources: The 

Corporation for National and Community Service through the West Virginia Commission for 

National and Community Service, the West Virginia Department of Education and the Arts, and 

the West Virginia Department of Education, Office of Child Nutrition. The focus of the program 

is to promote school success and, in order to do so, Energy Express sets both nutritional and 

scholastic goals for the program each year. The program attempts to meet nutritional goals by 

providing the children who participate with a nutritious breakfast and lunch each day. Scholastic 

goals are addressed through activities that aim to maintain or improve the participants’ reading 

skills over the summer break. Past evaluations of the program’s effectiveness with respect to its 

scholastic goals have found that Energy Express is successful in helping a majority of 

participating children to maintain or improve their scores on reading achievement tests (as 

measured using three subtests of the Woodcock-Johnson Test of Individual Achievement-III

[WJ-III]). This result is important, given the evidence showing particularly steep learning losses 

for low-income children as a result of out-of-school-time (what has been referred to as the 

summer slide). In addition to goals concerning the children who participate in the program, 

Energy Express also aims to engage a significant number of AmeriCorps volunteers who staff 

the program in this community service effort. The AmeriCorps members are largely made up of 

college students who volunteer at Energy Express sites either as mentors or community 

coordinators. 
3 

Energy Express: The Program and its Goals 

Energy Express 2019 served 79 low-income communities in West Virginia, with 

participants ranging from kindergarteners to incoming sixth graders. Each summer, the reading

program is attended by as many as 3000 children. The centers provide programming on differing 

schedules of either four or five days per week for either five or six total weeks. The Energy 

Express program was conceptualized and implemented as a six-week program lasting four hours 

per day. Subsequent action of the West Virginia State Board of Education allowing county 

boards of education the option of establishing calendars with shorter summer breaks presented a 

problem for the Energy Express program. To combat the effect this change would have on the 

program, in 2015 Energy Express sought and received approval to give individual sites the 

option of reducing the program to five weeks with five-hour days rather than six weeks with 

four-hour days. As a result, over half of the programs were offered in the shorter five-week

format in the summer of 2019. 

The program attempts to meet goals concerning healthy nutrition for the participating 

children by providing participants with a family-style breakfast and lunch each day they attend. 

In addition to receiving two nutritious meals per day, the children take part in a literature-based 

language arts curriculum. This curriculum is implemented by the AmeriCorps members who

volunteer at the different Energy Express sites each summer. These members complete a threeday statewide site team training regimen before participating in the program. The Energy 

Express curriculum focuses predominantly on reading, and the AmeriCorps members engage the 

children in various activities, such as drama, writing, and art, all of which are intended to provide 

participants with multiple opportunities to practice their reading skills. 
4 

Three goals were established for the 2019 Energy Express program, all of which concern 

the children. The program sought to have at least 3060 children enroll in the program (objective 

1). In addition, the program aimed to have at least 3000 of the enrolled children (objective 2) 

attend at least 50% of the days the program is in session. For the children who participate, the 

program aimed to ensure that at least 300 children improve their reading achievement by one 

month (objective 3). 

(1) The Energy Express program will enroll at least 3060 children entering first through 

sixth grade 

(2) Over 3000 children will complete participation in Energy Express, where a child who 

attends at least 50% of the scheduled days is considered to have completed the 

program 

(3) This summer 300 of the children completing the program will increase their reading 

achievement by one month as measured by the Woodcock-Johnson III achievement 

test

For 2019 the program added a new dimension to the evaluation of its effectiveness. In 

addition to evaluating the goals stated above for children participating in Energy Express, the 

program decided to compare reading achievement of participants to that of a control group; that 

is, similar children who did not participate in Energy Express. To that extent, the program 

focused on the following four research questions: 

(1) Do children that participate in Energy Express maintain their reading skills 

throughout the summer? 

(2) Do children that participate in Energy Express increase their reading skills throughout 

the summer? 
5 

(3) To what extent do Energy Express's participating children increase their reading skill? 

(4) Do children that do not participate in Energy Express experience the effects of the 

"summer slide" and, thereby not maintain reading skills throughout the summer? 

In order to make a valid comparison, the program strove to select children for the control 

group with demographics comparable to those enrolled in Energy Express. The sampling 

procedures used to select the control group were the same as those used to establish the sample 

of Energy Express students who were administered the WJ-III. The sample was stratified based 

on geography to accurately represent SES. Twelve sites within five counties were used to draw 

students for the control group. These counties represented the range of SES found throughout the 

state of West Virginia, thus representing income diversity found in the Energy Express program. 

Additional efforts were made to ensure similar representation of children with special needs, 

children of color and children in the same grades. 

The data collected from the Energy Express children and the control group were analyzed 

in order to determine whether these three objectives had been successfully met for 2019 and to 

answer the four research questions posed in the grant. 

Data Collection: Instruments and Procedures

Reading Achievement: Woodcock-Johnson Test of Individual Achievement-III (2001 

Edition) 

The WJ-III (2001 Edition) was used to measure the reading achievement of the children 

who participated in Energy Express 2019 and of those nonparticipants selected for the control 

group. This test comprises a wide-ranging set of tests of academic achievement, which can be 

administered individually. To assess reading achievement, three reading subtests were 
6 

administered: word/letter identification, Reading Fluency, and Passage Comprehension. The 

Letter/Word Identification subtest requires that children name letters or words printed in large 

type (test reliability = .91). The Reading Fluency subtest measures reading ease and speed (test 

reliability = .87). Finally, the Passage Comprehension subtest asks children to read a short 

passage and to provide a missing key word (test reliability = .83), thereby measuring the 

student’s comprehension of the passage. A final score is calculated using scores from each of the 

three subtests and, as such, provides a comprehensive measure of reading achievement. This 

final score is known as the Broad Reading composite and is not a simple average of the scores 

achieved on the three subtests. The WJ-III has a specially designed standard score, the W score, 

which is used to derive all the other scores in their reporting system (Jaffe, 2009). The equal 

interval property of the W score makes it well suited for measuring progress over time. For this 

reason, the W score was used in the analysis of student achievement.

These three subtests were administered to a stratified random sample of 559 children 

participating in Energy Express and to a control group of 240 children. The subtests were given 

to Energy Express participants during the first two days of the summer session and again during 

the final two days of the program. Children in the control group were tested at the end of the 

school year and then again at the beginning of the following school year, approximating the same 

test/retest interval as the Energy Express participants. Of the 559 Energy Express participants, 

546 completed the program (i.e. attended at least 50% of the scheduled days the program was in 

session). Of the 240 children recruited for the control group, 238 completed both the pre-and 

post-tests. 

AmeriCorps Surveys 
7 

AmeriCorps mentor service survey. The same online service survey was administered 

to AmeriCorps mentors both immediately after they had completed the three-day statewide 

training program but before the Energy Express program had started (pre-service survey) and 

after they had finished the Energy Express program (post-service survey). The survey questions 

were divided into four sections: Confidence, Knowledge, Personal Efficacy, and “about you” 

(background questions). In the first section, mentors were asked to rate how confident they felt 

with respect to 18 skills using a 5-point Likert scale; a response of “1” indicated that the 

respondent was “not very confident,” whereas a score of “5” meant s/he was “very confident” 

with respect to the skill in question. The second section presented nine true/false questions 

intended to address knowledge of subject matter necessary for the effective implementation of 

the Energy Express curriculum. Section three was comprised of nine statements intended to 

measure attitudes towards community service, and to which mentors were asked to respond using 

a Likert scale, with responses ranging from “1,” which would indicate that the statement was of 

little importance to the respondent, to “5,” which would mean that the respondent found the 

statement to be very important. Finally, all participants responded to six background questions. 

The AmeriCorps mentor service survey can be found in Appendix A1

. 

AmeriCorps community coordinator service survey. The community coordinator 

survey was also administered immediately after training (pre-service survey) and at the end of 

the Energy Express program (post-service survey) via the Internet. This questionnaire examined 

the same four areas as the mentor questionnaire: Confidence (14 questions), Knowledge (7 

questions), Personal Efficacy (9 questions), and “about you” (6 questions). As was the case for 

1 In surveys administered to both AmeriCorps mentors and community coordinators, the labels for sections one and 

three did not appear to match the actual areas being measured. In particular, although section one was labeled 

“Confidence” on the survey, the items seemed designed to measure personal efficacy, and whereas section three was 

labeled “Personal efficacy”, the items it contained aimed to measure attitudes towards community service. 
8 

the mentor surveys, the questions on Confidence and Personal Efficacy required a response on a 

Likert scale, whereas the seven knowledge questions were True/False. The content of the 

questions, however, changed for all sections except for section three (Personal Efficacy), which 

reflects the fact that the AmeriCorps mentors and community coordinators played distinct roles 

in the Energy Express program. For the section on Personal Efficacy, all AmeriCorps members

— both mentors and community coordinators — saw and responded to the same questions. The 

service survey for the AmeriCorps community coordinators can be found in Appendix B. 

Results

Children

Each summer, over 3000 children take part in the Energy Express program across the 

state of West Virginia. A total of 3092 children enrolled for the 2019 program. Of those enrolled, 

3009 attended with 2368 (78.7%), attending Energy Express for at least 50% of the scheduled 

days in session. This year’s attendance was nearly the same as last year’s with only three fewer 

children having completed participation in Energy Express 2019. Following is a discussion of the 

demographics of the children in the program and the control group along with results of the 

Energy Express program and the comparison study. 

Demographics. Table 1 lists grade level, disability, and gender for all children attending 

Energy Express 2019. To assess whether the program met its reading achievement goal a subset 

of children was administered the WJ-III. The WJ-III was also administered to a control group to 

compare the performance of children attending Energy Express with non-participants. The 

demographic characteristics for the children belonging to the stratified random sample and who 

completed the program (n = 546) and children in the control group (n = 238) are reported in 
9 

Table 2. A comparison of the data for all Energy Express participants and the sample indicates a 

high degree of demographic similarity between the groups. Likewise, a comparison of the 

demographics of children in the sample with children in the control group suggests a high degree 

of similarity (see Table 2). The control group was comprised of children in grades K-5 who 

attended 12 of the same schools attended by Energy Express participants and were all eligible to 

attend Energy Express. 

Table 1. Demographic Characteristics of All Children Enrolled in Energy Express 2019. 

Characteristic Frequency Percent

Grade 

 Kindergarten

 First

 Second

 Third

 Fourth 

 Fifth

 Sixth

 Not Reported 

110 

492 

629 

600 

550 

402 

254 

55 

3.6 

15.9 

20.3 

19.4 

17.8 

13.0 

8.2 

1.8 

Ethnicity

 White 2673 86.4 

 Afr. American 190 6.1 

 Asian 22 .7 

 Pacific Islander 3 .1 

 Mixed 129 4.2 

 Other 47 1.5 

 Not Reported 28 .9 

IEP Status 

 None 

 Speech

 LD

 BD

 MI

 GI

 Other

2243 

271 

203 

16 

18 

30 

99 

72.5 

8.8 

6.6 

.5 

.6 

1.0 

3.2 

 Not Reported 212 6.9 

Gender

 Female

 Male

 

1541 

1551 

49.8 

50.2 

Note. LD = learning disability; BD = behavioral disability; MI = mental impairment; GI = gifted.
10 

Table 2. Demographic Characteristics of the Children in the Samples and Control Group.

Energy Express 

Participants

Control Group 

Characteristic Frequency Percent Frequency Percent

Grade 

 Kindergarten

 First

 Second

 Third

 Fourth 

 Fifth

 Sixth 

14 

102 

131 

101 

103 

54 

41 

2.6 

18.7 

24.0 

18.5 

18.9 

9.9 

7.5 

30 

46 

54 

55 

38 

15 

12.6 

19.3 

22.7 

23.1 

16.0 

6.3 

Ethnicity

 White 470 86.1 190 79.8 

 Afr Amer 40 7.3 14 5.9 

 Mixed 20 3.7 19 8.0 

 Asian 4 .7 

 Other 11 2.0 

 Not Reported 15 6.3 

IEP Status 

 None 

 Speech

 LD

 BD

 MI

 GI

 Other

381 

55 

45 

3 

3 

14 

19 

69.8 

10.1 

8.2 

.5 

.5 

2.6 

3.5 

217 

6 

3 

12 

91.2 

2.5 

1.2 

5.0 

 Not Reported 26 4.8 

Gender

 Female

 Male

272 

274 

49.8 

50.2 

120 

118 

50.4 

49.6 

Note. LD = learning disability; BD = behavioral disability; MI = mental impairment; GI = gifted.

Reading achievement of Energy Express participants. Three sections of the WJ-III 

were administered to a subset of children taking part in Energy Express during the first two days 

of the program and again during the final two days. Records of the 546 children in the subset 

who completed the program are used in this analysis. All 546 children included in the random 

sample completed both the pre- and post-tests for all three subtests. In what follows, presented 

first are the descriptive statistics from the Broad Reading composite scores and from each of the 

three subtests, before analyzing the described trends using Wilcoxon Signed Ranks tests. 
11 

The results of the WJ-III test are slightly better than last year’s in all areas except 

Reading Fluency. A comparison of the Broad Reading composite scores from before and after 

the Energy Express summer program showed that 68.7% of the children of this sample 

maintained (4.2%) or improved (64.5%) their scores on reading achievement over the course of 

the Energy Express 2019 session (see Table 3). Likewise, children performed better in 

Letter/Word Recognition with 69.0% either maintaining (4.8%) or improving (64.3%) their 

scores and in Passage Comprehension with 67.6% either maintaining (9.0%) or improving 

(58.6%). Unfortunately, the scores for Reading Fluency did not follow this trend. Only 56.6% of 

the children maintained (18.1%) or improved (38.5%). Table 3 indicates the percentage of 

children who increased or maintained their reading scores on each of the three subtests and on 

the Broad Reading composite over the Energy Express program in 2019. On two of the three 

subtests and on the overall score, these results show that two-thirds of participants either 

maintained or improved their scores from the beginning of the program to the end. These results 

point to the efficacy of the reading achievement program put into place by Energy Express. 

The trends described were further analyzed using the Wilcoxon Signed Ranks test, which 

is a non-parametric test used when assumptions of normality are not met (Field, 2005). The 

Wilcoxon test ranks the differences between the post- and pre-test scores, retaining the sign (+ or 

-), and reports the number of positive ranks, negative ranks, and ties. This ranking system allows 

Table 3. Percent and Number of Children who Increased or Maintained Reading Achievement

Scores. 

Increase Maintain Total

Woodcock-Johnson III N n % n % n % 

Subtest 

 Letter/word identification 546 351 64.3 26 4.8 377 69.0 

 Reading fluency 546 210 38.5 99 18.1 309 56.6 

 Passage comprehension 546 320 58.6 49 9.0 369 67.6 

Broad reading composite 546 352 64.5 23 4.2 375 68.7 
12 

for the identification of the number of participants who have improved (higher scores on the 

post-session tasks or positive ranks), maintained (similar scores on the pre- and post-session 

tasks or ties), or lost ground (lower scores on the post-session tasks or negative ranks) in reading 

achievement. The test statistic used in the Wilcoxon test is based on the smaller number of ranks 

(positive or negative), and scores are converted to z-scores. This means that a z-score based on 

negative ranks indicates more post-session gains, whereas one based on positive ranks indicates 

the opposite. All the z-scores reported in this evaluation are based on negative ranks unless 

otherwise noted, and thus denote an overall trend towards gains from pre- to post-session. The 

resulting z-scores are displayed in Table 4. Improvement was found to be statistically significant 

on two of three subtests. Overall reading improvement, or the Broad Reading composite, was

statistically significant at the .001 level. 

Table 4. Results of the Wilcoxon Signed Ranks Test in z-Scores on the Woodcock-Johnson. 

Woodcock-Johnson III z-scores

Subtest 

 Letter/word identification -9.038* 

 Reading fluency -.451**

 Passage comprehension -7.014* 

Broad reading composite -8.805* 

Note. *p < .001, **Based on positive 

ranks

In addition to providing information concerning maintenance and improvement, WJ-III 

Broad Reading composite scores can be used to derive an age equivalent score. An age 

equivalent score indicates at what developmental age a child is reading. By comparing the pre- to 

the post-test age equivalent scores for Broad Reading it was found that 336 children (62%) 

gained at least 1 month in reading. Furthermore, the children participating in the program 

experienced on average an age equivalent gain of 3.1 months. 
13 

Comparison study. In response to a grantor’s request, the Energy Express program 

conducted a comparison study to determine whether children completing the program made 

larger reading gains over the summer than children who did not participate. The program 

recruited 240 children with demographic characteristics similar to those of the participants to act 

as a control group. Of those recruited, 238 completed both the pre- and post-tests and were used 

as the control group. 

The methodology outlined in the grant calls for the use of a t-test to determine whether a 

significant difference exists between the two groups; however, the data collected violated the 

assumptions of normality. Thus, a nonparametric measure, the Mann-Whitney U, was employed 

to compare the groups (Kirk, 1999). Like the Wilcoxin signed-rank test, the Mann-Whitney test 

bases its calculations on ranks of scores rather than the scores themselves. Instead of comparing 

means, the Mann-Whitney ranks the dependent variable across groups and then uses the sum of 

ranks for each group for the calculation of its statistic. In order to use this test, gain scores were

calculated by subtracting the pre- from the post-test W score for the three subtests and the Broad 

Reading composite. 

Table 5 contains the results of the Mann-Whitney test. The Mann-Whitney test ranks 

scores from lowest to highest; therefore, the group with the higher mean rank has the greater 

number of higher scores. The Energy Express children had higher mean ranks in all subtests and 

in the Broad Reading composite. The difference between the groups was significant in all cases 

except Passage Comprehension. 
14 

Table 5. Comparison of Groups Conducted Using the Mann-Whitney U test on Gain 

Scores for W

Mean Ranks

Control 

Energy 

Express U Z 

Asymptotic Sig.

(2-tailed)

Letter/Word ID 345.68 412.91 53830.0 -3.824 0.000 

Reading fluency 366.45 403.86 58773.5 -2.137 0.033 

Passage comp. 386.68 395.04 63589.0 -0.475 0.635 

Broad reading 357.95 407.56 56750.5 -2.823 0.005 

Age equivalent gains were also calculated for the control group. A gain of at least 1 

month was recorded for 132 of the 238 children (55%). On average the group increased their 

reading by.5 months. 

AmeriCorps Members

AmeriCorps members were recruited to fulfill two distinct roles during the Energy 

Express program: mentors and community coordinators. The mentors worked directly with the 

children, implementing the language arts based curriculum adopted for the Energy Express 

program. The community coordinators, on the other hand, were responsible for overseeing 

volunteers (generally family members or people from the community) at the Energy Express 

sites and ensuring that they interacted effectively with the participants. All AmeriCorps members 

were to individually complete a pre-service survey and a post-service survey. 

Records for 472 members were made available by the Energy Express office for analysis. 

All the members completed at least one of the surveys; however, only 232 completed both the 

pre- and post-surveys. Additionally, not all surveys submitted were usable. The following 

statistics refer only to those members who responded to the survey items with usable data. 

Demographics. The background information provided by AmeriCorps members on their 

pre- and post-service surveys revealed that 41 percent of the members fulfilling the role of 

mentor had had previous contact with the Energy Express program, either as volunteers and/or as 
15 

participants when they were children (see Table 6). Moreover, it was found that more than 78 

percent of AmeriCorps volunteers were women. As for the ages of the volunteers, the mentors 

were somewhat younger than the community coordinators. The mentors fell in the three youngest 

age bands with 96% of their group falling in the 18 to 24-year range while the community 

coordinators had members falling in all six age bands and only 66% fell into the 18 to 24 age 

range. 

Table 6. Demographic Characteristics of the AmeriCorps Volunteers.

Mentors Community Coordinators 

Characteristic Frequency Percent Frequency Percent

Gender

 F 

 M 

148 

34 

81 

19 

27 

14 

66 

34 

Previous experience 74 41 15 37 

Attended Energy Express as a child 25 14 N/A N/A

Pre- and post-service surveys. Surveys administered before (pre-service survey) and 

after (post-service survey) the program were designed to assess the domains of Confidence, 

Knowledge, and Personal Efficacy for both mentors and community coordinators. For the current 

assessment, the focus will be on the overall results for each set of questions. For the first set of 

items, members had to rate their confidence with respect to a series of statements using a Likert 

scale that ranged from 1 (“not very confident”) to 5 (“very confident”). The second set of 

questions included True/False items assessing specific knowledge about the Energy Express 

reading program. The final set of questions also used a Likert scale to rate the importance of 

statements about community service from 1 (little importance) to 5 (very important)2. After 

2 The post surveys provided to this evaluator for inclusion in the report asks the respondent to rate the final section 

on a three-point scale; however, the data provided responses in a five-point spread. Therefore, the analysis was 

carried out using the five-point format.
16 

matching all results, there were 182 usable surveys from mentors3

, and 41 surveys from

community coordinators. Throughout this analysis, the results from the mentors and community

coordinators will be kept separate; given that these different roles necessitated different skills, 

this variable may very well lead to different outcomes on the pre- and post-service surveys. 

Mentors. The first section of the survey developed for the mentors was labeled 

Confidence and asked them to reflect on 18 items dealing with their role as a reading mentor4. 

The items cover skills germane to working directly with the children participating in Energy 

Express, as well as the ability to plan lessons, and mentors were asked to judge how confident 

they felt in their ability to carry out each item. The second section, labeled Knowledge, consisted 

of nine True/False statements on specific aspects of the reading program. Examples include 

“Word walls can help with vocabulary and writing.” and “Children must be quiet during reading 

time.” The final section was labeled Personal Efficacy and contained nine statements about 

community involvement for the respondent to rate the degree of importance such as “Helping 

others in my community.” 

Results from each survey were first analyzed by summing scores on each of the 18 

Confidence items, each of the nine Knowledge items and each of the nine Personal Efficacy 

items in order to create an overall score for each mentor in each domain. Confidence scores 

could range from 18 (response of 1 “not very confident” on each question) to 90 (response of 5 

“very confident” on each question). Knowledge scores could range from 0 to 9 (correct response 

= 1 point). Personal Efficacy scores could range from 9 (response of 1 “little importance” on 

each question to a high of 45 (response of “very important” on each question). 

3 191 mentor records had matching pre- and post-surveys, however, nine of them had responses out of range (>5) 

and were removed.

4 The pre-service survey contained 19 items, one of which was not included on the post-service survey. The analysis 

was conducted only using the 18 matching items.
17 

When overall scores on the pre- and post-service surveys were compared, it was found 

that the mentors had grown significantly in all three domains: Confidence, Knowledge, and 

Personal Efficacy. In fact, 79.2% of mentors either maintained (19.8%) or increased (56.6%) 

their overall level of reported Confidence over the course of Energy Express 2019. Likewise, 

90.1% of mentors either maintained (23.6%) or increased (66.5%) their scores on the Knowledge 

section and 73.6% maintained (29.1%) or increased (44.5%) their reported Personal Efficacy. 

The Wilcoxon Signed Ranks test was then conducted to assess the overall growth in each of the 

three domains among the mentors. On average, these individuals reported significantly greater 

Confidence, Knowledge, and Personal Efficacy on the post-service survey than on the preservice survey (see Table 7). 

Table 7. Percentage of Mentors who Maintained or Increased Scores in a Domain

Survey Section N Maintain Increase Total z sig. 

Confidence 182 19.8 56.6 79.2 —5.60 .000* 

Knowledge 182 23.6 66.5 90.1 -8.21 .000* 

Personal Efficacy 182 29.1 44.5 73.6 -2.82 .005** 

Note. *p < .001, **p < .01 

Community coordinators. The survey for the community coordinators was very similar 

to the one used with the mentors. It, too, was composed of three sections covering the domains of 

Confidence, Knowledge, and Personal Efficacy. The Confidence section included 14 items 

designed to assess how they perceived their level of confidence in carrying out their 

responsibilities in their role as coordinator. Seven of the items focused on direct interaction with 

volunteers, and the remaining seven items concerned the community coordinators’ own personal 

attributes and their ability to execute tasks related to the position of community coordinator. The 

Knowledge section was composed of seven True/False statements regarding policies and 
18 

practices of the Energy Express reading program that a community coordinator should know to 

perform their duties. The Personal Efficacy section was identical to the one used on the mentor 

survey. 

As was the case for the mentors, results from each survey were first analyzed by 

summing scores on each of the 14 Confidence items, each of the seven Knowledge items and 

each of the nine Personal Efficacy items in order to create an overall score in each domain for 

each mentor. Confidence scores could range from 14 (response of 1 “not very confident” on each 

question) to 70 (response of 5 “very confident” on each question). Knowledge scores could range 

from 0 to 7 (correct response = 1 point). Personal Efficacy scores could range from 9 (response 

of 1 “little importance” on each question to a high of 45 (response of “very important” on each 

question). 

When overall scores on the pre- and post-service surveys were compared, it was found 

that the results for the community coordinators were somewhat different from those for the 

mentors. While the Knowledge score increased significantly, the Confidence and Personal 

Efficacy scores did not. For the domain of Confidence, 73.2% of community coordinators either 

maintained (31.7%) or increased (41.5%) their overall level of reported Confidence over the 

course of Energy Express 2019. on the Knowledge section 100 % of community coordinators 

either maintained (29.3%) or increased (70.7%) their scores. Unlike the mentors, a significant 

increase was not seen in their reported Personal Efficacy although 75.6% maintained (39.0%) or 

increased (36.6%) their scores. The Wilcoxon Signed Ranks test was then conducted to assess 

the overall growth in each of the three areas among the community coordinators. On average, 

these individuals reported significantly greater Knowledge on the post-service survey than on the 

pre-service survey (see Table 8). 
19 

Table 8. Percentage of Community Coordinators who Maintained or Increased Scores in a 

Domain

Survey Section N Maintain Increase Total z sig. 

Confidence 41 31.7 41.5 73.2 -.981 .327 

Knowledge 41 29.3 70.7 100 -4.791 .000* 

Personal Efficacy 41 39.0 36.6 75.6 -.757 .449 

Note. *p < .001 

Discussion and Recommendations

The Energy Express summer program brings together children from low-income 

communities and AmeriCorps members (made up predominantly of college students), and the 

organizers expect that all involved will show a positive effect from the experience. In particular, 

program organizers established three objectives for the 2019 Energy Express program focused on 

student enrollment, participation, and achievement. For the first objective, it was expected that a 

minimum of 3060 children in grades 1-6 would enroll in the program. The second objective 

specified that at least 3000 children would complete the program (i.e., to have attended at least 

50% of scheduled days). The third objective targeted reading achievement, stating specifically 

that at least 300 children who complete the program would demonstrate a 1-month improvement 

in reading, as measured by the WJ-III. 

Additionally, this year the program posed four research questions. Three of the four 

questions centered on the reading achievement of children participating in Energy Express. The 

first two questions asked whether those children maintained or improved their reading. The third 

question asked to what extent they improved reading. The final question focused on the 

nonparticipants and asked whether they experienced “summer slide” in reading achievement. 
20 

The results presented in the previous sections will now be discussed considering each one of 

these objectives and research questions. 

With respect to the first objective, there was clear evidence that the goal of having 3060 

children entering first through sixth grade enroll in the Energy Express program was not met. To 

be counted as enrolled for the purpose of this objective, a child must have registered and 

attended at least one day. Although there were 3092 children registered, the records provided by 

the program did not have any attendance for 83 of the children. Therefore, the program fell short 

of the enrollment goal by 51 children. 

 Energy Express 2019 was not able to meet the second objective of having over 3000 

children complete the program. Attendance data provided by program staff indicate that 3009 

children attended at least one day with 2368 children attending at least 50% of their scheduled 

days in the program for a completion rate of 79%, 3.0 points higher than the completion rate for 

the 2018 program. A quick look at two demographic attributes, income and race, indicated that 

while race did not play a role in the completion rate, income did. Children of color were just as 

likely to complete the program as their White peers; however, children on free or reduced 

lunches were significantly less likely to complete the program than those not on the lunch 

program. 

The final objective, to have 300 of the children completing the program increase their 

reading achievement by at least 1 month was met. When Broad Reading composite scores from 

the WJ-III were analyzed, 337 (61.7%) of children increased their Broad Reading age equivalent 

scores by at least 1 month. The Broad Reading age equivalent for all children in the sample 

increased an average of 3.1 months over the course of the program. 
21 

The first two research questions posed in the grant asked whether children participating 

in Energy Express maintain or increase their reading performance over the summer. The 

information in Table 3 suggests that the answer to both questions is Yes. Looking first at the 

overall Broad Reading, which is the best indicator of general achievement reported, it is noted 

that over two-thirds of children in the program either maintained (4.2%) or improved (64.5%) 

their reading; a rate of improvement that is highly significant (p < .001). Furthermore, when the 

Broad Reading gains made by the Energy Express participants were compared to the control 

group it was found that program participants significantly outperformed nonparticipants (See 

Table 5). When looking closer at the components of the Broad Reading score the picture changes 

slightly. Considering the performance of the participants, significant gains were made in 

Letter/Word Identification and passage completion, but not in Reading Fluency. In fact, a 

troubling finding is that more children had lower post-test scores (43.4%) in Reading Fluency 

than higher scores (38.5%). Comparing subtest performance with the control group, it was found 

that the Energy Express participants had significantly greater gains in Letter/Word Identification

and Reading Fluency. 

The third research question asks to what extent Energy Express participants increase their 

reading skill. Using the age equivalent score for the Broad Reading composite, it was determined 

that participants increased their reading an average of 3.1 months over the six-week-long 

program. This is an especially promising result when one considers that this increase occurred 

during the summer, a time when little academic progress is traditionally made. In fact, the 

control group increased their reading skill by only 0.5 months during the same period. 

The final research question asks whether children who do not participate in Energy Express 

experience the phenomenon known as summer slide, which results in not maintaining their 
22 

reading skills throughout the summer. This question is addressed in this report by comparing the 

progress of the Energy Express participants with the control group. As noted earlier, the Energy 

Express participants significantly outperformed the control group on two of the three WJ-III

subtests as well as the Broad Reading composite. Furthermore, the participating children 

experienced 3.1 months of growth over the summer while the control group children only 

advanced 0.5 months. A Chi square analysis was performed on the age equivalent gains of the 

two groups and it was found that the Energy Express group had significantly more children (p < 

.05) who maintained or gained in reading than the control group. While this is not definitive 

affirmation that the control group had a loss of reading skills over the summer, it is ample 

evidence that they were significantly outperformed by the Energy Express children. These results 

suggest that the Energy Express program is effective in making a difference in the reading levels 

of the low-income children attending the program. 

Recommendations 

1) Energy Express was successful again this year in providing the disadvantaged youth 

of West Virginia with high-quality summer programming that helped children make 

significant gains in their reading levels over the summer. It is recommended that they 

continue providing such a program. 

2) As the program strives to compare the progress of participants to nonparticipants, it is 

of importance that proper sampling of children occurs. It is recommended that the 

program review and revise its sampling procedures. Specifically, the variables upon 

which stratification of the sample are based should be meaningfully selected, 

procedures for selection of the sample should ensure randomization, and procedures 

for obtaining the sample should be standardized. 
23 

3) Reading fluency scores show a troubling trend of decrease despite participation in the 

program. It is recommended that the program review its curriculum and consider 

some of the fluency-building interventions suggested by the publishers of the WJ-III

(Wendling, Schrank, & Schmitt, 2007). These interventions include repeated reading, 

passage previewing, assisted reading, and practicing words in isolation. These 

interventions may be beneficial for use with students individually or as part of a small 

group instructional program. 

4) Developing strong objectives upon which to determine the effectiveness of the 

program is critical to the evaluation process. When working with samples, the use of 

absolute numbers in the goal-setting process can be problematic (see Objective #3). It 

is recommended that the program reconsider the wording of such goals. This 

evaluator can be of assistance if needed. 

5) The information collected on participation of the AmeriCorps volunteers is an 

important part of the overall evaluation of the program. The gathering of accurate and 

usable data is critical to provide the program with the information it desires. It is 

recommended that the program review and revise its data collection instruments. 

Specifically, alignment of items on the pre- and post-service mentor surveys, 

alignment of response scales and directions for both post-service surveys, and 

response values for post-service mentor surveys need attention.
24 

References

Field, A. (2005). Discovering statistics using SPSS (2nd ed.). Thousand Oaks, CA: Sage 

Publications. 

Jaffe, L. E. (2009). Development, interpretation, and application of the W score and the relative 

proficiency index (Woodcock-Johnson III Assessment Service Bulletin No. 11). Rolling 

Meadows, IL: Riverside Publishing.

Kirk, R. (1999). Statistics: an introduction (4th ed.). Harcourt Brace: New York. 

Wendling, B. J., Schrank, F. A., & Schmitt, A. J. (2007). Educational Interventions Related to 

the Woodcock-Johnson III Tests of Achievement (Assessment Service Bulletin No. 8). 

Rolling Meadows, IL: Riverside Publishing 
25 

APPENDIX A

(Double click on document image to retrieve entire document.) 
26 
27 
28 

APPENDIX B

(Double click on document image to retrieve entire document.) 
29