AmeriCorps

Tutoring Outcomes

Study

Prepared by:
Prepared for the

Corporation for National Service

under

Contract #97-743-1004

Task Order 4

by

Abt Associates Inc.

55 Wheeler Street

Cambridge, MA 02138

Created in 1993, the Corporation for National Service engages more than 1.5 million Americans

annually in improving communities through service. The Corporation supports service at

national, state, and local levels through:

• AmeriCorps, whose members serve with local and national organizations to meet community

needs while earning education awards to help finance college or training;

• Learn and Serve America, which helps link community service and learning objectives for

youth from kindergarten through college as well as youth in community-based organizations;

and

• The National Senior Service Corps (Senior Corps), the network of programs that helps

Americans age 55 and older use their skills and experience in service opportunities that

address the needs of their communities. Senior Corps includes the Retired and Senior

Volunteer Program (RSVP), Foster Grandparent Program, and Senior Companion Program.

Upon request, this material will be made available in alternative formats for people with

disabilities.
AmeriCorps Tutoring Outcomes Study

February 28, 2001

Acknowledgements

This report is the final product of the AmeriCorps Tutoring Outcomes Study of the

AmeriCorps*State/National Literacy Programs, prepared by Abt Associates for the Corporation

for National Service’s Office of Evaluation. The report presents the study’s findings about the

effects of AmeriCorps tutoring programs on children’s reading achievement. 

The report represents substantial contributions from many individuals during the course of the

evaluation. Most importantly, the authors of this report—Marc Moss, Janet Swartz, Dawn

Obeidallah, Gerrie Stewart and Diane Greene—gratefully acknowledge the participation and

input of the AmeriCorps*State/National Program Directors who participated in the study. In

addition, we thank the principals and teachers in the participating schools who graciously

allowed us to come into their schools and classrooms and conduct the study. We thank them for

their contributions of time and information.

The authors wish to thank several other important contributors to the report. Throughout the

evaluation, the Corporation’s evaluation staff provided valuable insights and guidance. In

addition, the Technical Experts Wayne Osgood, Suzanne Graham and Barbara Wasik provided

valuable guidance on the reading research on tutoring programs and on key elements of the study

design.

Contents

Executive Summary.................................................................................. i

AmeriCorps......................................................................... i

Study Design ....................................................................... i

Major Findings.................................................................... ii

Chapter 1 Introduction....................................................................... 1

National Focus on Reading Achievement ............................ 1

Research on Effective Practices........................................... 2

AmeriCorps......................................................................... 3

The Tutoring Outcomes Study............................................. 6

Chapter 2 Description of Tutoring Programs and Activities............ 11

Student Characteristics ........................................................ 11

Tutor Characteristics............................................................ 14

Effective Tutoring Practices................................................. 16

Chapter 3 Program Effects on Student Achievement and

Classroom Behavior .......................................................... 25

Student Reading Performance.............................................. 25

Classroom Behaviors........................................................... 30

Summary............................................................................. 34

Chapter 4 Factors Affecting Student Reading Achievement............. 35

Introduction......................................................................... 35

Effective Tutoring Practices................................................. 35

Summary............................................................................. 39

Chapter 5 Summary and Conclusions................................................ 41

Appendix A Sample Selection and Weighting Procedures

Appendix B Study Measures

Appendix C Analytic Procedures and Results of Effective Practices and

Student Gains

AmeriCorps Tutoring Outcomes Study Executive Summary i

Executive Summary

This executive summary highlights the findings from the AmeriCorps Tutoring Outcomes

Study about the effects of the AmeriCorps tutoring program. The summary begins with a

description of AmeriCorps, followed by a brief discussion of the study design. The

remainder of the summary presents descriptive information about the tutoring programs,

tutors and students; the effects of the program on student reading achievement and classroom

behavior; and the relationship between program components and student gains in reading.

AmeriCorps

The AmeriCorps program is the largest initiative of the Corporation for National Service (the

Corporation), a public corporation established by the National Service Trust Act of 1993.

The Corporation’s mission is to provide opportunities for Americans of all ages and

backgrounds to engage in service that addresses the nation’s unmet needs in education,

public safety, the environment, and other areas.

Education-related activities have been a priority for the Corporation since its inception. Most

recently, the Corporation has directed substantial resources toward addressing the national

issue of children’s reading performance. This focus supports a national mandate as

articulated in the America Reads initiative: to help ensure that every child can read well and

independently by the end of the third grade. Toward that goal, the Corporation has

encouraged its programs to provide literacy services to young readers. A typical AmeriCorps

tutoring program in reading places members in schools and after-school programs to provide

tutoring to elementary students. During the 1999-2000 academic year, over 330 AmeriCorps

programs nationwide provided tutorial assistance in reading to more than 100,000 students in

grades one, two and three.

Study Design

To inform program staff, Congress, and other stakeholders about the effects of AmeriCorps

tutoring programs, the Corporation contracted with Abt Associates to conduct the

AmeriCorps Tutoroing Outcomes Study.

Research Questions

The study addressed the following broad research questions:

• What is the effect of participation in AmeriCorps’ tutoring programs on children’s

reading proficiencies and other classroom behaviors?

• Which components of the reading tutoring programs, considered to be effective practices,

are associated with improved student outcomes in reading and classroom behavior? Do
ii Executive Summary AmeriCorps Tutoring Outcomes Study

students in programs implementing more practices identified by research as effective,

show greater gains in reading than students in the other AmeriCorps tutoring programs?

Measures

The study collected four types of information to address these research questions:

• Reading Performance. Field staff tested students on the Woodcock-Johnson PsychoEducational Battery-Revised. Scores from four subtests were combined to yield two key

components of reading achievement: reading comprehension, based on passage

comprehension and reading vocabulary subtests; and reading skills, based on word attack

and letter-word identification subtests. In addition, teachers reported their view of

changes in students’ reading performance and attitudes in the classroom.

• Classroom Behavior. Teachers rated students using two short, standardized ratings

scales: the Behavioral Academic Self-Esteem (BASE) and the cooperation sub-scale from

the Social Skills Rating System (SSRS).

• Tutor Experiences. Tutors completed a brief questionnaire and an activity log to

describe their experience and training as well as the nature of tutoring sessions.

• Program Characteristics. AmeriCorps program directors completed a brief

questionnaire about their tutoring program.

Data Collection Schedule and Sample

The pretest took place in the late fall 1999/early winter 2000; the post-test occurred in spring

2000. Field staff collected information from 869 first, second and third grade students who

received tutoring from a nationally representative sample of 68 AmeriCorps programs

operating in schools and after-school programs.

Analytic Approach

Analysis of program effects examined student gains from pretest to post-test (post-test scores

minus pretest scores). For the standardized measures, these gains were compared to scores

reported by the test publisher to see whether students receiving tutoring gained more or less

than expected for students in their grade level. To examine the relationship of student gains

to program characteristics, a statistical technique called hierarchical linear modeling was

used to incorporate student-level and program-level information.
AmeriCorps Tutoring Outcomes Study Executive Summary iii

Major Findings

Tutoring Programs and Activities

• Students’ reading levels at pretest were generally in the low average to average range.

Across all grades, approximately 20 percent of students scored in the low average range

and 57 to 64 percent were average on reading comprehension. On reading skills,

approximately 30 percent of students scored in the low average range and about half were

average.

• Most AmeriCorps tutors were female (85 percent) and ranged in age from 17 to 80,

with the average age of 27. Nearly half of the tutors were between 20 and 25 years of

age. Most of the tutors have completed some college (82 percent) but only about onethird had a college degree.

• The majority of AmeriCorps tutoring programs implemented tutoring practices that the

research has found to be effective in helping struggling students learn to read. More

than 70 percent of programs were moderately or fully implemented, conducted formal

evaluations, coordinated tutoring activities with classroom reading instruction, and

provided training to tutors both before and during the course of tutoring.

Program Effects on Students’ Reading Skills

• Tutored students at all grade levels improved their reading performance from pretest to

post-test more than the gain expected for the typical child at their grade level. Reading

comprehension and reading skills started out below grade-level; by year-end, students

closed the gap and were reading at or near the grade-level expectation.

• The magnitude of reading gains was the same for students of different ethnic/racial

backgrounds. White students and non-White students showed the same level of gains.

• Boys in first and second grade showed greater gains than girls on some subtests.

Among first graders, boys made greater gains than girls in reading comprehension;

among second graders, boys had greater gains than girls on reading skills.

• According to the ratings of classroom teachers, most students at all three grades

improved their reading skills, at least to some degree over the course of the year as a

result of the AmeriCorps tutoring. However, these ratings had fairly low correlations

with the standardized tests of student reading performance.
iv Executive Summary AmeriCorps Tutoring Outcomes Study

Program Effects on Students’ Classroom Behavior

• Only boys in first and third grades showed significant gains on the BASE. Girls made

no measurable gain from pretest to post-test. However, at pretest, girls at all grade levels

received scores that were average to above average for elementary age students. Boys at

all grades scored slightly below average at pretest; those in the first and third grade

achieved significantly higher post-test scores (gains of 5 to 7 points).

• On the cooperation subscale of the SSRS, most students stayed at the same level from

pretest to post-test. The majority of students were rated average or above average at

pretest. However, of those students rated below average at pretest, 40 percent improved

their skills to the average or above average level at post-test.

Program Factors Affecting Student Reading Achievement

• Four effective practices were significantly related to gains in students’ reading skills:

1) tutors met with students at least three times a week; 2) programs conducted formal

evaluations; 3) tutors were trained both prior to and during the tutoring program; and 4)

programs were moderately or fully implemented. For example:

8 Students in programs where they met with their tutors at least three times per

week increased their reading skills scores between pretest and post-test by 2.1

points more than their peers in programs that met less frequently.

8 Students in programs where tutors received training both during and prior to

tutoring obtained gains of 2.3 points more on the reading test than students in

programs where tutors did not have such training.

• Students in programs implementing these four effective practices showed larger gains

in reading skills (5.4 points) than their counterparts in programs that had three of these

elements (2.5 points).

• Students in programs implementing none of the four effective practices showed no

significant gains in reading skills from pretest to post-test.

• Only one effective tutoring practice was associated with significant gains in reading

comprehension but not in the expected direction. Students in programs where tutors

coordinated activities with classroom reading instruction were less likely to show gains in

reading comprehension than students in programs where tutors did not engage in

coordinating activities. This unexpected finding suggests that tutors who did not

coordinate with classroom instruction may be qualitatively different from those tutors

that did (e.g., less experienced), since their students had smaller gains in reading

comprehension.
AmeriCorps Tutoring Outcomes Study Executive Summary v

Summary

On standardized tests of reading, students tutored by AmeriCorps members improved their

reading performance from pretest to post-test more than the gain expected for the typical

child at their grade level. This finding holds for students at all grade levels tested. The gains

were statistically significant and large enough to indicate meaningful improvements in

reading performance. However, the fact that several effective tutoring practices were related

to student gains increases our confidence in attributing the results to the AmeriCorps

program.

Program staff anticipated that the special attention students received through tutoring would

have positive effects on their classroom behavior. However, there were only modest effects

in this area. In general, teachers rated most students in the normal or expected range at

pretest.

AmeriCorps Tutoring Outcomes Study Chapter 1 1

Chapter 1

Introduction

Over the past several years, there has been increasing concern among educators, policy

makers, and corporate leaders about the reading ability of America’s youth. This has

culminated in a national mandate, the America Reads initiative, whose goal is: to help ensure

that every child can read well and independently by the end of the third grade. Efforts to

support this initiative have been launched by schools, states, federal agencies and businesses.

The Corporation for National Service, through its AmeriCorps program, also is working to

improve the reading skills of young students through its tutoring programs in elementary

schools.

This report describes the effects of AmeriCorps tutoring programs on students’ reading

achievement and classroom behavior. Chapter 1 provides a context for the study, describes

the AmeriCorps tutoring program, and summarizes the study design and methods. Chapter 2

describes the students, tutors, and tutoring activities in the AmeriCorps program. The effects

on students’ reading performance and classroom behavior are presented in Chapter 3; the

program characteristics affecting student reading achievement are discussed in Chapter 4. The

report concludes with a summary of findings in Chapter 5.

National Focus on Reading Achievement

As we move into the 21st century, improving the educational achievement of American

children and youth has become increasingly important due to the globalization of the economy

and the expanding demands of a technological society. To succeed in this new environment,

children and adults alike must be able to read and comprehend text without difficulty. Equally

important, the ability to read well has other intangible benefits, including reading books for

one’s own enjoyment, reading to one’s children, and reading to obtain information and to

continue one’s own learning, to name just a few. As a result, it is a well-accepted notion that

to read well is a critical cornerstone for successful academic performance and meaningful

professional and personal development. Thus, the reading performance of our children is a

major issue not only among educational practitioners and researchers, but also among

national, political and economic leaders, as well.

A major study on reading difficulties in young children noted that “large numbers of schoolage children, including children from all social classes, have significant difficulties learning to

read.”1 Further evidence comes from the results of the National Assessment of Educational

Progress (NAEP) describing the educational attainment of the nation’s students. The most

 

1

Snow, C., Burns, M.S., and Griffin, P. (Eds.). (1998). Preventing Reading Difficulties in Young

Children. Washington, DC: National Research Council.
2 Chapter 1 AmeriCorps Tutoring Outcomes Study

recent NAEP results on reading indicate that there has been no substantive improvement in

the achievement among 9-year-olds in the last eight to ten years. Furthermore, the average

reading score for 9-year-olds obtained in 1999 was lower than the corresponding score from

1985.

Reading performance in the early grades is particularly important since it is a key determinant

of future academic success. One researcher reported that a student’s chances of graduating

from high school are strongly related to his/her reading skills at the end of third grade.2

Elementary school teachers also recognize the importance of reading; in another national

study, more than half of the teachers surveyed responded that “building basic literacy skills is

the most important goal in education.”3

It is not surprising then, that the federal government has devoted significant attention and

resources to the topic of reading achievement, particularly for students in the early elementary

grades (K-3), both through major initiatives in the Department of Education and in other

federal agencies. For example, in 1999, Congress passed the Reading Excellence Act (REA),

a major federal initiative aimed at improving reading instruction in the early grades in schools

where large numbers of children are struggling to learn to read. During the first year of

implementation for REA (the 1999-2000 school year), seventeen states received REA grants

with the requirement to use “scientifically-based” research methods in their approaches to

training teachers to provide high-quality, effective reading instruction.

Research on Effective Practices

The concern with the reading ability of the nation’s school children has led to increased

attention on identifying effective practices to teach reading. While much of this research

focuses on classroom instruction, there is also discussion of the role of volunteers. Educators

and researchers believe that the use of volunteers to support student learning is valuable and

fosters improved learning outcomes for children. Tutoring students in one-to-one or small

group settings is generally perceived as an effective means of providing instruction because

lessons can be tailored to individual students’ specific needs. Volunteer tutoring may be

particularly useful to teachers who have a large number of students in their classrooms and are

unable to provide the same level of individualized attention that a tutor can provide.

There are few large, well-controlled studies, however, that examine the effectiveness of

school-based volunteer activities,4 nor is there overwhelming evidence about the effectiveness

 

2

Slavin, R.E., Karweit, N.L., Wasik, B.A., Madden, N.A., and Dolan, L. (1994). Success for All: A

comprehensive approach to prevention and early intervention. In Preventing Early School Failure, R.E.

Slavin, N.L. Karweit and B.A. Wasik (Eds.). Boston: Allyn and Bacon.

3 National Center for Education Statistics’ Schools and Staffing Survey, 1990-1991.

4 National Research Council (1990). Volunteers in Public Schools. B. Michael (Ed.). Washington, D. C.:

National Academy Press.
AmeriCorps Tutoring Outcomes Study Chapter 1 3

of tutoring as a way to teach reading to students who are having difficulty.5 Nevertheless, in

a review of several reading interventions, each of which includes a central tutoring

component, reading researcher Barbara Wasik has identified a number of features or

guidelines that may represent critical ingredients for an effective reading tutoring program.6 

At the same time, the America Reads initiative identified a similar set of program

characteristics. Examples of these effective practices include:

• Intensity of tutoring:

ÿ Tutoring sessions occur at least three times a week

ÿ Tutoring sessions total at least 1.5 hours per week

ÿ Most of the tutoring is conducted one-to-one with students

• Administration/Implementation:

ÿ Tutors use a formal curriculum model in their sessions

ÿ Tutors coordinate their activities with classroom reading instruction

ÿ Tutors meet regularly with the school’s reading specialist

ÿ Programs evaluate the effectiveness of their tutoring activities

• Tutor training:

ÿ Tutors receive training both prior to and during the course of tutoring

In addition, Wasik identified a number of other program features as important for positive

reading outcomes: well-structured, carefully planned tutoring sessions; frequent assessment of

students’ progress; regular attendance by tutors and students; and fostering of positive, caring

relationships among students, staff and tutors.

AmeriCorps

The AmeriCorps program is one initiative of the Corporation for National Service (the

Corporation), a public corporation established by the National Service Trust Act of 1993.

The Corporation’s mission is to provide opportunities for Americans of all ages and

backgrounds to engage in service that addresses the nation’s unmet needs in education, public

safety, the environment, and other areas. The goals are to encourage all Americans to engage

in such service and to achieve direct and demonstrable results. In addition to AmeriCorps, the

Corporation provides funding to two other major initiatives: Learn and Serve America and

the National Senior Service Corps. Across these three programmatic areas, examples of

service projects include:

 

5 Wasik, B. A. (1997). Volunteer tutoring programs: Do we know what works. Phi Delta Kappan, 79,282-

287. See also Wasik, B. A. (1998). Volunteer tutoring programs in reading: A review. Reading Research

Quarterly, 33, 266-291.

6 Wasik, B. A. (1998). Using volunteers as reading tutors: Guidelines for successful practices. Reading

Teacher, 51, 562-570.
4 Chapter 1 AmeriCorps Tutoring Outcomes Study

• tutoring disadvantaged students;

• organizing neighborhood crime watches;

• converting vacant lots into neighborhood parks;

• leading community health awareness campaigns; and

• operating food banks.

AmeriCorps is composed of three programs: State/National; National Civilian Community

Corps (NCCC) and Volunteers in Service to America (VISTA). The focus of this report is on

the AmeriCorps*State/National program during 1999-2000, which enrolled 35,000 members

nationwide in 961 sites. State/National programs are operated through state commissions and

national service organizations. They support a national network of community-based

programs providing opportunities for participants, referred to as "members," to engage in

community service. In exchange for a year of full-time service, AmeriCorps members receive

a stipend and earn an education award that may be used to pay for higher education or to help

pay back existing student loans. Members serving part-time receive pro-rated stipends and

education awards.

Education-related activities have been a priority for the Corporation since its inception.

Indeed, education programs comprise the largest single area of service (of the four issue areas

of need) as well as a significant portion of the Corporation’s funding. Most recently, the

Corporation has directed substantial resources toward the national issue of children’s reading

performance, and has encouraged its programs to provide literacy services to young readers.

Members offer these services through a variety of strategies, including tutoring, mentoring,

volunteering as classroom assistants, and the provision of other literacy-related activities (e.g.,

conducting trips to the library or organizing book distributions).

AmeriCorps Tutoring

Educational services and tutoring represent a major focus of the AmeriCorps*State/National

programs. Slightly more than half of the AmeriCorps*State/National programs provide

educational services in their communities (Exhibit 1.1). More than one-third (37 percent)

provide direct instruction in the form of tutoring in reading.
AmeriCorps Tutoring Outcomes Study Chapter 1 5

Exhibit 1.1

Frequency of Educational and Literacy Services in AmeriCorps*State/National Programs

A typical AmeriCorps tutoring program in reading places members in schools and after-school

programs to provide reading tutoring to elementary students. During the 1999-2000

academic year, AmeriCorps programs nationwide provided tutorial assistance in reading to

more than 100,000 students in grades one, two and three. On average, participating students

received between 3 and 3.5 hours of tutoring a week, some part of which was individualized

instruction. To ensure that the tutoring is as effective as possible, most AmeriCorps programs

provide training to tutors before they begin working with students, covering topics such as

how to teach reading and how to work with young children. Individual schools, classroom

teachers, and district reading resource specialists also provide ongoing support and assistance

to the members and their volunteers.

Source: Corporation for National Service.

Direct Literacy

Tutoring

37%

(n=360)

Other

Literacy

5%

(n=46)

Education:

Other Services /

Not Specified

12%

(n=111)

Non-Education

Services:

• Environmental

• Public Safety

• Other Human

Needs

46%

(n=444)

n=(961)
6 Chapter 1 AmeriCorps Tutoring Outcomes Study

The AmeriCorps Tutoring Outcomes Study

To inform staff, programs, Congress, and other stakeholders about its literacy activities, the

Corporation contracted with Abt Associates Inc. to conduct two related studies of

AmeriCorps’ literacy and tutoring programs. The first of these, the Descriptive Study of

AmeriCorps Literacy Programs: State and National, had several goals: 1) to describe the

programmatic structures and activities of AmeriCorps*State/National programs engaged in

literacy development and reading tutoring; 2) to identify programs using effective instructional

models likely to improve children’s reading abilities; and 3) to describe the target populations

receiving services. The findings were presented in The Descriptive Study of the Corporation

for National Service’s Literacy Programs: Final Report (November 1999).

This report focuses on the second study, the AmeriCorps Tutoring Outcomes Study, which

was designed to measure the effects of AmeriCorps tutoring programs on the reading

performance of students in grades one, two and three. AmeriCorps tutoring programs are

grounded in the assumption that the provision of additional direct instruction in reading will

produce positive effects on children's reading skills by the end of one year's participation.

Furthermore, program designers also expect that students will improve their classroom

behavior as a result of receiving tutoring services. That is, the special attention and adult

interest that the students receive will increase their sense of worth and interest in the

educational process, thereby leading to improved classroom behaviors. Additionally, as

students improve their reading skills, their self-esteem and interest in learning will improve,

which will also reinforce more adaptive classroom behaviors.

Study Design

Two broad research questions are addressed in this study:

• What is the effect of participation in AmeriCorps’ tutoring programs on children’s

reading proficiencies and other classroom behaviors?

• Which components of the reading tutoring programs, considered to be effective

practices, are associated with improved student outcomes in reading and classroom

behavior? Do students in programs implementing more research-based effective

practices show greater gains in reading than students in the other AmeriCorps tutoring

programs?

To address the first question, trained field staff administered a standardized reading test at the

beginning of tutoring (pretest) and at the end of the school year (post-test) to students in

grades one, two and three. Teachers also rated students’ classroom behaviors at the same

time points. We then compared changes in students’ scores with the standardization sample
AmeriCorps Tutoring Outcomes Study Chapter 1 7

upon which the test norms are based.7 This tells us the extent to which students in the

tutoring program gained more or less than expected based on the performance of the

standardization sample, a nationally representative sample of students.

To address the second research question, we examined the strength of the relationship

between characteristics of the tutoring program and student outcomes in reading achievement

and classroom behavior.

The Study Sample

Results are based on pretest and post-test data from 869 students in grades one, two and three

receiving tutoring services in 68 programs serving 93 schools nationwide. The design

included a nationally representative of all AmeriCorps programs that provide tutoring to

students at these grade levels.8 The final sample included eligible programs willing to

participate in the study and students whose parents provided written permission for testing.

(We present a technical discussion of the sampling procedures and the construction of

sampling weights in Appendix A.)

Measures

Four types of information were collected in this study. A brief description of each measure is

provided below. More detailed discussion of the measures appears in Appendix B.

• Reading performance. We measured student reading performance using a set of subtests from the Woodcock-Johnson Psycho-Educational Battery-Revised, a well-known

standardized battery of achievement tests. In addition, teachers completed a very

short questionnaire designed specifically for this study to describe changes in students’

reading in five areas: comprehension proficiency, mechanics proficiency, amount of

time on task during reading instruction, interest in reading, and general interest in

school.

• Classroom behavior. Teachers rated students using two short standardized rating

scales:1) the Behavioral Academic Self-Esteem rating scale (BASE), which measures

children's academic self-esteem, and 2) the cooperation sub-scale from the Social

Skills Rating System (SSRS), which assesses children’s academic competence,

problem behaviors, and social skills.

 

7 The standardization sample is the sample of students whose test results are used to construct a set of

norms to score a particular test or rating scale. For example, it is these scores that are used to calculate

the percentile ranks and grade equivalent of each score. The standardization sample consists of a

nationally representative group of students, not necessarily limited to low-achieving students.

8 To construct a nationally representative sample from the smaller study sample, the actual numbers of

programs and students are multiplied by a weighting factor, which is described in Appendix A. These

weights were used in all analyses. However, for ease of interpretation, the actual sample sizes are shown

on all exhibits.
8 Chapter 1 AmeriCorps Tutoring Outcomes Study

• Descriptions of Tutoring Program Components. All of the participating AmeriCorps

program directors completed a short questionnaire in spring 2000 that provided

updated information about the nature of the tutoring provided to their students (e.g.,

program intensity, type and quality of instruction). In addition, the AmeriCorps tutors

also completed a brief questionnaire about their training, education, age, and prior

experience.

• Tutor experiences with students. To assess the nature of the tutoring activities at the

student level, AmeriCorps tutors completed a short set of questions at both the pretest

and post-test in a Record of Reading Activities or tutor activity “log.” Information

included the amount of tutoring both scheduled and actually received, the types of

literacy activities that took place, and their impressions of student interest, motivation

and engagement during the tutoring sessions.

Data Collection

The pretest occurred in late fall 1999/early winter 2000, with the post-test conducted in spring

2000. In addition to student test data, 293 tutors and 369 teachers provided further

information through the rating scales and activity records collected at the same two points in

time. A group of 65 trained field staff administered the reading tests individually to students

and assisted in the collection of the tutor records and teacher ratings.

Analysis

Students’ gains from pretest to post-test provided the basis for estimating the impact of

AmeriCorps tutoring on reading skills and classroom behavior. Statistical tests of significance

at each grade level indicated whether these gains were greater than would be expected for

typical children in these grades. A multiple regression technique known as hierarchical linear

modeling, which estimates the effects of program-level and student-level variables

simultaneously, was used to investigate the relationship between program characteristics and

student reading gains.

Limitations of the Study Design and Methodology

The confidence with which we can interpret the results of a study is related to the research

design and methodology. Two features of this study are important to keep in mind when

reading the results presented in subsequent chapters of this report.

First, the study design used national norms to determine grade-level expectations for the

development of reading skills. The norms functioned as a comparison group which allowed us

to estimate how non-tutored students were likely to have performed in reading during the

school year. A study that collected student-level information on a variety of reading-related

characteristics from a matched comparison group would have enabled a more detailed analysis

of reading gains for different types of students and would have increased the confidence in
AmeriCorps Tutoring Outcomes Study Chapter 1 9

attributing gains to the program. However, it was not possible to identify and recruit an

appropriate set of matched comparison schools and students to participate in the study. Thus,

the study design relied on national norms as a reasonable and practical way to obtain an

indication of students' expected gains in reading performance over a school year.

The second limitation of this study concerns the teacher ratings. One type of teacher rating,

designed specifically for this study, asked teachers to make global judgments about student

reading and therefore may be a less reliable measure than a standardized reading test that

provides a direct and objective assessment of students’ skills. A second type of teacher rating

used in the study, measured students’ classroom behaviors. This information was obtained for

a substantially smaller sample than the one used in the analysis of reading performance (492

versus 869) because many teachers did not complete the student rating scales. The overall

response rate is only 57 percent, with a response rate of 65 percent at first grade, 58 percent

at second grade and 48 percent at third grade. The much-reduced samples jeopardize the

representativeness of the group of students rated by teachers.

AmeriCorps Tutoring Outcomes Study Chapter 2 11

Chapter 2

Description of Tutoring Programs and Activities

This chapter presents information on the AmeriCorps tutoring programs in the AmeriCorps

Tutoring Outcomes Study. We begin with a brief description of the students who were

tutored as part of the AmeriCorps program, followed by a more detailed picture of the tutors

and tutoring programs. The last section of the chapter describes the use of research-based

effective practices among this sample of AmeriCorps programs.

Student Characteristics

This study collected limited information about each student—the children were too young to

reliably provide family demographic information and parents were not interviewed as part of

the study. Thus, demographic information was restricted to gender and ethnicity, information

that testers or teachers were able to provide. Students pretest scores on reading achievement

tests also were collected as part of the study; these scores provide an indication of how these

students compare with their peers nationally at or near the beginning of tutoring.

Demographic Characteristic

The study sample includes 869 children with reading test scores at pretest and post-test,

nearly evenly distributed across grades: 294 in first grade, 292 in second grade and 283 in

third grade. (For a detailed discussion of how these students were selected for the study, see

Appendix A.) At grades two and three, the sample was fairly evenly split between boys and

girls; however, at grade one, the sample was 59 percent female and 41 percent male (Exhibit

2.1). Across the three grades, 51 percent of children in the sample were white, 23 percent

African-American, 20 percent Hispanic, and 7 percent Native American, Pacific Islander,

Asian, or mixed ethnicity.

Exhibit 2.1

Gender of Students by Grade

Gender

Grade 1

(n = 294)

Grade 2

(n = 292)

Grade 3

(n = 283)

Male 41% 48% 51%

Female 59% 52% 49%

Source: Field staff documentation.
12 Chapter 2 AmeriCorps Tutoring Outcomes Study

Pretest Reading Achievement Level

Student reading levels are based on four subtests of the Woodcock-Johnson Psychoeducational Battery—Revised (WJ-R): letter-word identification, word attack, passage

comprehension, and reading vocabulary (examples of each of these subtests appear in

Appendix B). The subtests combine to create two key components of reading achievement:

• Reading skills (word attack and letter-word identification)

• Reading comprehension (passage comprehension and reading vocabulary).

Taken together, these two scores represent the building blocks needed to become competent

readers; they measure critical skills involved in the mechanics of reading as well as

comprehension of text.

At pretest, 57 to 64 percent of students obtained reading comprehension scores in the average

range for their grade, with minimal differences across the three grades (Exhibit 2.2).9

Approximately 20 percent of the students’ reading comprehension scores were “low average”

and another 10 percent of scores were “low” or “very low” for their grade. Only about 10

percent of the sample scored in the high average to superior range.

Exhibit 2.2

Students’ Reading Comprehension Scores at Pretest, by Grade

9%

17%

64%

9%

1%

6% 3%

20%

60%

8%

4% 3%

10%

20%

57%

8%

<1% 2%

0%

10%

20%

30%

40%

50%

60%

70%

Very Low Low Low Average Average High Average Superior/Very

Superior

Proficiency Level

Percent of Students

Grade1 Grade 2 Grade 3

Source: Woodcock-Johnson Psycho-Educational Battery-Revised.

 

9 These classifications are based on information provided by the test publisher, which enables students’

standard scores to be classified as below average to above average for each grade level.
AmeriCorps Tutoring Outcomes Study Chapter 2 13

Students’ reading skills scores at pretest were, on average, lower than their reading

comprehension scores. These differences are greatest at first grade, where 35 percent of the

students scored in the “low average range” and another 7 percent scored “low” or “very low”

(Exhibit 2.3). Again, only about 10 percent scored in the high average to superior range at

pretest.

Exhibit 2.3

Students’ Reading Skills Scores at Pretest, by Grade

2% 5%

35%

47%

10%

1% 3%

7%

26%

56%

4% 4% 2%

7%

29%

48%

5% 8%

0%

10%

20%

30%

40%

50%

60%

70%

Very Low Low Low Average Average High Average Superior/Very

Superior

Proficiency Level

Percent of Students

Grade1 Grade 2 Grade 3

Source: Woodcock-Johnson Psycho-Educational Battery-Revised.

Thus, the reading ability of the students in the AmeriCorps tutoring programs at the start of

the school year seem to be in the low average to average range. These students do not display

superior reading skills for their grade level, nor do they display very low levels of reading

performance.
14 Chapter 2 AmeriCorps Tutoring Outcomes Study

Tutor Characteristics

Tutors in AmeriCorps programs can be AmeriCorps members or volunteers recruited by

members. However, most (86 percent) of the tutors involved in the AmeriCorps tutoring

program during the 1999-2000 year were AmeriCorps members (Exhibit 2.4).

Exhibit 2.4

Affiliation of AmeriCorps Tutors

AmeriCorps member

86%

Volunteer recruited 

by member

5%

Unrelated volunteer

9%

Source: Tutor Questionnaire (n=293).

The tutors were primarily female (85 percent) and ranged in age from 17 to 80, with the

average age 27. Nearly half of the tutors were between 20 and 25 years of age, and another

17 percent were between 26 and 30 years of age (Exhibit 2.5). There were more tutors who

are AmeriCorps members (70 percent) in this 20-30 age bracket than volunteers (42 percent).

Volunteers were more likely than AmeriCorps members to be less than 20 years of age or

more than 40 years of age.

The majority of AmeriCorps tutors do not have a college degree (Exhibit 2.6). Half of the

tutors have some college education prior to a degree and 15 percent have only a high school

diploma or GED certificate. About one-fifth of the tutors have a college degree and 11

percent have attended graduate school or attained a graduate degree. A larger percentage of

AmeriCorps members (34 percent) than volunteers (18 percent) have a college degree.
AmeriCorps Tutoring Outcomes Study Chapter 2 15

Exhibit 2.5

Age of AmeriCorps Tutors During 1999-2000

20-25 years old

49%

26-30 years old

17%

31-40 years old

12%

More than 40 years 

old

10%

Less than 20 years 

old

12%

Source: Tutor Questionnaire (n=293).

Exhibit 2.6

Level of School Completed by AmeriCorps Tutors

College

21%

Some grad school

8%

Graduate school

3%

Some high school

3%

High school/GED

15%

Some college

50%

Source: Tutor Questionnaire (n=293).
16 Chapter 2 AmeriCorps Tutoring Outcomes Study

Two-thirds of the tutors have prior tutoring experience in reading or another subject (Exhibit

2.7), while one-third have no prior tutoring experience. There were no differences in prior

tutoring experience between AmeriCorps members and volunteers.

Exhibit 2.7

Tutors’ Prior Experience

Tutored other subject

10%

Tutored reading

57%

No prior tutoring

33%

Source: Tutor Questionnaire (n=293).

Effective Tutoring Practices

As described in Chapter 1, the research on reading achievement identifies several factors of

classroom instruction and individual tutoring as effective in improving students’ reading skills.

This section presents information on the prevalence of these effective practices among

AmeriCorps programs. The study collected information from program directors and tutors

about eight effective practices noted in the literature, which can be grouped into three broad

categories:

• Intensity of tutoring:

ÿ Tutoring sessions occur at least three times a week

ÿ Tutoring sessions total at least 1.5 hours per week

ÿ Most of the tutoring is conducted one-to-one with students

• Administration/Implementation:

ÿ Tutors use a formal curriculum model in their sessions

ÿ Tutors coordinate their activities with classroom reading instruction

ÿ Tutors meet regularly with the school’s reading specialist

ÿ Programs evaluate the effectiveness of their tutoring activities

• Tutor training.

ÿ Tutors receive training both prior to and during the course of tutoring
AmeriCorps Tutoring Outcomes Study Chapter 2 17

In addition to the eight effective practices listed above, we also asked program directors about

the status of implementation. A fully implemented program is one in which all of the

components are available and functioning as planned (e.g., staff are hired and working with

students, materials are selected and on-site, participants are recruited and involved in

activities). We would expect that programs near or at full implementation are more likely to

be successful. Thus, for this study, we were interested in examining the level of

implementation and added this to our list of effective practices:

ÿ Program is at least moderately or fully implemented.

The prevalence of these nine specific practices among the AmeriCorps programs in the study

is summarized in Exhibit 2.8. More detailed information about each of these broad areas of

effective practices is presented below.

Exhibit 2.8

Tutoring Characteristics Prevalent Among AmeriCorps*State/National Programs

48%

53%

53%

66%

69%

73%

80%

80%

87%

0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%

Tutors meet regularly with the school's reading specialist

Tutoring is conducted one-to-one at least half the time

Tutoring sessions occur at least three times per week

Tutors use a formal curriculum model in sessions

Tutoring sessions last at least 1.5 hours

P rograms evaluate the effectiveness of their tutoring activities

P rogram is moderately or fully implemented

Tutors receive training both prior and during the course of tutoring

Tutors coordinate activities with classroom reading instruction

Percent of Programs

Source: Program Director Questionnaire (n=68) and Tutor Logs averaged across 68 programs.
18 Chapter 2 AmeriCorps Tutoring Outcomes Study

Intensity of Tutoring

Frequency of Tutoring Sessions

Nearly all of the AmeriCorps tutors met with the students at least twice a week (Exhibit 2.9).

About half (53 percent) of the program directors reported that tutors met with students at

least three times a week, which is the amount suggested in the literature for an effective

tutoring program. A quarter of the program directors indicated that tutors met daily with

students.

Exhibit 2.9

Frequency of Tutoring Sessions in AmeriCorps Tutoring Programs

Three to four times a 

week

29%

Once a week

3%

Daily

24%

Twice a week

44%

Source: Program Director Questionnaire (n=68).

Length of Sessions

Nearly half (47 percent) of the program directors reported that tutoring sessions lasted

between 30 and 60 minutes; another 44 percent of program directors indicated that sessions

lasted between 15 and 30 minutes (Exhibit 2.10). When we consider the length of each

session in combination with the number of sessions per week, we find that the majority of

program directors (66 percent) reported that tutoring was provided more than 1.5 hours per

week. The tutors’ own estimates of the length of tutoring sessions were slightly higher.

When asked about the length of tutoring sessions, 69 percent of tutors indicated that they

tutored students more than 1.5 hours per week. Across all tutors, the average length of

tutoring was 2.7 hours per week, with a range from 15 minutes to more than 11 hours per

week. In just over half of the programs (52 percent), the program director’s report was less

than the estimate provided by the tutors in that program.10 It is not possible to determine

 

10 For this analysis, the estimates of the length of tutoring sessions were averaged across tutors in each

program in order to obtain a program estimate that could be compared to the program director’s report of

intensity.
AmeriCorps Tutoring Outcomes Study Chapter 2 19

whether the program directors were underestimating the intensity of tutoring or the tutors

were overestimating this figure. However, the differences do not appear to be large enough

to raise concerns about either estimate.

Exhibit 2.10

Length of Tutoring Sessions in AmeriCorps Tutoring Programs

30-60 minutes

47%

More than 90 minutes

5%

60-90 minutes

4%

15-30 minutes

44%

Source: Program Director Questionnaire (n=68).

Size of Tutoring Group

More than a third of program directors reported that tutors met with students one-to-one and

another 40 percent indicated that tutoring took place in small groups of two to four students

(Exhibit 2.11). Tutors also provided information about the size of the tutoring group by

indicating the amount of time each week they tutored students individually or in groups.

Using this information, we computed the proportion of time that tutors met individually with

students and averaged this for each program. In 53 percent of programs, tutors spent at least

half of their time tutoring in individual sessions with students. The use of either one-to-one or

small-group tutoring is generally perceived to be an effective means of providing instruction

because lessons can be tailored to individual students’ needs.

In the vast majority of programs, students had the same tutors consistently from session to

session—in 77 percent of programs there was only one tutor per student and in 14 percent of

programs two tutors shared this responsibility. In only nine percent of programs were there

more than two individuals providing tutoring to the same student across sessions. Tutors did

not always work with the same students for the whole year. Although most students (85

percent) had one tutor, a small proportion of students had two (13 percent) or more (2

percent) tutors over the course of the year.

Taken together, this information suggests that most students met with a tutor consistently

over the course of the year either individually or in small groups, which would increase the

likelihood of developing a close relationship with the tutor. In fact, tutors generally rated
20 Chapter 2 AmeriCorps Tutoring Outcomes Study

students as having a very positive working relationship during the tutoring session, with an

average rating on this dimension of 3.8 on a 4-point scale. In addition, tutors rated students

high on their motivation to learn (average rating of 3.5 out of 4) and their focus and

involvement during the tutoring session (average rating of 3.6 out of 4).

Exhibit 2.11

Number of Students per Tutor in AmeriCorps Tutoring Programs

3-4

30%

1

36%

5-6

17%

7 or more

7%

2

10%

Source: Program Director Questionnaire (n=68).

Program Administration and Implementation

Program directors and tutors answered several questions about the tutoring program (e.g., the

type of curriculum used, the extent of program implementation, coordination with classroom

instruction), which are discussed below.

Level of Program Implementation

Program directors assessed whether their tutoring programs had reached full implementation,

defined as having all components in place and operating as planned. One-third of the program

directors indicated that their program was fully implemented, with another 47 percent

indicating moderate implementation (i.e., most but not all components in place and operating

as planned). Thus, 80 percent of programs could be characterized as either fully or

moderately implemented. Sixteen percent of program directors indicated their program was

partially implemented and another three percent rated their program as only at a low level of

implementation.

Use of Formal Instructional Models

There are a number of commercially available instructional models that can be used as part of

a tutoring program in reading. Nearly two-thirds of AmeriCorps programs used some sort of

formal model, with many incorporating more than one model. The most commonly used
AmeriCorps Tutoring Outcomes Study Chapter 2 21

models were Reading Recovery, used by 24 percent of programs, and Read*Write*Now, used

by 18 percent of programs (Exhibit 2.12).

Exhibit 2.12

Instructional Models Used in AmeriCorps Turing Programs

13%

24%

34%

12%

6%

18%

8%

0% 10% 20% 30% 40% 50%

Write-to-Read

Reading One-to-One

Success for All

Book Buddies

Read*Write*Now

Reading Recovery

No formal model

Percent of Programs

Source: Program Director Questionnaire (n=68).

Tutors provided information about the reading and academic activities they undertook with

students. Virtually all reported helping students with reading comprehension (97 percent) and

mechanics (94 percent), as well as listening to students reading (98 percent). Somewhat less

frequently, tutors read aloud to students (80 percent). Many tutors (73 percent) also helped

students with skills in other areas besides reading.

Coordination with Reading Specialist and Classroom Instruction

In about half of the programs (48 percent), tutors planned their instructional activities in

coordination with the school’s reading specialist. Among this subset of programs, program

directors indicated that the reading specialist provided the following assistance:11

ß Helped plan reading activities for students’ tutors (39 percent);

ß Provided ongoing monitoring or consultation during the school year (64 percent);

ß Supplied materials (49 percent); and

ß Assisted with instruction (53 percent).

The tutors in most programs (87 percent) also coordinated their tutoring with classroom

instruction, most often through regular meetings with the classroom teacher (100 percent) and

less frequently (24 percent) through formal written contact.

 

11 Percentages add to more than 100 percent because program directors could indicate more than one type of

assistance.
22 Chapter 2 AmeriCorps Tutoring Outcomes Study

Program Evaluation

The program director questionnaire included one question about whether programs assess the

effectiveness of their tutoring activities. The majority of program directors (73 percent)

reported that they do conduct a formal evaluation. This information can contribute to

program improvement, which is why conducting an evaluation is one of the effective practices

cited in the research literature.

Tutor Training

Training tutors, particularly those with limited prior experience tutoring reading, is a critical

component of a successful tutoring program. The majority of program directors (81 percent)

indicated that training sessions were conducted before tutoring sessions began; 86 percent

reported that training took place during the provision of services; and 80 percent noted that

training occurred both prior to and during the program year. More than 90 percent of

directors reported that AmeriCorps tutors received training in reading and reading materials.

In addition, approximately 70 percent of directors reported that tutors received materials or

training in child development.

Training Prior to Tutoring

Tutors answered more detailed questions about the length, provider and subject area of their

training. Prior to the program year, 77 percent of tutors reported participating in training,

which is fairly similar to program directors’ reports. There were slight differences in prior

training experiences for AmeriCorps members and volunteers; nearly 80 percent of

AmeriCorps members reported receiving training prior to the program year, compared with 67

percent of volunteers.

AmeriCorps staff most often provided this training to tutors. More than one-third of tutors

(37 percent) received more than 15 hours of training from AmeriCorps and another 15 percent

received between 6 and 15 hours of training (Exhibit 2.13). In contrast, more than half of the

tutors (56 percent) indicated that they did not receive training from any source other than

AmeriCorps prior to the start of tutoring, and 23 percent received more than five hours of

such training.

Prior to the start of tutoring, tutors received training primarily in reading or general tutoring

skills. About three-quarters (77 percent) of tutors indicated that they received training in

reading prior to tutoring, and 67 percent learned about general tutoring skills. In addition, a

smaller percentage of tutors received training in math (11 percent) or another subject (16

percent). Most tutors received training in more than one subject (e.g., reading and general

tutoring), and very few (3 percent) did not receive training in either reading or general

tutoring skills.
AmeriCorps Tutoring Outcomes Study Chapter 2 23

Exhibit 2.13

Amount of Training Received by Tutors Prior to Program

29%

13%

9%

6%

37%

56%

15%

8%

1%

14%

0%

10%

20%

30%

40%

50%

60%

None 1-5 Hours 6-10 Hours 11-15 Hours More than 15

hours

Provided by AmeriCorps Other Source

Source: Tutor Questionnaire (n=293).

Training During the Program Year

The majority of tutors (85 percent) indicated that they participated in training during the time

they were tutoring.12 There were no differences in this training between AmeriCorps

members and volunteers.

Almost half of the tutors (47 percent) reported receiving more than 15 hours of training from

AmeriCorps during the 1999-2000 program year (Exhibit 2.14). However, 15 percent

indicated that they received no training from AmeriCorps during the program year. Most

tutors also received training from other sources during the program year, although of lesser

duration than AmeriCorps training, with about half of the tutors receiving between six and ten

hours of such training.

 

12 This question was answered by only 183 of the 293 tutors, perhaps because the question was on the second

page of the survey and respondents did not continue beyond the first page. However, since the proportion

reporting training is quite similar to the percentage of project directors indicating that training took place,

the tutor estimates do not appear to be biased by the lower response rate.
24 Chapter 2 AmeriCorps Tutoring Outcomes Study

Exhibit 2.14

Amount of Training Received by Tutors During 1999-2000

15% 14% 14%

10%

47%

15%

20%

48%

5%

12%

0%

10%

20%

30%

40%

50%

60%

None 1-5 Hours 6-10 Hours 11-15 Hours More than 15 Hours

Percent of Tutors

Provided by AmeriCorp Other Source

Source: Tutor Questionnaire (n=183 of the 293 tutors).

Summary

Many AmeriCorps tutoring programs have program characteristics consistent with the

research on effective practices. For example, 80 percent or more of the programs coordinate

tutoring activities with reading instruction, are moderately or fully implemented, and train

tutors prior to and during the course of tutoring. However, not all effective practices are as

prevalent among these programs. Only about half of the programs tutor students individually

and at least three times a week, or have tutors meet regularly with the school’s reading

specialist.

The relationship of these effective practices to student outcomes in reading and classroom

behavior will be examined in Chapter 4. The next chapter presents overall student gains from

the start to end of the 1999-2000 program year.
AmeriCorps Tutoring Outcomes Study Chapter 3 25

Chapter 3

Program Effects on Student Achievement and

Classroom Behavior

This chapter presents the findings about the tutoring programs’ effects on students’ reading

performance and classroom behavior. Results include gains in reading achievement, teachers’

ratings of student reading gains, and analyses of changes in students’ classroom behavior.

Student Reading Performance

This section first reports on students gains from pretest to post-test on the WoodcockJohnson Psycho-Educational Battery-Revised, the standardized test of reading achievement

used in the study. The second part of the section presents teachers’ judgments about changes

in students’ reading performance and classroom behavior.

Standardized Reading Tests

The tutored students at all grade levels improved their reading performance from pretest to

post-test more than the gain expected for the typical child at their grade level. Reading

comprehension and reading skills started out below grade-level; by year-end, students closed

the gap and were reading at or near the grade-level expectation.13 As presented in Exhibit 3.1,

the reading gains, on average, range from 3.7 to 6.6 points, depending on the grade and

reading score. All of the gains are statistically significant, most at the .01 or .001 level.

Exhibit 3.2 displays the average pretest to post-test gains relative to grade-level expectations

for the three grades in the study. The gains are most pronounced for second graders who, on

average, are performing at or slightly above expectations on both reading measures at posttest. Reading scores can also be characterized in terms of percentile ranks that represent each

student’s standing relative to the norming sample. For example, on average, first grade

students performed at the 38th percentile at pretest (in other words, 38 percent of the norming

sample obtained lower scores), while they performed at the 45th percentile at post-test, on

average.

These results indicate that students at all grade levels made statistically significant gains in

reading skills and reading comprehension. At post-test, students in all grades were performing

at or near their grade-level expectations. However, as discussed in Chapter 2, the students

 

13 Grade-level expectations are based on the grade-standardized scores for the norming sample. These

scores have a mean of 100 and a standard deviation of 15 at each grade level. At any point in time, by

definition, the expected level of performance for the typical child (i.e., the norming sample) would be a

score of 100. The Woodcock-Johnson tests allow for the conversion of raw scores to several types of

scores. For this analysis, grade-standardized scores and percentile ranks were used.
26 Chapter 3 AmeriCorps Tutoring Outcomes Study.

Exhibit 3.1

Student Gains on Reading Skills and Reading Comprehension by Grade

Grade 1 Grade 2 Grade 3

Pretest Post-test Gain Pretest Post-test Gain Pretest Post-test Gain

Reading Skills

(word attack and letter/word

identification)

N=283 N=283 N=278

Mean 94.5 98.3 3.7*** 95.0 101.7 6.6*** 95.4 99.4 4.0**

Std error 1.8 1.3 1.2 1.4 2.4 1.8 1.6 2.4 1.5

Reading Comprehension

(passage comprehension and

reading vocabulary)

N=257 N=275 N=278

Mean 94.1 98.8 4.8*** 96.4 101.5 5.1* 92.7 98.2 5.5**

Std error 1.7 1.6 1.5 1.4 2.0 2.2 1.3 1.7 2.0

* p<.05, ** p<.01, *** p<.001

Source: Woodcock-Johnson Psycho-Educational Battery-Revised.
AmeriCorps Tutoring Outcomes Study. Chapter 3 27

Exhibit 3.2

Student Performance on the Woodcock-Johnson Reading Test

91

92

93

94

95

96

97

98

99

100

101

102

103

Pretest Post-test

Grade-Level

Expectation

 (100)

GradeStandardized

Scores

First Second Third

Percentile Ranks

 Pretest Post-test

Gr1 38 45

Gr2 38 50

Gr3 38 50

Percentile Ranks

 Pretest Post-test

Gr1 38 49

Gr2 40 52

Gr3 31 45

91

92

93

94

95

96

97

98

99

100

101

102

103

Pretest Post-test

Source: Woodcock-Johnson Psycho-Educational Battery-Revised.

Reading Skills

(word attack and letter word identification)

Reading Comprehension

(passage comprehension and vocabulary)
28 Chapter 3 AmeriCorps Tutoring Outcomes Study

participating in the tutoring program were, on average, not far below grade-level expectations

at pretest.

In addition to being statistically significant, these gains are large enough to indicate

meaningful changes in reading performance. One way to determine this is to look at the

standard deviation (or variation around grade-level means) reported by the test publisher. The

AmeriCorp results represent gains ranging from .25 to .33 of a standard deviation. According

to the convention in the field, gains of this size are “small but educationally significant,” which

indicate real improvement in reading skills.14

We also conducted analyses to see whether gains in reading varied by students’ ethnicity or

gender. With regard to ethnicity, there were no differences in the reading score gains between

White students and non-White students. This was true for both measures, reading skill and

reading comprehension.15 Further, these results are consistent across the three grades.

There were, however, some differences in results associated with gender. Second grade boys

showed larger gains in reading comprehension than girls; boys gained 7.4 points from pretest

to post-test, compared to 3.1 points for girls. In addition, first grade boys obtained larger

gains than girls in reading skills (5.4 points from pretest to post-test for boys compared to 2.6

points for girls).

Although there were significant gains on average across all students, there also were

substantial differences in student gains across the 68 tutoring programs (Exhibit 3.3). The

distribution of average scores is similar across both test measures. Approximately 30 percent

of the programs had an average gain of 5 or more points. In contrast, about one-fifth of the

programs showed no gain or a small loss on both tests. Possible explanations for differences

across programs are discussed in the next chapter.

Teacher Ratings of Reading Gains

Teachers rated students’ improvement on five dimensions: reading comprehension, reading

mechanics, time on task, interest in reading, and interest in school. This was not a

standardized measure, but rather a series of questions designed specifically for this study.

Teachers indicated that the majority of students at all grade levels improved “a lot” or “some”

over the course of the year as a result of the AmeriCorps tutoring (Exhibit 3.4). For grade 1

 

14 We applied the definitions suggested by Cohen in Statistical Power Analysis for the Behavioral Sciences

(1988), who proposed that differences of .20 to .50 of a standard deviation correspond to a small effect.

For instance, a five-point gain from pretest to post-test corresponds to a gain of one-third of the standard

deviation (the five-point gain is divided by 15, the standard deviation of the distribution of gradestandardized test scores in the norming sample).

15 The non-White group includes students who are Black, Hispanic, of mixed race, or a member of another

ethnic group. These groups were combined in order to have a large enough sample to conduct statistical

tests of significance.
AmeriCorps Tutoring Outcomes Study Chapter 3 29

students, teachers reported that more than 40 percent of students improved “a lot” on reading

mechanics, time on task and interest in reading. The same pattern did not hold for second and

third grade students. According to teachers, the extent of improvement in these areas was

Exhibit 3.3

Distribution of Average Program-Level Gains

Source: Woodcock-Johnson Psycho-Educational Battery-Revised, averaged across students within each of 68

programs.

more modest for these students. For reading comprehension, students at each grade level

were more likely to be rated as improving “some” (43 to 57 percent) than improving “a lot”

(20 to 31 percent).

Exhibit 3.4

Teacher Ratings of Student Gains by Grade

Area of Improvement

Grade 1

(n-225)

Grade 2

(n=215)

Grade 3

(n=189)

A Lot Some A Lot Some A Lot Some

Reading Comprehension 31% 43% 20% 46% 22% 57%

Reading Mechanics 42 34 29 42 27 52

Time on Task 42 50 19 53 11 76

Interest in Reading 46 36 26 42 23 54

Interest in School 39 40 19 49 24 49

Source: Classroom Teachers.

To assess the accuracy of the teacher ratings, we matched the ratings with student gains on

the Woodcock-Johnson. For first grade students, teacher ratings of improved reading

comprehension were moderately correlated (.30) with student gains on the reading

comprehension subtest of the Woodcock-Johnson. It is interesting to note that student gains

11%

15%

22%

32%

20%

10%

21%

21%

28%

20%

0% 5% 10% 15% 20% 25% 30% 35%

10.0 or more

5.0 - 9.99

3.01 - 4.99

.001 - 3.00

Zero or less

Average Point Gain

Reading Skills Reading Comprehension
30 Chapter 3 AmeriCorps Tutoring Outcomes Study

in reading comprehension also were correlated .35 with improved interest in reading and .31

with increased interest in school. However, teacher ratings of improvements in reading

mechanics were modestly correlated .20 with gains on the reading skills subtest of the

Woodcock-Johnson. At the second and third grade levels, teacher ratings of student

improvements generally had weaker correlations of less than .20 with gains on the

achievement test. These low correlations may be due to the reliability of the teacher rating

measure, which was designed for this study and may have been too general to capture small

student gains on the reading test. Another factor may be the accuracy of teachers’

perceptions of student performance. Either way, these low correlations suggest that the

ratings are not a robust measure and should be interpreted with caution.

Classroom Behaviors

An expected outcome of the AmeriCorp tutoring program is that students would improve

their classroom behavior. That is, the special attention that students receive during tutoring

would increase their sense of self-worth and their interest in school, thereby leading to

improved classroom behavior. To explore these anticipated outcomes on students’ classroom

behavior, the study included two standardized teacher rating scales: the BASE and the

cooperation subscale of the SSRS.

Behavioral Academic Self-Esteem (BASE)

The BASE assesses students’ self-esteem across five dimensions:

• Student initiative (e.g., willing to undertake new tasks, asks questions when does not

understand);

• Social attention (e.g., quiet in class, speaks in turn);

• Success/failure (e.g., deals with mistakes easily and comfortably, takes criticism in

stride);

• Social attraction (e.g., child’s company is sought by peers, acts as leader with peers);

and

• Self-confidence (e.g., readily expresses opinions, appreciates own work).

Exhibit 3.5 presents the findings for the BASE for each grade level. The findings are

expressed as standardized scores, where a score of 100 represents the average score obtained

by the standardization sample; this score can be interpreted as the ‘normal’ or ‘average’ level

of academic self-esteem. This measure has separate norms for boys and girls, acknowledging
AmeriCorps Tutoring Outcomes Study. Chapter 3 31

Exhibit 3.5

Behavioral Academic Self-Esteem (BASE) Standardized Scores

Grade 1 Grade 2 Grade 3

Pretest Post-test Gain Pretest Post-test Gain Pretest Post-test Gain

Boys n=116 n=103 n=65

Mean 90 95 5* 96 95 -1 94 95 7***

Standard Error 3 1 3 3 3 1 5 3 2

Girls n=76 n=80 n=74

Mean 103 106 3* 103 101 -2 101 103 2

Standard Error 4 5 2 3 4 3 4 2 3

*p<.05, **p<.001

Source: BASE completed by Classroom Teachers.
32 Chapter 3 AmeriCorps Tutoring Outcomes Study

that expected behaviors are different depending upon gender. For this reason, the findings are

presented separately for boys and girls.

The patterns of results on the BASE differ for boys and girls across all grades. At pretest, on

average, girls in all grades scored in the average or expected range for their grade level.

However, boys scored slightly below average at each grade. At post-test, girls in first grade

made statistically significant, but substantively small gains in self-esteem, while girls in the

second and third grades showed no meaningful change. In contrast, boys in the first and third

grade achieved significantly higher post-test scores than at pretest, making gains of 5 to 7

points.

Social Skills Rating System (SSRS)

At pretest and post-test, teachers rated students on the following 10 items that make up the

cooperation subscale of the SSRS:

• Uses free time in an acceptable way.

• Finishes class assignments within time limits.

• Uses time appropriately while waiting for help.

• Produces correct schoolwork.

• Follows your directions.

• Puts work materials or school property away.

• Ignores peer distractions when doing class work.

• Keeps desk clean and neat without being reminded.

• Attends to your instructions.

• Easily makes transition from one classroom activity to another.

As with the BASE, the SSRS has a separate set of norms for boys and girls. Student

performance on the cooperation subscale was analyzed using two types of scores: 1) raw

scores—a summed score across the 10 items in the subscale, each of which could be scored at

0,1 or 2, for a total maximum score of 20; and 2) behavior levels (constructed by the

developer of the SSRS) that convert students’ raw scores into three classifications: average,

below average, or above average skills.16

Exhibit 3.6 presents the results of pretest and post-test ratings by gender and grade level.

There are not consistent and substantively meaningful changes in students’ performance;

however, there are several observations:

 

16 The SSRS does not provide standardized scores for the subscales, although the responses of the

standardization sample are used normatively to classify subscale total raw scores into average, above, or

below average.
AmeriCorps Tutoring Outcomes Study Chapter 3 33

• The majority of students, regardless of gender or grade, were classified as having

average or above average skills on the cooperation subscale at pretest (a range of 54-

75 percent for boys, 75-92 percent for girls).

• On average, based on the raw scores, boys scored lower than girls at all three grades

and at both pretest and post-test.

Exhibit 3.6

SSRS: Scores on the Cooperation Subscale

Boys Girls

Pretest Post-test Pretest Post-test

Grade 1 (n=113) (n=73)

 Mean (standard error) 11.1 (1.0) 12.3 (0.8) 15.5 (1.2) 16.5* (1.0)

 Behavior Levels

 Above Average 8% 4% 22% 33%

 Average 52 67 63 55

 Below Average 40 29 15 12

Grade 2 (n=95) (n=75)

 Mean (standard error) 11.2 (1.2) 11.5 (0.7) 16.1 (0.7) 15.8 (0.7)

 Behavior Levels

 Above Average 10% 4% 24% 16%

 Average 44 63 68 64

 Below Average 46 33 8 20

Grade 3 (n=63) (n=73)

 Mean (standard error) 11.6 (1.1) 13.2** (1.1) 15.0 (0.9) 15.6 (0.7)

 Behavior Levels

 Above Average 11% 11% 10% 13%

 Average 64 75 65 71

 Below Average 25 14 25 16

*p<.05, **p<.001

Source: SSRS completed by Classroom Teachers.

• Third grade boys made small but statistically significant gains in their ratings. Only 14

percent are rated as having below average skills at post-test compared to 25 percent at

pretest.

• First grade girls also made small but statistically significant gains in their ratings. The

distribution of ratings using the behavior levels indicates that these gains were, for the

most part, the result of an increase in the proportion of girls classified as having above

average skills (22 percent at pretest versus 33 percent at post-test).

Another way to characterize change from pretest to post-test on the cooperation subscale is to

use the behavior levels to identify the proportion of students who improved from below

average to average or above average. As Exhibit 3.7 shows, 71 percent of the students

obtained average or above average ratings at pretest, indicating that most of these students

were already displaying the appropriate classroom behaviors. It is noteworthy, however, that
34 Chapter 3 AmeriCorps Tutoring Outcomes Study

40 percent of the students whose behavior levels were rated as below average at pretest

improved their skills to average or above average at post-test.

Exhibit 3.7

Changes in Students’ Behavior Levels on SSRS from Pretest to Post-test

Pretest Behavior Levels Post-test Behavior Levels Pre-Post Status

Average or

Above Average

Average or Above

Average

= 92% Remain Average or

Above Average

Below Average = 8% Decline

Below Average Average or Above

Average

= 40% Improve

Below Average = 60% Remain Below Average

Source: SSRS completed by classroom teachers.

Summary

This study included an individually administered test of student reading ability and teacher

ratings of reading performance and classroom behavior. On the standardized reading test, the

Woodcock-Johnson, students involved in the AmeriCorps tutoring program made gains in

reading skills and reading comprehension at all grade levels. These gains were large enough

to be statistically significant and also represent meaningful changes in reading performance.

Teachers indicated that the majority of students improved in reading and other classroom

behaviors over the course of year in the AmeriCorps tutoring program. However, these

global teacher ratings did not necessarily correspond to students’ actual scores on the

Woodcock-Johnson.

Two standardized teacher ratings, the SSRS and the BASE, assessed students’ self-esteem

and cooperation in the classroom. In contrast to the changes observed in student reading

performance, there were no such corresponding changes in students’ classroom behaviors.

Girls displayed appropriate or typical behaviors at pretest on both the SSRS and the BASE.

Thus, one would not expect substantive gains at post-test. Boys, on the other hand, were

rated by their teachers as displaying somewhat lower than average classroom behaviors at

pretest. On average, only the third grade boys improved their classroom behaviors on both

the BASE and the SSRS at post-test.

71%

29%

65%

 6%

12%

17%
AmeriCorps Tutoring Outcomes Study Chapter 4 35

Chapter 4

Factors Affecting Student Reading Achievement

Introduction

The previous chapter reported gains in students’ reading achievement overall and by student

characteristics. This chapter focuses on the nine effective practices described in Chapter 2

and examines the relationship of each of these program elements, individually and in

combination, to students’ gains in reading skills and reading comprehension.

It is important to point out that the program-level information was gathered primarily from

program directors’ reports. Tutors provided information on only a few items relevant to

effective practices. Thus, this information addresses what was intended to occur in tutoring

programs.

Effective Tutoring Practices

In Chapter 2 we identified the following group of effective tutoring practices:

• Intensity of tutoring:

ÿ Tutoring sessions occur at least three times a week

ÿ Tutoring sessions total at least 1.5 hours per week

ÿ Most of the tutoring is conducted one-to-one with students

• Administration/Implementation:

ÿ Tutors use a formal curriculum model in their sessions

ÿ Tutors coordinate their activities with classroom reading instruction

ÿ Tutors meet regularly with the school’s reading specialist

ÿ Programs evaluate the effectiveness of their tutoring activities

ÿ Program is at least moderately or fully implemented

• Tutor training:

ÿ Tutors receive training both prior to and during the course of tutoring

Below we describe the extent to which each of these nine effective practices was associated

with gains in students’ reading achievement.17

 

17 Some of the programs did not supply all the information requested. Rather than exclude such programs

from analyses, we replaced missing scores with the sample’s average score. This strategy allowed us to

include the maximum number of programs available and increase the statistical power necessary to detect

differences. Our final analyses are based on 64 programs and their students.
36 Chapter 4 AmeriCorps Tutoring Outcomes Study

Relationship of Effective Practices to Gains in Reading Skill

To investigate the relationship between programs’ use of effective tutoring practices and

students’ reading achievement, we conducted multiple regression analysis using a technique

known as Hierarchical Linear Modeling (HLM).18 This technique is used when the data to be

analyzed are at two levels; for example, in this study, information was collected about

programs and students. Developing models with this technique allows us to estimate the

effects of variables at both the student and program levels simultaneously (see Appendix C

for more details).

Of the nine effective practices, four were significantly related to gains in students’ reading

skills:

ÿ Tutors receive training both prior to and during the course of tutoring

ÿ Tutoring sessions occur at least three times a week

ÿ Program is at least moderately or fully implemented

ÿ Programs evaluate the effectiveness of their tutoring activities

For each of these effective practices, students showed significantly higher gains in reading

skills than students in programs without these practices (Exhibit 4.1). For example, students

who met with their tutors at least three times per week had increases between pretest and

post-test of 2.1 points more than their peers in programs that met less frequently; this

difference is statistically significant (p<.05).

The findings that individually, these four effective practices were related to student gains on

the reading skills subtest raises the question: Do students in programs implementing only

one of these effective practices make the same gains as students in programs with two, three

or all four of these effective practices? Approximately 28 percent of AmeriCorps programs

reported all four effective practices, 43 percent reported three, 13 percent reported two, 9

percent reported only one, and 6 percent did not report any of these program-level

components. Thus, the majority of programs (71 percent) included at least three of these

effective practices, and very few programs did not engage in any of these key components.

 

18 Bryk, A.S., and Raudenbush, S.W. (1992). Hierarchical Linear Models: Applications and Data Analysis

Methods. London: Sage Publications.
AmeriCorps Tutoring Outcomes Study Chapter 4 37

Exhibit 4.1

Additional Gains in Student Reading Skills Associated with Effective Practicesa

1.6

1.8

2.1

2.3

0 1 2 3 4 5

Program has formal

evaluation

Programs are moderately

or fully implemented

Tutor meets at least

three times per week

Tutors trained both

during and prior to

tutoring

Student Gains in Reading Skills

a

Only those effective practices associated with statistically significant (p<.05) gains are displayed.

Source: Woodcock-Johnson Psycho Education Battery-Revised for student-level gains; Program Director

Questionnaire and Tutor Logs for program-level information.

To further investigate the relationship between these effective practices and gains in reading

skills, we examined whether students in programs implementing all four practices exhibited

greater gains than students in programs that reported three or fewer elements. As Exhibit 4.2

shows, students in programs with all four effective practices gained an average of 5.4 points

between pretest and post-test, which is significantly more than students in programs with

three practices, who gained only 2.5 points; students in programs that reported none of the

four effective practices gained less than one point on the reading test. This suggests that

while each of the four practices are important individually with respect to students’ gains in

reading skill, students in programs with all four effective practices are likely to show the

greatest gains in reading skills compared to their counterparts in programs that had three or

fewer of these tutoring practices.

Students in programs with three effective practices showed greater gains than did those in

programs that did not report any of the program components. There were no statistically

significant differences in reading skill gains between students in programs with three versus

two, two versus one, one versus none of the effective practices, or any other combinations.
38 Chapter 4 AmeriCorps Tutoring Outcomes Study

Exhibit 4.2

The Relationship Between Multiple Effective Practices and Student Gains in Reading Skills

0.3

2.3

2.3

2.5

5.4

0 1 2 3 4 5 6

Program has no effective

practices

Program has 1 effective

practice

Program has 2 effective

practices

Program has 3 effective

practices

Program has 4 effective

practices*

Student Gains in Reading Skills

* Programs with four effective practices are associated with significantly higher gains scores than all other groups (p<.05).

Source: Woodcock-Johnson Psycho Education Battery-Revised for student-level gains; Program Director

Questionnaire and Tutor Logs for program-level information.

Relationship of Effective Practices to Gains in Reading Comprehension

We also examined the relationship between each of the nine effective practices and gains in

reading comprehension. Only one of the program-level components was associated with

significant student gains—coordination with classroom reading instruction—but in the

opposite direction than expected. Specifically, students in programs that reported

coordinating their tutoring practices with classroom reading instruction were less likely to

show gains in reading comprehension than students in programs whose tutors did not engage

in coordinating activities. This unexpected finding may suggest that tutors who did not

coordinate with classroom instruction may be qualitatively different from those tutors that

did engage in coordination. For example, it is possible that tutors who do not engage in

coordinating activities have more experience and thus may not seek external input from

classroom teachers. It may be having more experience, in this hypothetical example, not the

lack of coordination per se, that is responsible for the greater gains in reading

comprehension. Unfortunately, we do not have the information necessary to test this

hypothesis empirically.

As only one program-level component was significant, we did not examine the pattern of

effective practices in relation to students’ reading comprehension.
AmeriCorps Tutoring Outcomes Study Chapter 4 39

Summary

Overall, the relationship between effective tutoring practices and student reading gains is

different for the two reading scores. With respect to reading skills, four effective practices

were each significantly associated with gains—programs that met at least three times a week,

programs that reached fuller implementation, programs that conducted formal evaluations,

and tutors who received prior and ongoing training. Students in programs with all four

practices exhibited greater gains in reading skills than students in programs that reported

three or fewer components. In addition, students in programs with three of the effective

practices showed greater gains than did those in programs with none of the key practices. No

statistically significant differences were detected among programs that reported three versus

two, two versus one, or one versus none of these program elements. This suggests that there

may be a “threshold” effect, where students who experience at least three effective practices

are more likely to show reading skill gains than are those in programs that have less than

three of these practices.

In terms of reading comprehension, programs where tutors did not coordinate their tutoring

practices with classroom instruction were associated with larger student gains. This

relationship was counterintuitive and merits additional exploration and replication. No other

program-level components were significantly related to gains in reading comprehension.

Further summary and implications of these findings are reported in Chapter 5.

AmeriCorps Tutoring Outcomes Study Chapter 5 41

Chapter 5

Summary and Conclusions

The evaluation of AmeriCorps’ tutoring program was conducted at a time when there is

considerable public attention on the issue of our students’ reading achievement, especially

those in primary grades. Several significant federal initiatives seek to improve reading

instruction and reading achievement, and numerous state and local efforts do as well. Further,

tutoring activities are an integral part of many of these initiatives. Given this context, the

findings from this study become even more important.

Reading Achievement

The results from this study indicate that students participating in AmeriCorps tutoring

programs improved their reading performance from pretest to post-test more than the gain

expected for the typical child at their grade level. This finding holds for students at all grade

levels tested. Reading comprehension and reading skills were below grade-level at pretest; by

year-end, students closed the gap and were reading at or near grade-level expectations. Not

only are gains statistically significant, they are also large enough to signify real improvement in

students’ reading abilities. In addition, the magnitude of the reading gains was the same for

students of different ethnic/racial backgrounds; there were no statistically significant

differences in the reading score gains of White and non-White students.

Classroom Behavior

The study examined the effects of participation in the tutoring program on changes in

students’ classroom behaviors. On two classroom behavior measures, girls displayed

appropriate or typical behaviors at pretest, thus minimizing the likelihood of observing a

positive change at post-test. In general, teachers rated boys as displaying somewhat lower

than average classroom behaviors at pretest. Only the third grade boys improved their

classroom behaviors on both the BASE and the SSRS at post-test.

Program staff hypothesized that students would improve their classroom behavior as a result

of receiving tutoring services, because the special attention students received through tutoring

would have positive effects on children's sense of worth as well as their interest in learning.

Because the majority of students in our sample were rated as in the normal or expected range

at pretest, however, our results do not provide evidence to support this hypothesis.

Effective Tutoring Practices

Four program-level effective tutoring practices were related (both individually and in

combination) to student gains in reading skills:
42 Chapter 5 AmeriCorps Tutoring Outcomes Study

ÿ Tutoring sessions occur at least three times a week

ÿ Tutors receive training both prior to and during the course of tutoring

ÿ Program is at least moderately or fully implemented

ÿ Programs evaluate the effectiveness of their tutoring activities

Students participating in programs implementing all four effective practices are likely to show

the greatest gains in reading skills compared to their counterparts in programs that had three

or less of these tutoring practices. The fact that effective tutoring practices were related to

student reading skill gains increases our confidence that these gains can be attributed to the

AmeriCorps program, rather than other program factors that might also explain the

differences in student reading outcomes.

It is interesting to note that even though students made significant gains in reading

comprehensive, these gains were not related to program-level effective practices. Instruction

for struggling readers often focuses on the mechanics of reading, which is what the reading

skills subtest of the Woodcock-Johnson measures. These skills are the precursors to reading

comprehension. These results from the AmeriCorp evaluation suggest that effective practices,

such as the frequency of tutoring sessions, may have greater impact on students’ reading

mechanics than on reading comprehension.
APPENDIX A

SAMPLE SELECTION AND WEIGHTING PROCEDURES

AmeriCorps Tutoring Outcomes Study Appendix A 1

APPENDIX A

This appendix has two sections; first we discuss the sampling procedures, then we present the

techniques used to construct program- and student-level weights in order to generalize the

findings to the population of AmeriCorps tutoring programs and participating first, second

and third grade students.

Sample Selection

In order to produce estimates of the AmeriCorps tutoring program nationwide, a multi-stage

sampling design was implemented that selected programs, schools and students to participate

in the study. After completing this process, our effective sample included 68 programs, 96

sites (76 schools and 20 after-school programs), and 869 children. Since it was

representative, this sample allowed us to generalize the findings to the entire population of

AmeriCorps programs and tutored students through the use of sampling weights at the

program and student level.

Each stage in the sampling procedure is described below.

Stage 1: The Sample of Tutoring Programs

Using information provided by the Corporation, we identified 271 AmeriCorps programs that

provided tutoring in reading to students in some combination of grades one, two and three.

Exhibit A1 summarizes the recruitment of programs into the study.

A nationally representative sample of 145 programs was selected. We stratified the sample

by geographic region and size of tutoring program (i.e., number of students tutored). During

the recruitment of programs, we learned that 37 programs were ineligible to be in the study

and an additional 18 either refused or could never be contacted. The two primary reasons for

ineligibility at this stage centered on programmatic changes made between the 1998-1999

school year in which the Descriptive Study was conducted and the 1999-2000 school year in

which the data were collected for the AmeriCorps Tutoring Outcomes Study. We found that

either the program was no longer conducting reading tutoring at all or that the program no

longer supported tutoring in grades 1 through 3. This left us with a sample of 90 tutoring

programs.

1

 

1 Tests were conducted to assess the representativeness of the program sample. Overall, there were no

significant differences between our program sample and the national population of programs with one

exception, there were slightly more small programs and slightly fewer medium/large programs in the

sample studied.
2 Appendix A AmeriCorps Tutoring Outcomes Study.

Exhibit A1

Summary of Recruiting Status

Sampled Programs for the Tutoring Outcomes Study

Sampled

Programs

n = 145

Excluded Ineligible

Programs Programs

n = 18* Agreed to n = 37

Participate

n = 90

Program/Site Research Pending Complete Incomplete Not Serving No Longer Scheduling

Refusal Refusal Callbacks Grades 1-3 Tutoring Conflict

n = 12 n = 3 n = 3 n = 71 n = 19 n = 14 n = 17 n = 6

Other details:

1. Nationally representative sample selected from 21 Primary Sampling Units.

2. Current program level response rate = 66%. (Completes/Total sample - Ineligibles)

3. Programs that are labeled "incomplete" never provided signed consent forms or were not able to be scheduled for testing by January 14th.

4. Of the programs that agreed to participate, 44 (49%) are designated "high quality" programs.

13-Nov-2000 5. Current program level refusal rate = 11%. (Refusals/Total sample - Ineligibles)

6. * 12 of the excluded programs are refusals

 * 3 of the excluded programs are refusal to conduct research in schools

 * 3 of the excluded programs are final non-contact
AmeriCorps Tutoring Outcomes Study Appendix A 3

Stage 2: The Sample of Schools

Our sample of 90 AmeriCorps programs provided tutoring services in 459 sites (363 schools,

96 after-school programs). From this population, we selected a sample of 164 sites, 137

schools and 27 after-school programs. Of these, 128 sites initially agreed to participate in the

study. However, we were not able to collect data at 35 locations (representing 22 of the

original 90 programs), because the schools and after-school sites were unable to obtain

sufficient parental consents in time to be included in the baseline data collection. Thus, our

effective sample at the school/site level consists of 93 sites (73 schools and 20 after-school

programs) representing 68 AmeriCorps tutoring programs.2 Exhibit A2 summarizes the

participating schools and after-school programs.

Stage 3: Selection of students

Determining Sample Size. Before selecting a representative sample of participating

students, we had to determine the appropriate sample size needed to address the research

questions. Several factors were considered as part of this decision. First, we determined the

power we wanted to have to detect differences from pretest to post-test. Power is defined as

the ability to detect a difference on some measure between two groups, when there is, in fact,

such a real difference. Consistent with the standards in the field, we wanted to have at least

80 percent power. In addition, we set the size of the sample to allow for high precision; our

criterion was the ability to detect a five point pretest to post-test difference at the 95 percent

confidence level.

Two other factors were involved in determining the necessary sample size. One involved the

design effect. Because the sample was a clustered rather than a simple random sample, we

needed to increase the sample size in order to adjust for a reduction in variation due to the

effects of clustering. Since we could not know the true design effect until all data were

collected, we took a conservative approach by estimating a large design effect of 2.5

although we expected the true design effect to be somewhat less (perhaps 2.0). Finally, it

was important to allow for some attrition from the sample from pretest to post-test.

Interestingly, while we estimated an attrition rate of 20 percent, our final attrition rate was

only 8 percent. (Most of the students whom we were unable to test at the end of the school

year had moved and transferred to other elementary schools, often located out of the area.)

Selection of Students. To recruit the study sample from the selected schools and afterschool programs, consent forms were sent home to parents of all students participating in the

tutoring program. The original methodology called for obtaining a complete set of

permissions from each site and then randomly selecting the student sample. However, since

only a subset of students returned signed consent forms, we could only sample from that

subset. In schools/sites where we had more students with signed consent forms than called

 

2 While 71 AmeriCorps programs participated in the data collection, we were able to obtain a

complete set of data for only 68 of the programs, which was used for the various analyses.
4 Appendix A AmeriCorps Tutoring Outcomes Study.

Exhibit A2

Summary of Recruiting Status

Schools/Sites

Sampled

Schools/Sites

n = 164

Non-participating Ineligible

Schools/sites Schools/Sites

n = 19 Agreed to n = 17 Schedule

Participate Conflict

n = 128 n = 1

School/Site Pending Final Testing Pending Not Serving No Longer Research

Refusals Callbacks Other complete No Consent Grades 1-3 Tutoring Conflict

n = 16 n = 2 n = 1 n = 96 n = 32 n = 3 n = 6 n = 7

Other details:

1. Nationally representative sample selected from 21 Primary Sampling Units.

**************** 2. Current school/site response rate = 65%. (Completes/Total sample - Ineligibles)

3. 32 or 25% of the 128 consenting schools never completed parent consent forms and scheduling.

4. Current site level refusal rate = 11%. (Refusals/Total sample - Ineligibles)
AmeriCorps Tutoring Outcomes Study Appendix A 5

for by the sampling plan, a random selection of students was made.3 In sites where there

were fewer students than necessary for the sampling plan, all students with signed consent

forms were selected. Finally, only those students with data at both pretest and post-test were

included in our analyses. Exhibit A3 displays counts for the population of students

participating and our study sample for each grade.

Exhibit A3:

Population and Sample Counts of Tutored Students

Grade

Population of Students

across all Tutoring

Programs

Students in

Sample

First 31,584 294

Second 34,647 292

Third 35,734 283

Total 101,965 869

The fact that we selected our sample from those students with consent forms raised the issue

of whether the students not returning consent forms (i.e., the non-respondents) differed in

some way from students for whom we had forms. Unfortunately, since we had no

information about the non-respondents, we could not conduct analyses comparing them to

the study sample.

Below, we describe the procedures used to construct the sampling weights needed to

compute national estimates of program effects.

Sampling Weights

To generate national, population-based estimates (either descriptive or impact), each

responding program and student in the sample received a sampling weight. The final

sampling weight combined the basic sampling weight, which reflects the probability of

selection, and an adjustment for non-responding programs, schools/sites, and students. In

this section, we describe the procedure for computing these sampling weights. The

construction of weights was directly linked to the sampling design used to select programs,

sites and students for participation in the study. Hence, we begin with a brief description of

the sampling methods that have been used.

 

3 As a consequence of this over-sampling, we have data on more students than called for in the

original sampling plan. These data were not used in computing the estimates of program effects.

However, they were used to strengthen the relational analyses that examine relationships among

program characteristics and student outcomes.
6 Final Report: Appendix A AmeriCorps Tutoring Outcomes Study

Selection of the Sample

A sample of students in each grade was selected using a four-stage sampling design.

• At the first stage, the AmeriCorps tutoring programs were grouped into primary sampling

units (PSUs) or clusters within each census division. A sample of PSUs was selected within

each of these census divisions with probability proportional to the number of participating

students in each PSU.

• At the second stage, a systematic sample of programs was selected from each selected PSU.

• At the third stage, a systematic sample of schools/sites was selected from each selected

program.

• At the last stage, a systematic sample of students was selected from each of the three grades.

Construction of the sample weights

The basic sampling weight for a student selected in the sample reflects the probability of selection of a

PSU, the probability of selection of a program within a selected PSU, the probability of selection of a

school within a selected program and finally the probability of selection of a student within a selected

school.

Let Sij denote the number of students in the j th PSU in the i th census division. Let M i denote the

number of PSUs in the i th census division. Let mi denote the number of PSUs selected in the sample in

the i th census division. The probability of selecting the j th PSU in the i th census division is

s

s

m

i

ij

p ij = i

where ∑==

M

s s

i

j

i ij

1

 is the total number of students in the i th census division. The weight to be attached to

the j th PSU is 

p ij

wijk

1

= . Let Nij denote the number of programs in the j th PSU in the i th census

division. Let nij be the number of programs selected in the sample from the j th PSU in the i th census

division. The conditional weight to be attached to a selected program in the j th PSU in the i th census

division is 

n

N

ij

ij . This weight will be adjusted for program nonresponse. If out of nij programs, only nij

*

programs respond and nij

1

 are eligible nonrespondents, then the nonresponse adjustment is 

n

n n

ij

ij ij

*

* 1 +

.
AmeriCorps Tutoring Outcomes Study Appendix A 7

The overall program weight for a respondent program “k” in the j th PSU in the i th census division is

given by

N

n n

n

N

w

ij

ij ij

ij

ij

ij

ijk *

* 1

1 +

=

p

All national estimates of the characteristics of AmeriCorps tutoring programs were obtained using these

weights. f or example, let p denote the estimated proportion of programs that have a certain

characteristic. Then

w

m n

w y

m n

p

ijk

i j k

ijk ijk

i j k

ij i

ij i

∑ ∑ ∑

∑ ∑ ∑

= = =

= = =

=

1 1

9

1

1 1

9

1

gives the estimated proportion.

Let the number of schools in the population in the k th program in the j th PSU in the i th census division

be Hijk . Let the number of schools in the sample be hijk . The conditional weight for a selected school in

the k th program is given by 

h

H

ijk

ijk . If we assume that out of hijk schools, hijk

*

 schools respond, and hijk

1

eligible schools do not respond, then the nonresponse adjustment is given by 

h

h h

A

ijk

ijk ijk

ijk *

* 1 +

. The

non-response-adjusted conditional school weight is 

h

H

A

ijk

ijk

ijk . The overall school weight is given by the

overall program weight multiplied by the nonresponse adjusted school weight.

For determining student weights, we consider each grade as a separate stratum, since samples were

selected independently from each grade. Let the number of students in grade 1 in the l th school in the kth

program, j th PSU and the i th census division be S1ijkl . Let the number in the sample be s1ijkl . The

conditional basic student sampling weight is 

s

S

ijkl

ijkl

1

1

.
8 Final Report: Appendix A AmeriCorps Tutoring Outcomes Study

The basic student weight was adjusted for nonresponse using the same approach described in the

construction of the school weights. The overall student weight is the overall school weight multiplied by

the nonresponse adjusted conditional student weight.

In summary, the program and student weights will be used to compute the estimates of population

parameters relating to AmeriCorps’ tutoring programs and the student participants in these programs.
APPENDIX B

STUDY MEASURES

AmeriCorps Tutoring Outcomes Study Appendix B 1

APPENDIX B

This appendix describes the measures used in the study along with the rationale used in the

selection process.

The Measurement Battery

The measurement battery used in the study collected information in three broad areas: 1)

student outcomes; 2) characteristics of the tutoring programs and 3) background information

about tutors and students. Each measure is described below. Copies of the instruments

developed specifically for this evaluation are included at the end of the Appendix.

Student Outcomes

To address the research questions, two kinds of student behaviors were measured: 1) reading

proficiency; and 2) classroom behavior. Each measure had to be appropriate for use with

students in grades 1, 2 and 3. Each of the outcome measures is summarized below.

Reading skills. One of the central literacy services provided by AmeriCorps members is

the direct tutoring of reading skills to children at risk of becoming poor readers. The goal of

this activity is to improve children’s reading skills. Therefore, one of the primary outcome

measures to be selected was a standardized measure of reading ability. The selected test had

to: a) measure several reading subskills; b) have an adequate norming sample; c) have

acceptable psychometric properties (e.g., reliability and validity); d) be easy to administer;

and e) be well-accepted in the field (i.e., used in comparable studies of at-risk students).

We selected the Woodcock-Johnson Psycho-Educational Battery--Revised (WJ-R), a group

of subtests, many of which address the basic skills for reading.1 The WJ-R is an individually

administered test battery designed to assess the intellectual and academic development of

individuals from preschool through adulthood. This recent version includes a norming group

for which data were collected from 1986 through 1988. There are 41 subtests that may be

administered separately or in combination with other subtests; each subtest takes

approximately five minutes to complete. Reliabilities for the subtests are generally above

.90, and the tests have demonstrated acceptable concurrent and construct validity.

A selected group of six subtests was administered to the study sample:

• reading skills using the Letter-Word Identification and Word Attack subtests;

• reading comprehension using the Passage Comprehension and Reading Vocabulary

subtests; and

 

1 Woodcock, R.W., and Johnson, M.B. (1989, 1990). Woodcock-Johnson Psycho-Educational BatteryRevised. Itasca, IL: Riverside.
2 Appendix B AmeriCorps Tutoring Outcomes Study

• phonological awareness using the Incomplete Words and Sound Blending subtests.

Each subtest is briefly described below.

• Letter-Word Identification: The first five items involve symbolic learning, or the

ability to match a rebus (pictographic representation of a word) with an actual picture

of the object. The remaining items measure the participant’s reading identification

skills in identifying isolated letters and words. In this test it is not necessary that the

participant know the meaning of any word correctly identified. The items become

more difficult as they present words that appear less and less frequently in written

English.

• Example: Item 13 is the letter “j”. The examiner points to it and says,

“What is the name of this letter?” Item 14 is the word “to.” The

examiner points to it and says, “What is this word?”

• Word Attack: This subtest measures a participant’s skill in applying phonic and

structural analysis skills to the pronunciation of unfamiliar printed words. The

participant is asked to read aloud letter combinations that are linguistically logical in

English (or Spanish) but that form nonsense words or low-frequency words.

• Example: Items 1-3 are the nonsense words “tiff,” “nan,” and “rox.”

After having explained to the child that some of these words are not real

words, the examiner asks the child to “Read each of these words to me.”

• Passage Comprehension: The first four items are presented in a multiple-choice format

requiring the participant to point to the picture represented by a phrase. The remaining

items measure the participant’s skill in reading a short passage and identifying a

missing key word. The task requires the subjects to state a word that would be

appropriate in the context of the passage. The subject must exercise a variety of

comprehension and vocabulary skills.

• Example: Item 7 shows a blue book resting on a yellow chair and a red

ball resting on a green box. The sentence next to the illustration reads,

“Something is on the chair. It is a _____.” The child is instructed to

“Read this to yourself and tell me one word that goes in the blank space.”

The correct response would be “book.”

• Reading Vocabulary: This subtest measures the participant’s skill in reading words

and supplying appropriate meanings. In Part A: Synonyms, the participant must state

a word similar in meaning to the word presented. In Part B: Antonyms, the participant

must state a word that is opposite in meaning to the word presented. Only one-word

responses are acceptable.
AmeriCorps Tutoring Outcomes Study Appendix B 3

• Example: Item 2 of the Synonyms is the word “puppy.” The examiner

says, “Read each of these words out loud and tell me another word that

means the same.” The correct responses for Item 2 would be “dog,”

“doggy,” “pup.” The instructions for the Antonyms are similar except the

child is told to “tell me another word that means the opposite.” Item 1 is

the word “on.” Correct responses would be “off,” “under,” while incorrect

responses would be “yes,” “in” or “out.”

• Incomplete Words: This is a tape-recorded test that measures auditory closure. After

hearing a recorded word that has one or more phonemes missing, the subject identifies

the complete word. This test primarily measures auditory processing.

• Example: The child hears Item 2 as “sti_er,” repeated once. The

examiner says to the child, “Tell me the whole word that she is trying to

say.” The correct response is “sticker.”

• Sound Blending: This test measures the ability to integrate and then say whole words

after hearing parts (syllables and/or phonemes) of the words. An audio tape is used to

present word parts in their proper order for each item. This test primarily measures

auditory processing.

• Example: Item 15 is “fi-sh”. The child is asked to tell the examiner what

the word is (fish) after hearing the two phonemes of the word.

Preliminary analyses of the scores obtained on the last two subtests, incomplete words and

sound blending, indicated that they were not reliable measures related to students’ reading

skill levels. Therefore, we excluded them from any subsequent analyses.

Classroom behaviors. For measures of classroom behavior, we selected ones that: 1) use

teacher ratings (rather than parent or child); 2) focus on adaptive school behaviors such as

cooperation, ability to take turns, rather than manifestations of clinical problems; and 3) are

normed using a standardization sample. Based on these criteria, we selected three measures:

1) the Behavioral Academic Self-Esteem rating scale (BASE);22) the cooperation subscale

from the Social Skills Rating System (SSRS);3and 3) several additional questions for the

classroom teacher about students’ reading behaviors in the classroom.

BASE. The BASE measures children’s academic self-esteem by using direct observations

of their classroom behaviors. Teachers rate their students on 16 items that capture the

school behaviors of interest to the study. The measure has adequate technical properties:

1) measures of reliability (ranging from .36 to .75); 2) predictive validity (correlation of

 

2 Coopersmith, S., and Gilberts, R. (1982). BASE: Behavioral Academic Self-Esteem. A Rating Scale. Palo

Alto: Consulting Psychologists Press.

3 Gresham, F.M., and Elliott, S.N. (1990). Social Skills Rating System. Circle Press, MN: American

Guidance Service.
4 Appendix B AmeriCorps Tutoring Outcomes Study

.50 with the composite score on the Comprehensive Test of Basic Skills); and 3) adequate

norming sample of 4,000 children grades K-8.

• Example: Item 1 is “This child is willing to undertake new tasks.” Item

14 is “This child refers to himself or herself in generally positive terms.”

Teachers rate the frequency of the behavior as “Never”, “Seldom,”

“Sometimes,” “Usually,” or “Always.”

SSRS. The SSRS is designed for teachers to rate children in the areas of Academic

Competence, Problem Behaviors, and Social Skills. The SSRS has been widely used and

also has been nationally normed with a sample of 2,400 children in grades K-6. We used

one part of the Social Skills section of the test, the 10 items comprising the Cooperation

subscale. This information provided an additional measure of students’ school behavior,

cooperation in a school setting. The Cooperation subscale has strong reliability properties:

1) internal consistency, coefficient alpha=.92; 2) test-retest reliability r=.88 and 3)

interrater reliability, r=.54,

• Example: Item 15 is “Uses time appropriately while waiting for help.”

Item 29 is “Easily makes transition from one classroom activity to

another.” Teachers rate the frequency of the behavior as “Never”,

“Sometimes,” or “Very often.”

Teacher perceptions of students’ reading behaviors. We developed a questionnaire

specifically for this study that asked teachers to make judgments about change in

comprehension proficiency, changes in mechanics proficiency, time on task while engaged

in reading activities, student interest in reading, and interest in school in general. (See

measure at the end of the appendix.)

• Example: Teachers are asked to circle one response for each of a number

of items. Item 2a reads, “Since participating in the tutoring program, I

think my student is more interested in reading.” Item 2b is “Since

participating in the tutoring program, I think my student is more interested

in school.” Responses are “A lot,” “Some,” “A little,” “Not at all” and

“Don’t know”.

Characteristics of the Tutoring Programs

We collected two types of information about the literacy services: 1) program-level

information about the operation and implementation of tutoring activities; and 2) studentlevel information about the intensity of the tutoring services they receive. Each type of

information is described below.

Program-level information. The Descriptive Survey provided a basic set of information,

at the program-level, on each literacy program in the study sample. These data were
AmeriCorps Tutoring Outcomes Study Appendix B 5

available to use analytically in this evaluation. However, the information was collected in

the spring of 1999 and described the program as implemented in the 1998-99 year. Thus, it

was necessary for us to update our database on a subset of key items relating to the delivery

of tutoring services in spring 2000 to ensure that we had data that reflected tutoring programs

as implemented in the 1999-2000 year. (A copy of the Program Director Questionnaire

appears at the end of the appendix.)

Student-level information. To assess the extent of implementation at the student level,

we collected information from the AmeriCorps tutors about their tutoring experiences with

students. Tutors completed a Record of Reading Activities with Students twice during the

1999-2000 school year. For two designated weeks, tutors provided information about 1) the

amount of tutoring scheduled and actually received during a week; 2) the types of literacy

activities that took place; and 3) their estimates of student interest, motivation and

engagement in the literacy activities. (A copy of the Tutor Log appears at the end of the

appendix.)

Background Information

We collected two kinds of background information: 1) demographic characteristics of the sample of

students receiving tutoring services; and 2) demographic characteristics of the tutors.

Student characteristics. Field staff recorded students’ gender at the same time that they

administered the Woodcock-Johnson. Student ethnicity was provided by teachers as part of

the SSRS/BASE instrument.

Tutor characteristics. We designed a Tutor Questionnaire specifically for this study that

collected information from tutors about their: 1) age; 2) gender; 3) education level; 4)

previous experience working in an educational setting; 5) previous experience with tutoring

children; and 6) amount of training they have received from AmeriCorps and other sources.

(See measure at the end of the appendix.)
Introduction to the AmeriCorps Tutoring Outcomes Study:

Information for Teachers

The Corporation for National Service, the government agency that sponsors AmeriCorps programs,

has contracted with Abt Associates Inc., a nationally recognized research firm with a long history of

studies in education, to assess the impact of AmeriCorps literacy tutoring on the reading

proficiency and classroom behavior of children. AmeriCorps literacy activities are one important

way to respond to President Clinton=s America Reads initiative that calls on all Americans to help

ensure that every child can read well and independently by the end of third grade. The AmeriCorps

Tutoring Outcomes Study is designed to tell us about the effects on students of direct literacy

tutoring through AmeriCorps programs.

In this study, we are following a group of students in first, second, and third grades who are

receiving direct tutoring through AmeriCorps. The students will be followed for a year; at the end of

the year, we will look at whether, over a single school year, students have improved on their reading

skills or their classroom behavior. The students in this study are a nationally representative

sample of students across the country who are receiving this tutoring, which means that what we

learn from the sample of students can be assumed to tell us about all children nationwide who are

receiving tutoring through AmeriCorps.

Your school has been selected at random to be part of this study and several of your students who

are receiving reading tutoring from the AmeriCorps program have been selected to participate from

the school. This means that we will collect information about student skills at the beginning and end

of the school year. Data on students= reading proficiencies will come from a standardized measure

administered individually to students. The AmeriCorps tutors will be providing information on the

nature, content, and duration of their tutoring sessions. These data will be used to help explain

differences in children=s growth in reading skills over the year. 

The study also depends on information from teachers about student=s classroom behavior. 

Information from the teachers about student=s classroom behaviors is a critical component for

learning about the effects of the tutoring services. These data will come from two brief teacher

ratings using standardized scales, that will also be collected twice during this school year. 

Parent permission has been obtained for all of the students in the study, which allows us to ask you

for this student information. In the analyses and reporting, your name and the names of the

students in the study will never be identified.

We are asking teachers to provide us with the ratings twice during this school year Bonce in the

fall, and again in the springBusing the following three measures:

$ Social Skills Rating System8: Cooperation Subscale (10 items);

$ Behavioral Academic Self-Esteem Rating Scale BASE8 (16 items); and

$ Teacher Perceptions of Students= Reading Behaviors (5 items).

You may be providing information on only one student or on many. The names of the students who

have been selected for the study will be provided to the tutors at the time you are asked to complete

the ratings. These forms are very user-friendly and should take no more than 5 - 10 minutes to

complete for each student. We are aware that we are placing an additional burden on busy teacher

schedules. In recognition of this fact, Abt Associates Inc. will pay you $10.00 for each completed

student packet of 2 forms. Thank you in advance for your cooperation.

Any questions about the study or how to fill out the form--please call Shana Cook toll-free at

1 - (888) 390-1214
AmeriCorps Tutoring Outcomes Study

Instructions for Teachers

Please read the sheet titled "Introduction to the AmeriCorps Tutoring Outcomes Study: Information

for Teachers," before completing either of the two teacher ratings for each student in our study.

What you should receive from the AmeriCorps coordinators:

ÿ A list of students included in the study

ÿ "Information for Teachers" (1 page)

ÿ "Instructions for Teachers" (1 page)

ÿ Teacher Payment Information Form (1 page)

ÿ Teacher Rating Packets containing copies of the three ratings for each child:

Social Skills Rating System: Cooperation Subscale (1 page, front and back)

Behavioral Academic Self Esteem Rating Scale, BASE (1 page, 3 sides)

Teacher Perceptions of Students’ Reading Behaviors (1 page)

Please make sure that you have all of these materials before you begin. If you have any questions or are

missing materials, please do not hesitate to ask the AmeriCorps coordinator.

What we need you to do:

Please complete the Teacher Rating Packet, containing the three forms, for each of your students on the

study list. The forms should take no more than 5 – 10 minutes to complete for each student. When you

have completed all forms for every study student, please return all the forms and packets to the

AmeriCorps coordinator as quickly as possible.

1. Teacher Instructions for the Behavioral Academic Self Esteem Rating Scale, BASE

Complete one form of the Behavioral Academic Self Esteem Rating Scale, BASE for each

student to be evaluated. Complete the front and inside pages only according to directions

on the front page. The inside pages contain 16 items relating to the student's academic

self-esteem. Please do not skip any items. You do not need to complete the back page.

2. Instructions for the Social Skills Rating System: Cooperation Subscale

You should complete one Social Skills Rating System: Cooperation Subscale, both front and

back for each student to be evaluated. When you have entered the Student and Teacher

Information, please turn the form over and complete rating scale of the student's classroom

behavior found on the back of the page. Please do not skip any items.

3. Instructions for the Teacher Perceptions of Students’ Reading Behaviors

This is a simple one-page form to be filled out once for each student. Please do not skip any

items.

Please list the names and grades of all students for whom you have completed a packet on the Teacher Payment

Information form so that we can accurately compensate you for your time in helping with our study.
Designed by Abt Associates Inc. Not to be used without permission from Abt Associates Inc. 1

Teacher Perceptions of Students' Reading Behaviors

1. In your opinion, as a result of the tutoring, how much has the student’s time on task, when

engaged in reading activities, changed. Would you say that student’s time on task is... (Circle

one response)

1. Very Improved, 147/

2. Somewhat improved,

3. The same i.e., no changes, or

4. Worse than when the student started in tutoring.

8. Don’t Know

2. Please circle one response for each item

below. A lot Some A

little

Not at

all

Don’t

Know

Since participating in the tutoring program, I

think my student:...

a. Is more interested in reading. 1 2 3 4 8

 148/

b. Is more interested in school. 1 2 3 4 8

 149/

c. Has improved his/her reading

comprehension skills. 1 2 3 4 8

 150/

d. Has improved his/her reading mechanics

skills (i.e., phonics, decoding, and

vocabulary). 1 2 3 4 8

 151/

SiteID#: _________________________________

School Name: ____________________________

Student Name: ___________________________

Student ID:______________________________
Introduction to the AmeriCorps Tutoring Outcomes Study:

Information for Tutors

As you already know, the Corporation for National Service has contracted with Abt Associates

Inc., a nationally recognized research firm with a long history of studies in education, to assess

the impact of AmeriCorps literacy tutoring on the reading proficiency and classroom

behavior of children. AmeriCorps literacy activities are one important way to respond to

President Clinton=s America Reads initiative that calls on all Americans to help ensure that every

child can read well and independently by the end of third grade. The AmeriCorps Tutoring

Outcomes Study is designed to tell us about the effects on students of direct literacy tutoring

through AmeriCorps programs.

In this study, we are following a group of students in first, second, and third grades who are

receiving direct tutoring through AmeriCorps. The students will be followed for one school year;

at the end of the year, we will look at whether, over a single school year, students have

improved on their reading skills or their classroom behavior. The students in this study are a

nationally representative sample of students across the country who are receiving this

tutoring, which means that what we learn from the sample of students can be assumed to tell us

about all children nationwide who are receiving tutoring through AmeriCorps.

Your school has been selected at random to be part of this study along with a sample of the

students who receive tutoring. We will collect information about these student’s reading skills at

the beginning and end of the school year. Data on students= reading proficiencies will come

from a standardized measure administered individually to students in fall and spring; data on

classroom behavior will come from teacher ratings using standardized scales, also collected

twice during this school year.

The study also depends on information about your tutoring activities with your students.

We need tutors to provide information on the nature, content, and duration of the tutoring

sessions for the students they tutor who are participating in the study. These data will be used

to help explain differences in children=s growth in reading skills over the year. Information

about the tutoring activities with your students is critical to the study and you are the ideal

source of this information. Parent permission has been obtained for all of the students in the

study, which allows us to ask you for this student information. In the analyses and reporting,

your name and the names of the students in the study will never be identified.

We are asking tutors to provide us with this information two times during this school year

Bonce in the fall and again in the springBusing a form we have developed called the Record of

Reading Activities with Students. This form asks you to describe the tutoring activities with

specific studentsBthe ones who were randomly selected from the school to be in the study. This

means that you may be providing information on only one student or on many. The names of

the students who have been selected for the study will be provided at the time tutors are asked

to complete the Record of Reading Activities with Students.

We know you are very busy, so we have tried to make the form as short and user-friendly as

possible. It should take no more than 5 - 10 minutes for each student you are describing. The

Record of Reading Activities with Students is attached, along with more detailed instructions on

how to complete it.

Any questions about the study or how to fill out the form--please call Shana Cook toll-free at

1 - (888) 390-1214.
Tutor Instructions

for

Record of Reading Activities with Students and

Tutor Questionnaire

Introduction to the AmeriCorps Tutoring Outcomes Study Information for Tutors: Please read

this sheet carefully, as it will save you time while answering our questions. If there is anything you

do not clearly understand, please call Shana Cook at (888) 390-1214, the toll-free number listed at

the bottom of the page, before you begin.

Record of Reading Activities with Students: Each Activities Record form allows space for

describing the tutoring activities for up to 5 students. If you have 5 or fewer students selected for the

study, you will only need to use one of the Record forms. If you have more than 5 students in the

study, you will need to use a second Record form. If you have more than 10 students in the study,

you will need to use a third Record form.

Also, if you need to describe some of your students for the designated week and the rest of your

students for another week, please use a separate Activities Record form for each different week.

We have sent you ten Record forms; you may need only one, or two, etc., depending on the number

of students selected for the study. If you need more than the four Record forms, please call the

number listed on the letter and more will be sent to you overnight.

1. In the first box, complete the information about yourself and your program.

2. In the second box, please write the names of the 5 students who are being described.

SECTIONS:

I-II. DURATION AND STRUCTURE OF TUTORING SESSIONS:

Please fill in each section, using one box for each student, if possible in the same order you

filled the Record. From now on refer to the dates assigned by Abt Associates and written in

this section.

III. CONTENT OF TUTORING SESSIONS:

For each student circle Y or N to indicate whether you worked on the following literacy

activities. Again, fill in one box for each student, if possible in the same order you filled the

Record.

IV. STUDENT INVOLVEMENT:

For each student, enter a number (1-4) that best describes that student during the tutoring

week selected by Abt Associates. Enter a 9 if you do not know.

Tutor Questionnaire:

1. Write your name and last name

2. Write your age

3. Check only one

4. Check only one

5. Check only one

6. If your answer is YES, answer questions 6a, 6b

If your answer is NO, then go directly to question 7

7. Check only one
Designed by Abt Associates Inc. Not to be used without permission from Abt Associates Inc. SiteID#______________

Record of Reading Activities with Students batch 234-236/

On the Record of Reading Activities with Students, you will answer a set of questions about the tutoring

activities during a particular school week for each of the students you tutor who have been selected for the

study. We want you to complete these questions during a designated week that is determined in advance by

Abt Associates. For the fall data collection, please fill out the form on your students for the school week

beginning: 237-238 245-246/

239-240// Monday, May 8th through Friday, May 12th, 2000 247-248/

241-244/ 249-252/

1. Please write your name and the date you fill out this record below. We will assign you a unique

identification number. Your name is linked to this ID number only on a list that is kept at Abt Associates in

a locked file that cannot be accessed by anyone other than senior study staff.

Tutor Name:___________________________________1-25/

ID:__________________26-28/ (To be assigned by Abt Associates)

 233/

Program ID:____________________________________29-31/

School/Site ID:_________________________________32-34/

Date you are filling out this record:_____/____/______35-42/

 MM / DD / YYYY

Posttest: Cycle 2

2. If any student misses more than 25%, or a quarter, of his/her tutoring sessions during the specified

week, please fill out the form on these students based on activities during the next 5-day week of

tutoring. Cross out the preprinted date on your form and enter the new dates of the week selected on

the form and in the box above.

On each Record form, the tutoring activities described for each child should all occur during the

same 5-day week. If, for some of your students, you need to describe tutoring activities during

another week, always use a new Record form and substitute the dates of the new week on the form

and in the box above.

3. Before you begin this Record form, please list below the students who are being described. On

the form, you will use one column for each student. You should never be describing two

students in the same column.

Student Number Student Name Student ID#

Student 1 43-49/

Student 2 50-56/

Student 3 57-63/

Student 4 64-70/

Student 5 71-77/

Thank you for taking time to complete this questionnaire as part of the Tutoring Outcomes Study!
Designed by Abt Associates Inc. Not to be used without permission from Abt Associates Inc. SiteID#______________

I. DURATION OF TUTORING SERVICES

Thinking about the entire school year, 1999-2000 ... Student 1

Name

Student 2

Name

Student 3

Name

Student 4

Name

Student 5

Name

1. Is this student still receiving tutoring services in reading? If

student is no longer tutored, answer Q.s 1a & 1b. If student is still

tutored, GO TO Q. 2

Y N

253/

Y N

294/

Y N

335/

Y N

376/

Y N

417/

1a. (If Student is no longer tutored) When did reading tutoring

end for this student? Please enter date in column.

254-261/

-----/-----/-----

295-302/

-----/-----/-----

336-343/

-----/-----/-----

377-384/

-----/-----/-----

418-425/

-----/-----/-----

1b. (If Student is no longer tutored) Why did reading tutoring

end for this student? Circle number(s) for reason code(s) in

column. Or, write in reason & student # as other specify below.

1. Student no longer at school

2. Student no longer needed assistance

3. Behavioral problem

4. Tutor Unavailable (e.g., Left Program, Illness)

5. Tutoring program ended

6. Other Specify:______________________/Student #____

GO TO NEXT STUDENT & ANSWER ALL OF Q.1

1 262/

2 263/

3 264/

4 265/

5 266/

6 

267/

268-269/

1 303/

2 304/

3 305/

4 306/

5 307/

6

 308/

309-310/

1 344/

2 345/

3 346/

4 347/

5 348/

6 349/

350-351/

1 385/

2 386/

3 

387/

4 388/

5 389/

6 390/

391-392/

1 426/

2 427/

3 428/

4 429/

5 430/

6 431/

432-433/

2. Were you the tutor who worked with this student when we tested

him/her the first time? If no, answer Q.s 2a & 2b. If tutor DID NOT

change, GO TO Section II.

Y N

270/

Y N

311/

Y N

352/

Y N

393/

Y N

434/

2a. (If Tutor changed) When did reading tutor change for this

student? Please enter date in column.

271-278/

-----/-----/-----

312-319/

-----/-----/-----

353-360/

-----/-----/-----

394-401/

-----/-----/-----

435-442/

-----/-----/-----

2b. (If Tutor changed) Why did reading tutor change for this

student? Circle number(s) for reason code(s) in column. Or,

write in reason & student # as other specify below.

1. Behavioral problem

2. Personality conflict with student

3. Tutor Unavailable (e.g., Left Program, Illness)

4. Scheduling Conflict

5. Other Specify:______________________/Student #____

GO TO NEXT STUDENT & ANSWER ALL OF Qs.1 & 2

1 279/

2 280/

3 281/

4 282/

5 283/

284-285/

1 320/

2 321/

3 322/

4 323/

5 324/

325-326/

1 361/

2 362/

3 363/

4 364/

5 365/

366-367/

1 402/

2 403/

3 404/

4 405/

5 406/

407-408/

1 443/

2 444/

3 445/

4 446/

5 447/

448-449/
Designed by Abt Associates Inc. Not to be used without permission from Abt Associates Inc. SiteID#______________

II. DURATION AND STRUCTURE OF TUTORING SESSION(S)

During the week of:

Monday, May 8th - Friday, May 12th, 2000 ...

Student 1

Name

Student 2

Name

Student 3

Name

Student 4

Name

Student 5

Name

1. Number of hours I was scheduled/expected to work with student.

Indicate total number of hours scheduled, regardless of whether the

student was absent or otherwise unable to participate:

78-81/ 109-112/ 140-143/ 171-174/ 202-205/

2. Number of hours I actually worked with student: 82-85/ 113-116/ 144-147/ 175-178/ 206-209/

 Of the total number of hours I worked with student:

2a. Number of hours I worked with him/her one-on-one: 86-89/ 117-120/ 148-151/ 179-182/ 210-213/

2b. Number of hours I worked with him/her as part of

a small group (2-4 students): 90-93/ 121-124/ 152-155/ 183-186/ 214-217/

 2c. Number of hours I worked with him/her as part of

 a larger group (5 or more students): 94-97/ 125-128/ 156-159/ 187-190/ 218-221/

III. CONTENT OF TUTORING SESSION(S)

For each student, circle Y or N to indicate whether you worked on the following literacy activities with him/her.

During this week (Refer to Dates in Section I.) Student 1 Con’t. Student 2 Con’t. Student 3 Con’t. Student 4 Con’t. Student 5 Con’t.

3. (i) I helped with the mechanics of reading (phonics, decoding)

Y 98/ N Y 129/ N Y 160/ N Y 191/ N Y 222/ N

(ii) I helped with comprehension of text (vocabulary development,

word recognition, discussing meaning of a passage or story) Y 99/ N Y 130/ N Y 161/ N Y 192/ N Y 223/ N

(iii) I read aloud to student

Y 100/ N Y 131/ N Y 162/ N Y 193/ N Y 224/ N

(iv) I listened to student read aloud

Y 101/ N Y 132/ N Y 163/ N Y 194/ N Y 225/ N

(v) I helped with reading of language homework

Y 102/ N Y 133/ N Y 164/ N Y 195/ N Y 226/ N

(vi) I worked with the student on other activities not related to

literacy or reading Y 103/ N Y 134/ N Y 165/ N Y 196/ N Y 227/ N
Designed by Abt Associates Inc. Not to be used without permission from Abt Associates Inc. SiteID#______________

IV. STUDENT INVOLVEMENT

For each student, enter a number (1-4 or 8) that best describes that student during the tutoring week of interest (Refer to dates in Section 1):

1 = Usually/most of the time

2 = Some of the time

3 = A few times/every now and then

4 = Almost never/never

8 = Don't know

During this week (Refer to Dates in Section I.) Student 1 Con’t. Student 2 Con’t. Student 3 Con’t. Student 4 Con’t. Student 5 Con’t.

4. Student and I developed a positive working relationship:

("Positive relationship" includes getting along with me, being

friendly to me, not opposing or arguing with me, cooperating with

me, and going along with my suggestions/activities.) 104/ 135/ 166/ 197/ 228/

5. Student focused on and was actively involved in activities with me. 105/ 136/ 167/ 198/ 229/

6. Student appeared to have (or said that he/she had) learned

something as a result of our work together. 106/ 137/ 168/ 199/ 230/

7. Student seemed motivated/interested in learning. 107/ 138/ 169/ 200/ 231/

8. Student seemed confident about his/her reading/literacy abilities.

108/ 139/ 170/ 201/ 232/

V. STUDENT TUTORING END DATE:

11. When will reading tutoring end for this student? Please enter

date in column.

286-293/

 

-----/-----/-----

327-334/

-----/-----/-----

368-375/

-----/-----/-----

409-416/

-----/-----/-----

450-457/

-----/-----/-----

Thank you for taking the time to complete this questionnaire!
Site ID#:___________________________ 1-3/ Tutor ID#:_______________________4-6/(To be assigned by Abt Associates)

Designed by Abt Associates Inc. Not to be used without permission from Abt Associates Inc.

TUTOR QUESTIONNAIRE I

1.___________________________________________________________________________________

Your First Name Your Last Name

2. Your age: _________7-8/ 3. Are you: (check one) 1. î Female 2. î Male 9/

4. What is the last level of school you completed? (check one)

1. î Some high school 10/

2. î Completed high school (or GED)

3. î Some college (or Associates Degree)

4. î Completed college

5. î Some graduate study

6. î Completed graduate study

At First Administration of Questionnaire:

5. You are: (check one) 11/

1. î An AmeriCorps member

2. î A volunteer recruited by AmeriCorps

 3. î A volunteer not recruited by AmeriCorps (although you may be supervised by AmeriCorps staff)

6. Before you started tutoring this year, did you receive any training from either AmeriCorps or another source in how to tutor

children? (check one)

1. î Yes o Go to question 6a 2. î No o Go to question 7 12/

6a. If you did receive training, in what subject area? (check all that apply)

1. î Reading 13/ 2. î Math/Science 14/ 3. î Other subject area 15/ 4. î General tutoring skills 16/

6b. Estimate the amount of training you have received this school year on how to tutor children

from AmeriCorps and from other sources : (check one in each column)

From AmeriCorps From other sources

1. î 17/ Less than 5 hours 1. î 18/

2. î 6-10 hours 2. î

3. î 11-15 hours 3. î

4. î More than 15 hours 4. î

7. Prior to this school year, had you ever tutored before? (check one) 1. î Yes 2. î No 19/

7a. If you had tutored before, was it in reading? (check one) 1. î Yes 2. î No 20/
Site ID#:___________________________ 1-3/ Tutor ID#:_______________________4-6/(To be assigned by Abt Associates)

Designed by Abt Associates Inc. Not to be used without permission from Abt Associates Inc.

TUTOR QUESTIONNAIRE II

1.___________________________________________________________________________________

Your First Name Your Last Name

2. Your age: _________7-8/ 3. Are you: (check one) 1. î Female 2. î Male 9/

4. What is the last level of school you completed? (check one)

1. î Some high school 10/

2. î Completed high school (or GED)

3. î Some college (or Associates Degree)

4. î Completed college

5. î Some graduate study

7. î Completed graduate study

At Last Administration of Questionnaire:

5. Since you started tutoring this school year, have you received any training on how to tutor?

1. î Yes o Go to question 6 2. î No o Go to the next page 21/

6. Estimate the amount of training you received this school year on how to tutor children from AmeriCorps and from other

sources : (check one in each column)

From AmeriCorps From other sources

1. î 22/ Less than 5 hours 1. î 23/

2. î 6-10 hours 2. î

3. î 11-15 hours 3. î

4. î More than 15 hours 4. î
<PROGRAM NAME>

ID NUMB: #

Program Director Questionnaire

AmeriCorps Tutoring Outcomes Study

The purpose of these questions is to collect updated information about several aspects of AmeriCorps

State/National programs that are participating in the AmeriCorps Tutoring Outcomes Study during the

1999-2000 year. As you are aware, this study is limited to programs like yours that are providing

tutoring in reading, i.e., activities intended to foster the development of children’s reading ability,

provided either directly (e.g., one-to-one instruction) or in group settings.

1. At the current time, approximately, how many students are served at each grade level across all your

program sites for grades 1 through 3?

1. GRADE 1 # of students _____________________ 4-6/

2. GRADE 2 # of students _____________________ 7-9/

3. GRADE 3 # of students _____________________ 10-12/

2. How many members and volunteers tutor children in reading in grades 1 to 3 across all your program

sites?

1. All Sites # of members tutoring in reading _____________________ 13-15/

2. All Sites # of volunteers tutoring in reading ____________________ 16-18/

3. Typically, how many days of tutoring PER WEEK are planned for each student to receive tutoring?

(Circle only one response #.)

1. Each day, 19/

2. Once a week,

3. Twice a week, or

4. Three to four times a week

4. On average, how long is each tutoring session planned to last? (Circle only one response #.)

1. Less than 15 minutes, 20/

2. 15 - 30 minutes,

3. 30 - 60 minutes,

4. 60 - 90 minutes, or

5. More than 90 minutes

5. Do students typically receive tutoring in reading from ...

1. One individual, consistently from session to session, 21/

2. Two individuals consistently from session to session, or

3. More than two individuals across sessions
<PROGRAM NAME>

ID NUMB: #

6. On average, how many children does a member or volunteer work with in a typical session? (Circle

only one response #.)

1. One student 22/

2. Two students

3. Three to four students

4. Five to six students, or

5. Seven or more students

7. Do members receive any materials about and/or training in teaching reading to children?

Reading Materials:

1. YES 2. NO 23/

Reading Training:

1. YES 2. NO 24/

8. Do members receive any materials about and/or training in child development?

Child Development Materials:

1. YES 2. NO 25/

Child Development Training:

1. YES 2. NO 26/

9. Were training sessions conducted before members began provision of literacy services?

1. YES # hours of training _____________________________ 27/

2. NO 28-30/

10. Were sessions conducted during the time in which members are delivering services (e.g., from the

beginning of the academic year to the present)?

1. YES # hours of training _____________________________ 31/

3. NO 32-34/

11. What, if any, instructional models are used? (Circle response # for all that apply.)

1. No formal “brand name” model is used 35-43/

2. Success For All

3. Howard Street Tutoring Model

4. Reading Recovery

5. Reading One-to-One

6. Book Buddies

7. Read*Write*Now

8. Write-to-Read

9. Other (Please specify.) __________________________ 44-45/
<PROGRAM NAME>

ID NUMB: #

12. Is there an effort to coordinate and align reading tutoring services with classroom instruction?

1. Yes !Go to Question 12a. 46/

2. No

12a. If "Yes," then how is this accomplished? (Circle response # for all that apply.)

1. Regular meetings between the tutor and teacher 47-49/

2. Formal reporting procedures, or

3. Other (Please specify.) __________________________________ 50/

13. Typically, do tutors have contact with the reading specialist at the school?

1. Yes ! Go to Question 13a. 51/

2. No

13a. If "Yes," what form does this contact with the reading specialist take? (Circle response # for

all that apply.)

1. Reading Specialist helps plan reading activities for the children’s tutor, 52-55/

2. Reading Specialist provides ongoing monitoring or consultation during the school year,

3. Reading Specialist supplies materials, or

4. Reading Specialist assists with instruction for participating students

14. Do you conduct a formal evaluation to assess the effectiveness of your reading tutoring program?

1. YES 2. NO 56/

15. Please assess the extent to which your reading tutoring program has reached full implementation.

That is, are all the components in place and being conducted as envisioned? Please select the box

that best describes the level of implementation you think your program has reached at the present

time. (Circle only one response #.)

1. Low level of implementation: most or all parts of the program are not yet fully in place, 57/

2. Partial implementation: some components of the tutoring program are in place and

operating as planned,

3. Moderate implementation: most components of the tutoring program are in place and

operating as planned, or

4. Full implementation: all components of the tutoring program are in place and operating as

planned.

Please FAX completed questionnaire to Shana Cook @ (617) 520-2954. Or, mail it back in the

postage-paid envelope provided as soon as possible.

Thank you for your continued support for our AmeriCorps Tutoring Outcomes Study. We look forward to

our spring post-test period with the children in your program.

APPENDIX C

ANALYTIC PROCEDURES AND RESULTS OF EFFECTIVE

PRACTICES AND STUDENT GAINS

AmeriCorps Tutoring Outcomes Study Appendix C 1

APPENDIX C

Procedures and Results of Effective Practices and Student Reading Outcomes

This appendix details the procedures used to examine the effects of program-level tutoring

components on students’ reading achievement using a multi-level regression technique

known as Hierarchical Linear Modeling (HLM).1 Given the data structure of this study,

HLM is preferable to single-level statistical techniques (e.g., Ordinary Least Squares)

because HLM accounts for the bias associated with data structures that are nested (e.g.,

students are nested within programs) and allows the regression equations estimating the

effects of variables at both the student and program levels to be modeled simultaneously.

With the exception of gender, all variables were centered at their grand mean to adjust for

differences among units in the percentage of the given variable. In addition, third grade

observations were used as the reference point, thus the intercept represented the expected

outcome for third graders.2 For all analyses, data were initially fit to a “no covariate” (i.e.,

baseline or null) model for each of the two achievement gain scores (reading comprehension

and reading skill). These models tell us whether gains in the reading scores varied across

programs. Analysis of the variance components revealed that there was variation in scores to

be explained at the program-level. Thus, in all analyses, data were fit to a two-level HLM

model. Level one represented student-level information, including students’ grade and

gender. Level two represented program-level information, such as indicators of effective

practices.

Forty programs had complete data for every program-level predictor. Rather than discard the

remaining 28 schools, we imputed the average program-level score for each missing

characteristic. Programs that were missing information for more than two program-level

characteristics were excluded from analyses. As a result, the final analytic sample included

64 programs. In all analyses involving imputed data, we modeled a dummy variable that

represented whether or not a particular characteristic had been imputed for that program.

This allowed us to determine whether the imputation itself was associated with gains in

reading achievement.

Analyses were performed for each outcome variable (i.e., reading skills gain and reading

comprehension gain). Exhibit C1 presents the regression coefficients and levels of

significance for the models that predict gains in reading skills. Exhibit C2 displays the results

of one multi-level model that examined the relationship between four effective tutoring

practices and gains in reading skills. The reference group in this analysis (i.e., the omitted

variable) represented the programs that employed all four of the effective practices identified

 

1 Bryk, A.S., and Raudenbush, S.W. (1992). Hierarchical Linear Models: Applications and Data Analysis Methods.

London: Sage Publications.

2

To explore the effects of grade further, we conducted two additional sets of analyses, centering data at grades 1 and 2,

respectively. Results of these analyses were similar to those when data were centered at grade 3.
2 Appendix C AmeriCorps Tutoring Outcomes Study.

Exhibit C1

Two-Level Hierarchical Linear Modeling of the Effects of Program Characteristics on Student Gains in Reading Skills

Parameters Model 1 Model 2 Model 3 Model 4 Model 5 Model 6 Model 7 Model 8 Model 9 Model 10

Fixed Effects

Intercept 3.52**** 2.35** 2.37** 2.45** 2.32** 2.37** 2.39** 2.38*** 2.38*** 2.61**

(L1) Grade -1.05** -1.02** -.99** -.99** -.99** -1.02** -1.03** -1.00** -1.02**

(L1) Gender .24 .23 .30 .25 .27 .27 .27 .27 .26

(L2) Whether had a formal

evaluation

2.37**

(L2) Fully implemented 1.91*

(L2) Meet more than twice a

week

2.07*

(L2)Training both prior and

during

2.21*

(L2)Dummy for imputation on

training

 1.74

(L2) No formal program used -1.40

(L2) Dummy for imputation on

formal program

1.50

(L2) Number of hours tutored -.05

(L2) Coordinated program with

instruction

-.50

(L2) Dummy for imputation on

coordination

.57

(L2) Tutor had contact with

reading specialist

.99

(L2) Dummy for imputation on

reading specialist

.52

(L2) Proportion of tutoring time

that is one to one

-.93

(L2) Dummy for imputation on

one to one tutoring

-1.68

* p < .05, ** = p < .01, *** p < .001,   p < .0001.
AmeriCorps Tutoring Outcomes Study Appendix C 3

Exhibit C2

Two-Level Hierarchical Linear Modeling of the Effects of a Combination of Effective

Program Characteristics on Student Gains in Reading Skills

Parameters Model 1

Fixed Effects

Intercept 2.35** 

(L1) Grade -1.00**

(L1) Gender 0.28

(L2) Program had three effective components -3.10*

(L2) Program had two effective components -3.66**

(L2) Program had one effective component -3.67*

(L2)Program had none of the identified effective components -5.54***

* p < .05, ** p < .01, *** p < .001,   = p < .0001.

Note: The reference group is programs that had 4 effective components.

in Exhibit C1. Other multi-level models were fitted to examine whether other combinations

of effective practices differed from each other (e.g., to compare the gain scores of students in

programs with three versus two effective practices). Results of the post-hoc analyses showed

that those in programs with three effective practices showed greater gains than did those in

programs with none of the four effective practices (p =.04).

Exhibit C3 displays coefficients and significance levels for the multi-level models predicting

students’ gains in reading comprehension.
4 Appendix C AmeriCorps Tutoring Outcomes Study.

Exhibit C3

Two-Level Hierarchical Linear Modeling of the Effects of Program Characteristics on Student Gains in Reading Comprehension

Parameters Model 1 Model 2 Model 3 Model 4 Model 5 Model 6 Model 7 Model 8 Model 9 Model 10

Fixed Effects

Intercept 3.77** ** 1.73* 1.74* 1.74* 1.72* 1.71* 1.74* 1.75* 1.78* 2.23**

(L1) Grade -1.23** -1.22** -1.23** -1.21** -1.21** -1.24** -1.27*** -1.22*** -1.22**

(L1) Gender 1.70*** 1.69*** 1.70*** 1.70*** 1.71*** 1.70*** 1.73*** 1.70** 1.71***

(L2) Whether had a formal

evaluation

2.10

(L2) Fully implemented 1.18

(L2) Meet more than twice a

week

-0.27

(L2) Training both prior and

during

2.41

(L2) Dummy for imputation on

training

0.43

(L2) No formal program used -1.01

(L2) Dummy for imputation on

formal program

1.19

(L2) Number of hours tutored -2.33

(L2) Coordinated program with

instruction

-3.71*

(L2) Dummy for imputation on

coordination

-2.09

(L2) Tutor had contact with

reading specialist

.80

(L2) Dummy for imputation on

reading specialist

-2.24

(L2) Proportion of tutoring time

that is one-to-one

-2.45

(L2) Dummy for imputation on

proportion of one-to-one

tutoring

-3.47*

* p < .05, ** = p < .01, *** p < 001, **** p < .0001.