###### 

###### 

###### 

###### 

###### 

###### 

###### 

###### 

###### 

###### 

**21st Century Community Learning Centers Descriptive Study of Program Practices**

****

**U.S. Department of Education**

**Office of Planning, Evaluation and Policy Development**

**Policy and Program Studies Service**

**Prepared by:**

**William R. Penuel**

**Raymond McGhee, Jr.**

**SRI International**

**Menlo Park, Calif.**

2010

This report was prepared for the U.S. Department of Education under Contract Number ED-01-CO-0059/0005 by SRI International. Beth Yeh served as the contracting officer’s representative. The views expressed herein do not necessarily represent the positions or policies of the U.S. Department of Education. No official endorsement by the U.S. Department of Education is intended or should be inferred.

**U.S. Department of Education**

Arne Duncan

*Secretary*

**Office of Planning, Evaluation and Policy Development**

Carmel Martin

*Assistant Secretary*

**Policy and Program Studies Service**

Alan Ginsburg

*Director*

July 2010

This report is in the public domain. Authorization to reproduce it in whole or in part is granted. While permission to reprint this publication is not necessary, the suggested citation is: U.S. Department of Education, Office of Planning, Evaluation and Policy Development, Policy and Program Studies Service, *21st Century Community Learning Centers Descriptive Study of Program Practices*, Washington, D.C., 2010.

To order copies of this report, write to

ED Pubs

Education Publications Center

U.S. Department of Education

P.O. Box 22207

Alexandria, VA 22304

Or fax your request to 703-605-6794.

Or call in your request toll-free to1-877-433-7827 (1-877-4-ED-PUBS). Those who use a telecommunications device for the deaf (TDD) or a teletypewriter (TTY) should call 1-877-576-7734. If 877 service is not yet available in your area, call 1-800-872-5327 (1-800-USA-LEARN).

Or order online at http://edpubs.gov.

On request, this publication is available in alternate formats, such as Braille, large print, or computer diskette. For more information, please contact the U.S. Department of Education’s Alternate Format Center at 202-260-0852 or 202-260-0818.

**Contents**

**Acknowledgments		vii**

**Executive Summary		ix**

Study Background and Overview		ix

Key Findings		x

Nature and Quality of Reading and Mathematics Activities		x

Student Attendance in Center Activities		xiii

Organizational Supports for Instructional Quality		xv

Conclusions		xvii

**1.	Study Background, Goals and Methods		1**

The 21st Century Community Learning Centers Program		2

Background		4

Conceptual Framework for the Study		5

Study Design and Data Collection Methods		7

Evaluation Questions		7

Study Design		8

Data Sources and Data Collection Procedures		8

Weighting		10

Limitations of the Study		10

Presentation of Survey Data in the Report		12

**2.	Reading and Mathematics Activities in Centers Serving Elementary School Students		13**

Key Findings		13

Exposure to Academics After School		14

Reading Activities		15

Reading Skills Emphasized		15

Reading Skills Emphasized in Observed Activities		16

Use of Instructional Strategies for Teaching Vocabulary, Fluency and Comprehension		17

Mathematics Activities 		20

Mathematics Skills Emphasized		20

Mathematics Content Emphasized in Observed Activities		21

Use of Instructional Strategies for Teaching Mathematics		22

Mathematics Instructional Strategies in Observed Activities		24

Instructor Education and Experience		26

Instructor Guidance in Observed Activities		27

Feedback to Students and Parents on Student Progress		28

Summary		29

**3.	Student Targeting and Attendance		31**

Key Findings		31

Priorities Regarding Targeting		32

Policies Regarding Attendance		33

Regular Attendance		34

Student Engagement in Observed Activities		39

Summary		39

**4.	Organizational Supports for Instructional Quality		41**

Key Findings		41

Center Staffing		41

Recruiting and Retaining High-Quality Staff		41

Professional Development		45

Alignment to Student Needs		47

Coordination With the School Day		47

Use of Student Assessment Data 		48

Adult Support: Fostering Positive Relationships Between Adults and Students		52

Coordination with Supplemental Educational Services		53

Summary		53

**5.	Conclusions and Implications for Future Research and Evaluation		55**

Nature and Quality of Reading and Mathematics Activities 		55

Attendance and Student Participation		55

Organizational Supports for Instructional Quality		55

Key Differences Between Different Types of Centers		56

Recommendations for Future Research and Evaluation		56

**References		59**

**Appendices**

A: Descriptions of Academic Instruction for Centers Serving All Students		65

B: Case Study Observations		79

C: Sampling Approach and Weighting of Surveys		85

D: Survey Instruments		95

E: Supplementary Tables for the Surveys		131

F: 21st CCLC Quality Study—Quality Observation Instrument		151

**Exhibits**

Exhibit ES-1:	Percentage of Centers Serving Elementary School Students That Reported Participation in Activities by a Typical Student, by Type of Activity and Amount of Time of Engagement per Week		xii

Exhibit 1-1:	Characteristics and Type of and Grade Levels Served by Funded and Sampled Centers		4

Exhibit 1-2:	Framework for the Study: Supports for Student Participation in Quality Academic Instruction That Leads to Increased Achievement		6

Exhibit 2-1:	Percentage of Centers Serving Elementary School Students That Reported Participation in Activities by a Typical Student, by Type of Activity and Amount of Time of Engagement per Week		15

Exhibit 2-2:	Percentage of Centers Serving Elementary School Students That Reported Reading Activities Emphases, by Reading Skill Emphasized and Inclusion Level		16

Exhibit 2-3:	Percentage of Observations of Reading Activities in Centers Serving Elementary School Students, by Reading Skill Emphasized		17

Exhibit 2-4:	Percentage of Centers Serving Elementary School Students That Reported Student Participation in Instructional Strategies in Reading Activities, by Strategy and Inclusion Level		18

Exhibit 2-5:	Percentage of Observations of Reading Activities in Centers Serving Elementary School Students, by Instructional Strategy Emphasized		19

Exhibit 2-6:	Percentage of Centers Serving Elementary School Students That Reported Emphases in Math Activities, by Type of Content Emphasized and Inclusion Level		21

Exhibit 2-7:	Percentage of Observations of Mathematics Activities in Centers Serving Elementary School Students That Emphasized Skills, by Type of Math Skill Emphasized		22

Exhibit 2-8:	Percentage of Centers Serving Elementary School Students That Reported Student Participation in Instructional Strategies in Math Activities, by Strategy and Inclusion Level		24

Exhibit 2-9:	Percentage of Observations of Mathematics Activities in Centers Serving Elementary School Students That Emphasized Instructional Strategies, by Strategy Emphasized		25

Exhibit 2-10:	Percentage of Observations in Reading and Mathematics Activities in Centers Serving Elementary School Students That Emphasized Use of Active Teaching Strategies, by Strategy Emphasized and Activity Type		27

Exhibit 2-11:	Percentage of Centers Serving Elementary School Students That Provided Feedback About Their Progress, by Type of Feedback and Center Type		28

Exhibit 3-1:	Percentage of Centers That Reported Top-Priority Target Groups, by Target Groups and Center Type		33

Exhibit 3-2:	Percentage of Centers That Reported Attendance Requirements, by Grade Level and Attendance Requirement		34

Exhibit 3-3:	Percentage of Centers That Reported That They Required Participation of All Students in Specific Activities When Students Were Present, by Type of Activity		35

Exhibit 3-4:	Percentage of Centers That Reported the Number of Days of Attendance in 2005–06, by Grade Level and Days of Attendance		37

Exhibit 3-5:	Percentage of Centers That Reported Consequences for Student Absences, by Consequence Type ......................................................		38

Exhibit 4-1:	Percentage of Centers That Paid Hourly Rates to Staff Who Led Instructional Activities, by Type of Staff and Rates		42

Exhibit 4-2:	Percentage of Staff Who Led Instructional Activities at Centers, by Hours Worked per Week		43

Exhibit 4-3:	Percentage of Centers That Reported Offering Benefits to Full- and Part-Time Staff Who Led Instructional Activities, by Benefit Offered and Staff Status		44

Exhibit 4-4:	Percentage of Centers That Reported Reasons for Staff Departures, by Reason for Departure and Center Type		45

Exhibit 4-5:	Percentage of Centers That Reported That They Offered Paid Professional Development Opportunities for Full- and Part-Time Staff Who Led Instructional Activities, by Staff Status and Type of Development		46

Exhibit 4-6:	Percentage of Centers That Reported That State Assessments Data Was Available for Use in Tracking Student Outcomes, by Subject Tested and Center Type		48

Exhibit 4-7:	Percentage of Centers That Reported Student Assessment Sources, by Type and Frequency of Assessment		49

Exhibit 4-8:	Percentage of Centers That Reported Uses of Student Assessment Data, by Type of Use		50

Exhibit 4-9:	Percentage of Centers That Reported Challenges in Implementing High-Quality Programming, by Type of Challenge and Center Type		51

Exhibit 4-10:	Percentage of Observations in Reading and Math Activities That Demonstrated Student-Staff Relationship-Building Efforts, by Relationship-Building Effort and Type of Activity 		52

Appendix A: Descriptions of Academic Instruction for Centers Serving All Students

Exhibit A-1:	Percentage of Centers That Reported Participation in Activities by a Typical Student, by Type of Activity and Hours of Engagement per Week 	 	68

Exhibit A-2:	Percentage of Centers Serving Middle School Students That Reported Reading Activities Emphases, by Reading Skill Emphasized and Inclusion Level		69

Exhibit A-3:	Percentage of Centers Serving High School Students That Reported Reading Activities Emphases, by Reading Skill Emphasized and Inclusion Level		70

Exhibit A-4:	Percentage of Centers That Reported Student Participation in Instructional Strategies in Reading Activities, by Strategy and Inclusion Level		71

Exhibit A-5:	Percentage of Centers Serving Middle School Students That Reported Emphases in Math Activities, by Type of Content Emphasized and Inclusion Level		72

Exhibit A-6:	Percentage of Centers Serving High School Students That Reported Emphases in Math Activities, by Type of Content Emphasized and Inclusion Level		73

Exhibit A-7:	Percentage of Centers That Reported Student Participation in Instructional Strategies in Math Activities, by Strategy and Inclusion Level		75

Exhibit A-8:	Percentage of Centers That Provided Feedback About Their Progress, by Type of Feedback and Center Type		76

Appendix B: Case Study Observations

Exhibit B-1:	Ratings of Observations in Enrichment Activities		81

Exhibit B-2:	Ratings of Observations in Reading Activities		82

Exhibit B-3:	Ratings of Observations in Mathematics Activities		83

Appendix E: Supplementary Tables for the Surveys

Exhibit E-1:	Centers Serving Elementary School Students That Reported Participation in Activities by a Typical Student, by Type of Activity and Amount of Time of Engagement per Week		134

Exhibit E-2:	Centers Serving Elementary School Students That Reported Reading Activities Emphases, by Reading Skill Emphasized and Inclusion Level		135

Exhibit E-3:	Centers Serving Elementary School Students That Reported Student Participation in Instructional Strategies in Reading Activities, by Strategy and Inclusion Level		136

Exhibit E-4:	Centers Serving Elementary School Students That Reported Emphases in Math Activities, by Type of Content Emphasized and Inclusion Level		137

Exhibit E-5:	Centers Serving Elementary School Students That Reported Student Participation in Instructional Strategies in Math Activities, by Strategy and Inclusion Level		138

Exhibit E-6: 	Centers Serving Elementary School Students That Provided Feedback About Their Progress, by Type of Feedback and Center Type and		139

Exhibit E-7: 	Centers That Reported Top-Priority Target Groups, by Target Groups and Center Type		140

Exhibit E-8:	Centers That Reported Attendance Requirements, by Grade Level and Attendance Requirement		141

Exhibit E-9: 	Centers That Reported That They Required Participation of All Students in Specific Activities When Students Were Present, by Type of Activity		142

Exhibit E-10:	Centers That Reported the Number of Days of Attendance in 2005–06, by Grade Level and Days of Attendance		142

Exhibit E-11:	Centers That Reported Consequences for Student Absences, by Consequence Type......................................................		143

Exhibit E-12:	Centers That Paid Hourly Rates to Staff Who Led Instructional Activities, by Type of Staff and Rates		144

Exhibit E-13: 	Staff Who Led Instructional Activities at Centers, by Hours Worked per Week		145

Exhibit E-14:	Centers That Reported Offering Benefits to Full- and Part-Time Staff Who Led Instructional Activities, by Benefit Offered and Staff Status		145

Exhibit E-15:	Centers That Reported Reasons for Staff Departures, by Reason for Departure and Center Type		146

Exhibit E-16:	Centers That Reported That They Offered Paid Professional Development Opportunities for Full- and Part-Time Staff Who Led Instructional Activities, by Staff Status and Type of Development		147

Exhibit E-17:	Centers That Reported That State Assessments Data Was Available for Use in Tracking Student Outcomes, by Subject Tested and Center Type		147

Exhibit E-18:	Centers That Reported Student Assessment Sources, by Type and Frequency of Assessment		148

Exhibit E-19:	Centers That Reported Uses of Student Assessment Data, by Type of Use		149

Exhibit E-20:	Centers That Reported Challenges in Implementing High-Quality Programming, by Type of Challenge and Center Type		150

Contents	vi

**Acknowledgments**

The authors wish to thank the many individuals who assisted with this study. 

SRI International: Nancy Adelman, Sarah Bardack, Nancy Kamprath Bier, Sara Carriere, Reina Fujii, Larry Gallagher, Bowyee Gong, Bonnee Groover, Catherine Hall, Meredith Ittner, Karla Jones, Christine Korbak, Carlin Llorente, Bladimir Lopez-Prado, Nicolette Mayes, Samantha Murray, Natalie Nielsen, Ron Orpitelli, Melia Repko, Angeline Reyes, Patrick Shields, Michael Smith and Stephanie Stone.

Policy Studies Associates: Kolajo Afolabi, Monica Mielke, Ellen Pechman, Christina Russell, Dwayne Smith and Mark Wilson.

U.S. Department of Education: Adriana de Kanter, Alan Ginsburg, David Goodwin, Daphne Kaplan, Jessica Ramakis, Margery Yeager and Beth Yeh. 

Technical Work Group and Consultants: David Fleming, Andria Fletcher, Denise Huang, Priscilla Little, Neil Naftzger, Mary Parella, Marguerite Peebles and Karyl Resnick.

And finally, our deep appreciation to the center directors and center staff who participated in the study’s data collection efforts. 

Contents	viii

Executive Summary

Study Background and Overview

The 21st Century Community Learning Centers (21st CCLC) program is a federal program authorized in 1994 under Title IV, Part B, of the *Elementary and Secondary Education Act of 1965* (*ESEA*), as amended. The program was originally intended to provide funding to school districts to support continuing education and lifelong learning opportunities to children and adults to help keep the country’s workforce competitive for the 21st century. The program’s authorizing statute was amended by the reauthorization of *ESEA* in 2002 to provide before- and after-school and summer academic enrichment opportunities for children, particularly children attending low-performing public schools, and to eligible private school students. The goal of the program is to help students meet local and state academic standards in core subjects, such as reading and mathematics. In addition to adding to academic content, programs may also provide youth development activities, drug and violence prevention, technology education, art and music activities, character education, counseling and recreation. The federal program is administered by the United States Department of Education, which awards grants to states by formula. States, in turn, award subgrants to eligible entities on a competitive basis. Eligible entities include education agencies, community-based organizations and other entities operating in either school or community settings.

The previous national evaluation of the 21st Century Community Learning Centers program (Dynarski et al., 2003; James-Burdumy et al., 2005), which examined centers funded under the pre-2002 legislative authority, found that the centers did not focus on academic achievement and had no effects on participants’ academic outcomes. These findings raised questions about the level of program quality in after-school programs. The requirements changed to focus more on academics, so a study conducted today of program impacts might not find the same results.

In 2004, the U.S. Department of Education’s Policy and Program Studies Service contracted with SRI International and its partner, Policy Studies Associates, to undertake an evaluation of the 21st Century Community Learning Centers The following evaluation questions informed this study: 

1. What is the nature of activities in centers that are designed to promote the academic development of students?

2. How do centers vary with respect to regular attendance?

3. How do center leaders staff their centers, coordinate with other service providers and use data to improve programming? 

The sources of data for the study were surveys and site visits. A sample of 516 center directors intended to be nationally representative of centers offering academic activities completed a written survey in the 2006–07 school year. A subsample of administrators and program staff members from 122 centers completed a more in-depth telephone survey on attendance and staff characteristics in the same school year. Site visits in fall 2006 and spring 2007 provided data on the nature of instruction to compare with the survey data; the visits also provided observational data on instruction and student participation. The study team interviewed program staff members and observed after-school programming at 12 sites (11 served elementary school students; one served middle school students). The site visit data are not nationally representative and observation protocols used have not been related to any outcomes of importance to the program. The study also used grantee- and center-level data collected by the 21st CCLC program office at the U.S. Department of Education, through the Profile and Performance Information Collection System (PPICS) database, to identify the basic center characteristics nationwide and to construct the survey and case study samples. 

The study provides descriptive information on the 21st CCLC program; it does not provide information on program outcomes or impacts. In addition, the study’s original sampling strategy and its reliance on self-reported data from surveys limit the generalizability of the findings and provide no basis for making causal inferences. To ensure that all centers surveyed could respond to questions about academics, the sampling plan limited the sample to centers that were funded at the time of the study, and that offered instruction in reading, mathematics and technology; thus, the sample was not nationally representative of all 5,122 centers funded at the time of the study. To address this limitation, the data were poststratified to reflect the full population of centers. Respondents’ self-reported answers to survey questions may reflect unreliable memory of past events and may include responses on academic instruction that center staff considered socially desirable. This report presents findings on academic instruction only for centers serving elementary school students in which the study team collected observation data. Findings on academic instruction for centers serving middle and high school students are reported in Appendix A.

Key Findings 

This report on the 21st Century Community Learning Centers program focuses on the implementation of reading and mathematics activities, student attendance and hiring and retaining qualified staff in centers from which data were obtained. 

Nature and Quality of Reading and Mathematics Activities

The program statute requires that centers focus on academics and use research-based strategies for instruction. The law requires that students participate in academic activities at a frequency that is “sufficient to influence their learning.”

**According to grantee performance data, nearly all centers funded offered reading[footnoteRef:1] and mathematics activities.** [1:  Reading enrichment activities are defined as structured activities designed to build students’ literacy skills. Reading enrichment may include scheduled time for independent student reading, writing and literacy enrichment activities but not homework assistance. However, homework assistance activities could include reading activities.] 

Ninety-eight percent of all centers funded as of the 2006–07 school year (the time of the study) reported that they offered activities in reading, and 94 percent of all centers offered activities in mathematics. Whether students are required to participate in these activities, however, varies by center.

**Three-quarters of the centers reported that a typical student participated in reading activities (75 percent) and mathematics activities (81 percent) for less than 4 hours per week.**

Centers serving elementary school students reported that the average student spent the most amount of time per week doing homework in a group setting (36 percent) or participating in arts or recreation activities (33 percent) (Exhibit ES-1). One-quarter of centers reported that a typical student received instruction in reading or practiced reading skills, and 19 percent of centers indicated that a typical student engaged in mathematics activities for more than 4 hours per week. Thirty-six percent of centers reported that a typical student worked on homework in a group setting (which could also include reading and mathematics activities), and 33 percent said that the typical student was involved in arts/recreation activities for more than 4 hours per week. Because centers were open for about 16 hours per week, on average, student participation for 4 hours per week in a particular activity represented 25 percent of the available time.

Exhibit ES-1Percentage of Centers Serving Elementary School Students That Reported Participation in Activities by a Typical Student, by Type of Activity and Amount of Time of Engagement per Week

Note: Percentages may not equal 100 percent due to rounding.

Source: National Survey, Item 5.

*n*= 389

Exhibit reads: Thirty-six percent of centers serving elementary school students reported that a typical student participated in homework activities for more than 4 hours per week. 

**A majority of centers serving elementary school students reported that reading activities included the five essential components of reading instruction.**

Many centers serving elementary school students reported emphasizing comprehension, fluency, vocabulary, phonics and phonemic awareness in at least some activities. Seventy-four percent of centers serving elementary school students reported that they focused on comprehension in all or most instructional activities in reading, compared with 52 percent that concentrated on phonics skills in all or most activities. Observational data were consistent with the pattern reported by centers serving elementary school students: 86 percent of observed activities focused on comprehension, compared with 46 percent that focused on phonics. However, the observation data were collected from a small number of purposively sampled sites.[footnoteRef:2] Thus the comparison may not be useful. [2:  It is important to note that the study team asked center directors to obtain these data from someone familiar with the activities, but they may have responded without consulting an expert. Additionally, center directors may have had reasons to provide socially desirable responses to a U.S. Department of Education (ED) survey. ] 

**Centers reported that mathematics activities focused on basic skills.**

Seventy-one percent of centers surveyed serving elementary school students reported stressing operations with whole numbers in all or most mathematics activities. Centers serving elementary school students also reported that they were more likely to engage students in tasks that required simple rather than complex problem solving. Sixty-eight percent of centers reported asking students to practice basic facts in all or most instructional activities in mathematics. Observational data from a small, purposive sample of case study sites were consistent with the pattern of emphasizing basic mathematics facts: 83 percent of observed activities involved practice with basic facts. 

**Observers in case study sites found that staff providing instruction used active teaching strategies in academic activities, communicating goals clearly to students in most activities.**

However, staff were more likely to use multiple teaching strategies in reading than in mathematics. Staff providing instruction communicated the goals, purposes and expectations of activities to students more often in mathematics activities (89 percent) than in reading activities (60 percent). Staff providing instruction used multiple strategies in 22 percent of the mathematics activities observed, compared with 53 percent of the reading activities observed.

Student Attendance in Center Activities

Researchers have linked regular participation to better outcomes for students in after-school programs (e.g., Lauer et al., 2006). Although ED’s annual performance reporting guidelines define regular attendance as 30 days or more per year, the number of days required to have an effect on academic achievement is not known. 

**Centers reported that about half of their students attended roughly 2 days a week or more.**

The study team asked a random subset of 140 centers in the study to report on student attendance and participation. Just 75 of the 119 centers (63 percent) that completed surveys indicated they could track these data. The centers that could track attendance indicated that 44 percent of all center students attended 60 days (roughly 2 days per week) or more in the last year.

**More than half of all centers reported having policies that required students to attend at least 2 days a week, but attendance policies were not associated with greater attendance.**

Centers’ attendance policies varied. About half of all centers (56 percent) had policies requiring student attendance at least 2 to 3 days per week in order to remain enrolled in the after-school program.[footnoteRef:3] Thirty-eight percent of centers reported requiring attendance daily. Twenty-six percent did not require attendance. There were no significant differences in attendance requirements between school-based and nonschool-based centers. A higher percentage of centers serving elementary school students (41 percent) were more likely to have policies requiring attendance than those centers not serving this age group. In contrast, 31 percent of centers serving middle school students and 22 percent of centers serving high school students required attendance every day. Beyond basic attendance requirements, more than half of the centers required the participation of all students in specific academic activities, including homework help (73 percent), reading activities (60 percent), mathematics activities (58 percent) and tutoring (14 percent).  [3:  Although requiring attendance was not defined on the survey, respondents may have varied in their interpretations of this question, depending upon the consequences attached to nonattendance; therefore, data are included in a separate question about consequences of nonattendance. “Require attendance” may mean that center directors have a rule that only students who attend five days per week may enroll in the program; in other instances, requiring attendance could simply mean to center directors that there is a norm that students will attend regularly. ] 

In centers that required attendance, 66 percent of students attended for more than 30 days; 62 percent of students attended for the same duration in centers that did not require attendance. No relationship was found between any kind of attendance policy and actual attendance for centers overall****or for centers serving elementary school students. For the middle grades, centers that required attendance every day had higher attendance than those that did not require attendance. In high school, centers that required attendance 2 to 3 days per week had higher attendance than those that did not require attendance. 

**Although attendance rates varied little by center type, elementary school students were more likely to attend center programming than older students.**

The pattern of higher attendance for elementary-serving centers than for secondary-serving centers mirrors that of the previous national evaluation. 

Centers that served elementary school students and had adequate tracking systems reported that 48 percent of students attended 60 days or more in the 2005–06 school year, or roughly 2 days per week. Centers serving the middle grades indicated that 36 percent of students attended this often, and centers serving high school students cited 30 percent of their students’ meeting this attendance level. Researchers conducting the previous national evaluation found a similar pattern of results for elementary school students but not for middle school students: In their study, 55 percent of elementary students attended 51 or more days, but just 20 percent of middle school students attended that often (Dynarski et al., 2004). Their study used different methods to study attendance, however.

Organizational Supports for Instructional Quality 

Offering high-quality instruction in reading and mathematics requires recruiting, developing and retaining high-quality staff, as well as developing policies and programs that attract and retain students and reflect students’ academic needs.

**Centers reported that they relied primarily on part-time staff, who were unlikely to receive benefits from their work in the center. This latter finding is not surprising, as the centers are open for an average of only 16 hours per week.**

Seventy-six percent of program staff members in centers who led instructional activities reported working fewer than 20 hours per week. Centers infrequently offered job benefits for part-time staff members. Fourteen percent of centers reported offering a retirement savings plan to part-time staff, 11 percent offered paid time off for vacation and sick leave, 8 percent offered health insurance and 4 percent offered tuition reimbursement. It is important to note that part-time staff may include individuals who have full-time teaching positions in addition to working at the 21st CCLC center. 

**To provide professional development to staff, about half of centers reported offering opportunities through training courses or conferences.**

Centers indicated that the school-day teachers they employed may have had increased opportunities for staff development through activities offered by their districts and schools than did other types of center employees. Sixty-two percent of centers offered other paid training or professional development to full-time staff, and 55 percent did so for part-time staff. Fifty-three percent of centers offered paid conference attendance to full-time staff; 39 percent offered this opportunity to part-time staff. In the case studies, center directors in school-based programs said they relied on professional development opportunities the staff received through the district. They also reported that they encouraged staff to share their ideas for innovative programming through weekly staff meetings that served as school-based professional development opportunities. 

**Across all centers, 29 percent of staff had worked at the center for less than 1 year.**

About half of centers (48 percent) reported that the primary reason for staff turnover was graduation from school or completion of a program of study. Other commonly reported reasons for staff turnover were lack of benefits and the centers’ inability to offer full-time positions.

**Centers reported that more than two-thirds of staff providing instruction in reading and mathematics had prior experience as certified classroom teachers or as instructional specialists in reading or mathematics.**

Fifty-five percent of the staff who provided instruction in reading or mathematics had been or were, at the time of the study, regular classroom teachers, and 23 percent had been instructional specialists in reading or mathematics. Twenty-three percent of staff providing instruction in reading or mathematics were currently or had served as classroom aides. 

**Centers reported that nearly two-thirds of reading and mathematics instructors had a bachelor’s degree or higher (64 percent for reading and 63 percent for mathematics).**

Centers reported that a little more than a third of their instructors reported having only a bachelor’s degree (32 percent for reading and 33 percent for mathematics). 

**A majority of centers reported using assessment data to improve existing program offerings and evaluate program success.**

Seventy-one percent of centers reported having access to whole-school state assessment data. In addition, more than four-fifths reported that they received state assessment results for individual students at their centers (83 percent in reading and 82 percent in mathematics). Centers said that they used a variety of data to assess student academic growth, make program adjustments or evaluate program success periodically. Almost half (47 percent) of centers noted that once or twice a year they used results from tests administered at the students’ school, while 34 percent of centers said that written reports from students’ teachers were used to assess academic growth once or twice annually. 

**About 40 percent of centers reported some involvement with supplemental educational services (SES). Like 21st Century Community Learning Centers, SES is intended to provide after-school academic activities to students. A small percentage (9 percent) of centers said they coordinated their activities with SES providers. Fifteen percent of centers reported being authorized to provide SES themselves.**

On average, each center that was an SES provider reported offering supplemental instructional services to 38 students in reading and mathematics. Just 9 percent of all centers reported coordinating with one or two providers, and only 5 percent of all centers reported that their coordination activities focused on aligning schedules with the providers, while 4 percent indicated they coordinated their academic support activities with the providers. 

**Just under one-third of all centers reported that coordination with staff from the school-day instructional program was a challenge to implementing high-quality programming. However, the percentage was higher for nonschool-based centers.**

Lack of information about students’ academic needs, school-day teachers’ lack of responsiveness to requests from after-school staff for information and lack of information about the school-day curriculum were cited as barriers to implementing high-quality programming for 22 percent to 32 percent of all centers. Barriers to obtaining information about student needs were greater for nonschool-based centers than for school-based centers. Thirty-six percent of nonschool-based centers reported lack of information as a barrier, compared to 20 percent of school-based centers. Barriers were also greater for nonschool-based centers with respect to responsiveness of school staff: 39 percent of nonschool-based centers identified lack of responsiveness as a barrier, compared to 29 percent of school-based centers. 

Conclusions 

Survey data and site observations indicate that 21st Century Community Learning Centers focused on reading and mathematics enrichment. Compared with the breadth of reading skills emphasized, mathematics instruction covered a narrower range of basic skills. Centers reported that 44 percent of students attended 60 days or more in the last year, which amounts to roughly 2 days per week. Elementary school students attended after-school activities for more days in the school year than did middle and high school students. About half of the centers reported using data for a variety of purposes, including program evaluation and ongoing review of programming activities. The majority of centers reported having access to state assessment data results on individual students and many reported using this and other information to inform program practice.

Although there were few differences between school-based and nonschool-based centers, school-based centers were more likely to report emphasizing higher-order skills such as asking students to make predictions about something they were reading and talking or writing about answers to questions related to something they had read. At the same time, nonschool-based centers had students read teacher-selected books more often than school-based centers did.

The findings of this report suggest three challenges that centers face in implementing their programs: (a) staff departures after graduating from school or completing a program of study, as the lack of benefits makes it difficult to retain high-quality staff; (b) a lack of up-to-date information about students’ individual needs and (c) low attendance rates, the remedy for which requires more than simply having attendance policies. 

The study did not directly measure instructors’ knowledge or skills, but future studies could examine detailed measures of instructor knowledge to predict differences in instructional quality. Future studies could also examine the content of professional development provided for center staff. Finally, future work that examines the quality of reading programming could focus on how best to capture information on instruction in after-school programs since this study had challenges in this area. Future studies could also identify effective practices for improving attendance levels among participants.

Executive Summary	xi

1. Study Background, Goals and Methods 

Since the reauthorization of the *Elementary and Secondary Education Act* (*ESEA*) of 2002, a stronger emphasis has been placed on out-of-school learning to improve student academic achievement. As policy makers, schools and districts contemplate possible interventions, one option is to increase the amount of time spent on learning, such as through before- and after-school programs. Additional hours provide more opportunities for instruction, academic assistance and enrichment or experiential activities to support learning. 

The body of rigorous research on the effects of extended learning time on student academic achievement is not extensive, but it suggests the potential for a positive effect on academic performance (e.g., Lauer et al., 2006; Zimmer et al., 2007). How the additional time is used is what matters. For example, a review of the literature on time and learning found that there was no relationship between allocated time and student academic achievement, some relationship between engaged time and achievement and a greater relationship between time spent on academic learning and achievement (Aronson, Zimmerman, and Carlos, 1999). Large-scale survey studies find a similar relationship between time spent on academics in specific content areas and academic achievement in those areas (Rowan, Correnti, and Miller, 2002).

The 21st Century Community Learning Centers (21st CCLC) program is an example of a federal effort to improve academic achievement by extending learning opportunities. The program, which is funded by the U.S. Department of Education and administered by the states, seeks to provide opportunities for academic enrichment, youth development and other activities for students before and after school, as well as during the summer, and to offer the families of the students served opportunities for literacy and related educational development. Past program evaluations have generated questions about the impact of federally funded after-school services on student academic achievement. In the previous 2-year national evaluation study of the 21st CCLC program, which was conducted early in the program’s history, researchers found that attendance was irregular in many centers. In addition, the evaluators reported that academic opportunities in core subject areas such as reading and mathematics were limited and that participation in the program had no effect on student academic achievement (Dynarski et al., 2003; Dynarski et al., 2004; James-Burdumy et al., 2005).

The current study examined characteristics of programming, building on the findings from the earlier evaluation. In 2004, the U.S. Department of Education’s Policy and Program Studies Service contracted with SRI International and its partner, Policy Studies Associates, to evaluate the programming that the 21st CCLC program offers students and their families. One purpose of the evaluation was to provide evidence regarding how 21st CCLC program grantees, center directors and center staff implemented activities—in particular, academic activities focused on reading and mathematics. In addition, the study examined organizational supports that previous research suggests may be associated with high-quality programs as indicated, for example, by the hiring and retention of staff with experience and expertise relevant to their roles in the centers. This report presents the findings of the evaluation. 

The 21st Century Community Learning Centers Program

The *Elementary and Secondary Education Act* of 1965 (*ESEA*), as amended by the Improving America’s Schools Act of 1994 (P.L. 103-382), initially authorized the 21st CCLC program as a small discretionary grant program. In the authorizing legislation, Congress described local public schools—particularly those in rural areas and inner cities—as valued institutions to which communities already turned for educational and other essential services; thus, funding would go to school districts to develop centers. Centers were to provide continuing education and lifelong learning opportunities “to individuals of all ages” and were to help keep the country’s workforce “competitive and successful ... [in] the high technology, global economy of the 21st century.” Each center was required to offer no fewer than 4 of 13 authorized activities, which included: services for school-age students; senior citizen programs; children’s day care services; and employment counseling, training and placement for adults. In 1998, an absolute priority was added for “activities that offer expanded learning opportunities for children and youth in the community.” 

The 21st CCLC program underwent a significant transformation when Congress reauthorized it in January 2002 as Title IV, Part B, of the *ESEA* (P.L. 107-110). The reauthorization emphasized improving academic achievement for low-performing schools with high percentages of low-income students. 

The reauthorization also established a new set of authorized activities that centers could offer:

Remedial education activities and academic enrichment learning programs, including providing additional assistance to students to improve their academic achievement

Mathematics and science education activities

Arts and music education activities

Entrepreneurial education programs

Tutoring services (including those provided by senior citizen volunteers) and mentoring programs

Activities that emphasized language skills and academic achievement for students with limited English proficiency

Recreational activities

Telecommunications and technology education programs

Expanded library service hours

Programs that promoted parental involvement and family literacy

Programs that provided assistance to students who had been truant, suspended or expelled, to allow them to improve their academic achievement

Drug- and violence-prevention programs, counseling programs and character education programs

As significant as these changes to the goals and authorized activities were, changes relating to program administration following reauthorization were at least as important. Congress transferred the responsibility for administering the program from the U.S. Department of Education to each state when funding rose to $1 billion in fiscal year (FY) 2002. The U.S. Department of Education currently awards grants to states using a formula based on a state’s allocation under Title I, Part A, of the *ESEA*, which takes into account the state’s size and the proportion of students from low-income families. Local education agencies and a variety of community-based organizations, including for-profit businesses, nonschool-based organizations and faith-based organizations, are eligible to apply for subgrants from the state. Each state is required to submit an application for implementing the program, and the U.S. Department of Education is responsible for approving those applications. States then operate their own grant competitions, according to their approved applications, in which they determine the duration of program grants (between 3 and 5 years) and specify their own requirements for matching funds (up to a one-to-one maximum). 

Individual programs also have flexibility under the reauthorized program. States can allow programs (subgrantees) to apply to implement programs in various locations—for example, choosing to operate in elementary or middle school buildings, or in a location more accessible to students and their families. This policy change made it more likely that nonschool-based private and public entities could operate with support from the 21st CCLC program. In addition, applicants for subgrants have some flexibility in setting program goals. As part of their grant applications, subgrantees must identify the community need for their program and base their offerings on the need identified. As principal grantees, states are also required to select program- and student-level outcomes. 

Although the federal guidelines specify a range of measures for which grantees must report data, states have considerable flexibility in determining how to hold local programs accountable for results. Certain states allow local programs to set program goals and measures within the bounds of federal and state priorities. Centers are required to report data on the Government Performance Results Act measures for the program through the Profile and Performance Information Collection System (PPICS) system. They are given flexibility on whether they report on grades or state assessment results, but they are required to report on one or the other. 

To monitor program progress, the U.S. Department of Education contracted with Learning Point Associates in October 2003 to develop the PPICS. This online data collection and reporting system gathers grantee- and center-level data aligned with overall program goals to enable local programs to report on progress toward the outcomes they set out to achieve as part of their grants. This study used the PPICS database to identify basic characteristics of all centers to construct the survey and case study samples. In addition, PPICS was the source of contextual information for the survey data and the case study sites. 

In FY 2009, Congress appropriated more than $1.1 billion for the 21st CCLC program. Currently, the federal program supports 1,585 local programs within more than 9,500 centers in 53 states and U.S. territories. Roughly two-thirds of all centers serve students in each of the elementary grades; slightly fewer than half serve students in grades 6, 7 and 8 and one in six centers serve students in the high school grades. Exhibit 1-1 summarizes basic information about funded centers that were in the population at the time of this study, i.e. 2005–06, and about centers targeted by the study.

Exhibit 1-1Characteristics and Type of and Grade Levels Served by Funded and Sampled Centers

* For this study, nonschool-based centers are overrepresented to facilitate the examination of differences between nonschool-based centers and school-based centers.

Source: Profile and Performance Information Collection System database.

*n*= 5,122 for funded; *n*= 516 for sample

Exhibit reads: Seventy-four percent of centers are located in urbanized areas.

Background 

This study attempts to analyze the extent to which activities funded by the 21st CCLC program have characteristics known to be associated with positive student academic achievement. Throughout much of their history, after-school programs have been defined by their diversity of organizational sponsors, voluntary youth participation, modest levels of adult direction and commitment to meeting the social and developmental needs of low-income youth (Halpern, 2002; Muller and Frisco, 1998). Over time, however, programs have shifted away from voluntary and toward mandatory participation, and they have increased the level of adult direction, especially with respect to homework help (Halpern, 2002). The pressure on schools to meet the academic needs of disadvantaged youth has also led to increased emphasis on academics in after-school programs (Brown, 1999). 

This study focuses narrowly on academic activities in two areas, reading and mathematics, in after-school programs and does not address practices associated with outcomes other than academic achievement. This narrow focus is significant, since research syntheses of experimental and quasi-experimental research studies suggest that after-school programs may produce positive outcomes in areas other than student academic achievement (Durlak and Weissberg, 2007; Lauer et al., 2006). The authors of these syntheses also conclude that programs do not have to be exclusively academic in focus to have positive impacts on academic achievement. This study focuses on reading and mathematics activities, however, because the 2004 program evaluation identified these areas as ones where programs might be strengthened. 

Evidence has shown that programs that take place outside the school day can help improve reading and mathematics achievement. Lauer and colleagues (2006) conducted a meta-analysis of studies that examined the effectiveness of after-school and summer strategies for improving achievement in reading and mathematics. Their study, conducted under contract for the U.S. Department of Education’s Institute of Education Sciences, was limited to studies that measured achievement in reading or mathematics and used comparison-group designs. They found small but significant positive effects of programs on achievement in both reading and mathematics, and larger positive effect sizes for programs with specific characteristics such as tutoring in reading. 

Conceptual Framework for the Study

Because the enhanced focus on academics is relatively new in the field of after-school programming, few studies have been conducted of the program quality of after-school academic instruction. Until now, studies have principally relied on frameworks for studying youth development in community settings (e.g., Leffert, Benson, and Roehlkepartain, 1997; National Research Council & Institute of Medicine, 2002), rather than on frameworks that focused on specific academic content or instructional strategies. The framework that guided this study, however, drew on both research conducted in community settings and research conducted in schools about instructional quality. In this respect, the study framework was similar to Huang’s (2001) system of program quality indicators for middle and high school after-school programs, which were based on research carried out in both schools and after-school programs (see also Little, 2007). 

Each subsequent chapter first briefly describes the relevant components of the framework that guided the study (see Exhibit 1-2) and the research that supports the inclusion of each component of the framework. The chapters then set forth relevant study findings. It is important to note that while this study is informed by the broader universe of after-school programming, it is focused on only the 21st Century Community Learning Centers program, and its findings are not intended to be applied to other after-school programs.

The framework below helped organize the presentations of findings, but it has some limitations. Its principal limitation is its narrow focus on those elements needed to promote participation in reading and mathematics instructional activities in 21st CCLC programs, which were the focus of the study. The framework is by no means intended to be inclusive of the broad range of elements of program quality that may be needed to improve academic achievement or related school outcomes, such as attendance. Nor does the framework encompass all research relevant to judging program quality in after-school programs. Research about which program elements contribute to social and emotional outcomes for youth, for example, is excluded from the framework because those findings fall outside the focus of the study.

Exhibit 1-2Framework for the Study: Supports for Student Participation in Quality Academic Instruction That Leads to Increased Achievement

 (INSTRUCTIONALSUPPORTSCenter Staffing and PoliciesRecruit and retain quality staffProvide opportunities for staff developmentPolicies requiring attendanceAlignment to Student NeedsCoordinate activities with school and with support servicesUse of data for program improvementAdult SupportMaintain positive relationship with youthsQUALITY ACADEMIC INSTRUCTIONIntense exposure to academics after school on a consistent basisFocus on core academic contentUse of research-based instructional strategiesGuidance from instructors with education and experience related to their rolesFeedback to students and parents on academic progressStudent ParticipationRegular attendanceStudent engagement in academic activitiesParticipant OutcomesIncreased achievement)

Exhibit reads: In the framework for the study, student participation in quality academic instruction is hypothesized to lead to desired student outcomes for the program.

Chapter 2[footnoteRef:4] addresses the components of **high**-**quality academic instruction**, which is one of the most important influences on academic achievement (Cohen, Raudenbush, and Ball, 2003): [4:  The information presented in this chapter is limited to the study’s elementary school findings. Data from programs serving middle and high school students are presented separately in Appendix A. Comparisons between school-based and nonschool-based centers are based on elementary-level findings.] 

*Sufficient exposure to academics after school on a consistent basis.*The frequency and intensity of student participation in academic activities at centers needs to be sufficient to influence their learning. 

*Focus on core academic content.*Research in reading has identified core skills (e.g., phonemic awareness) that are essential to literacy. Likewise, research in mathematics has identified several content strands that build students’ mathematical proficiency (e.g., peer-assisted learning strategies, solving cognitively complex problems, using computers and graphing calculators to solve problems).

*Guidance from instructors with education and experience related to their roles.*Instructors with backgrounds in helping students in reading and mathematics are likely to be better prepared to offer high-quality instruction matched to student needs in these domains. Staff members’ active guidance of student learning is likely to be more effective than passive supervision of students.

*Feedback to students and parents on academic progress.*Feedback to students helps develop their metacognitive skills and motivates improvement. Parent feedback supports caregiver involvement in student learning, a key component of academic success.

Chapter 3 addresses the components of **student participation** necessary to affect academic achievement outcomes:

*Regular attendance.*Regular center attendance is necessary for students to benefit from after-school instruction. 

*Student engagement in academic activities.*In addition to attendance, being on task and focused on academics when present support students’ motivation to learn and to appreciate the importance of developing proficiency in reading and mathematics.

Chapter 4 addresses the necessary **supports** for high-quality instruction:

*Center staffing and policies.*Centers are responsible for providing pay and benefits at levels that help attract and retain qualified staff, and for providing opportunities for staff to develop and enhance skills relevant to their roles. 

*Alignment of activities to student needs.*High-quality instruction is aligned to student needs. To understand student needs, after-school programs need to be well coordinated with school-day activities and use data from student assessments (or conduct their own student assessments) to adjust program offerings accordingly. 

*Adult support*. After-school programs provide an opportunity to extend students’ learning beyond the school day, but the effectiveness of that help depends on the quality of the relationships between adults and students.

Study Design and Data Collection Methods 

Evaluation Questions 

This evaluation of program quality addressed three main questions: 

1. What is the nature of activities in centers that are designed to promote academic development of students?

2. How do centers differ with respect to regular attendance?

3. How do center leaders staff their centers, coordinate with other service providers and allocate their resources to provide quality programming?

Study Design

The evaluation included the following components: (a) a national survey of center directors, sent to a sample of 600 centers in 2006–07, (b) a focused telephone survey of center directors conducted among a subsample of 140 of those centers in 2006–07 and (c) case study visits to 12 sites in fall 2006 and spring 2007. The focused telephone survey sample was nested within the survey sample, which was nationally representative of centers offering instruction in reading, mathematics, and technology (see explanation below on weighting). Approximately 70 percent of the sample consisted of school-based sites. Nonschool-based sites were deliberately overrepresented to enable statistical comparison with school-based sites.[footnoteRef:5] [5:  School-based sites are not always administered by schools or school districts and nonschool-based sites are not necessarily administered by a community-based or nonprofit organization. Of the 516 schools in our final sample, nonschool grantees administered 210 programs (101 in school settings; 109 in nonschool settings) and schools or school districts administered 306 programs (266 in school settings; 40 in nonschool settings).] 

The following outline briefly describes the sources of data, data collection procedures and response rates, along with a brief explanation of how to interpret the data presented in the exhibits. (See Appendix C for information about the selection of samples for the national survey, focused survey and case studies.)

Data Sources and Data Collection Procedures 

National survey. The survey provided data from a nationally representative sample of 600 center directors to whom surveys were sent, of whom 516 responded. The survey elicited information regarding centers’ objectives and approaches and basic staffing information, as well as the indicators that they use to measure success, their methods for tracking outcomes and their methods for communicating with parents about student progress. In addition, the survey collected data on the organization of activities aimed at improving students’ reading and mathematics skills. In selecting centers, we did not distinguish among centers offering summer, before- or after-school programming.

The surveys were mailed in September 2006 to the center director in charge of after-school services at each of the 600 selected centers. Survey responses were accepted until January 19, 2007. After eliminating centers that could not be reached or were ineligible for the study (e.g., because funding had expired), the size of the sample for the national survey was 558. Directors of 516 centers completed the national survey, a response rate of 92.5 percent.

Focused telephone survey. To provide more in-depth information on program quality, a focused telephone survey collected data from a nationally representative subsample of 140 center directors from the 600 centers surveyed by mail. The telephone survey gathered information about the extent to which centers tracked students’ level of proficiency on state tests in reading and mathematics, the targeting of services and centers’ coordination with local supplemental educational services providers. Questions were also asked about student attendance and the qualifications of center directors and staff. To collect data on staff qualifications, each center director used a method provided by the research team to randomly select three staff members. All centers were asked to provide data on the chosen staff members. If centers had fewer than three staff members, directors completed information regarding all staff members. Printed versions of the focused telephone surveys were mailed, along with the national surveys, to the 140 selected centers, and respondents had the option of mailing in their completed surveys or communicating their responses by telephone during the data collection period. Survey responses were accepted until January 19, 2007. After eliminating ineligible and unreachable centers, 132 centers were chosen for the telephone survey. Center directors completed 122 focused telephone surveys—a response rate of 92.4 percent. 

Site visits. The study team visited 12 sites between fall 2006 and March 2007. Eleven of the site visits were to elementary schools; the 12th was to a middle school site. Two sites were visited in each of California, North Carolina, South Dakota and Maryland, and four sites were visited in Florida. To ensure comparability of data collected across sites, all site visitors received training at the beginning of the study. Two researchers then visited each site, where they conducted program observations, document review and interviews with center directors, staff and school-day instructors. After the site visits, each pair of researchers completed a debriefing form to ensure that they had captured comparable data aligned with the research questions. 

Site visit protocols included interviews with grant directors, center directors, program staff and school personnel. The protocols also included a structured observation of academic and enrichment activities. The case studies provided additional evidence about the nature of academic instruction and about how well center practices aligned with policy goals for the program and with previous research on program quality. The observation form used in the study was adapted from an earlier study Policy Studies Associates conducted of after-school programs funded by The After School Corporation (TASC) (Birmingham, et al., 2005). That instrument included three constructs used in this study: active teaching strategies, teacher relationships and engagement. Four items comprised the active teaching strategies construct: communicating goals, using multiple teaching strategies, challenging students to move beyond current skill levels and asking students to expand on answers and ideas. For the teacher relationships construct, there were also four items: using positive behavior management techniques, showing positive affect toward students, attentively listening to and/or observing students and encouraging students to share ideas, opinions and concerns. Student engagement had one item. For each observation, the study team rated each of these items on a four-point scale, ranging from “did not occur” to “highly characteristic.” A concurrent validity study of this instrument conducted in 2006 found ratings on these constructs to be significantly correlated with student ratings of settings observed by researchers (Pechman, et al., 2008). Each item is reported separately but in a group with other items in the constructs identified in the earlier research; no weights were assigned to the items. The observation instrument can be found in Appendix F. 

Because this study focused on academic instruction, the study team added content to the observation form related to reading and mathematics (in Appendix F). The items added focus on reading skills and activities and mathematics skills and activities asked about in the national survey. For each activity, observers rated whether there was an explicit focus on the skills and activities, an implicit focus or no focus. The distinction between explicit and implicit focus is an important one for some after-school programs, which may make academic goals implicit in an effort to attract participants and increase engagement (Vadeboncoeur, 2006). A limitation of these observational constructs is that separate validity data are not available, as this was the first time the constructs were used. 

SRI provided training to all case study researchers in the instrument, and researchers conducted joint observations in the field to establish the reliability of the instrument. The training session provided guidance on the definitions of categories used in the instrument; participants also reviewed a codebook with examples of types of activities to help them learn how to use it. A total of 13 joint observations took place on site visits. For the active teaching strategies, raters agreed on the exact rating in 64 percent of observations and were within one point on the four-point scale 94 percent of the time. For on-task ratings, raters agreed on an exact rating 69 percent of the time, and 100 percent of the ratings were within one point. Agreement was lower for reading skills (63 percent exact agreement) and activities (65 percent), but higher for mathematics content (96 percent) and skills (97 percent). 

Weighting 

All the survey data were weighted to reflect the full population of eligible centers and to take into account nonresponse. Centers included in the original sampling frame for this study had to meet all of the following four criteria: (a) receiving funding at the time the sample was drawn; (b) offering reading activities; (c) offering mathematics activities and (d) offering technology-related activities.[footnoteRef:6] Nonschool-based centers were deliberately oversampled and were weighted to reflect their percentage in the population of funded centers through the variables discussed below. To arrive at conclusions about the national population of centers, the data were poststratified to reflect the full population of funded centers by using the variables discussed below. The study team also used Deming’s procedure to weight the respondents so that the weighted marginal counts were the same as in the universe of sites funded at the time of the study and in operation for at least a year. (For example, the weighted number of responding elementary schools was the same as the number of elementary schools in the universe.)  [6:  Centers provided self-reported answers to these questions through the PPICS database.] 

The specific characteristics used to poststratify the respondents came from the PPICS database and from U.S. census data on characteristics of persons living in ZIP codes where centers were located. The variables from the PPICS database were the numbers of students served, program longevity, centers’ history of offering academic services, primary grade levels served and number of weeks per year centers were open. Data from U.S. census ZIP code files were the percentage of the ZIP codes’ population living in an urbanized area and the percentage of minority population in the area.

Limitations of the Study

One limitation of the study pertains to the representativeness of the sample. Because of the original emphasis on the quality of instructional services in reading, mathematics and technology, the study team excluded sites that did not offer all those services. Although nearly all sites offered reading and mathematics activities, 19 percent of centers did not offer technology services and thus were excluded from the sample. 

The specific characteristics used in poststratifying the respondents, using the PPICS database and U.S. census data from ZIP codes where centers were located, equalized the weighted sample and the universe of centers in regard to all characteristics expected to influence survey responses. The reweighting thus essentially eliminated the biases caused by these variables. A potential limitation is that reweighting did not eliminate biases not associated with these variables, nor did it eliminate biases that were functions of variable interactions, such as between the size of the community served and a center’s academic offerings. 

A second limitation was the study’s heavy reliance on self-reported data. Self-reported data on instruction can be inaccurate, and observation data do not always confirm what instructors report they are doing in their classrooms (Burstein et al., 1995). One source of bias is instructors’ memory, which has been found to be poor for instructional events that are rare or occur infrequently (Hoppe et al., 2000; Rowan, Camburn, and Correnti, 2004; Sudman and Bradburn, 1982). Response choices can also influence survey respondents’ answers, either by cueing respondents to supply socially desirable answers or by suggesting estimation strategies that systematically bias the obtained reports (Schwartz, 1999; Schwartz and Oyserman, 2001). Questions about frequency of events or the number of hours spent in certain academic activities may be particularly biased, because respondents tend to answer these questions quickly at a cost of accuracy (Burton and Blair, 1991; Marquis, Marquis, and Polich, 1986). 

The study team sought to reduce the impact of self-report bias by using established techniques. The questionnaire asked center directors to ask their instructors about specific pedagogies and behaviors, a strategy known to increase the reliability of teacher self-reported data (Garet, et al., 2001; Herman, Klein and Abedi, 2000; Koziol and Burns, 1986; Mullens, 1998; Ross et al., 1997). In addition, respondents were asked to skip items if earlier items on the survey established that respondents could not report accurately. If, for example, respondents to the telephone survey reported that they did not keep accurate records of the number of individuals who attended in the past week, they were asked not to report on the number of students who attended 90 days or more in the past year. This approach resulted in some additional missing data, but it likely reduced the bias in the responses obtained. 

When possible, the team checked the consistency of reporting across questions and against data reported by centers in their annual reports. For example, centers provided information about their operating hours in annual reports, and they also answered questions about the number of hours typical students participated per week. Despite these precautions, bias may still have affected the responses to some items, particularly those related to academic instruction. Center directors who were mailed the questionnaire may not have asked their instructors to complete it as instructed, but instead may have completed the surveys themselves. Center directors, given the roles they play in centers, may be reliable informants about record keeping, data use and organizational supports for program quality. They rarely provide instruction to students, however, so they may not have been good sources for information about instruction. Because we were able to compare survey results on instruction with observation results for the centers serving elementary school students, we have focused in Chapter 2 of this report on findings for centers serving elementary students. Descriptions of academic instruction for centers serving middle and high school students appear in Appendix A. Appendix E provides detailed statistics for data presented in the tables in Chapter 2. 

A third limitation is that the study team did not distinguish among before-, after- and summer school programs. This limitation is important, because a review by Lauer and colleagues (2006) found that summer programs were more likely to emphasize academics than programs offered during the regular school year. Because this study does not distinguish among different types of programs, the data collected in it may be of limited utility in addressing concerns about attendance raised within the 21st CCLC program. 

There are also three important limitations of the observation data. First, the lower agreement among ratings for reading skills and activity implies these data are less reliable than data associated with the other ratings. Raters had difficulty agreeing on which skills were explicit and which ones were implicit; thus, ratings of reading activities were measured with a higher level of error than other ratings and should be interpreted with caution. Second, observations were purposively sampled, so the data collected are not nationally representative and observation protocols used have not been related to any outcomes of importance to the program. We initially sampled sites for the study using the promising practices framework (Luce and Thompson 2005). The evaluation team analyzed the data from the case studies with the intent of comparing promising versus typical sites along such dimensions as center goals, staffing and professional development; attendance patterns and policies; academic program offerings and resource management. However, the analysis yielded few meaningful differences between typical and promising centers; therefore, the focus of data analysis shifted to examining consistency between observation and self-reported survey data. Because the observations were few and purposively sampled, we cannot draw inferences from these data about program practices nationwide. Data from the observations can be used, however, to qualitatively examine whether the patterns are similar between the observation and survey data. Descriptions of observation ratings for site visits appear in Appendix B. A third limitation is that although analyses were conducted that demonstrated the consistency of the OST scales, no analyses have been conducted to establish the predictive validity of the scales (i.e., there is no evidence about the relationship between the measure and outcomes of interest such as youth behavior and academic outcome measures).

Presentation of Survey Data in the Report

Throughout the report, differences among different types of centers are noted. Unless otherwise stated, all differences reported in the text are statistically significant. Statistically significant differences are identified by asterisks and the inclusion of *p* values in the report’s exhibits. “School-based” and “nonschool-based” refer to the location where grantees provide services for students. Moreover, although some centers serve only a single school level (elementary school, middle school or high school), many centers serve multiple school levels. All comparisons related to levels, therefore, analyze differences between centers serving a particular level (e.g., elementary) and those not serving that level (e.g., nonelementary).

12

1. Study Background, Goals and Methods	

2. Reading and Mathematics Activities in Centers Serving Elementary School Students

Key Findings

The following are key findings for centers serving elementary school students. Findings for centers serving middle and high school students appear in Appendix A.

· *Sufficient exposure to academics.* To have an impact on academic outcomes, centers need to focus on academics, and students need to participate in academic activities at centers at a frequency that is sufficient to influence their learning (Dynarski et al., 2003; Fleming and Zhang, 2005). Grossman and colleagues (2002) found that homework help activities implemented well provided adult support to students and helped them to build academic skills. A recent meta-analysis found that after-school programs that provided one-on-one tutoring to students had positive effects on student academic achievement. The effect sizes ranged from 1 and 4/5 of a standard deviation, which are relatively large effects for educational programs (Lauer et al., 2006). More than 70 percent of centers**serving elementary school students reported their students participated in reading activities on a weekly basis for at least two hours per week; 60 percent reported their students participated weekly in mathematics activities for at least two hours per week. Centers reported that a typical student spent 4 or more hours per week on reading activities (25 percent of centers) and mathematics activities (19 percent of centers). Centers also reported that a typical student spent 4 or more hours per week working on homework (36 percent of centers) and tutoring (12 percent of centers), both of which could include reading or mathematics activities. None of the observed activities involved one-on-one tutoring, but more than half of (58 percent) of observed reading activities involved some small-group (three to five students) tutoring activities (*n* = 23).

· *Focus on core academic content*. Research in reading has identified core skills that are essential to literacy: phonemic awareness, phonics, vocabulary, fluency and comprehension (Snow, Burns and Griffin 1998). Likewise, research in mathematics has identified content strands that build students’ mathematical proficiency: number sense, operations, measurement, geometry, data analysis, statistics, probability and algebra (National Research Council, 2001, 2005). The data from surveys indicated that reading activities in centers serving elementary school students included all skills emphasized by the National Reading Panel (2000). Instruction in mathematics covered a narrower range of skills, with little emphasis on more complex problem solving. 

*Guidance from instructors.* Instructors with backgrounds in helping struggling students in reading and mathematics are likely to be better prepared to offer high-quality instruction matched to student needs (Intercultural Center for Research in Education & National Institute on Out-of-School Time, 2005). Staff members’ active guidance of student learning is likely to be more effective than passive supervision of students (Brophy and Good, 1986; Good and Brophy, 2008; Rowan et al., 2002). Observers in case studies found that instructors in centers serving elementary school students used active teaching strategies in academic activities, communicating goals clearly to students in most activities (in 60 percent of reading activities and in 89 percent of mathematics activities). However, staff were less likely to use multiple teaching strategies in observed mathematics activities than in observed reading activities. The case studies were conducted in a small number of purposively sampled schools, so the data are not representative of all 21st CCLC sites.

*Feedback to students and parents.* Feedback to students helps develop their metacognitive skills and motivates improvement (Butler and Winne, 1995). Parent feedback supports caregiver involvement in student learning, a key component of academic success (Epstein, 2008; Sheldon, 2008).****Most centers serving elementary school students reported providing frequent verbal feedback to students and parents, but written feedback was less common. 

Exposure to Academics After School

Center activities need to provide sufficient exposure to academics if they are to increase the chances of improving student academic achievement (Dynarski et al., 2003; Fleming and Zhang, 2005). Reviews of academic time-on-task in schools (Wang, Haertel and Walberg, 1993) as well as evaluations of after-school programs (Baker and Witt, 1995; Huang, et al., 2000) have found a relationship between sufficient exposure to academic content and a variety of positive academic outcomes. Lauer and colleagues’ (2006) meta-analysis found that reading programs with durations of 44 to 84 hours and mathematics programs with durations of 46 to 75 hours had the largest effect sizes. They also found that more exposure does not always result in higher academic achievement, making it difficult to say how much exposure to academics in an after-school program is ideal to improve academic achievement. 

According to the PPICS database, only 2 percent of all centers reported that they did not intend to offer activities that focused on reading, and 6 percent did not intend to offer activities that focused on mathematics. By contrast, although all the centers included in the survey sample intended to offer services in both reading and mathematics, of the surveyed centers serving elementary school students, 1 percent indicated that they did not offer reading activities and 4 percent did not offer mathematics activities. Not all students in a center offering these activities are required to participate in both of these activities.

**Surveyed centers serving elementary school students reported that a typical student spent time weekly in academic support activities like homework and in reading or mathematics activities.**Centers reported that a typical student participated in homework help (36 percent), reading activities (25 percent), mathematics activities (19 percent) and tutoring (12 percent of centers) for more than 4 hours per week (see Exhibit 2-1; see also Exhibit E-1 in Appendix E). Additionally, 33 percent of centers reported that a typical student engaged in arts or recreation activities for more than 4 hours weekly. Because centers were open slightly less than 16 hours per week, on average, student participation of 4 hours per week in a particular activity amounted to about 25 percent of the available time. These data are different from and more detailed than those recorded in PPICS; one reason may be that PPICS data used for identifying the sample were recorded the year before the study team fielded the survey.

Exhibit 2-1Percentage of Centers Serving Elementary School Students That Reported Participation in Activities by a Typical Student, by Type of Activity and Amount of Time of Engagement per Week

Note: Percentages may not equal 100 percent due to rounding.

Source: National Survey, Item 5.

*n* = 389

Exhibit reads: Thirty-six percent of centers serving elementary school students reported that a typical student participated in homework activities for more than 4 hours per week. 

Reading Activities

Reading Skills Emphasized 

Proficiency in reading requires a range of skills, including awareness of print features, phonemic awareness, phonics, vocabulary, fluency and comprehension (Snow et al., 1998). Concentrating on only one of these reading skills is likely to be insufficient to improve reading achievement because evidence indicates that decoding and language comprehension skills make independent, but related, contributions to reading comprehension (Oakhill, Cain and Bryant, 2003). 

**A majority of centers serving elementary school students reported that most or all reading activities[footnoteRef:7] included the skills emphasized by the National Reading Panel (2000).** Center directors provided these data, which the study team asked them to obtain from someone familiar with the activities. Among centers serving elementary students, the largest percentage reported focusing on comprehension and vocabulary in all or most activities (74 percent and 69 percent, respectively) (see Exhibit 2-2; see also Exhibit E-2 in Appendix E).  [7:  Reading enrichment activities are defined as structured activities designed to build students’ literacy skills. Reading enrichment may include scheduled time for independent student reading, writing and literacy enrichment activities, but not homework assistance. ] 

Exhibit 2-2Percentage of Centers Serving Elementary School Students That Reported Reading Activities Emphases, by Reading Skill Emphasized and Inclusion Level

Note: Percentages may not equal 100 percent due to rounding.

Source: National Survey, Item 22.

*n*= 388

Exhibit reads: Seventy-four percent of centers serving elementary students reported emphasizing comprehension in most or all instructional activities in reading.

Reading Skills Emphasized in Observed Activities

Site visit observations of centers that served elementary school students, though not nationally representative, were consistent with the pattern of the survey data. Of the 35 reading activity observations conducted, 86 percent stressed reading comprehension, and 71 percent emphasized vocabulary (see Exhibit 2-3). The observation data revealed a greater emphasis on fluency than did the survey data: 77 percent of observed reading activities focused on fluency, but 59 percent of centers reported that all or most reading activities emphasized fluency (see Exhibit 2-2). 

Exhibit 2-3Percentage of Observations of Reading Activities in Centers Serving Elementary School Students, by Reading Skill Emphasized

Source: Site visit observations.

*n* = 35 observations

Exhibit reads: Eighty-six percent of observed reading activities in centers serving elementary school students focused on developing students’ comprehension skills.

Use of Instructional Strategies for Teaching Vocabulary, Fluency and Comprehension

The National Reading Panel (2000) summarized existing rigorous evidence on strategies for the teaching of vocabulary, fluency and reading comprehension.[footnoteRef:8] To develop vocabulary skills, children need rich, explicit instruction (Beck, McKeown and Kucan, 2002), but children can also learn vocabulary incidentally, as part of authentic reading activity (National Reading Panel, 2000). Both forms of instruction may be effective because children need extensive practice and multiple exposures to words in order to learn them (Coyne, et al., 2004). Frequent opportunities to read aloud with guidance from staff, peers or parents have been found to have a significant and positive effect on word recognition, fluency and comprehension across a range of grade levels (Coyne, et al., 2004). Finally, research suggests that strategies such as asking students to recall a text, answer and generate questions about it, make predictions about it and summarize it, improve comprehension (Biancarosa and Snow, 2006; National Reading Panel, 2000). Recent experimental research has also shown the advantages of selecting books at students’ reading level for them to read independently (Borman and Dowling, 2006). [8:  This study focused on these components of reading in order to select survey items from the National Assessment of Educational Progress Teacher Background Survey related to reading instruction, and to design interview items that could measure the extent to which programs addressed the major processes involved in learning how to read, according to the findings of the National Reading Panel. ] 

**Although 99 percent of centers serving elementary school students reported using at least one strategy in reading, the most commonly reported reading activities were designed to foster more independent reading and greater comprehension.**Roughly half of centers serving elementary school students indicated that in all or most reading activities, they asked students to explain or support statements about something they read (50 percent) or had students talk about or write answers to questions on something they had read (42 percent). More than a third of these centers also reported asking students to make predictions about a text they were about to read (38 percent), or to talk with peers about what they had read (36 percent) in all or most activities (see Exhibit 2-4; see also Exhibit E-3 in Appendix E).

Exhibit 2-4Percentage of Centers Serving Elementary School Students That Reported Student Participation in Instructional Strategies in Reading Activities, by Strategy and Inclusion Level

Note: Percentages may not equal 100 percent due to rounding.

****Source: National Survey, Item 21.

*n* = 388

Exhibit reads: Fifty percent of centers serving elementary school students reported having students explain or support their understanding of what they had read in most or all reading activities.

School-based centers were more likely than nonschool-based centers to report that their instructors asked students to make predictions about something they were reading, and talk about or write answers to questions related to something they had read. Observation data, while not nationally representative, were generally consistent with the survey data. Two instructional strategies were found in more than half of the observations conducted on site visits: having students help other students understand new words (80 percent) and asking students to make generalizations (63 percent) (see Exhibit 2-5). In contrast, as shown previously in Exhibit 2-4, only slightly more than a third of centers reported that most or all reading activities employed these strategies. At the same time, the proportions of observed activities in which students were asked to make predictions (49 percent) and read aloud (46 percent) were generally aligned with the survey data for these activities. 

Exhibit 2-5Percentage of Observations of Reading Activities in Centers Serving Elementary School Students, by Instructional Strategy Emphasized

 (Work on understanding new wordsMake generalizations and draw inferences about something they have readMake predictions about something they are readingRead aloud to peers or staffDo a group activity or project about what they have readDiscuss different interpretations of what they have readDescribe the style or structure of a text they have readWrite answers to structured questions about something they have read)

Source: Site visit observations.

*n*= 35 observations

Exhibit reads: Eighty percent of observations of reading activities in centers serving elementary school students focused on students helping other students understand new words.

Mathematics Activities

Mathematics Skills Emphasized

The content of mathematics instruction is a significant predictor of gains in student achievement in mathematics (Gamoran, et al., 1998; Porter, 2002; Rowan et al., 2002). Syntheses of research have also pointed to the central importance of content in mathematics, drawing attention, for example, to the foundational role of numbers and procedures with numbers in mathematics (National Research Council, 2001, 2005). Instruction needs to emphasize more than just one strand of mathematics content, however; it is also important for students in the elementary and middle grades to learn about the foundations for algebra, which include a focus on fractions, measurement and geometry (National Research Council, 2001; National Mathematics Advisory Panel, 2008, pp. xvii, 18). The National Mathematics Advisory Panel recommends that the content should build over grade levels; failure to address any of the foundations, even in the early grades, may cause many students to fall behind (Roschelle, et al., 2009). In addition, large-scale studies of mathematics instruction have shown that the use of more rigorous content was a significant predictor of gains in student academic achievement (Gamoran et al., 1998; Rowan et al., 2002). 

**Mathematics instruction focused on a relatively narrow range of content strands.**Among centers serving elementary students, 71 percent reported emphasizing operations such as addition, subtraction, multiplication and division in all or most activities[footnoteRef:9] (see Exhibit 2-6; see also Exhibit E-4 in Appendix E). Fifty-nine percent reported that number sense (i.e., understanding of numerical relationships as expressed in ratios, proportions and percentages) was stressed in all or most activities. Operations with decimals or fractions, a key target of instruction according to the National Mathematics Advisory Panel, ranked third as an area of emphasis in center programming. Fewer activities targeted complex mathematics: 15 percent of centers emphasized algebraic concepts, and 8 percent focused on data analysis in all or most activities.  [9:  Mathematics activities are defined as structured activities designed to build students’ mathematics skills. Mathematics activities may include scheduled time for independent problem solving and instructor-led enrichment activities, but not homework assistance.] 

Exhibit 2-6Percentage of Centers Serving Elementary School Students That Reported Emphases in Math Activities, by Type of Content Emphasized and Inclusion Level

Note: Percentages may not equal 100 percent due to rounding.

Source: National Survey, Item 24.

*n* = 387

Exhibit reads: Seventy-one percent of centers serving elementary school students emphasized operations with whole numbers in most or all mathematics activities.

School-based centers were more likely than nonschool-based centers to emphasize advanced topics in mathematics. In particular, school-based centers were significantly more likely than nonschool-based centers to have at least some emphasis on geometry (84 percent versus 70 percent), algebra (75 percent versus 59 percent) and data analysis (70 percent versus 60 percent). However, from both types of centers, the reported focus on foundational topics such as number sense and operations was similar. 

Mathematics Content Emphasized in Observed Activities

Observation data from the site visits, while not nationally representative, generally showed that centers primarily focused on teaching operations with whole numbers and number sense (see Exhibit 2-7). The emphasis on these basic skills may reflect a focus on building a strong foundation in basic math facts. In an interview at one of the centers visited, the mathematics instructor explained, “Because these kids are struggling, they need explicit systematic instruction. They have to get their foundation solid first.” In addition, a center’s ability to deliver instruction in higher-order mathematics may be limited by the knowledge of its tutors and staff. In another center, one classroom teacher commented that tutors struggled with the level of content knowledge necessary to successfully assist older students with more complex mathematics problems, but did not experience the same challenges in literacy or language arts. 

Exhibit 2-7Percentage of Observations of Mathematics Activities in Centers Serving Elementary School Students That Emphasized Skills, by Type of Math Skill Emphasized

Source: Site visit observations.

*n* = 18 observations

Exhibit reads: Eighty-three percent of observed mathematics activities in centers serving elementary schools focused on developing students’ skills in performing operations with whole numbers. 

Use of Instructional Strategies for Teaching Mathematics

In mathematics, research on effective instructional strategies has focused on how best to develop the full range of skills students need to be mathematically proficient (National Research Council, 2001, 2005; National Mathematics Advisory Panel, 2008). One strand of research has examined the relationship between student academic achievement and the cognitive complexity of tasks instructors in regular classrooms assign to students. Cognitive complexity refers to the demands instructors place on students to engage in thinking about particular academic content. A low-complexity task in mathematics might involve asking students to memorize facts, definitions or formulas; in contrast, a high-complexity task might involve asking students to prove a theorem or solve a nonroutine problem that requires many steps (Porter, 2002). Although an emerging consensus views both procedural fluency and conceptual understanding as fundamental, interwoven aspects of mathematical proficiency (National Research Council, 2001), strong experimental and correlational evidence indicates that when instructors require more complex tasks of students in a particular content area, student academic achievement gains are higher (Carpenter, et al., 1989; Gamoran et al., 1998). 

Other evidence for effective strategies in mathematics comes from the National Mathematics Advisory Panel (2008), which was established to promote American students’ knowledge of mathematics and improve their performance in the subject. Having analyzed relevant research, the panel concluded that providing students the opportunity to explain the reasoning behind a problem solution helps to build proficiency in computation and translation of word problems and helps teachers to ensure that students have the conceptual framework necessary for understanding grade-level mathematics. The National Mathematics Advisory Panel (2008) also concluded that cooperative learning strategies, including peer-assisted learning strategies and heterogeneous grouping, have significant effects on student outcomes, particularly in teaching young children mathematical operations. Finally, other reviews of experimental research studies indicate that computers and graphing calculators can positively affect mathematics achievement (Dixon, et al., 1998). 

**Although 99 percent of centers serving elementary school students reported using at least one strategy, the most common strategies used in all or most mathematics activities were practicing mathematical facts (68 percent) and asking students to explain their reasoning in solving a problem (46 percent).**Twenty-four percent of centers reported having students use computers to support mathematics instruction or to solve open-ended problems in mathematics (see Exhibit 2-8; see also Exhibit E-5 in Appendix E).

Exhibit 2-8Percentage of Centers Serving Elementary School Students That Reported Student Participation in Instructional Strategies in Math Activities, by Strategy and Inclusion Level

 (Inclusion Levels) (Practice basic mathematics factsExplain the reasoning behind how they solved a problemSolve problems in which students have to figure out what method to use to solve themWrite mathematical equations to solve a problemUse computers to practice mathematicsSolve problems that have many possible correct answersMake charts, tables or graphsUse graphing calculators to solve problems)

Note: Percentages may not equal 100 percent due to rounding.

Source: National Survey, Item 23.

*n* = 388

Exhibit reads: Sixty-eight percent of centers serving elementary school students reported asking students to practice basic mathematics facts in all or most instructional activities in mathematics.

Mathematics Instructional Strategies in Observed Activities

Observation data, though not nationally representative, showed a pattern similar to the survey data. Consistent with the mathematics strategies that centers reported on the survey, practicing basic mathematics facts was a strategy in 83 percent of the observed mathematics activities. Solving problems in which students had to select a method was assigned in 47 percent of observed activities, and students had to explain the reasoning behind their problem-solving strategy in 47 percent of observed activities. There were a few differences, however. For example, site visitors reported that students engaged in more complex tasks in mathematics than reported on the survey, but were asked to write mathematical equations less often than reported (see Exhibit 2-9). 

Exhibit 2-9Percentage of Observations of Mathematics Activities in Centers Serving Elementary School Students That Emphasized Instructional Strategies, by Strategy Emphasized

*(Practice basic mathematics factsSolve problems in which they figure out what method to useExplain the reasoning behind how they solved a problemUse computers, calculators to practice basic mathematics skillsSolve problems that have many possible correct answersWrite mathematical equations to solve problemsMake charts, tables or graphs)*

Source: Site visit observations.

*n* = 18 observations

Exhibit reads: Eighty-three percent of elementary school center mathematics activities observed involved students practicing basic mathematics facts.

Instructor Education and Experience

Research on after-school programs suggests that when students receive guidance from instructors with education and experience relevant to their roles, students are more likely to benefit from program activities. A case study comparing after-school programs assessed to be “high quality” and “low quality” found that these programs differed on key program features like staff commitment, amount of staff training and staff background (Vandell, et al., 2004). A study of after-school programs in Massachusetts found that the programs that demonstrated higher outcomes relied on staff with high levels of educational attainment and often employed a number of instructors with teaching certificates (Intercultural Center for Research in Education & National Institute on Out-of-School Time, 2005). In addition to providing relevant expertise, certified instructors who also teach after-school participants during the regular school day may have a deeper knowledge of students’ needs (Little, 2006).

Research on the need for certified instructors to offer high-quality instruction is by no means definitive, however. For example, a large-scale study of elementary schools found no relationship between student academic achievement gains and special certification for instructors for either reading or mathematics (Rowan et al. 2002). It is also important to note that the authorizing legislation for the 21st CCLC program does not require centers to use highly qualified teachers.

**Centers serving elementary school students reported that more than two-thirds of staff providing instruction in reading and mathematics had prior experience as certified classroom teachers or as instructional specialists in reading or mathematics.** Fifty-two percent of the staff who provided instruction in reading and mathematics were at the time of the study or in the past regular classroom teachers, and 16 percent had been instructional specialists in reading or mathematics. Twenty-two percent of staff providing instruction in reading or mathematics were at the time of the study or had previously been classroom aides. 

During site visits, instructors in observed reading activities were more likely to hold a teaching certificate than instructors in mathematics activities. In addition, the percentage of certified teachers in reading was higher than reported in the survey. Currently certified teachers were present in 94 percent (*n* = 35) of observed reading activities (compared with 56 percent reported by the center surveys). Currently certified teachers were present in 50 percent (*n*= 18) of observed mathematics activities. The discrepancies may reflect the fact that the data from the observations were not nationally representative. 

**Centers reported that nearly two-thirds of reading and mathematics instructors had a bachelor’s degree or higher (64 percent for reading and 63 percent for mathematics).** Approximately one third of the centers indicated that their instructors had only a bachelor’s degree (32 percent for reading and 33 percent for mathematics) and a little more than a third of the centers said that their instructors had some graduate training (36 percent for reading and 37 percent for mathematics). 

Instructor Guidance in Observed Activities

Research on active teaching has found that in classrooms where instructors actively supervise, instruct, support and provide feedback to students, students make greater learning gains in both reading and mathematics (Brophy and Good, 1986; Good and Brophy, 2008; Rowan, et al., 2002). Active teaching is contrasted with teaching in which students primarily work independently or on tasks unrelated to academic activities (e.g., generating a list of behavior rules for the class). 

**Instructors used active teaching strategies in 60 percent of all observed reading activities and in about 90 percent of all observed mathematics activities**(see Exhibit 2-10). In 60 percent of reading activities, observers noted that instructors communicated the goals, purposes and expectations about activities to students. In more than half of all reading activities (53 percent), instructors used two or more strategies to teach a topic. In 89 percent of mathematics activities, observers rated instructors as providing guidance to students about the goals, purposes and expectations of activities. In 44 percent of observed mathematics activities, instructors challenged students to extend their current level of skill. In 22 percent of mathematics activities, instructors were observed to use more than one strategy to teach a skill. 

Exhibit 2-10Percentage of Observations in Reading and Mathematics Activities in Centers Serving Elementary School Students That Emphasized Use of Active Teaching Strategies, by Strategy Emphasized and Activity Type

Source: Site visit observations.

*n* = 35 observations of reading activities; *n* = 18 observations of mathematics activities

Exhibit reads: Instructors communicated the goals, purposes and expectations of activities to students in 

60 percent of the reading activities observed in centers serving elementary school students.

Feedback to Students and Parents on Student Progress 

Research on how people learn (National Research Council, 1999, 2005) emphasizes the need to provide feedback to learners about their progress. Feedback supports students’ developing skill in monitoring their own learning (Butler and Winne, 1995). Feedback can also prompt learners to correct misunderstandings (Bangert-Downs, et al., 1991), motivate students’ interest in seeking help (Butler and Neuman, 1995) and increase students’ interest in improving their understanding or skill (Butler, 1987). Feedback to parents about students’ performance can also support improvement in learning and serve as an important vehicle for parent involvement (Epstein, 2008; Sheldon, 2008). Evidence suggests that delayed feedback—written comments on work as opposed to immediate verbal feedback—is better for long-term recall and retention (Butler and Winne, 1995). None of this research, however, was conducted specifically in after-school programs.

**Most centers serving elementary school students reported providing informal feedback to students; written or formal feedback was less common.** For instance, 87 percent of school-based centers reported providing verbal feedback to students on work assigned by the school, as well as on their behavior. In contrast, about a quarter of school-based centers gave students written feedback on work assigned by the school (22 percent) and on work assigned in the center (26 percent). More than two-thirds of nonschool-based centers reported offering some kind of reward or recognition for program accomplishments, more than did school-based centers (see Exhibit 2-11; see also Exhibit E-6 in Appendix E). 

Exhibit 2-11Percentage of Centers Serving Elementary School Students That Provided Feedback About Their Progress, by Type of Feedback and Center Type

Source: National Survey, Item 18.

*n* = 393

*p < 0.05

Exhibit reads: Eighty-six percent of nonschool-based centers serving elementary school students provided verbal feedback to students on work assigned by the students’ schools.

**A majority of centers serving elementary school students reported providing frequent, informal feedback about student progress to parents.**Sixty-two percent of centers serving elementary students reported talking with parents nearly every week when the parents picked up or dropped off their child. Four percent sent home written reports of students’ progress in the after-school program nearly every week, but 53 percent never did so or did so rarely. One percent of centers serving elementary school students indicated that they held parent–center staff conferences nearly every week, and 56 percent never or rarely held conferences. Nonschool-based centers were more likely to hold parent–center staff conferences than school-based centers, but less likely to contact parents by telephone or e-mail. These differences were statistically significant. 

Summary 

Overall, the reading and mathematics activities implemented in 21st CCLC programs that serve elementary students reflected a focus on academics. Most students were exposed to a variety of activities in reading and mathematics. Additionally, centers reported offering feedback to students and parents on a regular basis about their progress. The feedback was largely informal, however, and limited primarily to verbal exchanges about progress on work assigned by the school.

Reading activities covered a broader range of skills and used a wider variety of strategies than did mathematics activities. For example, instruction covered a broader range of reading skills, and centers were likely to use a range of strategies for teaching reading skills. In contrast, mathematics instruction focused primarily on two content strands and engaged students in tasks with low levels of cognitive complexity. The emphasis on operations and number sense in mathematics activities may have resulted from a need to build a strong foundation of basic mathematics skills among students. These differences in the quality of reading and mathematics activities cannot be explained using data from the surveys about staff qualifications, because instructors in reading and mathematics had similar levels of formal education. 

2. Reading and Mathematics Enrichment in Centers Serving Elementary School Students	28

3. Student Targeting and Attendance

In general, 21st Century Community Learning Centers operate before and after school as well as during the summer. According to a recent report (Naftzgeret, et al., 2006), the operating hours for centers (typically open Monday through Thursday during the school year) averaged 14.2 hours per week during the school year and 23.5 hours per week during the summer. On average, centers were open for 150 days during the school year. 

Key Findings

*Setting priorities and policies regarding services and participation*. Centers’ policies are important organizational supports, particularly for setting expectations regarding student attendance (Crollick, Zhang and Fleming, 2005). More than half of all centers had policies that required students to attend, but mandatory attendance policies were not found to relate to greater attendance.[footnoteRef:10],[footnoteRef:11] About two-thirds of centers had tracking systems capable of identifying how many days individual students participated in the program (i.e., could generate attendance reports easily, within 1–2 hours’ time).  [10:  The National Survey Item 10 asked center directors: “What policies regarding attendance does the center have for students?” Possible answers were: “Attendance is not required; students can drop in and out”; “We require students to attend at least 1 day per week”; “We require students to attend 2–3 days per week”; “We require students to attend every day we offer services”; and “Other.”]  [11:  Although requiring attendance was not defined on the survey, respondents may have varied in their interpretations of this question, depending upon the consequences attached to nonattendance; therefore, data are included in a separate question about consequences of nonattendance.] 

*Regular attendance.* Regular attendance by students who are most in need of after-school programming is necessary for them to benefit from the instruction (Reisner, et al., 2004). Although all centers served students who attended Title I schools, the centers did not focus solely on serving low-performing students. Most allowed any interested students to participate. School-based centers were more likely to give priority to low-performing students. 

*Attendance rates and grade level*. Of the centers that could generate daily attendance reports easily (within 1–2 hours’ time) (75 of 122 indicated that they could generate attendance reports easily), centers serving elementary school students reported that 48 percent of students attended 60 days or more in 2005–06. Centers serving middle grades reported 36 percent of students attended this often, and centers serving high school students reported 30 percent of their students met this standard. These findings are not generalizable to all centers; it is possible that centers that are capable of reporting attendance data may be more likely to have higher attendance.

*Student engagement in academic activities.* Beyond attendance, being on task and focusing on academics when present support students’ motivation to learn and appreciate the importance of developing proficiency in reading and mathematics (Chaput, Little and Weiss, 2004; Mahoney, Parente and Lord, 2007). Students were more engaged in observed reading activities during site visits than in observed mathematics activities. However, these data are not nationally representative. 

Priorities Regarding Targeting

Historically, economically advantaged students have had greater access to after-school programs than students from low-income families (Simpkins, et al., 2005). The legislation that authorizes the 21st CCLC program requires grantees to primarily target students who attend Title I schools in which 40 percent or more of the students are from low-income families and the families of such students. At the same time, the program gives local grantees some discretion in how their programming targets students who attend these schools.

In general, centers reported that they sought to serve a variety of students and used a mix of criteria to target students for participation. Overall, 85 percent of centers reported that they sought to serve students identified by the school as needing special assistance in reading and/or mathematics, and 76 percent sought to serve students scoring “below proficient” on state tests. Sixty-eight percent of centers sought to serve students who were eligible for free or reduced-price lunches. 

When centers were asked to rank their top three priority groups, the most commonly reported groups were students who scored below proficient on state assessments (44 percent of centers) and students whom the school identified as needing special assistance in reading and mathematics (21 percent of centers). By contrast, a relatively small proportion of centers (2 percent of school-based centers and 3 percent of nonschool-based centers) reported that students who received free or reduced-price lunch were among the top three priority groups. 

Top-priority groups differed in certain respects for school-based and nonschool-based centers (see Exhibit 3-1; see also Exhibit E-7 in Appendix E). Both types of centers targeted students scoring below proficient on state tests, but nonschool-based centers were more likely than school-based centers to target this group (79 percent versus 46 percent). Nonschool-based centers were also more likely than school-based centers to report their top priorities were students whom their teacher or counselor identified as in academic need (88 percent versus 55 percent).

Exhibit 3-1Percentage of Centers That Reported Top-Priority Target Groups, by Target Groups and Center Type

 (Center Type) (Open enrollment for all interested studentsStudents who are recommended by teachers or counselors at the feeder schoolsStudents who receive free or reduced price mealsStudents identified by the school as needing special assistance in reading and/or mathStudents who scored “below proficient” on state assessmentsOther)

Source: Focused Telephone Survey, Item 1.

*n* = 102

*p < 0.05

Exhibit reads: A higher percentage of centers in school-based settings (75 percent) reported they had open enrollment for all interested students than nonschool-based centers (60 percent).

Policies Regarding Attendance 

Data on general attendance policies were collected for all centers, but data on attendance were collected using the focused survey from a random sample of centers sampled for the national survey. The study team asked centers to report attendance only if they had adopted methods for tracking attendance and only if staff kept daily records of individual students’ attendance. Sixty-one percent of the centers tracked attendance in a way that allowed them to report easily (within 1–2 hours’ time) how many days an individual student attended in the past year. 

More than half of all centers (56 percent) in the full sample of centers reported having policies that require students to attend 2 days per week or more to remain enrolled in the program.[footnoteRef:12] Thirty-eight percent of centers required daily attendance, and 26 percent of centers reported that attendance was on a drop-in basis. Centers serving elementary school students were more likely to have policies that required attendance than did those serving older students****(see Exhibit 3-2; see also Exhibit E-8 in Appendix E).****Forty-one percent of centers serving elementary school students required students to attend the program every day services were offered, compared with 31 percent of centers serving students in middle school and 22 percent of centers serving high school students. These data are for the sample that completed the national survey.  [12:  Although requiring attendance was not defined on the survey, respondents may have varied in their interpretations of this question, depending upon the consequences attached to nonattendance; therefore, data are included in a separate question about consequences of nonattendance.] 

Exhibit 3-2 Percentage of Centers That Reported Attendance Requirements, by Grade Level and Attendance Requirement

 (Attendance Requirement)

Note: Percentages may not equal 100 percent due to rounding.

Source: National Survey, Item 10.

*n* = 499

*p < 0.05

Exhibit reads: Forty-one percent of centers serving elementary school students reported that they required the students to attend every day the center-offered services. 

**More than half the centers reported requiring participation for all students in specific academic activities when students were present,**including****homework help (73 percent), reading activities (60 percent), mathematics activities (58 percent) and tutoring (14 percent) (see Exhibit 3-3; see also Exhibit E-9 in Appendix E). Programs serving high school students were less likely than those not serving this age group to require participation in these activities. A higher proportion of school-based centers reported that they required participation in reading activities (10 percent, compared with 2 percent of nonschool-based centers) for all students attending the center. 

Exhibit 3-3Percentage of Centers That Reported That They Required Participation of All Students in Specific Activities When Students Were Present, by Type of Activity

Source: Focused Telephone Survey, Item 9.

*n* = 114

Exhibit reads: Seventy-three percent of centers reported that homework time was mandatory.

Center directors at case study sites noted that they felt it important to consider carefully what would engage students and encourage them to keep coming to the program. For example, in a Florida center, the director offered engaging recreational activities as a way to entice students to attend the academic components of the program, which were voluntary. “The main issue is if you only offer academics, then your attendance is affected.... We’ve offered a professional football player doing flag football games, and that generated so much interest among kids.... They’re dying to play and consequently are attending after school.”

Regular Attendance

Research on after-school programs generally has found a significant association between regular program attendance and positive student outcomes. For example, participation in after-school activities sponsored by The After School Corporation in New York City was linked to higher grades and increased school attendance, especially for students who participated regularly in the corporation’s programs over 2 consecutive years (Reisner, et al., 2004). Researchers studying attendance in before- and after-school programs have argued that it is important for evaluation studies to distinguish among intensity, duration and breadth of student participation (Chaput, et al., 2004). Intensity refers to how often students participate in a program during a given time period. Duration refers to the length of time (in days, months or years) students participate in programs. Other studies have found a relationship between regular attendance in after-school programs and educational attainment in high school (Fredricks and Eccles 2006), in-class attendance (Grossman et al., 2002; Huang et al., 2000) and reading and mathematics achievement (Lauer et al., 2006). 

Although most research tends to focus on just one aspect of participation, each type of measure has been associated with improved outcomes for youth. For example, in their evaluation study of two after-school programs, Baker and Witt (1995) found a relationship between intensity of participation and a variety of academic outcomes (see also Huang, et al., 2000). Duration appears to be particularly significant for arts-related programming, where interactions may not be intense but take place over many years (Catterall, 1998). A study examining the relationship between attendance and academic achievement of students participating in Los Angeles’ BEST after-school program found that elementary school students attending program activities for 50 or more days throughout the school year demonstrated positive academic achievement growth (Huang, et al., 2008). Other researchers have studied how attendance in after-school programs, in combination with other home-based and community activities, is linked to positive behavioral and academic outcomes (Vandell, Reisner and Pierce, 2007). More attendance is not always better, however, as Lauer and colleagues (2006) reported in their synthesis: Students who received more than 210 hours of instruction did not show gains in reading.

As stated earlier, 21st Century Community Learning Centers operate before and after school as well as during the summer. According to Naftzger and colleagues (2006), centers were open an average of 14.2 hours per week during the school year and 23.5 hours per week during the summer. On average, centers were open for 150 days during the school year. 

Regular attendance has been a concern within the 21st CCLC program, especially since the publication of the first national evaluation (Dynarski, et al., 2003). That study found that many programs had lower-than-expected attendance: 55 percent of students in the elementary school treatment group of this evaluation attended 51 or more days per year, and 20 percent of middle school students in the treatment group attended that often.

Although this study sought to better understand the nature of attendance in centers, the data collected from this study may be of limited utility in addressing concerns about attendance raised within the 21st CCLC program, since it was not collected in the same way as the national evaluation and did not distinguish among different types of programs (summer, before- or after-school programs). 

**Of the centers that reported on attendance in the current study (approximately 61 percent of centers surveyed), 44 percent of students attended 60 days or more in the last year.** Centers serving elementary school students reported higher attendance than did centers serving middle and high school students (see Exhibit 3-4; see also Exhibit E-10 in Appendix E). Centers serving elementary school students reported that 48 percent of students attended 60 days or more in the last year, which amounts to approximately 2 days a week. Centers serving middle grades reported 36 percent of students attended this often (60 days or more), and centers serving high school students reported 30 percent of their students met this standard. In the previous evaluation, the average elementary student attended programming 2 to 3 days a week or 56 days per year during the year of the study (Dynarski et al., 2004), and the typical middle school student attended 1 day a week, or 32 days a year (Dynarski et al., 2003). The results of this study cannot be compared directly with the results of the previous evaluation because the two studies did not measure attendance using the same method.[footnoteRef:13]  [13:  Data collection methods varied between this study and the previous study. The previous national evaluation collected program attendance records of individual students across 2 years, while the current study relied on survey reports of attendance during 1 year by program directors. Hence, the attendance rates in the previous evaluation are likely to be more accurate than the self-reported attendance rates reported in this study.] 

Exhibit 3-4Percentage of Centers That Reported the Number of Days of Attendance in 2005–06, by Grade Level and Days of Attendance

Note: Percentages may not equal 100 percent due to rounding.

Source: Focused Telephone Survey, Item 8.

*n* = 75

Exhibit reads: Centers serving elementary school students reported that 17 percent of students attended 120 days or more.

**The proportion of students attending 60 days or more was higher in school-based centers than in nonschool-based centers (52 percent versus 44 percent).**A third of students in school-based centers serving elementary school students attended fewer than 30 days, compared with 14 percent in nonschool-based centers serving elementary school students. About half of students in school-based centers serving high school students attended fewer than 30 days, compared with 89 percent in nonschool-based centers serving high school students. 

**Overall attendance rates were similar for centers that did and did not require some level of attendance to remain enrolled in the program, but attendance policies were related to attendance in centers serving secondary-level students.**In centers that required some level of attendance, 66 percent of students attended for more than 30 days; 62 percent attended for the same duration in centers that did not require attendance. There was no relation between any kind of attendance policy and actual attendance for centers overall or for centers serving elementary school students. For the middle grades, centers that required attendance every day had higher attendance than did centers with other attendance policies. In high school, centers that required attendance 2 to 3 days per week had higher attendance. 

**Staff members address absences from the programs primarily by talking with parents and students**(see Exhibit 3-5; see also Exhibit E-11 in Appendix E). Fifty-eight percent of staff members reported they notified parents or caregivers if the student did not meet the attendance requirements of the center. Thirty-eight percent said they spoke to students after several absences. Few centers punished students for absences by restricting privileges (6 percent) or by restricting participation in particular activities (10 percent).

Exhibit 3-5Percentage of Centers That Reported Consequences for Student Absences, by Consequence Type

 (A staff member notifies the student’s parents or caregiversA staff member speaks to the student about an absence after a certain number of absencesA staff member speaks to the student about an absence after each absenceNot applicable; we do not require attendanceA staff member notifies the student’s schoolThe student is restricted from participating in other activities at the center for a period of timeThe student receives “points” or some other kind of penalty that restricts privileges at the center)

Source: National Survey, Item 11.

*n*= 503

Exhibit reads: Fifty-eight percent of centers reported notifying parents when a student is absent from the program.

Student Engagement in Observed Activities

Beyond attendance, learning requires student engagement in after-school activities (Chaput et al., 2004; Mahoney et al., 2007). Cognitive engagement in learning tasks is closely related to self-regulation in learning: by engaging with classroom activities, students are better able to track new information and assimilate feedback from instructors (Corno and Mandinach, 1983). Engagement is also related to students’ beliefs about the value of effort in learning (Mahoney et al., 2007; Pintrich and Schrauben, 1992). Motivation and engagement are important to developing students’ disposition to see mathematics as valuable and worthwhile (National Research Council, 2001) and in developing students’ reading skills (National Reading Panel, 2000). 

**Observers in the case studies rated students as “on task” in most activities in reading, but not in mathematics.**In 80 percent of reading activities, observers judged students to be on task and engaged in those activities, compared with 42 percent of mathematics activities. 

Summary

Centers reported seeking to serve low-income students and students who scored low on state assessments or whom instructors judged to need special assistance in reading and mathematics. 

Centers’ reports on student attendance revealed a familiar pattern. On average, elementary schools attended programming more than middle and high school students did. With respect to regular attendance, this evaluation found that, on average, elementary students attended programming more than middle and high school students did. These findings replicated a general trend shown in other studies (Borman and Dowling, 2006; Grossman et al., 2002; Reisner et al., 2004; Simpkins et al., 2005): Elementary school students attended after-school activities for more days than did middle and high school students. These data underscore the challenge centers face in encouraging and sustaining regular student attendance. 

Engagement was generally high in reading activities but lower in mathematics activities. These data help underline the differences between these two subjects. Observers in the case studies rated students as “on task” in most activities in reading, but not in mathematics. Lower engagement in mathematics activities may be related to the narrower range of mathematics skills targeted by these activities. 

3. Student Targeting and Attendance	40

4. Organizational Supports for Instructional Quality

Key Findings

*Center staffing.* Research on staff compensation suggests that adequate pay and benefits can help attract and retain qualified staff to support high-quality after-school programming (Wechsler et al., 2001). Additionally, it is important for after-school programs to provide sufficient staff development to enhance staff skills and maximize their expertise relevant to their roles (Crollick et al., 2005). Centers relied primarily on part-time staff, who were unlikely to receive benefits as a result of their work in the center. This is not surprising, since centers offered programming 16 hours per week, on average. It is also possible that some of these part-time staff were employed full-time as teachers or paraprofessionals at the time of the study. The majority of centers reported providing paid professional development opportunities to full-time and part-time staff. Across all centers, 29 percent of the staff had worked at the center for less than 1 year. About half of centers (48 percent) reported that the primary reason for staff turnover was graduation from school or completion of a program of study.

*Alignment of activities with student needs.* After-school programs provide an opportunity to extend students’ learning beyond the school day, but the effectiveness of that help depends, in part, on the quality of the relationships between adults and students (Birmingham et al., 2005; Carruthers and Busser, 2000; Grossman et al., 2002; Huang et al., 2000; Intercultural Center for Research in Education & National Institute on Out-of-School Time, 2005; Pierce, Hamm and Vandell, 1999). Centers reported using state assessment data to assess student growth and to identify different strategies for matching activities intended to meet student needs. About one-quarter of the centers coordinated activities with supplemental educational services (SES) providers; 9 percent reported coordinating schedules and academic activities with providers. More than two-thirds of centers reported having access to whole-school achievement data for reading and mathematics. 

*Coordination with the school day.*High-quality instruction is aligned with student needs; however, to understand student needs, after-school programs need to be well coordinated with school-day activities and use data from student assessments to adjust program offerings (de Kanter, et al., 2003; Noam, 2003). Respondents from nonschool-based centers were more likely than school-based centers to report challenges aligning activities with the school day because they lacked information about students’ needs and because school staff were not responsive to the centers’ requests. 

Center Staffing 

Recruiting and Retaining High-Quality Staff

An important role for center directors in 21st Century Community Learning Centers programs is recruiting and retaining high-quality staff with expertise matched to their roles (Crollick et al., 2005). Past research has focused primarily on the role of competitive salaries in retaining staff (Halpern, Barker and Mollard, 2000). This section focuses on both staff pay and benefits at centers and on center directors’ perceptions about the relationship between staff pay and benefits and staff turnover in centers. 

**About half of centers paid staff with current teaching certificates $15 to $25 per hour (54 percent) and staff without current teaching certificates $10 to $15 per hour (52 percent)**(see Exhibit 4-1; see also Exhibit E-12 in Appendix E). Currently certified teachers who worked in nonschool-based sites were paid less than staff without teaching certificates working in school-based settings, but pay for staff without teaching certificates was comparable across center types. Data from the Bureau of Labor Statistics (Occupational Online Network, 2008) indicate that the median wage for paraprofessionals working in schools in 2008 was approximately $11.45 per hour (based on an annual salary of $22,500). Rates are not likely to be comparable across different regions of the country; however, no objective data are readily available about how competitive wages are for center staff.

Exhibit 4-1 Percentage of Centers That Paid Hourly Rates to Staff Who Led Instructional Activities, by Type of Staff and Rates 

 (Pay Rate) (Pay Rate)

Note: Percentages may not equal 100 percent due to rounding.

Source: National Survey, Item 27.

*n* = 453

Exhibit reads: One percent of all centers paid staff who are certified teachers less than $10 per hour.

**Seventy-six percent of staff in all centers reported working fewer than 20 hours per week at the center**(see Exhibit 4-2; see also Exhibit E-13 in Appendix E).****Ten percent of staff members offering academic instruction worked in full-time positions. Another 14 percent worked half time or more. Nonschool-based centers were more likely to employ staff in half-time (21 percent) or full-time (34 percent) positions, compared with school-based centers (11 percent half-time and 8 percent full-time). Centers serving high school students were more likely to have staff members who worked full-time than were centers not serving this age group. Center staff could be employed in other jobs, such as working as teachers during the school day.

Exhibit 4-2Percentage of Staff Who Led Instructional Activities at Centers, by Hours Worked per Week

 (Hours Worked Per Week)

Note: Percentages may not equal 100 percent due to rounding.

Source: Focused Telephone Survey, Items 26, 32 and 38.

*n* = 349

Exhibit reads: Ten percent of staff worked full-time at the center (more than 35 hours per week).

**About half of centers reported offering some type of benefits to either full- or part-time staff members.**The most commonly offered benefits related to health insurance (offered to full-time staff in 43 percent of centers and to part-time staff in 8 percent of centers) and retirement plans (offered to full-time staff in 41 percent of centers and part-time staff in 14 percent of centers) (see Exhibit 4-3; see also Exhibit E-14 in Appendix E). About half of centers reported that they did not offer health insurance (49 percent) or paid time off for vacation and sick leave (49 percent) to any staff members. Nonschool-based centers were more likely than school-based centers to provide their full-time staff with health insurance (66 percent versus 40 percent), a retirement savings plan (59 percent versus 39 percent) and paid time off for vacation and sick leave (72 percent versus 37 percent). Centers serving high school students were more likely to provide their full-time staff with health insurance and paid time off than centers not serving this age group, but less likely to provide this benefit to their part-time staff than centers not serving high school students. Part-time staff at centers serving high school students were also less likely than staff at centers not serving this age group to receive tuition reimbursement as a benefit. Finally, centers serving middle school students were more likely than centers not serving this group to provide health insurance to their part-time staff.

Exhibit 4-3Percentage of Centers That Reported Offering Benefits to Full- and Part-Time Staff Who Led Instructional Activities, by Benefit Offered and Staff Status

 (Staff Status)

Source: National Survey, Item 28.

*n* = 505

Exhibit reads: Forty-three percent of centers offered health insurance to full-time staff. 

**Over one-quarter of staff members had been in their positions for less than 1 year.** Across all centers, 29 percent of the staff had worked at the center for less than 1 year.[footnoteRef:14]  [14:  Since the National Survey Item 29 asked only about “all staff,” the tenure for certain types of staff may vary.] 

**The most commonly reported reasons for staff turnover were graduation from school or completion of a program, but staff also left for lack of benefits and because centers were unable to offer a full-time position.**More nonschool-based centers than school-based centers said that staff left the program because of the lack of benefits (44 percent versus 25 percent) and because of low pay (18 percent versus 8 percent) (see Exhibit 4-4; see also Exhibit E-15 in Appendix E). 

Exhibit 4-4Percentage of Centers That Reported Reasons for Staff Departures, by Reason for Departure and Center Type

 (Types of Centers)

Source: National Survey, Item 30b.

*n* = 466

*p < 0.05

Exhibit reads: Forty-nine percent of school-based centers reported that the primary reason why staff left was because they had completed a program (e.g., AmeriCorps) or graduated from school compared with 42 percent of nonschool-based centers. 

Professional Development 

Ensuring staff quality in schools is widely believed to require ongoing professional development in evidence-based practice (National Reading Panel, 2000; National Research Council, 2001). Similarly, researchers of after-school programs have argued that professional development is necessary for staff to learn principles of youth development (Huebner, Walker and McFarland, 2003), develop skills for building positive relationships with youth (Carruthers and Busser, 2000) and acquire strategies for supporting youth skill building (Carruthers and Busser, 2000). For programs seeking to enhance students’ academic achievement, professional development activities need to be aligned with student needs identified by school-day staff and administrators (Huang, 2001). An analysis of The After School Corporation’s most promising programs with respect to youth outcomes indicated that those programs all had a leader who provided ongoing opportunities for professional development to staff members (Birmingham et al., 2005).

**The majority of centers reported providing paid professional development opportunities to full-time and part-time staff**(see Exhibit 4-5; see also Exhibit E-16 in Appendix E). Professional development often included conference attendance paid for by the center (53 percent of centers for full-time staff and 39 percent of centers for part-time staff). Centers were more likely to pay for training or professional development to full-time staff (62 percent) than to part-time staff (55 percent). At the same time, for both full-time and part-time staff, school-based centers were less likely than were nonschool-based centers to pay for attendance at conferences. Centers serving students in the middle grades were less likely to pay for attendance at conferences to their part-time staff than were centers not serving this age group.

Exhibit 4-5Percentage of Centers That Reported That They Offered Paid Professional Development Opportunities for Full- and Part-Time Staff Who Led Instructional Activities, by Staff Status and Type of Development

 (Types of Benefits)

Source: National Survey, Item 28. 

*n* = 505

Exhibit reads: Sixty-two percent of centers reported that they offered paid professional development to full-time staff. 

**Site visit data suggest that professional development offered by centers may have focused on nonacademic areas.** Four of 12 case study centers provided just 1 to 2 days of training or orientation to staff members, and that training focused primarily on logistics and program implementation. At first glance, these findings appear to contradict the survey findings, but the survey did not include questions about the content of the professional development. Additionally, the case study data are not nationally representative.

Among the case study sites, few centers reported that they offered professional development to staff, beyond that which was available to regular school-day staff members. Center directors in the case studies said they relied on the professional development opportunities their instructional staff received through the district, and they thought additional professional development specific to the center’s program might be a burden for staff. For example, one director noted that most of the center’s instructional staff were “already certified instructors who have to go to so many workshops that we don’t feel the need to give them more professional development.” During site visits, directors reported that rather than participating in formal professional development workshops or conferences, staff were encouraged to share their ideas for innovative programming in weekly staff meetings.

Alignment to Student Needs

Coordination With the School Day

A central challenge for centers is to coordinate their activities with the school-day instructional program in ways that are aligned with student needs and with the school-day curriculum (de Kanter et al., 2003; Noam, 2003). Studies of after-school programs show that when programs align their activities to complement school-day activities, they can contribute to student success (Little, 2006). At the same time, studies also indicate that such coordination can be difficult, especially when programs are not based in schools, because the programs may have limited access to the curricular materials used in the school day and to school-day staff who could inform them about particular students’ progress and challenges. 

**More than two-thirds of the centers (71 percent) had access to whole-school state assessment**data in reading, and 69 percent had access to this information for mathematics.** Of those centers that received these data, 83 percent also had access to individual students’ level of proficiency on state assessments.

**Significantly more school-based centers than nonschool-based centers had access to whole-school data.**More school-based centers reported having access to state assessment data at the school level on reading (73 percent) and mathematics (71 percent) than did nonschool-based centers in those two subject areas (53 percent and 52 percent, respectively) (see Exhibit 4-6; see also Exhibit E-17 in Appendix E). Centers did not differ by type with respect to their access to individual-level data, however. 

Exhibit 4-6Percentage of Centers That Reported That State Assessments Data Was Available F for Use in Tracking Student Outcomes, by Subject Tested and Center Type

 (Types of Centers)

Source: National Survey, Item 13.

*n* = 467

*p < 0.05

Exhibit reads: Seventy-three percent of school-based centers reported that they had access to test results from No Child Left Behind (NCLB) state reading assessments for the whole school.

Use of Student Assessment Data 

Instructors should be able to assess the needs of individual students and tailor instruction to meet specific needs (National Reading Panel, 2000). Likewise, in mathematics, emphasis should be on assessing and engaging students’ prior knowledge (National Research Council, 1999, 2005). In terms of providing quality programming, a key use of assessment data, then, is to adjust instructional practice and design new offerings that better meet identified student needs. 

**Center directors reported using several sources of information to assess student academic growth, make program adjustments or evaluate program success.**Almost half of the centers reported using results from tests administered at students’ schools to assess students’ academic growth in the 21st CCLC program. More centers reported using informal staff observations (28 percent of centers reported using them almost daily) and verbal reports from teachers (13 percent reported using them almost daily) to assess student progress with greater frequency than more formal assessments. Only 6 percent of centers reported that they used results from tests administered by center staff or student self-assessments (17 percent) at least weekly to assess students or evaluate program success (see Exhibit 4-7; see also Exhibit E-18 in Appendix E). 

Exhibit 4-7Percentage of Centers That Reported Student Assessment Sources, by Type and Frequency of Assessment

*(Frequency of Assessment)* (Informal staff observationsVerbal reports from teachersStudent self-report or self-assessmentInformal discussions with family membersWritten reports from teachers Results from tests administered at students’ schools Results from tests administered by center staff)

Note: Percentages may not equal 100 percent due to rounding.

Source: National Survey, Item 15.

*n* = 502

Exhibit reads: Twenty-eight percent of centers reported using informal staff observations on an almost daily basis.

**More than half of the centers reported using assessments of students’ academic growth for at least one of the following purposes: to identify new offerings (59 percent) or improve practices in existing program offerings (69 percent), to assign students to particular staff members (54 percent) and to evaluate the success of their programs (66 percent)**Two-thirds of centers reported using assessment information to improve practices in existing program offerings (see Exhibit 4-8; see also Exhibit E-19 in Appendix E). 

Nonschool-based centers were significantly more likely than school-based centers (67 percent versus 52 percent) to report using formative assessment data collected by the center to assign students to particular staff members who could meet their specific academic needs. For example, at one site visited, staff used data on students’ mathematics assessments administered by the center to assign them to mathematics tutors who were also certified instructors. Centers serving elementary school students were significantly less likely than were centers not serving this age group to use center-administered assessment data to identify new program offerings (56 percent of centers serving elementary school students versus 60 percent of centers serving middle school students and 64 percent of centers serving high school students).

Exhibit 4-8Percentage of Centers That Reported Uses of Student Assessment Data, by Type of Use

Source: National Survey, Item 16.

*n* = 516

Exhibit reads: Sixty-nine percent of school-based centers reported using students’ assessments of academic growth to improve existing program offerings. 

Case study data illustrate ways that programs used data to target instruction. For example, one Florida grantee used extensive student-level data to track student progress and growth, including results from the two benchmark assessments administered by the district. Staff in this center looked at the subscale scores to identify areas of weakness in students’ skills and determine which skills to target during the after-school program. Similarly, a North Carolina grantee used the pre- and post-test data associated with the reading and mathematics intervention curricula provided by the center to monitor student progress from week to week. The center leadership worked with staff members to identify the trends in these data and adjust instruction to meet students’ needs. 

Beyond the use of individual assessment to target specific instruction to students, the case study data support the survey findings that staff in some centers used data for program-level improvement. For example, one Maryland grantee employed an onsite evaluator to monitor program activities. That program evaluator used a narrative reporting form to document weekly program visits, as well as surveys of parents, students and classroom instructors. The evaluator then used the data collected by these means to “zero in on the [progress related to] homework and academic performance and to assess how students are getting along” in order to “analyze it by school and communicate it back to the coordinators at each site.” On the basis of this feedback, the coordinators adjusted programming activities, targeting students in need. 

Use of data by some centers in the case studies was more limited. For instance, a site in South Dakota made little use of student assessment data. According to the center director, staff at this nonschool-based site did not use data in their decision-making processes. All data collected were used for public relations or for reporting to grantee agencies. The outreach director reported that he did not find the data collected to be useful: “I found next to no useful information from that evaluation.... For me, the purpose of evaluation is to make the program better, and if I’m not going to learn how to make the program better, I don’t need it.”

**Despite the achievement data available to many centers,****about a third of centers reported lacking information about students’ academic needs, a lack of responsiveness by school staff to center requests and a lack of information about the school’s curriculum as major challenges in implementing high-quality programming.**Nonschool-based centers were more likely to report these challenges than were school-based centers****(see Exhibit 4-9; see also Exhibit E-20 in Appendix E).****A significantly higher percentage of nonschool-based center directors (36 percent) reported they had difficulty obtaining information about student needs from schools than did directors of school-based centers (20 percent). Similarly, 39 percent of nonschool-based centers reported a lack of responsiveness by school staff to their requests, compared with 29 percent of school-based centers. Directors of centers serving elementary school students reported less difficulty getting information about student needs and the school’s curriculum than did directors of centers serving older groups. Difficulties getting this information were greater among centers serving middle school students than they were for centers serving other age groups.

Exhibit 4-9Percentage of Centers That Reported Challenges in Implementing High-Quality Programming, by Type of Challenge and Center Type

 (Lack of information about students' academic needsLack of responsiveness by school staff to center requestsLack of information about the school-day curriculum) (Types of Centers)

Source: National Survey, Item 31.

*n* = 499

*p < 0.05

Exhibit reads: Twenty percent of school-based centers cited lack of information about students’ academic needs as a challenge to implementing high-quality programming.

Adult Support: Fostering Positive Relationships Between Adults and Students

One of the most consistent findings about after-school programs is that youth engagement in activities depends on positive relationships between adult staff and participants (Birmingham et al., 2005; Carruthers and Busser, 2000; Grossman et al., 2002; Huang et al., 2000; Intercultural Center for Research in Education & National Institute on Out-of-School Time, 2005; Pierce et al. 1999). The after-school programs that researchers identified as more successful in offering quality programming hire staff who are able to create warm and friendly relationships with youth, and provide a challenging and motivating intellectual environment for them (Grossman et al., 2002). Positive staff relationships with students are key to retaining students in a program, especially among older youth (Roffman, Papagano and Hirsch, 2001), and one research study has linked such positive relationships to positive behavioral adjustment in school (Pierce et al., 1999). See Appendix F for the observation form that describes the dimensions of adult support.

In the large majority of observed activities in reading and mathematics (*n* = 59), staff used positive behavior management strategies and showed warm affect toward students, but infrequently sought to determine students’ concerns and ideas. In 88 percent of reading activities and 84 percent of mathematics activities, staff were observed using positive behavior management strategies, such as reinforcement of good behavior (see Exhibit 4-10). However, instructors encouraged students to share their ideas in only 30 percent of reading activities and 21 percent of mathematics activities.

Exhibit 4-10Percentage of Observations in Reading and Math Activities That Demonstrated Student-Staff Relationship-Building Efforts, by Relationship-Building Effort and Type of Activity

 (Type of Activity) (Use positive behavior management techniquesShow positive affect toward studentsAttentively listen to and/or observe studentsEncourage students to share their ideas, opinions, and concerns)

Source: Site visit observations.

*n* = 40 for reading; *n* = 19 for mathematics

Exhibit reads: In 88 percent of observed reading activities, staff used positive behavior management techniques.

Coordination with Supplemental Educational Services

To strengthen their links with schools, leverage their resources and enhance their services, some 21st Century Community Learning Centers coordinated with or became SES providers. 

**Fifteen percent of centers were authorized to provide supplemental educational services.** Each center providing SES did so for an average of 38 students in the core subjects of reading and mathematics while, according to the PPICS database, delivering services for a median of 100 students across all subjects and activities, including reading and mathematics. Just 9 percent of all centers reported coordinating with one or two providers, and only 5 percent of all centers reported that their coordination activities focused on aligning schedules with the providers, while 4 percent indicated they coordinated their academic support activities with the providers. 

The vast majority of instructional practices reported for centers providing SES and for centers not providing SES were similar. However, SES-providing centers responded differently regarding two practices in mathematics and one in tutoring than centers that were not providers. The SES providers were less likely than non-SES providers to have students practice basic mathematics operations in all or most activities (46 percent versus 68 percent). Furthermore, SES-providing centers were less likely to have students solve mathematical equations in some activities than non-SES providers (59 percent versus 85 percent). However, SES providers were more likely to offer 2–4 hours of tutoring weekly than non-SES providers (56 percent versus 36 percent). 

Summary 

Instructional supports for program quality were more widely available in school-based centers than in nonschool-based centers. School-based centers paid staff who held teaching certificates more than did nonschool-based programs. About one-third of the staff had joined the center within the past year. Low pay and the inability to offer staff full-time positions were greater challenges to retention in nonschool-based centers than in school-based centers. With respect to coordinating their activities with the regular school day, nonschool-based centers faced greater barriers to obtaining information about student needs than did school-based centers. This is not surprising, given the beneficial location that school-based centers possess. 

Professional development was not emphasized in the centers. Although more than half of all centers reported that they offered paid professional development opportunities to their staff, site visit data suggest that those offerings were limited in time and scope. Four of 12 case study centers provided just 1 to 2 days of training or orientation to staff members, and the training focused primarily on logistics and program implementation. Content-focused professional development is important because it has been linked, in correlational studies, to changes in teacher knowledge and practice (Garet et al., 2001). 

It was significant that half the centers reported using some form of student- or school-level data to improve program practice, given increasing attention to the use of data to monitor progress and enhance the quality of before and after-school programs (Yohalem, Wilson-Ahlstrom and Yu, 2005). The logic behind these efforts is straightforward: With greater accountability for results, educators pay more attention to those results and to their significance for ongoing efforts to improve practice. Research in this area, however, is sparse and has not been linked to outcomes. 

In nearly all observed activities in reading and mathematics (*n* = 59), staff used positive behavior management strategies and showed warm affect toward students. Such high-quality relationships are important because they are linked to greater student engagement with after-school programming activities and positive youth outcomes, including improved behavior in school (Pierce et al., 1999).

4. Organizational Supports for Instructional Quality	54

5. Conclusions and Implications for Future Research and Evaluation 

Nature and Quality of Reading and Mathematics Activities

Considering that the previous national evaluation noted the centers’ limited focus on academics (Dynarski et al., 2003; Dynarski et al., 2004), the fact that nearly all the centers funded at the time of this study reported providing activities in reading and mathematics may be regarded as a significant shift in approach. The center directors’ responses indicated that reading activities were aligned with scientifically based practices: Instruction focused on a broad range of skills and employed several strategies for teaching vocabulary, fluency and comprehension. Additionally, many centers reported that they asked students to talk or write about something they had read.

Survey findings and observations indicated that mathematics activities may be more limited than reading activities. Mathematics instruction focused primarily on two content strands and engaged students in tasks with low levels of cognitive complexity. Observation data, while not nationally representative, suggest that a more limited range of strategies was used in mathematics instruction. Furthermore, in the mathematics activities observed on site visits, students were less likely to be on task than were students in reading activities.

Attendance and Student Participation

With respect to regular attendance, centers in a nonrepresentative sample reported that 48 percent of students in centers serving elementary school students attended 60 days or more in the last year, or roughly 2 days per week. In this study, 36 percent of middle school students attended 60 days or more in the last year. Twenty-eight percent of middle school students in this study attended 30–60 days and 36 percent attended fewer than 30 days. In the earlier evaluation, 20 percent of middle school students attended 51 days or more. Attendance was higher than reported in the earlier study for school-based centers but not for nonschool-based centers. These data also replicated a general trend found in the other studies: Elementary students attend after-school activities for more days in the school year than do middle and high school students. These data are of limited utility, however, compared with other studies that measured attendance directly and distinguished among attendance in different types of programs (summer, before and after school). 

Organizational Supports for Instructional Quality

Two ways that centers’ organizational practices strongly supported instructional quality were their use of data and positive staff–student relationships. About half of all centers reported making use of data for a variety of purposes, including program evaluation and ongoing review of programming activities. Site visitors also observed that staff relationships with students were characterized by high levels of warmth and positive behavior management strategies.

At the same time, professional development was an area that needed greater attention. Although more than half of all centers reported offering paid professional development opportunities to their staff, site visit data suggested that, in some centers, those opportunities were limited in time and scope. 

Centers faced three additional challenges to offering high-quality reading and mathematics activities: (a) low pay and lack of benefits were barriers to retaining high-quality staff; (b) centers did not always have the information they needed about students’ needs and (c) attendance policies did not necessarily result in better attendance rates. 

Key Differences Between Different Types of Centers

There were few reported differences between school-based and nonschool-based centers’ math and reading instruction, except that school-based centers were generally more likely to report emphasizing higher-order skills. School-based centers were more likely than nonschool-based centers to report that their instructors asked students to make predictions about something they were reading, and answer questions, orally or in writing, on something they had read. At the same time, nonschool-based centers had students read teacher-selected books more often than school-based centers did. In mathematics as well, school-based centers were more likely than nonschool-based centers to report emphasizing advanced topics. School-based centers were also significantly more likely than nonschool-based centers to ask students to make charts, graphs and tables in all or most activities.

One of the key differences between school-based and nonschool-based centers was the level of instructional support for program quality. School-based centers paid staff with teaching certificates at higher rates than nonschool-based programs did. For example, school-based centers reported that low pay and lack of benefits were less important in retaining staff than they were for nonschool-based centers.. With respect to coordinating their activities with the regular school day, nonschool-based centers faced greater barriers to obtaining information about student needs and performance than did school-based centers. These barriers present formidable challenges to nonschool-based centers in partnering with the school-day instructional staff to deliver high-quality programming to address individual student needs.

Significant differences existed in attendance policies and patterns of attendance based on the grade level of students served. In centers serving the elementary grades, students were more likely to attend regularly; these centers were also more likely to have policies requiring students to attend the program for a minimum number of days to remain enrolled. In contrast, centers serving high school students were more likely to have a high proportion of students who attended fewer than 30 days in 2005–06 than centers not serving that age group. At the same time, centers that served middle and high school grades and that had attendance policies reported higher levels of attendance than did centers serving these grades that allowed students to participate on a drop-in basis.

Recommendations for Future Research and Evaluation

Recommended areas of future study include analyzing the role of instructors’ qualifications, the quality of reading programming and students’ and families’ needs for after-school programs. Finer-grained measures of instructor qualifications may prove more effective in predicting differences in instructional quality than the broader measure of teacher certification or specialist status. Measures of instructors’ relative confidence in teaching reading and mathematics and their knowledge of mathematics teaching may explain these differences (Hill, Rowan and Ball, 2005). The current study did not directly measure instructors’ knowledge or skills, but future studies could examine those aspects. The content of the professional development provided for center staff could also be considered as part of instructors’ qualifications. Future work examining the quality of reading programming could focus on how best to capture information on reading when respondents are after-school program administrators (rather than school-day administrators) who may be less familiar with specific instructional strategies used in academic programming. Future studies concerned with student and family demand for after-school programs could collect information on program enrollment capacity and waiting lists, as well as reasons why parents or students chose to enroll in after-school programs. Effective strategies that centers implement to increase participation in their programs could also be analyzed in this regard.

Last, because all students who take part in 21st Century Community Learning Centers programs also participate in learning experiences at home and at school, and some also participate in other after-school programs, longitudinal studies that follow participants over an extended period could provide a better understanding of how these learning experiences may complement one another and enhance academic achievement. Emerging frameworks (e.g., Barron, 2006) within the learning sciences that highlight the need to understand how activities, resources and interactions outside of school relate to learning within schools could guide such a study. Studying how learning experiences in one setting lead students to pursue learning in another setting could be the focus of a specific longitudinal study. Connections might be found at 21st Century Community Learning Centers programs with a strong focus on developing interest, knowledge and skill in a particular area and strong connections to the school day and other service providers. A 3- to 5-year study of students who participate in such programs could yield important new insights into how 21st Century Community Learning Centers programs can enhance outcomes for students.

5. Conclusions and Implications for Future Research and Evaluation	58

References 

Aronson, J., Zimmerman, J., & Carlos, L. (1999). *Improving student achievement by extending school: Is it just a matter of time?* San Francisco: WestEd.

Baker, D. A., & Witt, P. A. (1995). Evaluation of the impact of two after-school programs. *Journal of Park and Recreation Administration,*14(3), 60–81.

Bangert-Downs, R. L., Kulik, C.-L. C., Kulik, J. A., & Morgan, M. T. (1991). The instructional effect of feedback in test-like events. *Review of Educational Research,*61(2), 213–238.

Barron, B. J. S. (2006). Interest and self-sustained learning as catalysts of development: A learning ecology perspective. *Human Development,*49, 193–224.

Beck, I., McKeown, M., & Kucan, L. (2002). *Bringing words to life*. New York: Guilford.

Biancarosa, C., & Snow, C. E. (2006). *Reading next—A vision for action and research in middle and high school literacy: A report to Carnegie Corporation of New York*. Washington, DC: Alliance for Excellent Education.

Birmingham, J., Pechman, E. M., Russell, C. A., & Mielke, M. (2005). *Shared features of high-performing after-school programs: A follow-up to the TASC evaluation*. Washington, DC: Policy Studies Associates.

Borman, G. D., & Dowling, N. M. (2006). Longitudinal achievement effects of multiyear summer school: Evidence from the Teach Baltimore randomized field trial. *Educational Evaluation and Policy Analysis,*28(1), 25–48.

Brophy, J., & Good, T. L. (1986). Teacher behavior and student achievement. In M. C. Wittrock (Ed.), *Handbook of research on teaching* (3rd ed., pp. 328–375). New York: Simon & Schuster.

Brown, C. (1999). The role of schools when school is out. *The Future of Children,*9(2), 139–142.

Burstein, L., McDonnell, L., Van Winkle, J., Ormseth, T., Mirocha, J., & Guiton, G. (1995). *Validating national curriculum indicators.*Report prepared for the National Science Foundation. Santa Monica, CA: RAND. 

Burton, S., & Blair, E. (1991). Task conditions, response formulation processes, and response accuracy for behavioral frequency questions in surveys. *The Public Opinion Quarterly,*55(1), 50–79.

Butler, D. L., & Winne, P. H. (1995). Feedback and self-regulated learning: A theoretical synthesis. *Review of Educational Research,*65, 245–281.

Butler, R. (1987). Task-involving and ego-involving properties of evaluation: Effects of different feedback conditions on motivational perceptions, interest, and performance. *Journal of Educational Psychology,*79(4), 474–482.

Butler, R., & Neuman, O. (1995). Effects of task and ego-achievement goals on help-seeking behaviours and attitudes. *Journal of Educational Psychology,*87(2), 261–271.

Carpenter, T. P., Fennema, E., Peterson, P. L., Chiang, P. L., & Loef, M. (1989). Using knowledge of children's mathematics thinking in classroom teaching: An experimental study. *American Educational Research Journal,*26(4), 499–531.

Carruthers, C. P., & Busser, J. A. (2000). A qualitative outcome study of Boys and Girls Club program leaders, club members, and parents. *Journal of Park and Recreation Administration,*18(1), 50–67.

Catterall, J. S. (1998). *Involvement in the arts and success in secondary school*. Washington D. C., Americans for the Arts.

Chaput, S. S., Little, P. M. D., & Weiss, H. B. (2004). *Understanding and measuring attendance in out-of-school time programs*. Cambridge, MA: Harvard Family Research Project.

Cohen, D., Raudenbush, S., and Ball, D. (2003). Resources, Instruction, and Research *Educational Evaluation and Policy Analysis*, 25, 119-142.

Corno, L., & Mandinach, E. B. (1983). The role of cognitive engagement in classroom learning and motivation. *Educational Psychologist,*18(2), 88–108.

Coyne, M. D., Kameenui, E., Simmons, D. C., & Harn, B. A. (2004). Beginning reading intervention as inoculation or insulin: First grade reading performance of strong responders to kindergarten intervention. *Journal of Learning Disabilities,*37*,* 90–104.

Crollick, J., Zhang, J. J., & Fleming, D. S. (2005, April). Program staffing: A critical element for after school program quality. Paper presented at the 2005 AAHPERD National Convention and Exposition, Chicago.

de Kanter, A., Adair, J. K., Chung, A.-M., & Stonehill, R. M. (2003). Ensuring quality and sustainability in after-school programs: How partnerships play a key role. *Yearbook of the National Society for the Study of Education,*102(2), 201–220.

Dixon, R. C., Carnine, D., Lee, D.-S., Wallin, J., & Chard, D. (1998). *Report to the California State Board of Education and addendum to principal report: Review of high quality experimental mathematics research*. Eugene, OR, and Austin, TX: National Center to Improve the Tools of Educators, University of Oregon and University of Texas at Austin.

Durlak, J. A., & Weissberg, R. P. (2007). *The impact of after-school programs that promote personal and social skills*. Chicago, IL: Collaborative for Academic, Social, and Emotional Learning.

Dynarski, M., James-Burdumy, S., Moore, M., Rosenberg, L., Deke, J., Mansfield, W., et al. (2004). *When schools stay open late: The national evaluation of the 21st Century Community Learning Centers program: New findings*. Washington, DC: Institute of Education Sciences, National Center for Education Evaluation and Regional Assistance, U.S. Department of Education.

Dynarski, M., Moore, M., Mullens, J., Gleason, P., James-Burdumy, S., Rosenberg, L., et al. (2003). *When schools stay open late: The national evaluation of the 21st Century Community Learning Centers program: Year 1 report*. Princeton, NJ: Mathematica Policy Research.

Epstein, J. L. (2008). Research meets policy and practice: How are school districts addressing NCLB requirements for parental involvement? In A. R. Sadovnik, J. O'Day, G. W. Bohrnstedt & K. Borman (Eds.), *No Child Left Behind and the reduction of the achievement gap: Sociological perspectives on federal educational policy* (pp. 261-280). New York: Routledge.

Fleming, D. S., & Zhang, J. J. (2005, April). Program content and operation of after school programs. Paper presented at the 2005 AAHPERD National Convention and Exposition, Chicago.

Fredricks, J. A., & Eccles, J. S. (2006). Extracurricular involvement and adolescent adjustment: Impact of duration, number of activities, and breadth of activities. *Applied Developmental Science,*10(3), 132–146.

Gamoran, A., Porter, A. C., Smithson, J. L., & White, P. A. (1998). Upgrading high school mathematics instruction: Improving learning opportunities for low-achieving, low-income youth. *Educational Evaluation and Policy Analysis,*19(4), 325–338.

Garet, M. S., Porter, A. C., Desimone, L., Birman, B. F., & Yoon, K. S. (2001). What makes professional development effective? Results From a national sample of teachers. *American Educational Research Journal,*38(4), 915–945.

Good, T. L., & Brophy, J. (2008). *Looking in classrooms*. Boston: Allyn and Bacon.

Grossman, J. B., Price, M. L., Fellerath, V., Jucovy, L. Z., Kotloff, L. J., Raley, R., et al. (2002). *Multiple choices after school: Findings from the Extended-Service Schools Initiative*. Philadelphia and New York: Public/Private Ventures and Manpower Research and Development Corporation.

Halpern, R. (2002). A different kind of child development institution: The history of after-school programs for low-income children. *Teachers College Record,*104(2), 178–211.

Halpern, R., Barker, G., & Mollard, W. (2000). Youth programs as alternative spaces to be: A study of neighborhood youth programs in Chicago’s West Town. *Youth and Society,*31(4), 469–505.

Herman, J. L., Klein, D. C. D., & Abedi, J. (2000). Assessing students’ opportunity to learn: Teacher and student perspectives. *Educational Measurement: Issues and Practice*, 19(4), 16–24.

Hill, H. C., Rowan, B., & Ball, D. L. (2005). Effects of teachers’ mathematical knowledge for teaching on student achievement. *American Educational Research Journal,*42(2), 371–406.

Hoppe, M., Gillmore, M., Valadez, D., Civic, D., Hartway, J., & Morrison, D. (2000). The relative costs and benefits of telephone interviews versus self-administered diaries for daily data collection. *Evaluation Review,*24(1), 102–116.

Huang, D. (2001). An after-school evaluation system for middle and high school programs. *NAASP Bulletin,*45(626), 45–61.

Huang, D., Gribbons, B., Kim, K. S., Lee, C., & Baker, E. (2000). *A decade of results: The impact of the LA’s BEST after school enrichment program on subsequent student achievement and performance*. Los Angeles: CRESST.

Huang, D., Leon, S., La Torre, D., & Mostafavi, S. (2008). *Examining the relationship between LA’s BEST program attendance and academic achievement of LA’s BEST students*. Los Angeles: CRESST.

Huebner, A. J., Walker, J. A., & McFarland, M. (2003). Staff development for the youth development professional: A critical framework for understanding the work. *Youth and Society,*35(2), 204–225.

Intercultural Center for Research in Education & National Institute on Out-of-School Time. (2005). *Pathways to success for youth: What counts in after-school. Massachusetts After-School Research Study (MARS) report*. Boston: United Way of Massachusetts Bay.

James-Burdumy, S., Dynarski, M., Moore, M., Deke, J., Mansfield, W., Pistorino, C., et al. (2005). *When schools stay open late: The national evaluation of the 21st Century Community Learning Centers program*. Washington, DC: U.S. Department of Education, Institute of Education Sciences, and National Center for Education Evaluation and Regional Assistance.

Koziol, S. M., & Burns, P. (1986). Teachers’ accuracy in self-reporting about instructional practices using a focused self-report inventory. *Journal of Educational Research,*79(4), 205–209.

Lauer, P. A., Akiba, M., Wilkerson, S. B., Apthorp, H. S., Snow, D., & Martin-Glenn, M. L. (2006). Out-of-school time programs: A meta-analysis of effects for at-risk students. *Review of Educational Research*, 76, 275–313.

Leffert, N., Benson, P., & Roehlkepartain, J. (1997). *Starting out right: Developmental assets for children*. Minneapolis, MN: Search Institute.

Little, P. M. D. (2006). Beyond basic training: Promising strategies for connecting out-of-school time programs to schools: Learning what works [Electronic version]. *The Evaluation Exchange*, 12.  Retrieved May 15, 2010 from http://www.hfrp.org/evaluation/the-evaluation-exchange/issue-archive/building-and-evaluating-out-of-school-time-connections/promising-strategies-for-connecting-out-of-school-time-programs-to-schools-learning-what-works

Little, P. M. D. (2007). *The quality of school-age child care in after-school settings*. New York and Ann Arbor, MI: National Center for Children in Poverty, Columbia University, and Institute for Social Research, University of Michigan.

Luce, T., & Thompson, L. (2005). *Do what works: How proven practices can improve America’s public schools*. Dallas, TX: Ascent Education Press.

Mahoney, J. L., Parente, M. E., & Lord, H. (2007). Afterschool program engagement: Links to child competence and program quality and content. *The Elementary School Journal,*107(4), 385–404.

Marquis, K. H., Marquis, M. S., & Polich, J. M. (1986). Response bias and reliability in sensitive topic surveys. *Journal of the American Statistical Association,*81, 381–389.

Mullens, J. E. (1998, October). Validating teacher responses on instructional practice self-report surveys: Findings from a recent field test. Paper presented at the Association for Public Policy Analysis and Management Fall Research Conference, New York.

Muller, C., & Frisco, M. (1998). Social institutions serving adolescents. In K. Borman & B. Schneider (Eds.), *The adolescent years: Social influences and educational challenges* (pp. 142–159). Chicago: National Society for the Study of Education.

Naftzger, N., Kaufman, S., Margolin, J., & Ali, A. (2006). *21st Century Community Learning Centers (21st CCLC) analytic support for evaluation and program monitoring: An overview of the 21st CCLC Program: 2004–05*. Naperville, IL: Learning Point Associates.

National Mathematics Advisory Panel. (2008). *Foundations for success: The final report of the National Mathematics Advisory Panel.* Washington, DC: U.S. Department of Education.

National Reading Panel. (2000). *Report of the National Reading Panel: Teaching children to read*. Washington, DC: National Institute of Child Health and Human Development, and National Institutes of Health.

National Research Council. (1999). *How people learn: Brain, mind, experience*. Washington, DC: National Academies Press.

National Research Council. (2001). *Adding it up: Helping children learn mathematics*. Washington, DC: National Academies Press.

National Research Council. (2005). *How students learn: History, mathematics, and science in the classroom*. Washington, DC: National Academies Press.

National Research Council & Institute of Medicine. (2002). *Community programs to promote youth development*. Washington, DC: National Academies Press.

Noam, G. G. (2003). Learning with excitement: Bridging school and after-school worlds and project-based learning. In K. J. Pittman, N. Yohalem, & J. Tolman (Eds.), *When, where, what, and how youth learn: Blurring school and community boundaries* (Vol. 97, pp. 121–138). San Francisco: Jossey-Bass.

Oakhill, J., Cain, K., & Bryant, P. E. (2003). The dissociation of word reading and text comprehension: Evidence from component skills. *Language and Cognitive Processes,*18, 443–468.

Occupational Online Network (O*NET) Online Summary Report for: 25-9041.00 - Teacher Assistants,****in Bureau of Labor Statistics 2008 wage data, downloaded on August 13, 2009 from: http://online.onetcenter.org/link/summary/25-9041.00

Pechman, E. M., Mielke, M. B., Russell, C., White, R. N., & Cooc, N. (2008). *Out-of-school time (OST) observation instrument: Report of the validation study*. Washington, DC: Policy Studies Associates, Inc.

Pierce, K. M., Hamm, J. V., & Vandell, D. L. (1999). Experiences in after-school programs and children’s adjustment in first-grade classrooms. *Child Development,*70(3), 756–767.

Pintrich, P. R., & Schrauben, B. (1992). Students’ motivational beliefs and their cognitive engagement in classroom academic tasks. In D. Schunk & J. Meece (Eds.), *Student perceptions in the classroom: Causes and consequences* (pp. 149–183). Hillsdale, NJ: Erlbaum.

Porter, A. C. (2002). Measuring the content of instruction: Uses in research and practice. *Educational Researcher,*31(7), 3–14.

Reisner, E. R., White, R., Russell, C. A., & Birmingham, J. (2004). *Building quality, scale, and effectiveness in after-school programs*. Washington, DC: Policy Studies Associates.

Roffman, J. G., Papagano, M. E., & Hirsch, B. J. (2001). Youth functioning and experiences in inner-city after-school programs among age, gender, and race groups. *Journal of Child and Family Studies,*10(1), 85–100.

Roschelle, J., Singleton, C., Sabelli, N., Pea, R. D., & Bransford, J. D. (2009). Mathematics worth knowing, resources worth growing, research worth noting: A response to the National Mathematics Advisory Panel Report. *Educational Researcher,*37(9), 610-617.

Ross, S. M., Troutman, A., Horgan, D., Maxwell, S., Laitinen, R., & Lowther, D. (1997). The success of schools in implementing eight restructuring designs: A synthesis of first-year evaluation outcomes. *School Effectiveness and School Improvement, 8*(1), 95–124.

Rowan, B., Camburn, E., & Correnti, R. (2004). Using teacher logs to measure the enacted curriculum: A study of literacy teaching in third-grade classrooms. *Elementary School Journal,*105(1), 75–102.

Rowan, B., Correnti, R., & Miller, R. J. (2002). What large-scale, survey research tells us about teacher effects on student achievement: Insights from the Prospects Study of Elementary Schools. *Teachers College Record,*104(8), 1525–1567.

Schwartz, N. (1999). How the questions shape the answers. *American Psychologist,*54, 93–105.

Schwartz, N., & Oyserman, D. (2001). Asking questions about behavior: Cognition, communication, and questionnaire construction. *American Journal of Evaluation,*22(2), 127–160.

Sheldon, S. B. (2008). Getting families involved with NCLB: Factors affecting schools' enactment of federal policy. In A. R. Sadovnik, J. O'Day, G. W. Bohrnstedt & K. Borman (Eds.), *No Child Left Behind and the reduction of the achievement gap: Sociological perspectives on federal educational policy* (pp. 281-294). New York: Routledge.

Simpkins, S. D., Ripke, M., Huston, A. C., & Eccles, J. S. (2005). Predicting participation and outcomes in out-of-school activities: Similarities across social ecologies. In H. B. Weiss, P. M. D. Little, & S. M. Bouffard (Eds.), *Participation in youth programs: Enrollment, attendance, and engagement* (pp. 51–69). San Francisco: Jossey-Bass.

Snow, C., Burns, M. S., & Griffin, P. (1998). *Preventing reading difficulties in young children*. Washington, DC: National Academies Press.

Sudman, S., & Bradburn, N. M. (1982). *Asking questions. A practical guide to questionnaire design*. San Francisco: Jossey-Bass.

Vadeboncoeur, J. A. (2006). Engaging young people: Learning in informal contexts. *Review of Research in Education,*30(1), 239–278.

Vandell, D. L., Reisner, E. R., & Pierce, K. M. (2007). Outcomes linked to high-quality afterschool programs: Longitudinal findings from the Study of Promising Afterschool Programs. In *The afterschool impact: Making every minute count*. Policy briefing, U.S. Congress, Washington, DC.

Vandell, D. L., Shumow, L., & Posner, J. K. (2004). After-school programs for low-income children: Differences in program quality. In J. L. Mahoney, J. S. Eccles, & R. W. Larson (Eds.), *Organized activities as contexts for development: Extracurricular activities, after-school, and community programs* (pp. 437–456). Mahwah, NJ: Erlbaum.

Wang, M. C., Haertel, G. D., & Walberg, H. J. (1993). Toward a knowledge base for school learning. *Review of Educational Research,*63(3), 249–294.

Wechsler, Samantha, et al. (2001). *Meeting the challenge: Financing out-of-school time programming in Boston and Massachusetts.* Parents United for Child Care.

Yohalem, N., Wilson-Ahlstrom, A., & Yu, D. (2005). *Youth program quality assessment and improvement: Celebrating progress and surfacing challenges. A meeting report*. Washington, DC: The Forum for Youth Investment, Impact Strategies.

Zimmer, R., Gill, B., Raquin, P., Booker, K., & Lockwood, J. R. (2007). *State and local implementation of the No Child Left Behind Act: Volume I—Title I school choice, supplemental educational services, and student achievement.* Washington, DC: U.S. Department of Education.

References	64

Appendix ADescriptions of Academic Instruction for Centers Serving All Students

References	66

This appendix contains descriptions of academic instruction for all centers, with separate descriptions for centers serving middle and high school students. The organization of the discussion parallels Chapter 2, which presents data for centers serving elementary school students only. Information presented on school-based versus nonschool-based is for all centers, including centers that serve elementary, middle or high school students or students at multiple levels. Data presented on centers serving middle school students cover any center that included middle school students, and data presented on centers serving high school students cover any center that served high school students. 

Exposure to Academics After School 

**Across all centers, students spent similar amounts of time per week in reading and mathematics activities, but they spent more time working on homework and in arts or recreation activities.**Overall, almost a quarter of centers reported that students participated in reading activities and 19 percent of centers conducted mathematics activities for more than 4 hours per week (see Exhibit A-1). In contrast, higher percentages of centers indicated that students participated in both homework (35 percent) and arts or recreation activities (35 percent) for more than 4 hours. Because centers were open slightly less than 16 hours per week, on average, student participation of 4 hours per week in a particular activity amounted to about 25 percent of the available time.

Exhibit A-1Percentage of Centers That Reported Participation in Activities by a Typical Student, by Type of Activity and Hours of Engagement per Week

 (Hours per Week)

Note: Percentages may not equal 100 percent due to rounding.

Source: National Survey, Item 5.

*n* = 504 

Exhibit reads: Thirty-five percent of all centers reported that a typical student participated in homework activities for more than 4 hours per week. 

Reading Activities

Reading Skills Emphasized 

**More than half of****centers reported that most or all reading activities[footnoteRef:15] included the skills emphasized by the National Reading Panel (2000).** Although phonemic awareness and phonics instruction were emphasized in secondary reading activities, centers reported very few differences in the skills that were emphasized in different grade levels. The largest percentage of centers serving middle schools reported emphasizing comprehension and vocabulary in all or most activities (65 percent and 62 percent, respectively). In particular, there was a greater-than-expected emphasis on phonics and phonemic awareness at the secondary level. Somewhat surprisingly, 39 percent reported focusing on phonics skills, and 37 percent reported focusing on phonemic awareness in all or most activities (see Exhibit A-2).  [15:  Reading enrichment activities are defined as structured activities designed to build students’ literacy skills. Reading enrichment may include scheduled time for independent student reading, writing and literacy enrichment activities, but not homework assistance. ] 

Exhibit A-2Percentage of Centers Serving Middle School Students That Reported Reading Activities Emphases, by Reading Skill Emphasized and Inclusion Level 

 (Inclusion Levels)

Note: Percentages may not equal 100 percent due to rounding.

Source: National Survey, Item 22.

*n* = 315 

Exhibit reads: Sixty-five percent of centers serving middle school students reported emphasizing comprehension in most or all enrichment activities in reading.

Centers serving students in high schools revealed few differences from centers serving younger students in the reading skills being emphasized. The largest percentage reported stressing comprehension and vocabulary in all or most activities (74 percent and 71 percent, respectively) (see Exhibit A-3). Although teaching phonics and phonemic awareness are typically not part of local and state standards at the secondary level, a large percentage of centers reported emphasizing those skills in their academic activities. Forty-five percent of centers serving high school students reported focusing on phonics in all or most reading activities, and 39 percent of centers serving this age group stressed phonemic awareness. The study team did not collect site visit data in high school–serving centers, so the reported data cannot be confirmed or challenged by observations. Nonetheless, the study team finds it likely that high school directors overreported the frequency with which academic activities focused on these skills. Another possible explanation for the pattern is that students in these high schools struggled with the most basic of reading skills; enrichment targeted to them reflected their needs. 

Exhibit A-3Percentage of Centers Serving High School Students That Reported Reading Activities Emphases, by Reading Skill Emphasized and Inclusion Level

 (Inclusion Levels)

Note: Percentages may not equal 100 percent due to rounding.

Source: National Survey, Item 22.

*n*= 107 

Exhibit reads: Seventy-four percent of centers serving high school students reported emphasizing comprehension in most or all enrichment activities in reading.

Use of Instructional Strategies for Teaching Vocabulary, Fluency and Comprehension

**Although 99 percent of all centers reported using at least one strategy in reading, the most commonly reported reading activities were designed to foster more independent reading and greater comprehension.**About half of centers (46 percent) reported that they asked students to explain or support statements about something they read in all or most reading activities. About a third of centers had students talk about or write answers to questions on something they had read (39 percent), asked students to help other students understand new words (35 percent) and asked students to make predictions about a text they are reading (35 percent) (see Exhibit A-4).

Exhibit A-4Percentage of Centers That Reported Student Participation in Instructional Strategies in Reading Activities, by Strategy and Inclusion Level

 (Explain or support their understanding of what they have readTalk/write about answers to questions about what they have readHelp other students understand new wordsMake predictions about something they are readingTalk with peers about what they have readGeneralize or make inferences about something they have readRead aloudDiscuss different interpretations of what they have readDo a group activity or project about what they have readDescribe the style or structure of a text they have readRead books selected by the leader)*(Inclusion Levels)*

Note: Percentages may not equal 100 percent due to rounding.

Source: National Survey, Item 21.

*n* = 501

Exhibit reads: Forty-six percent of centers reported having students explain or support their understanding of what they had read in most or all reading enrichment activities.

Reports of instructional strategies differed by grade level and center type. Centers serving elementary students were less likely than centers serving other grade levels to have students talk with peers about what they read and more likely to have students explain or support something they read. These centers were also more likely to have students make prereading predictions. Centers serving middle school grades were less likely than those serving other grades to have students explain or support something they read. All these differences are consistent with differences in reading skill and are most likely associated with student grade level. School-based centers were more likely than nonschool-based centers to report that their instructors asked students to make predictions about something they were reading. At the same time, nonschool-based centers had students read teacher-selected books more often than did school-based centers. 

Mathematics Activities

Mathematics Skills Emphasized

Among centers serving middle school students, 64 percent reported that they emphasized operations with whole numbers (such as addition, subtraction, multiplication and division) in all or most mathematics activities (see Exhibit A-5). About half targeted number sense in all or most mathematics activities (53 percent). Centers serving middle school students gave less attention to advanced content than to operations with numbers and other content: 18 percent emphasized algebraic concepts, 14 percent focused on geometry and 11 percent focused on data analysis in all or most activities. 

Exhibit A-5Percentage of Centers Serving Middle School Students That Reported Emphases in Math Activities, by Type of Content Emphasized and Inclusion Level

*(Inclusion Levels)*

Note: Percentages may not equal 100 percent due to rounding.

Source: National Survey, Item 24. 

*n* = 314 

Exhibit reads: Sixty-four percent of centers serving middle school students emphasized operations with whole numbers in most or all mathematics enrichment activities.

Among centers serving high school students, 69 percent reported that they emphasized operations with whole numbers in all or most mathematics activities (see Exhibit A-6). Fifty-eight percent stressed number sense in all or most activities. Geometry and data analysis received the least amount of attention among centers serving high school students: 20 percent focused on geometric concepts and 16 percent emphasized data analysis in all or most activities. 

Exhibit A-6Percentage of Centers Serving High School Students That Reported Emphases in Math Activities, by Type of Content Emphasized and Inclusion Level

*(Inclusion Levels)*

Note: Percentages may not equal 100 percent due to rounding.

Source: National Survey, Item 24.

*n* = 117

Exhibit reads: Sixty-nine percent of centers serving high school students emphasized operations with whole numbers in most or all mathematics enrichment activities.

Across all grade levels, school-based centers were more likely than nonschool-based centers to emphasize advanced topics in mathematics. In particular, school-based centers were significantly more likely than nonschool-based centers to give at least some emphasis to geometry (86 percent versus 72 percent), algebra (79 percent versus 65 percent) and data analysis (74 percent versus 62 percent). For foundational topics such as number sense and operations, however, both types of centers had similar emphases. 

Use of Instructional Strategies for Teaching Mathematics

The observed mathematics activities in the learning centers corresponded to instructional strategies. In particular, the incorporation of low-complexity cognitive tasks such as practicing basic mathematics facts as well as high-complexity tasks such as multistep problem solving in the learning centers was aligned with research identifying these methods as effective instructional strategies (Porter, 2002). Similarly, providing students the opportunity to explain the reasoning behind a problem solution helps to build proficiency in computation and translation of word problems. It also helps teachers to ensure that students have the conceptual framework necessary for understanding the grade-level mathematics (National Mathematics Advisory Panel, 2008). Experimental research studies have shown that computers and graphing calculators have a positive effect on mathematics achievement (Dixon et al., 1998). 

The National Mathematics Advisory Panel (2008), which was established to promote American students’ knowledge of mathematics and improve their performance in the subject, concluded that cooperative learning has significant effects on student outcomes, particularly in teaching young children mathematical operations. Strategies include team-assisted individualization and peer-assisted learning strategies in which heterogeneous students are grouped so that those who are weak in a specific skill can work with others who are strong in that area. Additional evidence suggests that teaching mathematics from a real-world perspective positively affects student academic achievement.

**Although 98 percent of all centers reported using at least one strategy, the most common strategies used in all or most mathematics activities were practicing mathematical facts (63 percent) and asking students to explain their reasoning in solving a problem (47 percent).**Twenty-three percent of all centers reported having students use computers to support mathematics activities or to solve open-ended problems in mathematics (see Exhibit A-7).

Exhibit A-7Percentage of Centers That Reported Student Participation in Instructional Strategies in Math Activities, by Strategy and Inclusion Level

*(Inclusion Levels)* (Practice basic mathematics factsExplain the reasoning behind how they solved a problemSolve problems in which students figure out solving methodWrite mathematical equations to solve a problemSolve problems that have many possible correct answersUse computers to practice mathematicsMake charts, tables or graphsUse graphing calculators to solve problems)

Note: Percentages may not equal 100 percent due to rounding.

Source: National Survey, Item 23.

*n* = 501

Exhibit reads: Sixty-three percent of all centers reported asking students to practice basic mathematics facts in most or all instructional activities in mathematics.

Mathematics activities differed in certain respects among center types and the grade levels served. For example, school-based centers were significantly more likely than nonschool-based centers to ask students to make charts, graphs and tables in all or most activities (20 percent versus 10 percent). Centers serving elementary school students were less likely than centers serving older groups to report using graphing calculators in all or most activities.

Instructor Education and Experience

**Centers reported that more than half of staff providing instruction in reading and mathematics had prior experience as certified classroom teachers.** Fifty-five percent of the staff who provided instruction in reading or mathematics had been regular classroom teachers, and 23 percent had been instructional specialists in reading or mathematics. Twenty-three percent of staff providing instruction in reading or mathematics had served as classroom aides. 

**Centers reported that nearly two-thirds of reading and mathematics instructors had a bachelor’s degree or higher (64 percent for reading and 64 percent for mathematics).** A third of instructors reported having a bachelor’s degree or a bachelor’s degree plus some graduate training (36 percent for reading and 37 percent for mathematics). 

Feedback to Students and Parents on Student Progress 

**The majority of centers providing academic instruction reported providing informal feedback to students; written or formal feedback was less common.** For instance, across all grade levels, 86 percent of school-based centers reported providing verbal feedback to students on work assigned by the school and 85 percent provided feedback on their behavior. In contrast, about a quarter of centers provided students written feedback on work assigned by the school (23 percent) and on work assigned in the center (25 percent). Nonschool-based centers were significantly more likely to provide written feedback on work assigned in the center than were school-based centers (35 percent versus 25 percent) (see Exhibit A-8). 

Exhibit A-8Percentage of Centers That Provided Feedback About Their Progress, by Type of Feedback and Center Type

Source: National Survey, Item 18.

*n* = 500

*p < 0.05

Exhibit reads: Eighty-six percent of all school-based centers provided verbal feedback to students on work assigned by the student’s school.

Centers serving elementary school students were significantly more likely than centers serving older students to report providing verbal feedback to students on work assigned at the center. Seventy-two percent of centers serving elementary school students reported providing this type of feedback, compared with 66 percent of middle school centers and 67 percent of high school centers. Significantly more centers serving elementary school students than centers serving other age groups also reported providing written feedback on student behavior. 

**A majority of centers reported providing frequent, informal feedback about student progress to parents.**More than half (55 percent) of centers reported talking with parents nearly every week when the parents picked up or dropped off their child. Five percent sent home written reports of students’ progress in the after-school program nearly every week, but 54 percent never did so or did so rarely. Similarly, 2 percent of centers held parent–center staff conferences nearly every week, and 56 percent never or rarely held conferences. There were no substantive differences by grade level served or type of center. Elementary-serving centers were more likely than centers not serving this age group to provide feedback to parents about student behavior. Nonschool-based centers were more likely than school-based centers to provide feedback on work assigned and to use points and certificates for accomplishments.

Appendix A: Descriptions of Academic Instruction for Centers Serving All Students	78

Appendix BCase Study Observations 

****

This appendix presents analyses conducted by coders working independently to characterize organizational goals and practices of 21st Century Community Learning Centers programs visited in fall 2006 and spring 2007. The focus of the analysis is identifying and describing practices in after-school programming, and identifying the factors that influence those practices. The observation instrument used to collect this data is available in Appendix F.

Case Study Observations: Identifying Practices 

Alignment of Observed Enrichment Activities With Practices

**Observed enrichment activities were partially aligned with practices.**Analysts rated enrichment activities observed during the site visits for their alignment with practices in four broad domains: content, instruction, engagement and relationships. For each of these domains, the observed activities were rated on a four-point scale. Raters assigned one point if the observed activity did not occur, two points if the activity focused on the feature but there was no alignment, 3 points if the activity focused on the feature and there was partial alignment and four points if the feature was a focus and there was good alignment with the definition of practice. 

The average ratings of quality were highest in engagement and relationships (see Exhibit B-1). The typical activity was partially aligned in terms of engagement (average rating = 3.6) and relationships (average rating = 3.3). There was more limited alignment in content (average rating = 2.9) and instruction (average rating = 2.7). 

Although different from classroom instruction, enrichment programming in the after-school setting maintains connections across learning environments, integrates learning goals and deepens exploration and skill acquisition for children (Noam, 2003). In addition to focusing on academics, the observed enrichment activities actively engaged learners in constructing knowledge and building skills, and targeted particular social or personal skills. Although it is not possible to generalize the case study findings to the entire population of 21st Century Community Learning Centers, these findings suggest that centers are employing practices and approaches that are aligned with research on program quality.

**Exhibit B-1**Ratings of Observations in Enrichment Activities

Source: Observation Activity Rating Rubric.

*n* = 57

Scale: 1 = did not occur; 2 = focus, no alignment; 3 = focus, partial alignment; 4 = focus, good alignment 

Exhibit reads: The level of student engagement for enrichment activities was rated 3.6 by observers in terms of alignment with practice.

Reading and Mathematics Activities 

Alignment of Observed Reading Activities With Practices

Analysts rated the reading activities observed during the site visits for their alignment with practices in four broad domains: content, instruction, engagement and relationships. For each of these domains, the observed activities were rated on a four-point scale as described earlier. The “content” and “instruction” categories are quality ratings made by observers and not simply occurrences of content. 

**Observed reading activities rated high in terms of student engagement**(see Exhibit B-2). The typical activity had good alignment in engagement (average rating = 3.8) and partial alignment in relationships (average rating = 3.3) and content (average rating = 3.3). There was more limited alignment with instructional practices (average rating = 2.9). 

**Exhibit B-2**Ratings of Observations in Reading Activities

Source: Observation Activity Rating Rubric.

*n* = 38

Scale: 1 = did not occur; 2 = focus, no alignment; 3 = focus, partial alignment; 4 = focus, good alignment 

Exhibit reads: The level of student engagement for reading enrichment activities was rated 3.8 by observers in terms of alignment with practice. 

Mathematics Activities 

Alignment of Observed Mathematics Activities With Practices

Analysts rated the mathematics activities observed during the site visits for their alignment with practices in four broad domains: content, instruction, engagement and relationships, as described earlier. The “content” and “instruction” categories are quality ratings made by observers and not simply occurrences of content. 

**The content of observed mathematics activities aligned with practice**(see Exhibit B-3). The typical activity had good alignment in content (average rating = 3.7) and partial alignment in engagement (average rating = 3.3) and relationships (average rating = 3.1). Alignment was more limited in instruction (average rating = 2.9).

Exhibit B-3Ratings of Observations in Mathematics Activities

Source: Observation Activity Rating Rubric.

*n* = 19

Scale: 1 = did not occur; 2 = focus, no alignment; 3 = focus, partial alignment; 4 = focus, good alignment 

Exhibit reads: The level of student engagement for mathematics enrichment activities was rated 3.3 by observers in terms of alignment with practice.

****

**Example of a Mathematics Activity**

In a mathematics activity that scored a 4 on the content and engagement observation rating scales, indicating good alignment with practices, a center used computerized instruction aligned with the state’s content standards. The program, which was also used during the school day, included a continuing assessment for students so that they were directed to new skills as they successfully mastered concepts. It also prompted students to call on a teacher when they repeated errors and could benefit from direct instruction. 

In the observed activity, after-school students worked at the computer under the supervision of their regular classroom teacher. Students worked at different skill levels on different topics, including number sense, expanded notation, measurement and place value. As they worked independently, the teacher circulated, explaining math concepts and procedures to individual students as necessary. When students mastered objectives, the program presented new material to them. If they stumbled, the computer alerted the teacher’s diagnostic screen, and the teacher moved to the struggling students with direct assistance. When students faltered, a pop-up “Teacher Time” window appeared, and the student summoned the teacher to diagnose the error. The teacher, together with the computerized program, kept students on task and moving swiftly through the instruction. When students made mistakes, the teacher assured them that “making mistakes is how we learn the most.” 

In summary, these findings suggest that reading activities address a broader range of processes than mathematics activities. Although reading instruction across center types focused on each of the reading processes emphasized by the National Reading Panel, the content strands of mathematics were not well represented in center activities. From the observation data, instructional strategies of mathematics activities appeared to be limited, suggesting a narrow range of instruction. Furthermore, the cognitive demand of the reading activities observed—both as part of formal instruction and in enrichment activities—was generally higher than the demand in activities where mathematics was explicitly taught or embedded. This trend was especially evident among nonschool sites, which also had fewer certified teachers offering instruction.

Appendix B: Case Study Observations	85

Appendix CSampling Approach and Weighting of Surveys

Appendix A: Descriptions of Academic Instruction for Centers Serving All Students	87

Center Director National Survey

Intended Survey Respondents

The national survey was administered to the center director in charge of after-school services at sampled sites. A “site” was generally defined as a location where 21st Century Community Learning Centers services were offered. 

Sampling Approach

The evaluation team used the Profile and Performance Information Collection System (PPICS) database, an online data collection and reporting system that collects grantee- and center-level data, to design the sample. To ensure adequate representation across characteristics of interest, the sample was stratified by the school levels the center served (elementary, secondary or both), the center location (school-based or nonschool-based) and the percentage of students eligible for free or reduced-priced lunch. The team identified 5,122 centers that met the following survey population-inclusion criteria:

1. At least one center was expected to be funded for the duration of the study.

2. The center offered academic enrichment services in mathematics, reading and technology.

3. The center had been in operation for at least 1 full school year by the time of the survey.

A total of 4,347 centers did not meet the survey population-inclusion criteria. Although nearly all sites offered reading and mathematics instruction, 19 percent of centers did not offer technology services and thus were excluded from the sample.

Degree of Precision

A simple random sample of 600 centers with 516 respondents allowed 95 percent confidence intervals to be developed for proportions (e.g., the proportion of centers that had organized homework activities) with a half-width of no more than 4.3 percentage points.[footnoteRef:16] However, because of the need to weight the sample to account for nonresponse, the target “effective” sample size was approximately 407, or about 79 percent of the number of respondents. That is, the 516 respondents for this study provided the same precision of estimation as 407 respondents who were selected via a simple random sampling method and were equally weighted. Thus, the 95 percent confidence intervals were designed to have a half-width of approximately 4.9 percentage points. If two approximately equally sized groups of centers were compared, differences in the means group would have a 95 percent confidence interval no larger than 9.7 percentage points. That is, descriptive differences between groups of more than 9.7 percentage points would be large enough to be practically important and statistically significant.  [16:  If the proportion of the sample with a certain characteristic is greater than or less than 50 percent, the half-width of the confidence interval is less than 4.3 percentage points. For example, for a proportion of 10 percent or 90 percent, the half-width would be 2.6 percentage points.] 

The effect size that is detectable with 80 percent power (using a two-sided 5 percent alpha level) is a function of the effective sample size for each subgroup. (In general, the study team expected a design effect of about 1.2 corresponding to an effective sample size that was about 85 percent of the actual number of respondents.) 

Selecting the Sample 

From the initial pool of 5,122 centers, the study team selected a random sample of 600 centers stratified on school-based or nonschool-based centers (with oversampling of the community-based centers to derive more precise estimates about the practices of those centers). 

Response Rates

During follow-up communication with survey nonrespondents, the project directors determined that 43 centers should be eliminated from the sample: 11 centers were eliminated because they had been defunded; 21 because they had not received program funds, did not serve the K–12 population or had no center at the school address supplied; 6 because they were duplicates of other centers and 4 because all efforts to obtain correct contact information failed. Thus, the size of the sampling frame after eliminating ineligible centers was 558 for the national survey. For the national survey, 516 surveys were completed, resulting in a response rate of 92.5 percent.

Weighting 

The national survey sample was weighted to reflect the stratified design. However, to be able to make statements about the national population of centers, the data were poststratified to reflect the full population of eligible centers. The study team also used Deming’s procedure to weight the respondents so that the weighted marginal counts were the same as in the universe of sites that were funded at the time of the study and that had been in operation at least a year (e.g., the weighted number of responding centers serving elementary school students was the same as the number of centers serving elementary school students in the universe). The specific characteristics used to poststratify the respondents came from the PPICS database and from U.S. Census data ZIP code files. The specific variables used from the PPICS database were the numbers of students served, program longevity, prior history with offering academic services, the primary grade levels served and the number of weeks per year that centers were open. Data from the U.S. census ZIP code files used to poststratify respondents were the percentage of a ZIP code’s population living in an urbanized area and the percentage of minority population in that area.

In addition, to correct for biases introduced by differences among some subpopulations (e.g., elementary and high schools may have different response rates), the evaluation team poststratified the respondents on the basis of characteristics available in the sampling frame and used Deming’s procedure to weight the respondents so that the weighted marginal counts were the same as in the universe (e.g., the weighted number of responding centers serving elementary school students was the same as the number of centers serving elementary school students in the universe).

Center Director Focused Telephone Survey

Intended Survey Respondents

The focused telephone survey targeted the center director (the same respondent to whom the national survey was sent) as the respondent. In addition to asking for information from the center director, the telephone survey asked a series of questions designed to elicit information about three staff members. The research team supplied the center director with an algorithm for randomly selecting appropriate staff members as targets for the questions in the survey. 

Sampling Approach

Determining the Sample Size

To obtain a response rate of 85 percent, 140 centers were selected to receive a focused telephone survey along with the national mail survey. Of these 140 centers, 116 completed both surveys. 

Degree of Precision

A simple random sample of 140 centers with 122 respondents would allow 95 percent confidence intervals to be developed for proportions (e.g., the proportion of centers that had organized homework activities) with a half-width of no more than 9.0 percentage points. However, because of the need to weight the sample to account for stratification (into school-based and nonschool-based strata) and differential nonresponse between the two types of centers, the target “effective” sample size was approximately 84 or about 69 percent of the number of respondents. That is, the 122 respondents for this survey would provide the same precision of estimation as 69 respondents who were selected via a simple random sample and were equally weighted. Thus, the 95 percent confidence intervals would have a half-width of approximately 12.0 percentage points. If two approximately equally sized groups of centers were compared, differences in the proportion of centers in each group with a certain characteristic would have a 95 percent confidence interval no larger than 24.3 percentage points.[footnoteRef:17] That is, differences of more than 24.3 percentage points would be considered statistically significant. Thus, differences that were large enough to be practically important would also be statistically significant. [17: 	These results are the consequence of general statistical principles. They would apply to any survey with 122 respondents and a sampling efficiency of 69 percent. Even if the sampling efficiency were 100 percent, the half-width would be (at most) 17.4 percentage points. Sampling efficiency could be as high as 100 percent, but would not be known until the responses were obtained (allowing us to ascertain the characteristics of respondents versus nonrespondents relative to information known about the universe).] 

Selecting the Sample 

The evaluation team used a sequential sampling approach (similar to that described for selecting the national mail survey sample) to select the sample of centers that were surveyed using the focused telephone questionnaire. 

The focused telephone survey collected data on staff qualifications for up to three staff members for each center. Statisticians used all the responses about staff in calculating staff characteristics. Because each center nominated three staff on average, no center effect had to be accounted for in the data. 

To select the target staff members, the evaluation team asked the center director to provide a list of staff whose primary responsibilities were planning and delivering academically oriented content to students. From that list, the evaluation team randomly selected three individuals and asked the center director to provide background information about their roles, credentials and qualifications. 

Response Rates

During follow-up procedures with nonrespondents, the project directors determined that 8 of the 140 centers were ineligible for inclusion. Thus, the size of the sampling frame after eliminating ineligible centers was 132 for the focused survey. For the focused survey, 122 surveys were completed, resulting in a response rate of 92.4 percent. A total of 116 centers completed both a national and a focused survey, for a response rate of 87.8 percent. 

Weighting 

The telephone survey sample was weighted to reflect the stratified design. In addition, the response rate among some subpopulations differed (e.g., elementary and high school centers had different response rates). To correct for biases that such differences could introduce, the statisticians poststratified the respondents on the basis of characteristics available in the sampling frame and used Deming’s procedure to weight the respondents so that the weighted marginal counts were the same as in the universe (e.g., the weighted number of responding centers serving elementary school students would be the same as the number of centers serving elementary school students in the universe).

Case Study Site Visits

Purpose of the Site Visits

The purpose of the case study data collection and analysis in fall 2006 and spring 2007****was to find after-school centers whose students’ academic outcomes had improved—“promising sites”—compare them with more typical sites, examine the activities and practices of the promising centers and make their successful strategies accessible to others. We visited 12 sites at elementary and secondary schools. Of particular interest to the Department of Education for the site visits was consideration of the links between the degree of academic focus and academic outcomes, alignment of student needs with academic activities and the links between schools and after-school centers. Overall, the analysis revealed few differences between promising and typical sites across the dimensions examined. The analysis used to distinguish between promising and typical sites did not provide sufficient warrant for making judgments about which goals, strategies and organizational practices were associated with more promising results. As a result, the case studies were analyzed with respect to how each case aligned with policy or research on best practices. Cross-case analyses were conducted to illustrate the different ways in which sites were aligned or seeking to align with policy and research.

Intended Respondents 

The site visits included at least three interviews with center staff and school personnel (i.e., with the center director, the lead instructor at the after-school center and at least one school staff member linked to the center). The site visits also included observations of a minimum of six after-school activities at the centers, with primary focus on reading and mathematics activities. Finally, as part of the site visit, the researchers collected relevant center artifacts (e.g., schedules, student worksheets) and existing student academic achievement data to assess whether test scores and academic skills were improving (e.g., data collected by the after-school center, feeder schools’ Adequate Yearly Progress status).

An initial planning conversation with the center director helped to identify the appropriate individuals for site visit interviews, and the study team developed interview guides for four types of staff associated with each center:

**Center director.** The center director was the best-placed individual to have salient information about the operations and organization of learning at a 21st Century Community Learning Centers program. 

**Lead center instructor.** With the assistance of the center director or main contact, the site visitors selected a key staff member who provided academically oriented programming at the center. 

**School staff.** With the assistance of the center director, the site visitors identified a staff member from the school who was familiar with the center’s after-school program and/or who provided coordination between the school and the center. ****

**Grant director.****In most cases, 21st Century Community Learning Centers have more than one location where services are offered, and a grant director coordinates the overall efforts of those locations. In those cases, the site visitors interviewed the grant director. When the center director and grant director were the same person, the site visitors combined the two interview protocols to avoid asking duplicate questions. 

Selecting Case Study Sites

A top priority in this study was to identify and visit centers that had a strong academic focus as evidenced by their student outcomes and other components of practice. Because survey data were not available to select cases, data reported in the PPICS database provided the best evidence about student outcomes for use in selecting promising sites to visit. Initially, the study team selected the following three data elements from PPICS as the primary criteria in this selection process: 

Percentage of students whose reading and/or mathematics assessment scores increased as of the 2003–04 reporting period 

Number of hours in reading and/or mathematics instruction per week

Student attendance at a center (for at least 30 days in the 2003–04 reporting period) 

Centers with these promising characteristics were selected as a subset of the stratified sample of centers to be surveyed. For the entire sample of centers in the PPICS database, the researchers first identified, separately for reading and for mathematics, a mean and standard deviation of percentage of students whose scores increased. Next, the researchers identified as “promising” two samples of centers: one in which the percentage of students was at least 1 standard deviation above the mean for reading, and a second in which the percentage of students was at least 1 standard deviation above the mean for mathematics. For sites in the promising category, the evaluation team then eliminated sites that offered fewer than 5 hours of instruction in its category (reading or mathematics) and that indicated that fewer than 75 students had attended for at least 30 days in the previous year. For each promising site, the researchers also identified a paired site to serve as a more typical one for purposes of comparison. Typical sites had to have similar ethnic diversity among students; be in a locale of similar size and in the same state and be within 1 standard deviation, plus or minus, of the average reported percentage of students whose test scores increased. Thus, each pair had two sites serving relatively similar populations, but with different levels of test score improvement.

Sorting against these three criteria resulted in a subset of possible centers from which sites were selected for further screening. From this subset, the evaluation team identified a pool of 40 sites for further screening: 10 pairs of reading sites and 10 pairs of mathematics sites. 

Telephone Screening Process

The next step in the site selection process consisted of conducting telephone screening interviews with center directors from the identified centers. In the interviews, trained callers collected information about each site’s availability for a site visit and validated the PPICS data used to identify the site initially. The callers gathered information about the center’s academic goals and activities and about its use of data for assessing program quality and student outcomes. They also explored instructional strategies, student participation, links with the school day, staffing, student–instructor relationships and staff development. Callers entered the responses directly into a database developed for this purpose. Research analysts summarized these data and used them to develop a one-page center profile for each of the initial pairs of sites contacted for presentation to the Department as candidate sites. 

Joint Review and Final Selection Process

The research team used the criteria described above to create a subset of centers to visit. However, many of the sites proved to be in the same state. To ensure a geographically dispersed group of sites, the project directors selected no more than one pair of sites for each subject area in a single state.[footnoteRef:18] This additional criterion reduced the ability to use the telephone screening data to select sites; the researchers then prepared a set of recommended sites on the basis of their availability and the initial PPICS screening for the Department to review. Twelve centers were selected for site visits.  [18:  Florida has two pairs of sites in the sample because the research team had visited one pair of sites in Florida and one in California before modifying the selection criteria. ] 

Site Notification Process

The Department contacted all the primary sites by letter and by telephone, inviting their participation in the case studies. Within a week of mailing the letters, the study team confirmed by telephone whether or not the center was willing to participate in the site visits. If a center declined to participate in the case studies, the researchers contacted a corresponding alternate center immediately to seek to replace the nonparticipating center.

Appendix C: Sampling Approach and Weighting of Surveys	96

Appendix DSurvey Instruments

### OMB No. 1875-0239 Approval Expires 10/31/07

**21st Century Community Learning Centers Program Quality Study**

## 

## 

## 

## 

## 

## 

## Center Director National Survey

 (Paperwork Burden Statement: According to the Paperwork Reduction Act of 1995, no persons are required to respond to a collection of information unless such collection displays a valid OMB control number. The valid OMB control number for this information collection is 1875-0239. The time required to complete this information collection is estimated to average 1.5 hours per response, including the time to review instructions, search existing data resources, gather the data needed, and complete and review the information collection. If you have any comments concerning the accuracy of the time estimate(s) or suggestions for improving this form, please write to: U.S. Department of Education, Washington, D.C. 20202-4651. If you have comments or concerns regarding the status of your individual submission of this form, write directly to: Margery Yeager, Policy and Program Studies Service, U.S. Department of Education, 400 Maryland Ave, S.W., Room 6W213, Washington, D.C. 20202. )

**

Center Director NATIONAL Survey 

March 2006

Appendix D: Survey Instruments	100

# 

# *Overall Description of Program*

## Program Goals

1.	To what extent is each of the following an objective or goal of the programs at your center? *(Mark ONE in each row.)*

2a.	Which of the goals or objectives listed in question 1 are your center’s **three****MOST IMPORTANT** ones? (Select THREE goals or objectives from question 1 and indicate their importance by marking the boxes under their corresponding letters in the rows below. *Mark ONE in each row.*)

2b. Which of the goals or objectives listed in question 1 do evaluation data show your center is making the **MOST PROGRESS** in achieving?(Select THREE goals or objectives from question 1 and indicate your progress toward them by marking the boxes under their corresponding letters in the rows below. *Mark ONE in each row.*)

3.	What indicators does your center use to measure your program’s success? ***(Mark ALL that apply.)***

a.	Staff observations of youth

b.	Student questionnaires

c.	Parent questionnaires

d. 	Teacher reports

e.	Assessments administered to students as part of the before- or afterschool program 

f.	Student grades 

g.	Student achievement test scores

h.	Other (specify): _________________________________________

## Program Approach

For the following questions, answer for the **most recent program year** in your program.

5a.	On average, for how many hours per week does a typical student participate in activities at the center? 

____ hours

5b.	On average, for how many hours per week does a typical student participate in the following activities at your center? *(Mark ONE in each row.)*

6.	What strategies do you use to help meet students’ academic goals? ***(Mark ALL that apply.)***

	The center I direct seeks to help students meet schools’ academic goals by:

a.	Adopting the same textbooks and strategies as used during the school day to help students with the academic subjects they find difficult

b.	Supplementing subject-matter instruction with exploratory activities, field trips, and/or applied learning

c.	Providing arts, physical fitness, field trips, and active enrichment learning opportunities that our feeder schools (i.e., schools students we serve attend) no longer offer

d. 	Providing opportunities for students to participate in subjects not traditionally offered in schools

e.	Providing leadership and/or community service opportunities for students

f.	Helping parents become more involved in their children’s education

g.	Providing students with a place where they have freedom to talk about their academic difficulties in a safe environment

h.	Other (specify):___________________________________________________________

7.	Which of the above statements **BEST** describes your approach to helping students meet their academic goals? *(Mark ONE.)*

## Center Structure and Policies 

8.	For how many years have before- or afterschool services been offered at this particular center (including before the beginning of the 21st Century Community Learning Centers grant)? *(Mark ONE.)*

a.	Never

b.	1-2 years

c.	3-5 years

d.	6-10 years

e.	More than 10 years

9.	For how many years have **academically focused** before- or afterschool services been offered at this particular center (including before the beginning of the 21st Century Community Learning Centers grant)? *(Mark ONE.)*

a.	Never

b.	1-2 years

c.	3-5 years

d.	6-10 years

e.	More than 10 years

10.	What policies regarding attendance does the center have for students? ***(Mark ONE.)***

a. 	Attendance is not required; students can drop in and out.

b. 	We require students to attend at least 1 day per week. 

c. 	We require students to attend 2-3 days per week.

d. 	We require students to attend every day we offer services.

e. 	Other (specify): _______________________________________

11.	What consequences are there for students who do not meet the general attendance requirements of the center? *(Mark ALL that apply.)*

a. 	Not applicable; we do not require attendance.

b. 	A staff member speaks to the student about an absence after each absence.

c. 	A staff member speaks to the student about an absence after a certain number of absences.

d. 	The student is restricted from participating in other activities at the center for a period of time.

e. 	The student receives “points” or some other kind of penalty that restricts privileges at the center.

f. 	A staff member notifies the student’s parents or caregivers.

g.	A staff member notifies the student’s school.

h. Other (specify): ____________________________________________	

12.	Does the program charge a fee for students to attend, and if so, how much is the fee (calculated on a monthly basis)? ***(Mark ONE.)***

a. 	The program does not charge a fee

b. 	Under $10 per month

c. 	Between $10 and $25 per month

d. 	Between $26 and $50 per month

e. 	More than $50 per month

## Tracking Outcomes

13.	What information is available to the center from school or district officials on student performance? *(Mark ALL that apply in each row.)*

14.	Do you assess students’ academic growth in the program as part of your program activities?	Yes	No	**If no, skip to Question 17.**

15.	How often does your center use the following information to assess students’ academic growth in the program, either for making adjustments to the program or for evaluating the program’s success? *(Mark ONE in each row.)*

16.	How does your center use the results of information from student assessments of academic growth? *(Mark ALL that apply.)*

a.	To identify needs for new program offerings.

b.	To make improvements to practices within existing program offerings.

c.	To assign students to particular activities. 

d.	To assign students to particular staff who are qualified to help students in identified academic areas of need.

e.	To assign students to particular groups or to older youth in the program.

f.	To evaluate the success of the program.

g.	We do not use these data in our programming.

17.	How do you use attendance data in your center? ***(Mark ALL that apply.)***

a.	We keep the attendance data on file for the grant director or evaluator to use.

b.	We track individual student attendance to ensure that they come regularly.

c.	We use data on attendance to adjust program offerings.

d.	Other (specify): _____________________________

18.	What kind of feedback do **students** receive about the progress they are making or their accomplishments at the center? ***(Mark ALL that apply.)***

a.	Verbal feedback from center staff on work assigned by the school

b.	Written feedback from center staff on work assigned by the school

c.	Verbal feedback from center staff on work they assign at the center

d.	Written feedback from staff on work they assign at the center

e.	Verbal feedback from center staff on behavior

f.	Written feedback from center staff on student behavior

g.	“Points” or rewards for program accomplishments 

h.	Certificates or awards for accomplishments

19.	What kind of feedback does the center send to **parents** about the progress their children are making in the before- or afterschool program? ***(Mark ALL that apply.)***

a.	Feedback from center staff on work assigned by the school

b.	Feedback from center staff on work assigned by the center

c.	Feedback from center staff on behavior

d.	“Points” or rewards for program accomplishments

e.	Certificates or awards for accomplishments

f.	We do not send written or verbal feedback to parents.

20.	How often do center staff communicate with parents about their students’ needs and progress in each of the ways listed below? ***(Mark ONE in each row.)***

## Student Academic Support Activities 

***Reading Support Activities***

Academic support in reading refers to structured activities designed to build students’ literacy skills. Academic support activities may include scheduled time for independent student reading, writing, and literacy enrichment activities, but NOT homework assistance. Note: Depending upon the age groups and needs of students you serve, some of these reading tasks may not be appropriate for your students. For these questions, **please consult with the staff member(s) who lead reading support activities at your center.**

21.	In what proportion of your reading activities are students asked to do the following? *(Mark ONE in each row.)*

22.	In what proportion of activities at your center aimed at promoting reading or literacy skills do instructors emphasize to the following reading processes? ***(Mark ONE in each row.)***

## Mathematics Support Activities

Academic support in mathematics refers to structured activities dedicated to increasing students’ mathematics skills. Academic support activities may include applied enrichment, skill building, mathematics games, and scheduled time for tutoring, but NOT homework assistance. Note: Depending upon the age groups and needs of students you serve, some of these mathematics tasks may not be appropriate for your students. For these questions, **please consult with the staff member(s) who lead mathematics support activities at your center.**

23.	In what proportion of your mathematics activities are students asked to do the following? *(Mark ONE in each row.)*

24.	In what proportion of activities at your center aimed at promoting mathematics skills do instructors give emphasis to the following? ***(Mark ONE in each row.)***

## Use of Technology Resources

25.	How many Internet-connected computers (desktop or laptop) are available for student use at this center? __________

a.	None

b.	Less than 5

c.	6 - 10

d.	More than 10

26.	On average, how often have students at your center used the following kinds of software or Internet applications in the past programming year? *(Mark ONE in each row.)*

# *Staffing*

## Staff Pay and Benefits 

27.	How many of the following types of staff does your center employ? ***(Mark ALL that apply.)***

Certified teachers: ____

School paraprofessional staff: ____

Other professionals (e.g., youth-development specialists): ____

Other paraprofessionals (e.g., clerks, administrative assistants): _

Unpaid volunteers: ___

28.	In addition to wages, what benefits are offered to full- and part-time staff? ***(Mark ALL that apply in each row.)***

## Staff Retention

29.	Please list the number of staff, aside from you, who have worked at the center (at least 5 hours per week) for the following durations. (Write a number on each line or check “Don’t know.”)

30a.	How many staff who work at least 5 hours per week have left the program since the start of the 2005–06 school year?

____ staff

30b. What are the primary reasons why staff leave the program? ***(Mark ALL that apply.)***

a.	We have not had staff leave our program.

b.	Low pay

c.	Lack of benefits

d.	Inability to offer a full-time position

e.	Completion/graduation from a program (e.g., Americorps) or from school

f.	Other (specify: )

g.	Don’t know

# *Overall Program Challenges*

31.	How much of a challenge to implementing high-quality programming at your center is each of the following? *(Mark ONE in each row.)*

# 

# Thank you for your participation.

Appendix D: Survey Instruments	116

**21st Century Community Learning Centers Program Quality Study**

**Center Director Focused Survey**

**(Paperwork Burden Statement: According to the Paperwork Reduction Act of 1995, no persons are required to respond to a collection of information unless such collection displays a valid OMB control number. The valid OMB control number for this information collection is 1875-0239. The time required to complete this information collection is estimated to average 1.5 hours per response, including the time to review instructions, search existing data resources, gather the data needed, and complete and review the information collection. If you have any comments concerning the accuracy of the time estimate(s) or suggestions for improving this form, please write to: U.S. Department of Education, Washington, D.C. 20202-4651. If you have comments or concerns regarding the status of your individual submission of this form, write directly to: Jessica Hausman, Policy and Program Studies Service, U.S. Department of Education, 400 Maryland Ave, S.W., Room 6W227, Washington, D.C. 20202. )21st Century Community Learning Centers Program Quality Study**

**Center Director FOCUSED Survey**

**March 2006**

***Student Participation***

***Recruitment***

1.	Which of the following groups of students does your center seek to serve? **(Mark ALL that apply.)**

a.	We have open enrollment for all interested students.

c.	Students who scored “below proficient” on state assessments.

d.	Students identified by the school as needing special assistance in reading and/or math.

e.	Students who are English-language learners.

f.	Students who receive free or reduced-priced meals.

g.	Students who are recommended by teachers or counselors at the feeder schools.

h.	Students with siblings already attending.

i.	Other (specify): _____________________________________________

2.	Which, if any, of the groups listed in question 1 have priority for the center’s services? (Select **THREE** groups from question 1 and indicate them by marking the corresponding letter in the rows below. List the top priority group first, followed by the second and third priority groups. **Mark ONE in each row.)**

3.	How many total students are currently enrolled at this center? ________ 

4.	What is the maximum number of students the center can serve? ________

5.	Which of the following strategies do you use to engage and retain students at the center? **(Mark ALL that apply.)**

a.	Offering targeted academic instruction in reading and mathematics. 

b.	Offering academic enrichment activities, such as free reading, hands-on projects in content areas (e.g., science, history, geography projects), or field trips. 

c.	Providing individualized or small group tutoring.

d.	Allowing students to choose the activities they participate in. 

e.	Providing both space and time at the center for recreation, snacks, and talking with friends.

f.	Providing a full meal after school.

g.	Providing family literacy assistance. 

h.	Providing transportation to the center. 

i.	Offering incentives (e.g., field trips, movie tickets) for participation.

***Attendance***

6.	How does this center track attendance? **(Mark ONE.)**

a.	Pen and paper records.

b.	Using a database management program on a single computer.

c.	Using a Web-based data system for recording and reporting data on a computerized network.

d.	Using student IDs with barcodes and an electronic scanner linked to computers.

e.	Other (specify): ______________________________________________________

7.	What kinds of attendance reports can your center generate easily (within 1-2 hours time)? **(Mark ALL that apply.)**

a.	Total number of students who attended the center in the past week. 

b.	Total number of students who attended the center in the past school year.

c.	Number of times each student attends, recorded by the week. 

d.	Number of times each student attends, recorded by the day.

e.	Number of special education students who have attended the program since the school year began.

8.	**If you selected “d” above in Question 7**, answer this question: Last year (2004–05), how many students attended this center for the following amounts of time? (Write a number on **each line** below. Write “0” if there are no students in a category.) 

	a. This information is not available in this format for our center.

9. For each of the activities below, mark the box that most accurately corresponds to your center’s policies regarding student attendance for the activities indicated. **(Mark ONE in each row.)**

	 a. This information is not available in this format for our center.

***Tracking of Student Scores on State Tests***

10.	Has your center received data from participants’ schools that tells how they performed on the state assessment test used for NCLB in **2004–05**? **(Mark ONE.)**

a.	I don’t know. Skip to Question 13.

b.	No. Skip to Question 13.

c.	Yes.

11.	Do you have knowledge of the level of proficiency of individual students on these tests? **(Mark ONE.)**

a.	No. Skip to Question 13. 

b.	Yes.

12.	Approximately what percent of participants at your center scored below proficient on the most recent assessment that the state administers for NCLB accountability?

a.	I don’t know.

b.	Percent scoring below proficient in READING: ____________ 

c.	Percent scoring below proficient in MATHEMATICS: ____________ 

13.	Have you or has the school administered any mid-year assessments that could help predict whether students scoring “below proficient” are likely to score “proficient” or higher on the 2005**–**06 state tests used for NCLB? **(Mark ONE.)**

a.	I don’t know. Skip to Question 15.

b.	No. Skip to Question 15.

c.	Yes. 

14.	Approximately what percent of participants enrolled in your center are likely to move from scoring “below proficient” to “proficient” on 2005**–**06 state tests, compared with last year?

a.	I don’t know.

b.	Percent likely to move from “below proficient” to “proficient” in READING: ____ 

c.	Percent likely to move from “below proficient” to “proficient” in MATHEMATICS: _____

***Coordination with Supplemental Services Providers***

15.	This year, in 2005**–**06, is your center a state-approved provider of supplemental educational services (SES)? **(Mark ONE.)**

a.	I don’t know. Skip to Question 17.

b.	No. Skip to Question 17.

c.	Yes. 

16.	If your center is a state-approved provider of supplemental educational services under NCLB, how many SES students does your center serve THIS SEMESTER?

a.	I don’t know.

b.	Our center serves _____________ students in READING under SES this semester.

c.	Our center serves _____________ students in MATHEMATICS under SES this semester.

17.	This year, in 2005**–**06, does your center collaborate with other supplemental education services providers to offer services to participants at your center? **(Mark ONE.)**

a.	No. Skip to Question 20.

b.	Yes. 

18.	With how many different supplemental service providers does your center collaborate? 

	______

19.	How do you coordinate services with those SES providers? **(Mark ALL that apply.)**

a.	The SES provider(s) help us decide the focus of our programs.

b.	We coordinate our schedule with the SES provider(s).

c.	We coordinate academic support activities such as tutoring for individual students.

d.	We help each other recruit students.

e.	We work together to communicate with students’ schools.

f.	Other (specify): _____________________________

20.	Do students who attend activities at your center receive SES as defined by NCLB from a provider other than your center? **(Mark ONE.)**

a.	I don’t know. 

b.	No. 

c.	Yes. 

***Staffing and Training***

***Center Director’s Background and Experience***

21.	What is the highest level of education you have completed? **(Mark ONE.)**

	a.	Less than high school

	b.	High school or GED

	c.	Some college, other classes/training not related to a degree

	d.	Completed 2-year college degree. Major: ____________________

	e.	Completed 4-year college degree. Major: ____________________

	f.	Some graduate work. Discipline: ____________________

	g.	Master’s degree. Discipline: ____________________

	h.	Ph.D. Discipline: ____________________

22.	Are you certified to teach in school? **(Mark ONE.)**

	a.	No.

	b.	I have a provisional or emergency teaching credential.

	c.	I have a single-subject certification.

	d.	I have a multisubject certification.

23.	What types of experience did you have before you assumed your current position at this 21st Century Center? How long were you in these positions? **(Mark ONE in each row.)**

24.	What roles and responsibilities do you have at the center? **(Mark ALL that apply.)**

a.	Supervising paid staff

b.	Helping develop or train staff

c.	Coaching and mentoring staff

d.	Coordinating with school day staff or activities

e.	Securing and maintaining space for activities

f.	Managing center budgets

g.	Fundraising or grant writing

h.	Community outreach and partnership development

i.	Internal evaluation of programs

j.	Instructor for reading activities

k.	Instructor for mathematics activities

l.	Instructor for other enrichment activities 

m.	Tutor for individual students

n.	Counselor or social worker 

***Staff Roles, Backgrounds, and Experience***

Please answer each of the following sets of questions for the three (3) staff members that we identified from the roster you submitted to us. For each, first indicate the staff member’s role at the center, and then answer the questions about his/her background and professional development experiences. If your center does not have three staff members, answer questions about as many staff members as we identified from the roster you submitted. If you do not have the information for a staff member, a researcher can contact that staff member directly by telephone or e-mail to set up a time to gather that information. 

25.**STAFF MEMBER 1:** What roles does this staff member play at the center? **(Mark ALL that apply.)**

	a.	Provides homework assistance

	b.	Provides literacy- or reading-related services or instruction

	c.	Provides mathematics-related services or instruction

	d.	Leads activities in (specify): _____________________

	e.	Other role (specify): _____________________

26.	Approximately how many hours each week is this staff member paid to work at this center, including any paid time for training and staff development? **(Mark ONE.)**

	a.	Full-time (35 hours or more per week)

	b.	Half-time (20-34 hours per week)

	c.	Part-time (10-19 hours per week)

	d.	Occasionally (less than 10 hours per week)

27.	How long has this staff member been in this position? **(Mark ONE.)**

	a.	Up to 1 school term

	b.	More than 1 school term up to 1 year

	c.	More than 1 year up to 2 years

	d.	More than 2 years up to 5 years

	e.	More than 5 years

28.	What is the highest level of education this staff member has completed? **(Mark ONE.)**

	a.	Less than high school

	b.	High school or GED

	c.	Some college, other classes/training not related to a degree

	d.	Completed 2-year college degree. Major: ____________________

	e.	Completed 4-year college degree. Major: ____________________

	f.	Some graduate work. Discipline: ____________________

	g.	Master’s degree. Discipline: ____________________

	h.	Ph.D. Discipline: ____________________

29.	Is this staff member certified to teach in public school? **(Mark ONE.)**

	a.	No.

	b.	He/she has a provisional or emergency teaching credential.

	c.	He/she has a single-subject certification.

	d.	He/she has a multisubject certification.

30.	What types of experience did this staff member have before joining the staff of the center? How long was the staff member in each position? **(Mark ONE in each row.)**

31.**STAFF MEMBER 2**: What roles does this staff member play at the center? **(Mark ALL that apply.)**

	a.	Provides homework assistance

	b.	Provides literacy- or reading-related services or instruction

	c.	Provides mathematics-related services or instruction

	d.	Leads activities in (specify): _____________________

	e.	Other role (specify): _____________________

32.	Approximately how many hours each week is this staff member paid to work at this center, including any paid time for training and staff development? **(Mark ONE.)**

	a.	Full-time (35 hours or more per week)

	b.	Half-time (20-34 hours per week)

	c.	Part-time (10-19 hours per week)

	d.	Occasionally (less than 10 hours per week)

33.	How long has this staff member been in this position? **(Mark ONE.)**

	a.	Up to 1 school term

	b.	More than 1 school term up to 1 year

	c.	More than 1 year up to 2 years

	d.	More than 2 years up to 5 years

	e.	More than 5 years

34.	What is the highest level of education this staff member has completed? **(Mark ONE.)**

	a.	Less than high school

	b.	High school or GED

	c.	Some college, other classes/training not related to a degree

	d.	Completed 2-year college degree. Major: ____________________

	e.	Completed 4-year college degree. Major: ____________________

	f.	Some graduate work. Discipline: ____________________

	g.	Master’s degree. Discipline: ____________________

	h.	Ph.D. Discipline: ____________________

35.	Is this staff member certified to teach in public school? **(Mark ONE.)**

	a.	No.

	b.	He/she has a provisional or emergency teaching credential.

	c.	He/she has a single-subject certification.

	d.	He/she has a multisubject certification.

36.	What types of experience did this staff member have before joining the staff of the center? How long was the staff member in each position? **(Mark ONE in each row.)**

37**.	STAFF MEMBER 3**: What roles does this staff member play at the center? **(Mark ALL that apply.)**

	a.	Provides homework assistance

	b.	Provides literacy- or reading-related services or instruction

	c.	Provides mathematics-related services or instruction

	d.	Leads activities in (specify): _____________________

	e.	Other role (specify): _____________________

38.	Approximately how many hours each week is this staff member paid to work at this center, including any paid time for training and staff development? **(Mark ONE.)**

	a.	Full-time (35 hours or more per week)

	b.	Half-time (20-34 hours per week)

	c.	Part-time (10-19 hours per week)

	d.	Occasionally (less than 10 hours per week)

39.	How long has this staff member been in this position? **(Mark ONE.)**

	a.	Up to 1 school term

	b.	More than 1 school term up to 1 year

	c.	More than 1 year up to 2 years

	d.	More than 2 years up to 5 years

	e.	More than 5 years

40.	What is the highest level of education this staff member has completed? **(Mark ONE.)**

	a.	Less than high school

	b.	High school or GED

	c.	Some college, other classes/training not related to a degree

	d.	Completed 2-year college degree. Major: ____________________

	e.	Completed 4-year college degree. Major: ____________________

	f.	Some graduate work. Discipline: ____________________

	g.	Master’s degree. Discipline: ____________________

	h.	Ph.D. Discipline: ____________________

41.	Is this staff member certified to teach in public school? **(Mark ONE.)**

	a.	No.

	b.	He/she has a provisional or emergency teaching credential.

	c.	He/she has a single-subject certification.

	d.	He/she has a multisubject certification.

42.	What types of experience did this staff member have before joining the staff of the center? How long was the staff member in each position? **(Mark ONE in each row.)**

***Organization of Instruction in Reading and Mathematics Activities***

Please describe one example of a READING activity at your center that you believe is particularly effective in reaching youth. Be sure to describe the content taught, the roles and responsibilities of staff, how students are grouped, student-to-adult ratio, and at least one lesson or activity that would help an outsider understand what it would be like to participate in the activity as a student. 

Please describe one example of a MATHEMATICS activity at your center that you believe is particularly effective in reaching youth. Be sure to describe the content taught, the roles and responsibilities of staff, how students are grouped, student-to-adult ratio, and at least one lesson or activity that would help an outsider understand what it would be like to participate in the activity as a student. 

**Thank you for your participation.**

Appendix D: Survey Instruments	134

Appendix E

Supplementary Tables for the Surveys

Background for This Appendix

This appendix presents exhibits with descriptive and inferential statistics that have been used as the source of data for the exhibits presented in the main text of the 21st Century Community Learning Centers Program Quality study. Evaluation team analysts constructed the exhibits from responses to both the national survey and the focused telephone survey. 

Many of these exhibits compare the responses of centers in different settings (school-based versus nonschool-based) and different levels of students served (elementary, middle and high school). The overall results are presented only for items for which there were no statistically significant differences between center types or among grade levels. When such statistically significant differences did occur, in comparing results for center types and levels with statistically significant differences (at the *p* < 0.05 level), separate exhibits were constructed. Statistically significant findings have been marked with an asterisk (*).

Interpreting differences among levels, however, was complicated because many centers served students at multiple levels. For example, some sites served both elementary and middle school students. A statistical comparison that did not take this fact into account would have been likely to lead to fewer observed differences among sites, and it would also have violated the assumption that observations were independent from one another. Therefore, when comparing levels, the team addressed this problem by statistically comparing sites that served elementary school children against all sites that did not serve children of that age range. Analysts similarly compared sites that did and did not serve middle school students and sites that did and did not serve high school students. An asterisk appearing in the elementary school column of an exhibit indicates that the responses of centers serving elementary school children were statistically significantly different from those centers that did not serve elementary school children. 

Exhibit E-1Centers Serving Elementary School Students That Reported Participation in Activities by a Typical Student, by Type of Activity and Amount of Time of Engagement per Week

Source: National Survey, Item 5.

Numbers in parentheses are standard errors.

Exhibit reads: Of those centers serving elementary school students, 24 percent of centers reported students participated in homework help in a group setting less than 2 hours a week.

Exhibit E-2Centers Serving Elementary School Students That Reported Reading Activities Emphases, by Reading Skill Emphasized and Inclusion Level

Note: Percentages may not equal 100 percent due to rounding.

Source: National Survey, Item 22.

Numbers in parentheses are standard errors.

Exhibit reads: Of those centers serving elementary school students, 43 percent or 168 of those centers reported students’ phonemic awareness was emphasized in some reading enrichment activities.

Exhibit E-3Centers Serving Elementary School Students That Reported Student Participation in Instructional Strategies in Reading Activities, by Strategy and Inclusion Level

Source: National Survey, Item 21.

Numbers in parentheses are standard errors.

Exhibit reads: Of those centers serving elementary school students, 57 percent of centers reported that students were asked to read aloud in some reading enrichment activities.

Exhibit E-4Centers Serving Elementary School Students That Reported Emphases in Math Activities, by Type of Content Emphasized and Inclusion Level

Source: National Survey, Item 24.

Numbers in parentheses are standard errors.

Exhibit reads: Of those centers serving elementary school students, 36 percent of centers reported that number sense was emphasized in some mathematics enrichment activities.

Exhibit E-5Centers Serving Elementary School Students That Reported Student Participation in Instructional Strategies in Math Activities, by Strategy and Inclusion Level

Source: National Survey, Item 23.

Numbers in parentheses are standard errors.

Exhibit reads: Of those centers serving elementary school students, 47 percent of centers reported that having students explain the reasoning behind how they solved a problem was emphasized in some mathematics enrichment activities.

Exhibit E-6Centers Serving Elementary School Students That Provided Feedback About Their Progress, by Type of Feedback and Center Type

Source: Focused Survey, Item 18.

*p<0.05 

Numbers in parentheses are standard errors.

Exhibit reads: Of those centers serving elementary school students, 87 percent of centers reported providing verbal feedback from center staff on work assigned by the school.

Exhibit E-7Centers That Reported Top-Priority Target Groups, by Target Groups and Center Type

Source: Focused Survey, Item 1.

*p<0.05 

Numbers in parentheses are standard errors.

Exhibit reads: Of nonschool-based centers, 60 percent reported having open enrollment for all interested students. 

Exhibit E-8Centers That Reported Attendance Requirements, by Grade Level and Attendance Requirement

Source: National Survey, Item 10.

Numbers in parentheses are standard errors. 

* Significant association (p<0.05) between grade level of school and distribution of attendance items.

Exhibit reads: Among centers serving elementary school students, 26 percent of centers reported that attendance is not required; students can drop in and out.

Exhibit E-9Centers That Reported That They Required Participation of All Students in Specific Activities When Students Were Present, by Type of Activity

Source: Focused Survey, Item 9.

Numbers in parentheses are standard errors.

Exhibit reads: Seventy-three percent of centers reported requiring participation in homework help in a group setting when students were present.

Exhibit E-10Centers That Reported the Number of Days of Attendance in 2005–06, by Grade Level and Days of Attendance

Source: Focused Survey, Item 8.

Numbers in parentheses are standard errors.

Exhibit reads: Thirty-two percent of elementary school students attended fewer than 30 days. 

Exhibit E-11 Centers That Reported Consequences for Student Absences, by Consequence Type 

Source: National Survey, Item 11.

*n =* 503

Numbers in parentheses are standard errors.

Exhibit reads: Fifty-eight percent of center directors report notifying parents when a student is absent from the program.

**Exhibit E-12Centers That Paid Hourly Rates to Staff Who Led Instructional Activities, by Type of Staff and Rates**

Source: National Survey, Item 27.

Numbers in parentheses are standard errors.

Exhibit reads: Among all centers, 1 percent reported that certified teachers leading instructional activities were paid less than $10 an hour. 

Exhibit E-13Staff Who Led Instructional Activities at Centers, by Hours Worked per Week

Source: Focused Telephone Survey, Items 26, 32 and 38.

Numbers in parentheses are standard errors.

Exhibit reads: Ten percent of centers reported staff leading instructional activities worked full-time (35 hours or more per week).

Exhibit E-14Centers That Reported Offering Benefits to Full- and Part-Time Staff Who Led Instructional Activities, by Benefit Offered and Staff Status

Source: National Survey, Item 28.

Numbers in parentheses are standard errors.

Exhibit reads: Forty percent of centers reported that full-time staff leading instructional activities were offered paid time off for vacation and sick leave.

Exhibit E-15Centers That Reported Reasons for Staff Departures, by Reason for Departure and Center Type

Source: National Survey, Item 30b.

*p<0.05 

Numbers in parentheses are standard errors.

Exhibit reads: Among school-based centers, 8 percent reported low pay as the primary reason that staff leading instructional activities left.

Exhibit E16Centers That Reported That They Offered Paid Professional Development Opportunities for Full- and Part-Time Staff Who Led Instructional Activities, by Staff Status and Type of Development

Source: National Survey Item 28.

Numbers in parentheses are standard errors.

Exhibit reads: Fifty-three percent of centers reported that paid attendance at conferences was a professional development opportunity for full-time staff leading instructional activities.

Exhibit E-17Centers That Reported That State Assessments Data Was Available for Use in Tracking Student Outcomes, by Subject Tested and Center Type

Source: National Survey, Item 13.

*p<0.05

Numbers in parentheses are standard errors.

Exhibit reads: Among school-based centers, 73 percent reported that centers could track student outcomes in reading using state assessment results.

Exhibit E-18Centers That Reported Student Assessment Sources, by Type and Frequency of Assessment

Source: National Survey, Item 15.

Numbers in parentheses are standard errors.

Exhibit reads: Twenty-six percent of centers reported informal staff observations are not used to assess students’ academic growth.

Exhibit E-19Centers That Reported Uses of Student Assessment Data, by Type of Use

Source: National Survey, Item 16.

Numbers in parentheses are standard errors.

Exhibit reads: One percent of centers reported not using assessment data in their program.

Exhibit E-20Centers That Reported Challenges in Implementing High-Quality Programming, by Type of Challenge and Center Type

Source: National Survey, Item 31.

*p<0.05 

Numbers in parentheses are standard errors.

Exhibit reads: Among nonschool-based centers, 40 percent reported lack of information about the school-day curriculum as a challenge in implementing high-quality programming. 

Appendix E: Supplementary Tables for the Surveys	155

Appendix F

21st CCLC Quality Study—Quality Observation Instrument

Appendix F: 21st CCLC Quality Study - Quality Observation Instrument	157

Appendix F: 21st CCLC Quality Study - Quality Observation Instrument	165

Reported Student Activities

Percent of Centers Reporting

Reported Student Activities

Percent of Centers Reporting

Reading Skills Emphasized

Percent of Centers Reporting

Reading Skills Emphasized

Percent of Observations

Reading Activity Strategies

Percent of Centers Reporting

Reading Instructional Strategies Emphasized

Percent of Observations

Math Activity Content Emphasized

Percent of Centers Reporting

Math Skills Emphasized

Percent of Observations

Instructional Strategies in Math

Percent of Centers Reporting

Instructional Strategies in Math

Percent of Observations

Active Teaching Strategies

Percent of Observations

79%*

88%*

46%*

55%*

Students Targeted

Percent of Centers Reporting

22%*

31%*

41%*

19%*

25%*

19%*

13%*

14%*

13%*

45%*

29%*

26%*

Type of School

Percent of Centers Reporting

Activity Required

Percent of Centers Reporting

Type of School

Percent of Centers Reporting

Consequences for Absences

Percent of Centers Reporting

Staff Who Are Certified Teachers

Other Staff

Benefits Offered

Percent of Centers Reporting

28%*

18%*

44%*

12%*

8%*

25%*

Reason for Departure

Percent of Centers Reporting

Staff Status

Percent of Centers Reporting

52%*

53%*

71%*

73%*

Subject Tested

Percent of Centers Reporting

Assessment Type

Percent of Centers Reporting

Use of Assessment Data

Percent of Centers Reporting 

39%*

36%*

29%*

20%*

Implementation   Challenges

Percent of Centers Reporting

Relationship-Building Efforts

Percent of Observations

Reported Student Activities

Percent of Centers Reporting

Reading Skills Emphasized

Percent of Centers Reporting

Reading Skills Emphasized

Percent of Centers Reporting

Reading Activity Strategies

Percent of Centers Reporting

Math Activity Content Emphasized

Percent of Centers Reporting

Math Activity Content Emphasized

Percent of Centers Reporting

Instructional Strategies in Math

Percent of Centers Reporting

Practices

Average Rating

Practices

Average Rating

Practices

Average Rating

# image6.png

# image3.jpeg

# image7.jpeg

# image8.jpeg

# image9.jpeg

# image10.jpeg

# image11.jpeg

# image12.jpeg

# image13.jpeg

# image14.jpeg

# image15.png