A Snapshot of OST Programs in Philadelphia: An Evaluation of Eleven 21st Century Community Learning Center Grantees
Author: Gao, Jian
Date: 2014-04-00
Type: Reports - Evaluative
Subjects: After School Programs,Community Centers,Grants,Program Effectiveness,Outcomes of Education,Grades (Scholastic),Elementary School Students,Middle School Students,Comparative Analysis,Individual Characteristics,Attendance,Scores,Mathematics Tests,Reading Tests,Reading Achievement,Mathematics Achievement,Suspension,Enrollment,Demography

This report examines the 21st Century Community Learning Center (CCLC) programs evaluated by Research for Action (RFA) and presents an aggregate analysis of key findings to inform broader discussions about OST programs. The report addresses the following questions: (1) Who participated in the 21st CCLC programs and how frequently did they participate?; (2) What organization-level elements of quality programming were in place to achieve the intended outcomes?; and (3) What was the relationship between program participation, organization-level elements of quality programming and academic, attendance and behavioral outcomes? RFA addressed these research questions through a mixed-methods study that analyzed the following: (1) Student outcomes data; (2) Program participation data; (3) OST staff survey responses; and (4) Qualitative data from local evaluations, including interviews with program staff, site visits, and a review of program documents. The study includes data from each of the 50 sites operated by the 11 grantee organizations. However, the analysis of student outcomes focuses on a smaller subset of 40 sites from 11 providers which operated programs in public or charter schools. Data from Catholic schools could not be integrated into the student outcome analyses because these schools utilize different grade structures and assessments (TerraNova rather than PSSA scores). In addition, the analysis of the organization-level elements of quality programming focuses only on 10 providers with staff who completed the staff survey. Student outcomes analyses considered the following: (1) Math, reading, and science course grades; (2) Math and Reading Pennsylvania System of School Assessment (PSSA) scores; (3) Developmental Reading Assessment (DRA) for 1st-3rd grade students; (4) HS credits earned for 9th-12th grade students; and (5) School attendance and suspensions Student outcomes analyses compared OST regular participants to students who did not regularly participate in the OST programs and non-participating students within their schools. The analysis also considered the level of program participation and the presence of key organization-level elements of quality programming. Overall, this study found promising evidence of the benefits of the 21st CCLC programs in two outcome areas--school attendance and course grades--for regularly participating elementary and middle school students. In addition, for students who participated in the program, the analyses found that outcomes in most areas improved as program participation increased. In addition, staff survey data revealed key organization-level elements of quality programming that may be related to more positive student outcomes. The analysis of 21st CCLC program content, staffing, and school partnerships found variation in the implementation of these organizational-level elements of quality programming and suggests ways in which programs could increase their impact on students' outcomes. The report is organized into the following main sections: (1) Context: 21st CCLC Providers, Schools, and Programs; (2) Participant Characteristics and Level of Participation; (3) Organization-Level Elements of Quality Programming: Promising Practices and Areas for Growth in Program Content, Staff, and School Partnerships; and (4) Student Outcomes and the Relationship between Student Outcomes with Level of Participation and Organization-Level Elements of Quality Programming. The following are appended: (1) Organization-Level Elements of Quality Programming Data Collection and Analysis; (2) Comparative Analyses: Regular Participants Versus Comparison Students; (3) Analyses of the Relationship between Program Participation Levels and Student Outcomes; (4) Results from Analyses using Catholic School Data; and (5) Provider Profile. [This report was funded by the Philadelphia Foundation and the Samuel S. Fels Fund.]