# 21st Century Community Learning Centers (21st CCLC) Analytic Support for Evaluation and Program Monitoring:

**An Overview of the\
21st CCLC Performance Data: 2005--06**

**November 2007**

**Submitted to**

**Sylvia Lyles, Program Director**

#### *21st Century Community Learning Centers*

***U.S. Department of Education***

**Prepared by**

**Neil Naftzger**

**Christina Bonney, Ph.D.**

**Tara Donahue, Ph.D.**

**Chloe Hutchinson**

**Jonathan Margolin, Ph.D.**

**Matthew Vinson**

***Learning Point Associates***

![](media/image1.jpeg){width="3.0027777777777778in"
height="0.6895833333333333in"}

> 1120 East Diehl Road, Suite 200
>
> Naperville, IL 60563-1486
>
> 800-356-2735 630-649-6500
>
> www.learningpt.org

Copyright © 2007 Learning Point Associates. All rights reserved.

2501_11/07

# Contents

Page

Executive Summary 1

Introduction 16

Section 1: Characteristics of 21st CCLC 18

> Partnerships 19
>
> Activities 24
>
> Staffing 36
>
> Attendance 41
>
> Summary of Characteristics of 21st CCLC Programs 44

Section 2: Student Achievement and Academic Behavioral Outcomes 45

> Evaluating the Efficacy of the Current Indicator System 47
>
> Outcome Indicators: Definitions, Limitations, and Proposed Changes 51
>
> Analysis of Grades, State Assessment, and Teacher Survey Results 56
>
> Changes in Student Behavior and Performance Across Years 57
>
> Changes in Student Behavior and Performance by Improvement Category 67
>
> Changes in Student Behavior and Performance by Student Attendance 70
>
> Changes in Student Behavior and Performance by Grade Level 79
>
> Next Steps 90

Summary of Key Findings and Next Steps 92

References 95

Appendixes

> Appendix A. State Tables 97
>
> Appendix B. Methods 106
>
> Appendix C. APR Reporting Options Afforded to States 114
>
> Appendix D. Glossary 116

**Executive Summary**

**Introduction**

The primary purpose of this report is to provide an overview of the 21st
Century Community Learning Center (21st CCLC) program, with a particular
emphasis placed upon exploring issues related to program quality and
performance measurement. More specifically, this report has been
structured to address three primary questions:

-   How can the current information collected about the characteristics
    of 21st CCLC programs be used to help inform issues related to the
    quality of programming being provided to youth at these centers?

-   To what extent did programs operating during the course of the
    2005--06 reporting period meet the Government Performance and
    Results Act (GPRA) performance targets established for the program?

-   What would be the effect of implementing recommendations made by the
    Profile and Performance Information Collection System (PPICS)
    Evaluation Task Force (ETF) for modifying some of the GPRA
    indicators?

## All of the information outlined in this report was obtained from the 21st CCLC PPICS. Funded by the U.S. Department of Education, PPICS is a Web-based data-collection system designed to capture information regarding state-administered 21st CCLC programs.

**Characteristics of 21st CCLC Programs**

As of December 2006, there were a total of 3,309 active 21st CCLC
grantees across the country that, in turn, operated a total of 9,824
centers. Note that the term "grantee" in this report refers to the
entity that serves as the fiscal agent for a given 21st CCLC grant,
while "centers" refer to the physical locations where grant-funded
services and activities are provided to participating students and
adults. (On average, a single grant supports three centers.) A center
offers academic, artistic, and cultural enrichment opportunities to
students and their families during nonschool hours (before or after
school), or periods when school is not in session (e.g., holidays,
weekends, or summer recess). A center can also be characterized by
defined hours of operation; a dedicated staff that plans, facilitates,
and supervises program activities; and an administrative structure that
may include a position akin to a center coordinator. Among the
characteristics associated with the current domain of active 21st CCLC
grantee and center populations are the following:

-   **School districts are still the most represented organizational
    type among grantees**, serving as the fiscal agent on 66 percent of
    all 21st CCLC grants. Community-based organizations (16 percent) and
    nationally affiliated nonprofit agencies (4 percent) collectively
    make up 20 percent of all grantees, with the remaining 14 percent
    representing a wide variety of other organization types. However, 89
    percent of centers are located in schools, indicating that even
    centers funded by a grant obtained by a non-school entity usually
    are housed in schools.

-   **Elementary school students are still the group most frequently
    targeted for services** by centers. About half of all centers serve
    elementary school students exclusively, and nearly two thirds of all
    centers serve at least some elementary students.

-   **Community-based organizations are still the most common type of
    organization to serve as partners on 21st CCLC-funded projects,
    providing centers with connections to the community** and additional
    resources that may not otherwise be available to the program,
    comprising 21 percent of all partners. For-profit entities are the
    next most frequent partner type (14 percent of partners), followed
    by nationally affiliated nonprofit agencies (11 percent) and school
    districts (10 percent). About 27 percent of all partners are
    subcontractors (i.e., under contract with the grantee to provide
    grant-funded activities or services).

-   **In terms of operations,** **nearly all centers at all school
    levels planned to provide programming after the school day**.
    Compared with those serving only elementary students, centers
    serving high schools or both middle and high schools were more
    likely to offer weekend hours.

**Using PPICS Data to Explore Issues Related to Program Quality**

> Although the aforementioned domain of characteristics is useful in
> understanding the basic context in which 21st CCLC programs are
> delivered, a fair degree of attention in the afterschool field has
> been directed at the identification of the features of high-quality
> afterschool programs (Granger, Durlak, Yohalem, & Reisner, 2007;
> Little, 2007; Wilson-Ahlstrom & Yohalem, 2007; Vandell et al., 2005;
> Yohalem & Wilson-Ahlstrom, 2007). Some of these efforts have focused
> on identifying those primary facets to which programs should be
> especially attentive when attempting to improve quality. Areas of
> program quality that have been given attention in these efforts
> include, but are certainly not limited to, the following: the
> intentional development of family, school, and community linkages;
> effective program administration and management practices; paying
> attention to issues related to activity planning and structure; and
> adopting processes to support the development of positive
> student-student and adult-student relationships.

Many of these categories include elements of program delivery that can
only be assessed in a review of organizational procedures and the nature
of the social processes at the point of service delivery (e.g.,
relationships between the adult activity leader and the youth
participating in the activity, the quality of interactions among youth);
however, some of the quality constructs receiving attention in recent
efforts include structural features and program characteristics that can
be informed, at least in part, by data collected in the Annual
Performance Report (APR) module in PPICS. Specifically, data captured in
relation to program partners, activities, staffing, and attendance as
part of the APR reporting process all have some relevance to one or more
categories identified by recent work in the area of quality assessment
and measurement in the field of afterschool.

**Partnerships**

Encouraging partnerships between schools and other organizations is an
important component of the 21st CCLC program. Many states required their
grantees to have a letter of commitment from at least one partner in
order to submit a proposal for funding. Partnerships provide grantees
connections to the community and additional resources that may not be
available to the program otherwise. Partner contributions vary greatly
depending on the available resources and the program's needs. In any
given program, one partner may deliver services directly to
participants, whereas another may provide goods or materials, evaluation
services, or a specific staff member.

Partners represented in the 2005--06 APR were heavily relied upon to
provide programming during center operations, with 69 percent of all
partners and 82 percent of all subcontractors providing this service.
With 21,806 partners represented in the 2005--06 APR, 31 percent of
which are subcontractors, this finding represents a large number of
staff not directly employed by the fiscal agent associated with the
grant who are working at the point of service with youth and adult
family members. Note that partners often operate at the center level and
thus may be a step removed from the grantee itself, which is ultimately
responsible for ensuring the quality of the services being provided to
participants. Although meaningful partnerships are seen as key to
expanding the domain of engaging activities available to participants,
providing a broader continuum of support, offering continuity of
services as participants mature, and contributing to program
sustainability (Jolly, Campbell, & Perlman, 2004), reliance on a network
of loosely controlled partners also makes the process of developing
systems to induct and train high-quality front-line staff more
challenging.

Stable partnerships may contribute to program sustainability, so an
effort was made to examine the extent to which partners identified
during the 2003--04 reporting period were still serving as partners
during the 2004--05 and 2005--06 reporting periods. Results from these
analyses demonstrated that school districts, colleges and universities,
nationally affiliated nonprofit agencies, and units of local government
were most likely to be retained as partners across both the 2004--05 and
2005--06 reporting periods. Partner types less likely to be retained
included faith-based organizations and for-profit entities, even when
controlling for whether or not the partner served as a subcontractor and
the type of support provided by the partner to the program in question.

**Activities**

The mission of the 21st CCLC program is to provide academic and other
enrichment programs that reinforce and complement the regular academic
program of participating students. Relying on information obtained as
part of the 2005--06 APR, an effort was made to assess the breadth of
programming provided by 21st CCLCs during the reporting period in
question and the relative emphasis that centers gave to providing
certain types of activities. Using information on individual activities
provided by centers in 22 states during the 2005--06 school year, we
were able to identify five primary types of centers based on the
relative emphasis they give to offering certain categories of
activities:

-   Centers that provide mostly tutoring activities (13 percent of all
    centers).

-   Centers that provide mostly homework help (14 percent of all
    centers).

-   Centers that provide mostly recreational activities (20 percent of
    all centers).

-   Centers that provide mostly academic enrichment (26 percent of all
    centers).

-   Centers that provide a wide variety of activities across multiple
    categories\
    (27 percent of all centers).

In light of the program's emphasis on the provision of academic
enrichment activities and the achievement of meaningful improvements in
student academic behaviors and achievement, an effort was made to
further explore the degree to which certain core academic subject areas
were addressed in the provision of programming across centers falling in
each of the five aforementioned program types. As shown in Figure 1,
centers falling within the mostly tutoring and mostly homework clusters
on average were more likely to dedicate a higher percentage of total
activity hours to the provision of reading/writing and mathematics
content, followed by centers found in the mostly academic enrichment and
variety clusters.

**Figure 1. Mean Percentage of Activity Hours Offered\
by Subject Area and Program Cluster**

![](media/image2.wmf){width="6.497916666666667in" height="4.05in"}

*Note.* Based on 2,142 centers reporting data in relation to the
2005--06 school year (23 percent of all centers in APR)

Of some interest in Figure 1 is the relatively low average percentage of
activity hours dedicated to reading/writing and mathematics content
among centers falling within the mostly recreation cluster. In light of
this finding, however, it should also be noted that when the relative
maturity of centers was examined, there was some evidence to suggest
that as centers mature over time, there is a movement away from an
overemphasis on recreation in programming to one more oriented toward
the provision of academic enrichment. This finding is shown in Figure 2
in which centers associated with the Cohort 1 bars represent the most
mature centers, whereas the least mature centers can be found in the
Cohort 3 bars. As shown in Figure 2, the percentage of centers falling
within the mostly recreation cluster increases as the length of time a
center has been in operation decreases.

Figure 2. Primary Program Clusters Based on Activity Data Provided\
in Relation to the 2005--06 School Year by Cohort

![](media/image3.wmf){width="6.502777777777778in"
height="3.488888888888889in"}

*Note.* Based on 2,142 centers reporting data in relation to the
2005--06 school year (23 percent of all centers in APR)

**Staffing**

Center staffing is a crucial factor in the success of afterschool
programming, and many of the quality assessment approaches being
developed and used in the field hone in on the capacity of staff
responsible for the delivery of programming to create positive
developmental settings for youth. The quality of staff can be the
difference between an effective program and a mediocre one. In this
regard, the success of afterschool programs is critically dependent on
students forming personal connections with the staff, especially for
programs serving older students in which a much wider spectrum of
afterschool options is available to these participants (Eccles &
Gootman, 2002; Rosenthal & Vandell, 1996).

Although school-day teachers are by far the highest proportion of total
afterschool staff delivering programming during both the summer and
school year, an effort also was made to classify centers into groups or
clusters based on the extent to which they relied upon different
categories of staff to deliver programming during the 2005--06 school
year. Five primary staffing models were identified:

-   Centers staffed mostly by school-day teachers (41 percent of all
    centers).

-   Centers staffed mostly by individuals with some or no college (24
    percent of all centers).

-   Centers staffed mostly by a combination of school-day teacher and
    other nonteaching school-day staff with a college degree (17 percent
    of all centers).

-   Centers staffed by college-educated youth development workers (9
    percent of all centers).

-   Centers staffed by a wide variety of staff types across multiple
    categories (9 percent of all centers).

Staffing data were then compared with the five primary types of centers
based on the relative emphasis they gave to offering certain categories
of activities (i.e., mostly academic enrichment, mostly tutoring, mostly
recreation, mostly homework help, and those centers that provide a
variety of activities). When this was done, as shown in Figure 3, it was
found that the percentage of paid school-year staff made up of teachers
is highest among centers that fall within the mostly tutoring (60
percent of paid staff on average) and mostly homework help (53 percent
of paid staff on average) clusters and then falls steadily across the
remaining clusters, with teachers representing 41 percent of paid staff
on average among centers represented in the mostly recreation cluster.
In addition, centers represented in the mostly recreation cluster are
likely to have a higher percentage of their paid staff fall within the
high school or college student and youth development worker categories
than centers represented in other clusters.

**Figure 3. Mean Percentage of Paid School-Year Staff Falling\
Within a Given Category by Activity Cluster**

![](media/image4.wmf){width="6.500694444444444in"
height="4.053472222222222in"}*Note.* Based on 2,030 centers reporting
individual activities and staffing data in relation to the 2005--06
school year (22 percent of all centers in APR)

**Attendance**

Attendance is an intermediate outcome indicator that reflects the
potential breadth and depth of exposure to afterschool programming.
Grantees completing the APR for the 2005--06 reporting period were asked
to identify both (1) the total number of students who participated in
the center's programming over the course of the year and (2) the number
of students meeting the definition of "regular attendee" by
participating 30 days or more in center activities during the 2005--06
reporting period. The former number can be utilized as a measure of the
breadth of a center's reach, whereas the latter can be construed as a
measure of how successful the center was in retaining students in
center-provided services and activities. Data obtained via the 2005--06
APR indicate that the median number of total attendees at a given center
was 122, and the median number of regular attendees was 72. Overall,
during the 2005--06 reporting period, 1,456,447 total students were
served by the 21st CCLC program, with 807,191 attending a given center
for 30 days or more.

An effort also was made to compare the average rate of regular
attendance across each of the primary program clusters outlined in
Figures 1--3. Although the average rate of regular attendance was found
to be fairly equivalent across programs that provided mostly homework
help, mostly academic enrichment, and a variety of activities (all of
which were above 60 percent in terms of the percentage of students
served meeting the definition of regular attendee), there was a
noticeable drop in the average rate of regular attendance among programs
that provided mostly recreational activities (54 percent of students
meeting the definition of regular attendee) and especially among
programs that focus primarily on the provision of tutoring services (46
percent of students meeting the definition of regular attendee).

The rate of regular attendance also was considered across program
cluster type based on the relative maturity of the centers in question.
As shown in Figure 4, for centers falling within the mostly homework
help, mostly academic enrichment, and variety clusters, there is a clear
trend that suggests that more mature programs (Cohort 1 represents the
most mature programs and Cohort 3 the least mature) have a higher rate
of average regular attendance than centers that are relatively new. This
trend is especially pronounced among centers represented in the mostly
academic enrichment cluster in which Cohort 1 centers have an average
rate of regular attendance of 71 percent, whereas the rate for Cohort 3
centers is 48 percent.

**Figure 4. Average Rate of Regular Attendance Among Centers by\
School Year Activity Cluster and Cohort**

![](media/image5.wmf){width="6.502083333333333in"
height="3.5097222222222224in"}

*Note.* Based on 967 centers operating only during the 2005--06 school
year providing attendance and individual activities

**Summary of Characteristics of 21st CCLC Programs**

From the perspective of further informing how quality assessment systems
should be constructed and implemented to help programs better meet the
purposes of the 21st CCLC program, the data highlighted in this report
suggest that consideration should be given to differentiating these
efforts in light of the staffing and activity models employed at a given
center. For example, the conversations related to improving the quality
of offerings among centers providing mostly tutoring services and that
largely employ school-day teachers are likely to be qualitatively
different than those in centers focusing mostly on recreation and that
employ a larger proportion of youth development workers and students
drawn from area high schools and colleges as staff.

Results outlined in this section clearly highlight differences among
centers based on their relative maturity. Over time, there is some
evidence to suggest that centers will increasingly move toward an
academic enrichment model in the delivery of programming and be less
dependent primarily on recreational activities to fill their programming
slate. In addition, there is also some evidence to suggest that programs
get better at retaining students in programming over time, especially
for those centers falling within the academic enrichment, homework help,
and variety clusters. Thinking about how this program evolution should
influence the development of quality assessment systems also would seem
to be a viable and worthwhile undertaking in terms of helping programs
get to a more ideal level of functioning and to get there more quickly.

**Student Achievement and Academic Behavioral Outcomes: Performance
Indicator Results**

In addition to collecting information on the operational characteristics
of 21st CCLC programs, one of the primary purposes of the APR is to
collect data to inform how well the program is meeting the Government
Performance and Results Act (GPRA) indicators established for the
program. The GPRA indicators associated with the 21st CCLC program are
the primary tools by which the U.S. Department of Education and other
agencies of the federal government evaluate the effectiveness of 21st
CCLCs operating nationwide. The indicators were established to gauge the
extent to which 21st CCLC programs could demonstrate changes in student
academic behaviors and achievement as well as the delivery of
statutorily required enrichment activities.

An overall summary of how the 21st CCLC programs that reported GPRA
indicator data for the 2005−06 reporting period is outlined in Table 1,
along with results from the previous two reporting periods. Generally,
the reported percentage of regular attendees with improvements in grades
was higher than the percentages for the 2004−05 reporting period,
particularly in relation to mathematics grades, but teacher-reported
improvements in student behaviors were generally lower than last year's
levels. There were also fairly dramatic drops in the percentage of
regular attendees who attained proficiency on state assessments,
although this change should not be overinterpreted because there was a
dramatic growth in the percentage of states reporting assessment data
for regularly participating students, increasing from 9 percent in
2004--05 to 38 percent in 2005--06. This development should be kept in
mind when comparing the most recent state assessment results with those
obtained from earlier reporting periods.

Table 1. Status of the GPRA Indicators Associated With the 21st CCLC
Program 2003−04, 2004−05, and 2005−06 Reporting Periods

+-----------------------+-----------+-----------+-----------+---------+
| **GPRA Performance    | **Pe      | *         | *         | **      |
| Indicator**           | rformance | *2003--04 | *2004--05 | 2005−06 |
|                       | Targets** | Reporting | Reporting | Re      |
|                       |           | Period**  | Period**  | porting |
|                       |           |           |           | P       |
|                       |           |           |           | eriod** |
+=======================+===========+===========+===========+=========+
| > Regular attendees   | 45%       | 44.55%    | 41.47%    | 42.52%  |
| > demonstrating       |           |           |           |         |
| > improved grades in  |           |           |           |         |
| > reading/language    |           |           |           |         |
| > arts                |           |           |           |         |
+-----------------------+-----------+-----------+-----------+---------+
| > Regular attendees   | 45%       | 40.84%    | 38.82%    | 42.49%  |
| > demonstrating       |           |           |           |         |
| > improved grades in  |           |           |           |         |
| > mathematics         |           |           |           |         |
+-----------------------+-----------+-----------+-----------+---------+
| > Regular attendees   | N/A       | Not       | 27.90%    | 20.63%  |
| > demonstrating       |           | reported  |           |         |
| > improved state      |           |           |           |         |
| > assessment results  |           |           |           |         |
| > in reading/language |           |           |           |         |
| > arts                |           |           |           |         |
+-----------------------+-----------+-----------+-----------+---------+
| > Regular attendees   | N/A       | Not       | 29.77%    | 20.82%  |
| > demonstrating       |           | reported  |           |         |
| > improved state      |           |           |           |         |
| > assessment results  |           |           |           |         |
| > in mathematics      |           |           |           |         |
+-----------------------+-----------+-----------+-----------+---------+
| > Regular attendees   | 75%       | 68.72%    | 74.98%\*  | 72.56%  |
| > demonstrating       |           |           |           |         |
| > improved homework   |           |           |           |         |
| > completion and      |           |           |           |         |
| > class participation |           |           |           |         |
+-----------------------+-----------+-----------+-----------+---------+
| > Regular attendees   | 75%       | 64.04%    | 71.08%\*  | 67.94%  |
| > demonstrating       |           |           |           |         |
| > improved student    |           |           |           |         |
| > behavior            |           |           |           |         |
+-----------------------+-----------+-----------+-----------+---------+
| > Centers emphasizing | 85%       | 97.73%    | 95.06%    | 95.49%  |
| > at least one core   |           |           |           |         |
| > academic area       |           |           |           |         |
+-----------------------+-----------+-----------+-----------+---------+
| > Centers offering    | 85%       | 65.61%    | 65.75%    | 64.32%  |
| > enrichment and      |           |           |           |         |
| > support activities  |           |           |           |         |
| > in technology       |           |           |           |         |
+-----------------------+-----------+-----------+-----------+---------+
| > Centers offering    | 85%       | 92.51%    | 94.09%    | 94.91%  |
| > enrichment and      |           |           |           |         |
| > support activities  |           |           |           |         |
| > in other areas      |           |           |           |         |
+-----------------------+-----------+-----------+-----------+---------+

\* The survey instrument was changed for the 2004--05 reporting period
to allow teachers to select "did not need improvement." This option was
not present in the 2003--04 survey. Efforts to validate the teacher
survey demonstrated that the items functioned differently depending upon
the grade level of the student in question.

**Evaluating Recommendations for Changing the GPRA Indicators**

In recent months, questions have been raised increasingly in meetings
sponsored by the U.S. Department of Education regarding both the
validity of the data supporting the results highlighted in Table 1 as
well as the utility of these metrics as the primary mechanisms by which
the efficacy of the 21st CCLC program is evaluated. In particular, two
design elements of the current PPICS application may complicate efforts
to obtain more meaningful data on grantee performance:

-   The vast majority of the data collected in PPICS that relates to
    student improvements in academic behaviors and achievement are
    reported directly by the grantees without consistent procedures in
    place to independently verify accuracy.

-   Information that is supplied to support the calculation of the
    student achievement and behavioral change performance indicators
    associated with the program is collected at the center level, as
    opposed to the individual student level. For example, when
    calculating the metrics associated with improvements in students'
    mathematics grades, fields are referenced in PPICS that contain data
    supplied by a grantee or center-level respondent in which the total
    number of regular attendees witnessing an improvement in mathematics
    grades has been reported. To do this, center officials (or perhaps
    their evaluator) collect and aggregate the student report card
    information and then report the aggregate results when completing
    the grades page in PPICS.

The data reported in PPICS used to support GPRA indicator calculations
are based on grantee self-reports and are obtained through aggregated
figures reported at the center-level, so the accuracy of the information
being supplied cannot be independently verified at a level of precision
that would be associated with a more finely controlled data-collection
effort. In some respects, these attributes of grantee-level data
collection through PPICS represent the domain of compromises that seemed
reasonable three years ago when PPICS was being designed, given concerns
raised by some state 21st CCLC coordinators relative to the collection
of individual student-level data and the inherent challenges in
obtaining data from close to 10,000 21st CCLCs in operation nationwide.

There has been an increasing recognition, however, that it may be
appropriate to reassess the viability and appropriateness of both the
current domain of performance indicators for the 21st CCLC program and
the manner in which that data are obtained. In light of this
recognition, the U.S. Department of Education asked Learning Point
Associates to do the following:

-   Undertake a process to obtain systematic input and advice from
    members of the 21st CCLC PPICS ETF and other experts in the field of
    afterschool research and evaluation on how the current domain of
    performance indicators associated with the program should be revised
    and modified.

-   Determine how these suggested changes to the indicators will impact
    data-collection approaches embedded in PPICS.

Meetings held to date for this purpose have yielded the following
recommendations from state 21st CCLC coordinators and afterschool
research and evaluation experts represented on the PPICS ETF:

-   *Indicators should be developed that demonstrate grantee success in
    helping students remain at an acceptable level of academic
    performance*. One of the flaws associated with the indicators
    related to assessing changes in academic behaviors and performance
    is that by themselves they fail to capture information about the
    extent to which students performing at an acceptable level remain at
    such a level, even in the face of increasingly difficult and
    challenging academic content. This flaw is probably most evident in
    relation to grades reporting in which students, for example, who
    receive a B+ in mathematics both at the end of the first and last
    marking periods of a given school year are not positively reflected
    in the indicator calculations for a given center. In this regard, a
    new set of indicators should be developed to capture instances in
    which a regular attendee has maintained performance at an acceptable
    level of functioning.

-   *Account for grade-level differences in the indicators that are
    assessed for performance measurement purposes.* There is emergent
    data both in the teacher survey validation studies performed by
    Learning Point Associates and in the program characteristic and
    performance data housed in PPICS that suggest that distinctions can
    be made across programs serving varying grade levels of students. In
    particular, it appears that 21st CCLCs serving high school students
    are meaningfully distinct from those programs serving students
    enrolled in elementary or middle school grades. In terms of program
    attributes, based on data housed in PPICS, 21st CCLCs exclusively
    serving high school students are more apt to serve a larger number
    of total students per year while demonstrating the lowest rate of
    regular attendance. Of some interest, however, is that high school
    programs on the whole perform better on some of the academic
    achievement indicators as compared to programs exclusively serving
    middle school students.

```{=html}
<!-- -->
```
-   Such results offer some interesting areas for further exploration,
    especially in terms of the afterschool programming that may be
    appropriate at the high school level, taking into consideration both
    the specific developmental needs and motivation of such students,
    given the plethora of other options they have in terms of how they
    spend their afterschool hours. In addition, in terms of behavioral
    change, some outcomes are certainly more relevant to high school
    populations as compared to programs serving students from other
    grade levels. An improvement in school-day attendance, for example,
    is significantly more relevant to high school students than either
    middle or elementary students. Similarly, given that most state
    assessment systems are less apt to test high school students
    annually, state assessment results are largely unavailable for this
    population, leaving a gap in terms of assessing how such programs
    may have positively impacted student achievement among high school
    attendees.

```{=html}
<!-- -->
```
-   *Include only student attendees who need improvement in terms of
    academic behaviors and achievement in indicator calculations*.
    Although this approach is already taken in relation to assessing
    regular attendee improvements in state assessment results and
    teacher-reported behaviors, the indicator calculation related to
    assessing changes in student grades has typically not factored out
    students who could not improve because they had attained the highest
    grade possible on the grading scale being employed at the end of the
    first marking period in the fall. Revised indicator calculations
    related to grades should remove such students from the denominator
    associated with these calculations.

-   *Base academic achievement indicators on a net change calculation.*
    At present, when the indicators related to improvements in student
    grades and self-assessment results are calculated, consideration is
    given only to those students who witnessed an improvement relative
    to the total regular attendee population with data reported in that
    area; however, without considering how many students also witnessed
    a decline in grades or state assessment results in calculating these
    respective indicators, the percentage of regular attendees
    witnessing an increase is overstated and less indicative of what the
    program's impact may have been on student achievement.

-   *Revise and expand the current domain of indicators associated with
    program content and quality.* Although there is widespread agreement
    that the current domain of indicators related to program content and
    delivery provide little in the way of useful information about the
    programming being provided by 21st CCLCs nationwide or the quality
    of such offerings, there has been much less consensus on what should
    replace them. Some suggestions in this area advise focusing such
    indicators on assessing center success in meeting more immediate and
    shorter term outcomes (e.g., increasing the acquisition of study
    skills among participants, improving school-day attendance, lowering
    disciplinary referrals). Others have suggested collecting additional
    information in PPICS regarding how states and grantees are engaging
    in quality assessment and program improvement processes that could
    serve as a foundation for a new series of metrics to assess how
    involved and vested states and grantees are in continuous
    improvement efforts.

Although many of these recommendations will require additional study and
discussion to make them more concrete and operationally viable, the
feasibility of several of the recommendations can be assessed relatively
easily by using the existing PPICS dataset to recalculate the indicators
related to teacher-reported changes in student behavior, grades, and
state assessment results.

Based on the reanalysis of PPICS data consistent with the
recommendations made in recent months regarding what steps should be
taken to revise the 21st CCLC performance indicators, some of the more
interesting and meaningful findings are presented below:

-   Revising the denominator associated with the improvement in grades
    indicator by excluding regular attendees that had already achieved
    the highest grade possible at the end of the first fall marking
    period had only a small impact on results associated with the net
    change approach to calculating the indicator. Although this result
    suggests that on the whole, 21st CCLC programs are not serving large
    percentages of straight-A students, it does not reveal the extent to
    which programs are serving students who generally are performing at
    an acceptable level of functioning or the role programs are playing
    in helping students remain in good academic standing during the
    course of the school year. It has been suggested that consideration
    should be given to adding an indicator to the performance
    measurement system that captures the role programs play in helping
    students maintain good grades.

-   As in previous years, the 2006 APR defined a regular attendee as a
    student who attended the center for 30 or more days during the
    reporting period. There has been some concern expressed by both
    grantees and SEA staff that this definition may be too low of a
    threshold of interaction with center programming to produce the
    expected outcomes in terms of grades, student achievement, and
    behavior. To further explore the relationship between levels of
    program attendance and student behavioral change and academic
    outcomes, states were afforded the option for the first time in
    2005--06 to require that their grantees submit APR grades data
    separately for three subgroups of regular attendees: (1) those
    attending 30 to 59 days during the reporting period, (2) those
    attending 60 to 89 days, and (3) those attending 90 days or more. An
    analysis of this information can provide insight into the
    relationship between program attendance and behavioral and
    achievement outcomes and contribute to the discussion around what an
    appropriate threshold may be when considering how to define a
    regular attendee.

```{=html}
<!-- -->
```
-   Once students reach the 60-day attendance threshold, there appears
    to be a noticeable increase in the percentage of students improving
    their grades. A similar finding was witnessed in relation to student
    gains on state assessments, although in this case, the jump in the
    percentage of regular attendees witnessing an increase occurs at the
    90 days or more attendance threshold. Such results seem to raise a
    series of questions around both the appropriateness of varying the
    definition of a regular attendee, depending upon the type of outcome
    being evaluated and the relationship between higher levels of
    attendance and the achievement of certain types of youth outcomes.

```{=html}
<!-- -->
```
-   Centers that exclusively served high school students demonstrate by
    far the lowest net percentage of regular attendees moving to a
    higher proficiency category between state assessments; however, such
    results may be more reflective of the fact that many states do not
    annually test high school students, resulting in a fairly small
    sample of students upon which these findings are based. In light of
    this, additional discussion seems warranted on excluding high school
    students from indicators predicated on an analysis of state
    assessment results.

-   In contrast to state assessment results, centers that exclusively
    served high school students demonstrated higher levels of
    performance on the grades indicator as compared to centers that
    served only middle school students or that served some combination
    of middle and high school students. This was the case even though
    centers exclusively serving high school students demonstrated a
    lower rate of regular attendance as compared to centers serving
    youth enrolled in middle school. Such results offer some interesting
    areas for further exploration, especially in terms of the
    afterschool programming that may be most appropriate at the high
    school level.

-   There appears to be a noticeable linear increase in the percentage
    of regular attendees witnessing an improvement in teacher-reported
    behaviors across each of the ten constructs under consideration when
    examining results for three subgroups of regular attendees: (1)
    those attending 30 to 59 days during the reporting period, (2) those
    attending 60 to 89 days, and (3) those attending 90 days or more.
    Although there are a number of concerns regarding the relevance of
    the overall percentage of regular attendees with teacher-reported
    improvements in academic-related behaviors, the linear increase in
    improvement across ascending levels of program attendance seems to
    warrant attention and adds weight to the benefits that could be
    obtained from a program assessment standpoint if student-level
    teacher survey and attendance data could be obtained. This would
    allow for further examination of the relationship between levels of
    program participation and teacher-reported changes in student
    behavior.

-   In terms of teacher-reported improvements in homework completion and
    homework quality, class participation, and overall academic
    performance, high school students generally demonstrate the lowest
    levels of improvement as compared to their peers in other grade
    levels. High school students, however, typically attained higher
    levels of improvement in terms of attending class regularly than
    students in other grade levels. As suggested by ETF members, such
    results may indicate which elements warrant attention when
    developing different teacher surveys by grade level.

**Learning Point Associates Recommendations Related to Acting on
Suggestions for Changing the GPRA Indicators**

It is clear that many of the recommendations made in the past few months
on how the performance indicators could be revised will require
additional study and discussion to make them more concrete and
operationally viable. In addition, in considering how these
recommendations could be incorporated into a revised set of indicators
for the 21st CCLC program and the federal reporting requirements housed
in PPICS, discussions held to date on this topic have increasingly
focused on the utility of collecting individual student-level attendance
and impact category data in PPICS. Generally, collecting data at the
individual level could offer the following benefits:

-   Alleviates the need for grantees to perform complex data aggregation
    tasks that can be easily misunderstood or subject to errors in
    compilation.

-   Allows for greater precision in exploring the relationship among
    program attendance, grade level, and student behavioral and academic
    outcomes.

-   Allows for the application of a broader array of psychometric
    techniques when analyzing teacher survey data.

Given the benefits that can be gained by collecting student-level
attendance and impact category data in PPICS, there have been
preliminary discussions about the possibility of sponsoring a pilot
during the course of the 2007--08 reporting period that would afford
states and grantees the option of reporting student-level attendance and
outcome data in PPICS. Data resulting from this pilot should further
inform what constitutes reasonable measures of student achievement and
academic behavioral change as they relate to the provision of the
activities and services provided as part of the 21st CCLC program and
how these metrics should be differentiated to account for various types
of programs.

Finally, revisions to the performance indicators should be viewed within
the context of supporting program improvement efforts, both at the state
and grantee levels. This may mean that consideration should be given to
adopting indicators that speak to the extent to which states and
programs are making strides in developing and participating in quality
assessment systems that lead to concrete efforts to improve program
quality, especially at the point of service delivery.

**Introduction**

For almost a decade, 21st CCLCs have provided students in high-poverty
communities across the nation the opportunity to participate in
afterschool enrichment programs designed to enhance their academic and
social progress. To ensure that comprehensive data on the 21st CCLC
program would be available, the U.S. Department of Education contracted
in 2003 with Learning Point Associates to design and implement PPICS, a
Web-based data-collection application that captures an extensive range
of descriptive and performance information from all state-administered
21st CCLC programs.

To monitor how well the 21st CCLC program is operating, centers funded
by the program are required to complete a yearly Annual Performance
Report (APR). The APR is a data-collection module (a subset of the
larger PPICS) that assesses the extent to which centers are achieving
the statutorily authorized purposes of the program. The purposes of the
APR module of PPICS are as follows: (1) to collect data from 21st CCLC
grantees on progress made during the preceding year in meeting their
project objectives, (2) to collect data on what elements characterized
center operation during the reporting period, including the student and
adult populations served and the activities provided, and (3) to collect
data that describe the extent to which regular participants in the
program improved their academic behaviors and achievement.

In response to a disappointing early evaluation of the 21st CCLC program
(James-Burdumy et al., 2004), greater attention was focused at the
federal, state, and local levels on creating and sustaining programs of
substantially higher quality that would be more likely to have
measurable, positive effects on participants' academic and social
progress. Since then, both individual 21st CCLC programs and the
afterschool field in general have matured in many respects, particularly
in better understanding how to implement high-quality afterschool
services and in measuring the impact of those services on the academic
achievement and behavioral patterns of regular attendees of such
programs.

In particular, recent efforts in the field have been focused in three
areas: (1) identifying promising programs and resources in core academic
content and service delivery areas,\
(2) identifying the common features of high-quality afterschool settings
and services, and\
(3) developing quality assessment tools and support systems to help
programs better understand how to enhance the quality of their
approaches and offerings (Granger et al., 2007; Little, 2007; Southwest
Educational Development Laboratory, 2007; Wilson-Ahlstrom & Yohalem,
2007; Yohalem & Wilson-Ahlstrom, 2007). In fact, the work done by the
Southwest Educational Development Laboratory (SEDL) to identify
promising content delivery strategies extensively utilized APR and PPICS
data, supplemented by onsite validation reviews.

These latter issues raise the questions of whether and, if so, how PPICS
data can be useful to inform questions around program quality. The
answer to these questions will depend in large part on how success is
defined and measured by the performance indicators that have been
established for the 21st CCLC program. These indicators, calculated each
year in accordance with the GPRA, are a primary tool by which ED
evaluates the effectiveness of the 9,824 21st CCLCs operating
nationwide. The indicators in question are predicated on measuring
changes in student academic behaviors and achievement, as well as
elements of program delivery and content, as reported by the 21st
CCLC-funded programs operating during the course of a given reporting
period.

The issues of program quality and performance measurement are of great
importance to U.S. Department of Education officials, to the state
educational agency (SEA) staff members who administer and monitor their
programs, and the grantees directly serving youth in the field. With
that expanded audience in mind, this report has been structured to
address three primary questions:

-   How can the current information collected about the characteristics
    of 21st CCLC programs be used to help inform issues related to the
    quality of programming being provided to youth at these centers?

-   To what extent did programs operating during the course of the
    2005--06 reporting period meet the GPRA performance targets
    established for the program?

-   What would be the effect of implementing recommendations made by the
    PPICS evaluation task force for modifying some of the GPRA
    indicators?

Each of these questions will be addressed by employing the data
collected through PPICS. Most of the data analyzed in this report were
collected as part of the 2005--06 APR process, which covers activities
undertaken by 21st CCLC-funded programs during the summer of 2005 and
the 2005--06 school year. Additional analyses include data obtained from
the two prior APR collections in 2003--04 and 2004--05.

**Section 1: Characteristics of 21st CCLC Programs**

As of December 2006, there were 3,309 active 21st CCLC grantees across
the country, which, in turn, operated a total of 9,824 centers. Note
that the term *grantee* in this report refers to the entity that serves
as the fiscal agent for a given 21st CCLC grant, while *center* refers
to the physical location where grant-funded services and activities are
provided to participating students and adults. (On average, a single
grant supports three centers.) A center offers academic, artistic, and
cultural enrichment opportunities to students and their families during
nonschool hours (before or after school) or periods when school is not
in session (e.g., holidays, weekends, or summer recess). A center can
also be characterized by defined hours of operation; a dedicated staff
that plans, facilitates, and supervises program activities; and an
administrative structure that may include a position akin to a center
coordinator. The following characteristics are associated with the
current domain of active 21st CCLC grantee and center populations:

-   **School districts are still the most represented organizational
    type among grantees**, serving as the fiscal agent on 66 percent of
    all 21st CCLC grants. Community-based organizations (16 percent) and
    nationally affiliated nonprofit agencies (4 percent) collectively
    make up 20 percent of all grantees, with the remaining 14 percent
    representing a wide variety of other organization types; however, 89
    percent of centers are located in schools, indicating that even
    centers funded by a grant obtained by a nonschool entity usually are
    housed in schools.

-   **Elementary school students are still the group most frequently
    targeted for services** by centers. About half of all centers serve
    elementary school students exclusively, and nearly two thirds of all
    centers serve at least some elementary students. (For a more
    detailed description of how centers are classified based on the
    grade level of students they serve, please see Appendix B.)

-   **Community-based organizations are still the most common type of
    organization that serve as partners on 21st CCLC-funded projects,
    providing centers with connections to the community** and additional
    resources that may not otherwise be available to the program,
    comprising 21 percent of all partners. For-profit entities are the
    next most frequent partner type (14 percent of partners), followed
    by nationally affiliated nonprofit agencies (11 percent) and school
    districts (10 percent). About 27 percent of all partners were
    subcontractors (i.e., under contract with the grantee to provide
    grant-funded activities or services).

-   **In terms of operations,** **nearly all centers at all school
    levels planned to provide programming after the school day**.
    Compared with those serving only elementary students, centers
    serving high schools or both middle and high schools were more
    likely to offer weekend hours.

> Although this domain of characteristics is useful in understanding the
> basic context in which 21st CCLC programs are delivered, recently much
> more attention and effort in the field has been directed at the
> identification of the features of high-quality afterschool settings
> (Granger et al., 2007; Little, 2007; Wilson-Ahlstrom & Yohalem, 2007;
> Vandell et al., 2005; Yohalem & Wilson-Ahlstrom, 2007). Some of these
> efforts have focused on identifying those primary facets of program
> quality that programs should be especially attentive to when
> attempting to improve the quality of their programs and offerings.
> Areas of program quality that have been given attention in these
> efforts would include, but are certainly not limited to, things like
> the intentional development of family, school, and community linkages;
> effective program administration and management practices; paying
> attention to issues related to activity planning and structure; and
> adopting processes to support the development of positive
> student-student and adult-student relationships.

Many of these categories include elements of program delivery that can
only be assessed in a review of organizational procedures and the nature
of the social processes at the point of service delivery (e.g.,
relationships between the adult activity leader and the youth
participating in the activity, the quality of interactions among youth).
Some of the quality constructs receiving attention in recent efforts,
however, include structural features and program characteristics that
can be informed, at least in part, by data collected in the APR module
in PPICS. Specifically, data captured in relation to program partners,
activities, staffing, and attendance as part of the APR reporting
process all have some relevance to one or more categories identified by
recent work in the area of quality assessment and measurement in the
field of afterschool. In the sections that follow, data captured in
PPICS for the 2005--06 reporting period are examined for each of these
four areas in greater detail with the intent to explore how the
information in question may be relevant to the current program quality
discussion.

**Partnerships**

Encouraging partnerships between schools and other organizations is an
important component of the 21st CCLC program. Many states required their
grantees to have a letter of commitment from at least one partner in
order to submit a proposal for funding. Partnerships provide grantees
connections to the community and additional resources that may not be
available to the program otherwise. Partner contributions vary greatly
depending on the available resources and the program's needs. In any
given program, one partner may deliver services directly to
participants, whereas another may provide goods or materials, evaluation
services, or a specific staff member. Figure 5 displays the percentage
of partners and subcontractors providing each contribution type tracked
in PPICS for the 2005--06 reporting period. A subcontractor is a type of
partner that is under contract with the grantee to provide 21st CCLC
grant-funded activities or services. 21st CCLC programs use both types
of partners to provide services and resources for their participants.

Figure 5 also describes the extent to which partners are relied upon to
provide programming, with 69 percent of all partners and 82 percent of
all subcontractors providing this service. With 21,806 partners
represented in the 2005--06 APR, 31 percent of which are subcontractors,
this finding represents a large number of staff not directly employed by
the fiscal agent associated with the grant that are working at the point
of service with youth and adult family members. Note that partners often
operate at the center level and thus, may be a step removed from the
grantee itself, which is ultimately responsible for ensuring the quality
of the services being provided to participants. Although meaningful
partnerships are seen as key to expanding the domain of engaging
activities to participating youth, providing a broader continuum of
support, offering continuity of services as a youth matures, and
contributing to program sustainability (Jolly et al., 2004), reliance on
a network of loosely controlled partners also makes the process of
developing systems to induct and train high-quality front-line staff
more challenging.

Figure 5 also shows that 10 percent of all partners were identified as
providing evaluation services during the 2005--06 reporting period.
Using data to examine the impact a program is having upon desired
outcomes and how well a program may be operating relative to defined
standards of quality is one of the categories identified by Little
(2007) in her scan of quality assessment tools; however, at this point,
PPICS cannot answer questions about how good the data from these local
evaluation efforts are or the extent to which these evaluations inform
program improvement efforts.

> Figure 5. Percentage of Partners and Subcontractors\
> Providing the Described Service

![](media/image6.wmf){width="6.499305555555556in"
height="4.106944444444444in"}

*Note.* Based on 2,705 grantees providing data (91 percent of all
grantees required to complete an APR)

In light of the role subcontractors play in the delivery of programming
across 21st CCLC programs nationwide, an effort was made to identify the
extent to which grantees subcontracted out their 21st CCLC grant
allocations during the 2005--06 program year. As shown in Figure 6, 40
percent of grantees reported having no subcontractors during the course
of the 2005--06 reporting period, whereas 28 percent reported
subcontracting out up to 10 percent of their grant amount for the year
in question. Generally, there were few grantees that reported
subcontracting large portions of their program to outside entities.

**Figure 6. Percentage of Grantees Subcontracting a Given\
Percentage of Their Grant Amount**

![](media/image7.wmf){width="6.500694444444444in"
height="3.571527777777778in"}

*Note.* Based on 2,687 grantees providing data (90 percent of all
grantees required to complete an APR)

In nearly all settings, long-term, stable partnerships are perceived as
a way to help ensure the quality of programming and enhance a program's
chances of sustaining itself when its 21st CCLC grant ends. For programs
that were initially funded in 2002--03, the 2005--06 APR represents the
third time they have reported on who their partners were and what role
they played in supporting their programs. For these programs, the issue
of sustainability is, or soon will become, a critical concern as these
grants approach their end and the grantees face the possibility of
losing their funding.

Stable partnerships may contribute to program sustainability, so an
effort was made to examine the extent to which partners identified
during the 2003--04 reporting period were still serving as partners
during both the 2004--05 and 2005--06 reporting periods. Figure 7
presents the percentage of partners first appearing in the APR for the
2003--04 reporting period and that continued to be associated with that
grantee in the 2004--05 and 2005--06 APR, organized by the type of
partner organization. This information can begin to answer questions
related to the extent to which partners continue to contribute to a
program over time and whether longevity of partnerships appears to be
related to the types of organizations that typically serve as partners.

As shown in Figure 7, school districts, colleges and universities,
nationally affiliated nonprofit agencies, and units of local government
were most likely to be retained as partners across both the 2004--05 and
2005--06 reporting periods. Partner types less likely to be retained
included faith-based organizations and for-profit entities. In Figures 7
and 8, the following codes are employed to represent the various
organizational types:

-   *CBO:* Community-based organizations

-   *College:* Colleges or universities

-   *Faith-based:* Faith-based organizations

-   *For-profit:* For-profit entities

-   *Health-based:* Health-based organizations

-   *NANPA:* Nationally affiliated nonprofit agencies

-   *Park District:* Local park or recreational district

-   *School District:* Public school district

-   *Gov Unit:* Unit of local government

-   *Other*

**Figure 7. Percentage of Partners Retained From the\
2003--04 Reporting Period by Organization Type**[^1]

![](media/image8.wmf){width="6.497916666666667in"
height="3.566666666666667in"}

*Note.* Based on 939 grantees providing data (32 percent of all grantees
required to complete an APR).

A similar chart is represented in Figure 8, but here, partners have been
categorized by those without subcontracts (identified as *Nonsub* in the
chart) and those serving as subcontractors (identified as *Sub* in the
chart). Generally, the trend in terms of longevity across organizational
type is the same as that noted in relation to Figure 7; however,
entities serving as subcontractors have, on the whole, a greater
likelihood of being retained during the 2004--05 and 2005--06 reporting
periods than partners without a subcontract.

**Figure 8. Percentage of Nonsubcontractors and Subcontractors Retained\
From the 2003--04 Reporting Period by Organization Type**

![](media/image9.wmf){width="6.498611111111111in"
height="3.609027777777778in"}

*Note.* Based on 939 grantees providing data (32 percent of all grantees
required to complete an APR)

The important thing to keep in mind in relation to Figures 7 and 8 is
that they both attempt to answer the following questions:

-   To what extent does a given partner, initially reported as being
    associated with a grantee during the 2003--04 reporting period,
    remain associated with that grantee in both the 2004--05 and
    2005--06 reporting periods?

-   To what extent does the answer to the first question depend upon the
    organization type associated with the partner in question?

Although differences are noted in the probability that a partner will
remain associated with a given program over time, one can only speculate
as to why some types of partners are more apt to drop out across APR
reporting cycles as compared to others in light of the fact that we do
not have any empirical evidence in PPICS that outlines the reason for
why a given grantee may have ended its relationship with a given
partner.

Despite these limitations, one additional way the partner data housed in
PPICS can be examined is to explore how the longevity of a
partner-grantee relationship may be related to the manner in which a
partner contributes to the afterschool program in question. This can be
done most effectively by collapsing the services provided by partners
outlined in Figure 5 into two categories: (1) one category that relates
to supporting the direct delivery of activities to participants
(providing programming or paid or volunteer staff), and (2) activities
that can be termed nonprogrammatic in nature (raising funds, providing
evaluation services and goods, or contributing in some other way). In
Figure 9, partners first appearing in the 2003--04 APR are categorized
by whether or not they provided services supportive of direct program
delivery or contributed in a nonprogrammatic fashion. Generally,
partners are more likely to be retained across APR years if they
provided programming or paid or volunteer staff (APR Program) as
compared to contributing to the project in a nonprogrammatic fashion
(APR Non-Program). Irrespective of these differences related to
contribution type, however, differences in the retention rate of
partners based on organization type remain and are consistent with the
findings highlighted in Figures 7 and 8.

**Figure 9. Percentage of Partners Retained From the 2003--04 Reporting
Period by Organization Type and by Contribution Type**

![](media/image10.wmf){width="6.495833333333334in"
height="3.767361111111111in"}

*Note.* Based on 939 grantees providing data (32 percent of all grantees
required to complete an APR)

**Activities**

The mission of the 21st CCLC program is to provide academic and other
enrichment programs that reinforce and complement the regular academic
program of participating students. Relying on information obtained as
part of the 2005--06 APR, this section outlines the breadth of
programming provided by 21st CCLCs during the reporting period in
question. Respondents to the 2005--06 APR were able to classify a single
activity both by category and subject area, so activities data can be
analyzed using two broad rubrics for describing programming: (1) the
category within which an activity fell and (2) the academic subject
areas addressed by the programming in question. For example, a center
may have offered a rocketry club in which participants learned to build
and launch rockets while also studying astronomy. In this case, this
activity would be classifiable as an academic enrichment learning
program (category of activity) *and* as a science educational activity
(subject area of activity).

When reporting activities in PPICS offered during 2005--06, respondents
were able to classify 21st CCLC programming by category employing the
following list:

-   *Enrich:* Academic enrichment learning programs

-   *Tutor:* Tutoring

-   *Homework:* Homework help

-   *Mentor:* Mentoring

-   *Rec:* Recreational activities

-   *CareerYouth:* Career/job training for youth

-   *Drug:* Drug and violence prevention, counseling, and character
    education programs

-   *Lib:* Expanded library service hours

-   *Suppl:* Supplemental educational services

-   *CommServ:* Community service or service-learning programs

-   *Lead:* Activities that promote youth leadership

-   *ParentInv:* Programs that promote parental involvement

-   *FamLit:* Programs that promote family literacy

-   *CareerAdult:* Career/job training for adults

In addition, respondents were able to identify whether any of the
following content areas were intentionally embedded in one or more
activities undertaken at a given site:

-   *Read:* Reading/literacy education activities

-   *Math:* Mathematics education activities

-   *Science:* Science education activities

-   *Arts:* Arts and music education activities

-   *Bus:* Entrepreneurial education programs

-   *Tech:* Telecommunications and technology education programs

-   *Culture:* Cultural activities/social studies

-   *Health:* Health/nutrition-related activities

In many respects, the spectrum of activities represented across these
categories and subject areas reflects the mandate of the 21st CCLC
program to promote academic achievement and to provide access to
enrichment and other youth development and support activities. Figure 10
summarizes the proportion of centers offering different categories of
activities and services for both the 2005--06 school year and the summer
of 2005. In terms of activities provided during the school year, the
vast majority of centers offered activities classified as academic
enrichment (83 percent), recreation (81 percent), tutoring (64 percent),
and homework help (63 percent). During the course of the summer, both
academic enrichment and recreational activities were predominant, with
81 percent and 69 percent of centers providing these types of activities
respectively.

Figure 10. Proportion of Centers Providing Programming by Category,\
School Year and Summer

![](media/image11.wmf){width="6.497222222222222in"
height="3.748611111111111in"}

*Note.* Based on 8,767 centers reporting data in relation to the
2005--06 school year (94 percent of all centers in APR) and 4,633
centers reporting data in relation to the summer of 2005 (50 percent of
all centers in APR)

Of some interest in Figure 10 is the extent to which centers reported
providing supplemental education services (SES) during the course of the
2005--06 reporting period, with 17 percent of centers reporting
providing these services during the school year and 12 percent during
the summer. It is unclear, however, how much confidence can be placed in
these figures. In PPICS, SES is defined in the following fashion:

"Supplemental Educational Services are a component of Title I of the
Elementary and Secondary Education Act (ESEA), as reauthorized by the No
Child Left Behind Act (NCLB). These services are meant to provide extra
academic assistance to increase the academic achievement of eligible
students in schools that have not met state targets for increasing
student achievement (adequate yearly progress). These services may
include tutoring and afterschool services. They may be offered through
public- or private-sector providers that are approved by the state, such
as public schools, public charter schools, local education agencies,
educational service agencies, and faith-based organizations. Students
from low-income families who remain in Title I schools that fail to meet
state standards for at least three years are eligible to receive
supplemental educational services."

It is hypothesized, however, that some PPICS respondents interpret *SES*
as a term to mean something more generic. The greatest evidence gleaned
from this study that seems to demonstrate that respondents may hold
multiple interpretations for what SES means is predicated on exploring
how APR activities data stack up with information collected in the
Grantee Profile module of PPICS, which asks respondents to identify
whether they are using other sources of funding in addition to 21st CCLC
to provide out-of-school time programming at their sites. One of the
options in relation to this series of questions is SES. In Table 2, the
extent to which a given center reported both providing SES activities
during the course of the 2005--06 reporting period and utilizing SES
funds to support programming delivered at their site is outlined. As
shown below, only 3 percent of centers reporting APR data indicated both
providing SES activities to students and using SES funds to support
programming, whereas 16 percent reported SES activities but no SES
funding. Twelve percent indicated they used SES funds, but no SES
activities were found to be associated with their APR submission. In
light of these results, it is quite difficult to determine the extent to
which there is true blending of 21st CCLC and SES programming at 21st
CCLC sites nationwide. These results suggest that either more robust
data validation provisions need to be added to PPICS in relation to the
reporting of SES data or that the attempt to track the intersection of
21st CCLC and SES programming should be abandoned within the confines of
PPICS given an assumption that there is little overlap between the two
programs.

Table 2. The Intersection of SES Activities and SES as a Program Funding
Source

  -------------------------------------------- ------------ --------------
  **Centers Reporting Activities Data as Part  **\# of      **% of
  of the 2005--06 APR by SES Activity and SES  Centers**    Centers**
  Funding Status**                                          

  Both SES activities and SES funding were     267          3.0%
  reported.                                                 

  SES activities were reported but not SES     1,378        15.6%
  funding.                                                  

  No SES activities were reported, but SES     1,035        11.7%
  funding was reported.                                     

  Neither SES activities nor SES funding were  6,145        69.6%
  reported.                                                 

  Total Centers                                8,825        100%
  -------------------------------------------- ------------ --------------

In terms of the intersection of 21st CCLC programming and academic
content areas, Figure 11 describes the extent to which centers
implemented activities during the 2005--06 reporting period to
intentionally cultivate skills in one or more academic subject area.
More than 80 percent of centers during the 2005--06 school year provided
activities focusing on reading, mathematics, and the arts. Similarly,
those three subject areas were also predominant in terms of the
activities provided during the course of the summer of 2005.

Figure 11. Proportion of Centers Providing Programming by Subject Area,\
School Year and Summer

![](media/image12.wmf){width="6.496527777777778in"
height="3.2430555555555554in"}

*Note.* Based on 8,630 centers reporting data in relation to the
2005--06 school year (93 percent of all centers in APR), and 4,548
centers reporting data in relation to the summer of 2005 (49 percent of
all centers in APR)

Given that one of the primary goals of the 21st CCLC program is to
support academic enrichment programming that cultivates student skill
development in core subject areas such as reading, mathematics, and
science, an effort was made to examine the extent to which core academic
subject areas were associated with academic enrichment activities
provided at centers during the course of the 2005--06 reporting period.
Generally, academic enrichment activities are meant to expand on
students' learning in ways that differ from the methods used during the
school day. They often are interactive and project-focused, and they
enhance a student's education by bringing new concepts to light or using
old concepts in new ways. Academic enrichment activities should be fun
for participating students, but they also should impart knowledge.

It is also important to note that the analyses oriented toward exploring
the degree to which academic subject areas were embedded in academic
enrichment offerings could only be undertaken in relation to the 22
states that opted to provide individual activities data during the
2005--06 reporting period. (SEAs are afforded two options for their
centers to report information about the activities provided at a 21st
CCLC site during the course of a given reporting period. For the
2005--06 reporting period, 22 of the 53 SEAs opted to use the individual
activities option. For a summary of the options SEAs are afforded in
customizing the APR for their state, please see Appendix C.)

In Figure 12, centers that reported providing academic enrichment
activities during the course of the school year and summer are
considered in terms of what percentage of centers operating during a
given timeframe reported providing one or more activity that addressed a
given subject area. As shown in Figure 12, the majority of centers
operating within a given timeframe\
(i.e., school year or summer) provided one or more academic enrichment
activity addressing reading, mathematics, and science.

**Figure 12. Proportion of Centers Providing Academic Enrichment
Programming by Subject Area, School Year and Summer**

![](media/image13.wmf){width="6.4944444444444445in"
height="3.7958333333333334in"}

*Note.* Based on 3,044 centers reporting data in relation to the
2005--06 school year (33 percent of all centers in APR), and 1,661
centers reporting data in relation to the summer of 2005 (18 percent of
all centers in APR)

Although the information outlined in Figures 10 through 12 provides some
insight into how centers are structuring their programs, these charts do
not describe the relative emphasis programs gave to one form of activity
or another during the span of the reporting period. For example, from
the information presented in the previous charts, there is no way to
tell how many centers spent 90 percent of their total activity hours on
tutoring and the remaining 10 percent on enrichment and how many centers
adopted a programming approach in which these two percentages were
reversed.

In order to explore these differences among programs, an attempt was
made to identify a series of "program clusters" based on the relative
emphasis given to providing certain categories of activities (e.g.,
academic enrichment, tutoring, service learning). Using the individual
activities data provided by the 22 states that selected this option
during the course of the 2005--06 school year, it was possible to
calculate the percentage of total hours of school-year programming
offered at a center estimated to be accounted for by each of the 12
categories of activities targeting youth (as outlined in Figure 10).
This was done by multiplying the number of weeks an activity was
provided by the number of days per week it was provided by the number of
hours provided per day. These products were then summed by activity
category for the center in question. These center-level summations by
category served as the numerator in calculations performed to derive the
percentage of activity hours offered that were dedicated to a given
category of activity. So, for example, what percentage of total activity
hours offered were dedicated to academic enrichment, tutoring, homework
help, and so on? The denominator for these calculations was predicated
on the total number of hours of activity the center offered during the
2005--06 school year. (See Appendix B for additional information on the
methods employed to undertake these calculations.)

In Figure 13, the percentage of hours dedicated to a given category of
activity relative to the total hours offered at a given center is
outlined for the 2005--06 school year. The largest percentage of
activity hours offered was dedicated to academic enrichment activities,
followed by activities identified as recreation. It is important to note
that the figures represented in Figure 13 are estimates at best given
some variation that is likely to have taken place in a center's schedule
from one week to the next, resulting in some activities being provided
less often than originally envisioned, whereas others were likely to
have been provided more frequently. In any event, there is undoubtedly
some error represented in the figures highlighted in Figure 13 because
of these likely deviations; however, these data provide a useful
representation of the relative emphasis centers placed upon the
provision of different categories of activities during the reporting
period in question.

**Figure 13. Mean Percentage of Activity Hours Dedicated to Offering
Given Activity Type During the School Year**

![](media/image14.wmf){width="6.500694444444444in"
height="3.9138888888888888in"}

*Note.* Based on 2,142 centers reporting data in relation to the
2005--06 school year (23 percent of all centers in APR)

Given the program's emphasis on supporting the provision of academic
enrichment activities and in cultivating student skills in core academic
subject areas, a similar approach to the one employed in relation to
Figure 13 can be undertaken in terms of exploring the relative emphasis
given to addressing different academic subject areas through academic
enrichment activities.\
In Figure 14, the average percentage of total academic enrichment hours
provided during the school year dedicated to a given subject area is
outlined. It is important to note in Figure 14 that a given academic
enrichment activity meant to support student skill development in more
than one subject area has been represented in each applicable subject
area bar appearing in the chart. So, for example, if a given academic
enrichment activity was meant to support the development of mathematics
and science skills, then the activity in question would be represented
in both the mathematics and science bars. As shown in Figure 14, the
majority of academic enrichment hours offered during the 2005--06 school
year involved content meant to improve the reading/writing skills of
participating students (61 percent), and just under half of all hours
were dedicated to imparting mathematics content (49 percent).

**Figure 14. Mean Percentage of Academic Enrichment Activity Hours
Dedicated to Addressing a Given Subject Area During the School Year**

![](media/image15.wmf){width="6.502083333333333in"
height="4.079861111111111in"}

*Note.* Based on 1,770 centers reporting data in relation to the
2005--06 school year (19 percent of all centers in APR)

While potentially informative, the data shown in Figures 13 and 14 still
fail to capture the diversity of programmatic approaches employed by
21st CCLCs during the 2005--06 reporting period across the 22 states in
question. For example, knowing more definitively about the extent to
which centers gave relative emphasis to providing different activities
is also relevant to the issue of identifying which quality constructs
warrant the greatest attention. For example, should a program that
almost exclusively provides tutoring pay as much attention to the same
quality-related constructs as a program that exclusively offers youth
leadership activities? In order to further summarize this programmatic
diversity and provide a basis for being able to answer questions of this
nature in the future, K-Means cluster analysis was employed using each
of the center-level percentages representing the relative proportion of
total hours of programming at a center accounted for by a given category
of activity. These analyses resulted in the identification of five
primary program clusters defined by the relative emphasis centers found
in that cluster gave to *offering* one or more programming areas during
the course of the 2005--06 school year. As shown in Figure 15, centers
operating during the summer of 2005 and the 2005--06 school year could
be classified into the following five primary clusters:

-   Centers mostly providing tutoring activities (13 percent of all
    centers).

-   Centers mostly providing homework help (14 percent of all centers).

-   Centers mostly providing recreational activities (20 percent of all
    centers).

-   Centers mostly providing academic enrichment (26 percent of all
    centers).

-   Centers that provided a wide variety of activities across multiple
    categories (27 percent of all centers).

Figure 15. Primary Program Clusters Based on Activity Data Provided\
in Relation to the 2005--06 School Year

![](media/image16.wmf){width="6.497916666666667in"
height="3.604861111111111in"}

*Note.* Based on 2,142 centers reporting data in relation to the
2005--06 school year (23 percent of all centers in APR)

In Table 3, the average percentage of total activity hours dedicated to
a given category of activity is outlined for each of the five primary
clusters highlighted in Figure 15. Note that across clusters there was
significant variation in the degree to which the average center focused
on the activity type that defined each cluster. For example, among
centers in the "mostly academic enrichment" cluster, the average
percentage of total activity hours dedicated to enrichment was 73
percent; by contrast, for the "mostly homework help" cluster, the
average was 49 percent of total activity hours dedicated to homework
help. For the "tutoring" and "recreation" clusters, the average
percentage of total activity hours dedicated to the cluster-defining
activity was found to be 67 percent and 56 percent respectively. Another
way to look at the data present in Table 2 is how the percentages differ
across columns for a given row. So, for example, while 73 percent of
hours offered by centers in the "mostly academic enrichment" cluster on
average are dedicated to offering academic enrichment activities, for
centers in the "mostly recreation" cluster, this percentage is 18
percent.

Table 3. Mean Percentage of Activity Hours Dedicated to Given Activity
Type by Cluster

+-----------+-----------+-----------+-----------+-----------+-----------+
| *         | **Mostly  | **Mostly  | **Mostly  | **Mostly  | **        |
| *Activity | Academic  | Rec       | H         | T         | Variety** |
| C         | Enr       | reation** | omework** | utoring** |           |
| ategory** | ichment** |           |           |           |           |
+-----------+-----------+-----------+-----------+-----------+-----------+
| Academic  | **73%**   | 18%       | 16%       | 13%       | 32%       |
|           |           |           |           |           |           |
| E         |           |           |           |           |           |
| nrichment |           |           |           |           |           |
+-----------+-----------+-----------+-----------+-----------+-----------+
| Tutoring  | 4%        | 6%        | 9%        | **67%**   | 12%       |
+-----------+-----------+-----------+-----------+-----------+-----------+
| Homework  | 6%        | 7%        | **49%**   | 5%        | 10%       |
| Help      |           |           |           |           |           |
+-----------+-----------+-----------+-----------+-----------+-----------+
| Mentoring | 1%        | 1%        | 1%        | 1%        | 3%        |
+-----------+-----------+-----------+-----------+-----------+-----------+
| R         | 10%       | **56%**   | 16%       | 8%        | 18%       |
| ecreation |           |           |           |           |           |
+-----------+-----------+-----------+-----------+-----------+-----------+
| C         | 0%        | 2%        | 1%        | 0%        | 3%        |
| areer/Job |           |           |           |           |           |
| Training  |           |           |           |           |           |
+-----------+-----------+-----------+-----------+-----------+-----------+
| Drug      | 2%        | 3%        | 2%        | 2%        | 6%        |
| /Violence |           |           |           |           |           |
| P         |           |           |           |           |           |
| revention |           |           |           |           |           |
+-----------+-----------+-----------+-----------+-----------+-----------+
| Extended  | 1%        | 1%        | 0%        | 1%        | 2%        |
| Library   |           |           |           |           |           |
| Hours     |           |           |           |           |           |
+-----------+-----------+-----------+-----------+-----------+-----------+
| Sup       | 1%        | 1%        | 2%        | 1%        | 6%        |
| plemental |           |           |           |           |           |
| Ed        |           |           |           |           |           |
| ucational |           |           |           |           |           |
| Services  |           |           |           |           |           |
+-----------+-----------+-----------+-----------+-----------+-----------+
| Service   | 1%        | 1%        | 1%        | 1%        | 2%        |
| Learning  |           |           |           |           |           |
+-----------+-----------+-----------+-----------+-----------+-----------+
| Youth     | 2%        | 4%        | 2%        | 1%        | 6%        |
| L         |           |           |           |           |           |
| eadership |           |           |           |           |           |
+-----------+-----------+-----------+-----------+-----------+-----------+

*Note.* Based on 2,142 centers reporting data in relation to the
2005--06 school year (23 percent of all centers in APR)

Of the data outlined in Figure 15 and Table 3, the area that some may
find of greatest interest pertains to the existence of a "mostly
recreation" cluster that encompasses 20 percent of the total number of
centers in operation during the course of the 2005--06 school year in
the 22 states in question. In light of the program's emphasis on the
provision of academic enrichment activities and the achievement of
meaningful improvements in student academic behaviors and achievement,
some may find the existence of this cluster strange or even problematic;
others may see this cluster as being completely congruent with the
desired outcomes of the 21st CCLC program depending upon how the
activities in question are actually delivered at the point of service.
Realizing that the existence of this cluster may engender such
discussions, an effort was made to further explore the degree to which
certain core academic subject areas are addressed in the provision of
programming across centers.

As shown in Figure 16, centers falling within the "mostly tutoring" and
"mostly homework" clusters on average were more likely to dedicate a
higher percentage of total activity hours to the provision of
reading/writing and mathematics content, followed by centers found in
the "mostly enrichment" and "variety" clusters. Of some interest in
Figure 16 is the relatively low average percentage of activity hours
dedicated to reading/writing and mathematics content among centers
falling within the "mostly recreation" cluster, so there is some
evidence to suggest that centers associated with the mostly recreation
cluster are less apt to be focused on core academic subject areas when
delivering activities.

**Figure 16. Mean Percentage of Activity Hours Offered by\
Subject Area and Program Cluster**

![](media/image2.wmf){width="6.497916666666667in" height="4.05in"}

*Note.* Based on 2,142 centers reporting data in relation to the
2005--06 school year (23 percent of all centers in APR)

Although the 2005--06 APR was the first reporting period in which
grantees were given the option to provide information about individual
activities offered at a given center, by comparing the programming
cluster in which a center was classified with the center's cohort
status, it is possible to explore how centers may change the relative
emphasis they place on a given activity type over time.

Centers reporting data as part of the 2005--06 APR can be classified as
falling within one of three cohorts. Cohort 1 comprises those centers
that first reported APR data for the 2003--04 reporting period. For this
set of centers, the 2005--06 APR represents the third wave of data they
have provided. Most of the grantees represented in this cohort received
their grant award in calendar year 2003 and represent the most mature
centers submitting data as part of the APR.

Cohort 2 centers represent those centers that first reported data as
part of the 2004--05 reporting period. For this cohort, the 2005--06 APR
represents the second time they have completed the reporting process.

Cohort 3 comprises those centers for which the 2005--06 reporting period
represents the first time they have provided APR data. Most of these
grantees received awards in late 2004 or calendar year 2005 and
represent the least mature programs providing data.

In Figure 17, the percentage of centers falling within each of the five
primary program cluster types is outlined separately for the three
center cohorts. The most interesting trend outlined in Figure 17 is the
extent to which the proportion of centers falling in the "mostly
recreation" cluster decreases as the centers get more mature; that is,
the relatively new Cohort 3 centers demonstrate a higher proportion of
centers that fall in the "mostly recreation" cluster and a lower
proportion of centers that fall within the "mostly enrichment" cluster.
This may suggest that as centers mature over time, there is a movement
away from an overemphasis on recreation in programming to one more
oriented toward the provision of academic enrichment. While this is
beyond our current analysis, it would be interesting to examine whether
this movement varies by state, depending upon the guidance and
priorities given in the state request for applications, the training and
professional development provided to new grantees, and the types of
metrics that are examined when monitoring grantee implementation of
programming at a given site.

Figure 17. Primary Program Clusters Based on Activity Data Provided in\
Relation to the 2005--06 School Year by Cohort

![](media/image3.wmf){width="6.502777777777778in"
height="3.488888888888889in"}

*Note.* Based on 2,142 centers reporting data in relation to the
2005--06 school year (23 percent of all centers in APR)

## Staffing

Center staffing is a crucial factor in the success of afterschool
programming, and many of the quality assessment approaches being
developed and used in the field hone in on the capacity of staff
responsible for the delivery of programming to create positive
developmental settings for youth. The quality of staff can be the
difference between an effective program and a mediocre one. In this
regard, the success of afterschool programs is critically dependent on
students forming personal connections with the staff, especially for
programs serving older students for which a much wider spectrum of
options are available to these youth in how they spend their time after
school (Eccles & Gootman, 2002; Rosenthal & Vandell, 1996).

The APR collected information on the number of 21st CCLC staff of
various types, based on background and training, who regularly staffed
the centers in operation during the summer of 2005 and the 2005--06
school year. To complete the staffing section of the APR, centers
indicated the number of staff in each category who were paid and the
number who served as volunteers during the reporting period. These data
are highlighted in Figures 18 and 19 for school year and summer staff
respectively. Note that the scales on the two charts are different
because there were many more school-year staff than summer staff. Staff
types represented in each of the charts are classified according to the
following categories:

-   *Teachers:* School-day teachers

-   *College:* College students

-   *High School:* High school students

-   *Parents:* Parents

-   *Youth Dev:* Youth development workers

-   *Commun:* Other community members

-   *Oth Sch Staff:* Other nonteaching school-day staff with a college
    degree of higher

-   *Coordinator:* Center administrators and coordinators

-   *Nonsch Staff:* Other nonschool-day staff with some or no college

-   *Other*

As demonstrated in Figures 18 and 19, school-day teachers are by far the
highest proportion of afterschool staff delivering programming during
both the summer and school year. In addition, school-day teachers, youth
development workers, and nonteaching school-day staff are often paid for
their afterschool time, whereas parents and other community members
generally serve as volunteers.

**Figure 18. Number of School-Year Staff by Type for the\
2005--06 Reporting Period**

![](media/image17.wmf){width="6.497916666666667in"
height="4.017361111111111in"}

*Note.* Based on 8,521 centers reporting data in relation to the
2005--06 school year (91 percent of all centers in APR)

**Figure 19. Number of Summer Staff by Type for the\
2005--06 Reporting Period**

![](media/image17.wmf){width="6.497916666666667in"
height="4.017361111111111in"}

*Note.* Based on 4,467 centers reporting data in relation to the
2005--06 school year (48 percent of all centers in APR)

Here again, Figures 18 and 19 provide some insight into the overall
involvement of various types of staff, but they do not answer the
question of whether there are typical staffing "models" or patterns. For
example, what proportion of centers relies almost exclusively on
school-day teachers to staff their program? What proportion employs
mostly college-educated youth development workers? To explore this
question, K-Means cluster analysis was employed with a series of
variables that identified the percentage of paid staff working in a
program during the course of the 2005--06 school year. (See Appendix B
for additional details on cluster analyses that were carried out.) To
simplify the analysis, paid staff working at a center during the school
year were grouped into the following categories:

-   *Teachers:* School-day teachers

-   *HS/College:* High school and college students

-   *Parents/Commun:* Parents and other community members

-   *Youth Dev:* Youth development workers

-   *Oth Sch Staff:* Other nonteaching school-day staff

-   *Other:* Other nonschool-day staff with some or no college and other

As shown in Figure 20, centers operating during the school year can be
classified into one of five primary clusters:

-   Centers staffed mostly by school-day teachers (41 percent of all
    centers).

-   Centers mostly staffed by individuals with some or no college (24
    percent of all centers).

-   Centers mostly staffed by a combination of school-day teacher and
    other nonteaching school-day staff with a college degree (17 percent
    of all centers).

-   Centers staffed by a wide variety of staff types across multiple
    categories (9 percent of all centers).

-   Centers staffed by college-educated youth development workers (9
    percent of all centers).

Figure 20. Primary Program Clusters Based on Paid Staffing Data
Provided\
in Relation to the 2005--06 School Year

![](media/image18.wmf){width="6.501388888888889in"
height="3.689583333333333in"}

*Note.* Based on 8,462 centers reporting data in relation to the
2005--06 school year (91 percent of all centers in APR)

In Table 4, the average percentage of paid staff falling within a given
staff type category is outlined for each of the five primary clusters.
Note that across clusters there was a large variation in the percentages
for the defining staff type. For example, among centers classified as
falling within the "mostly teachers" cluster, the average percentage of
paid staff that were teachers was 80 percent; in contrast, for the
cluster with "mostly staff with no college degree," the average
percentage of paid staff falling in this category was 67 percent. For
the "mostly school day staff" and "mostly youth development worker"
clusters, the average percentage of cluster-defining staff was found to
be 83 percent and 69 percent respectively. Here again, another useful
way to look at the data presented in Table 4 is how the percentages
differ across columns for a given row. So, for example, while 80 percent
of paid staff in the "mostly teachers" cluster on average are school-day
teachers, for centers in the "mostly staff with no college degree"
cluster, this percentage is 17 percent.

Table 4. Mean Percentage of Paid Staff Falling Within a Staff Type
Category by Cluster

+-------------+----------+----------+----------+----------+----------+
| *           | **Mostly | **Mostly | **Mostly | **Mostly | **V      |
| *Staffing** | Te       | Staff    | Sc       | Youth    | ariety** |
|             | achers** | With No  | hool-Day | Dev      |          |
| **Type**    |          | College  | Staff**  | elopment |          |
|             |          | Degree** |          | W        |          |
|             |          |          |          | orkers** |          |
+-------------+----------+----------+----------+----------+----------+
| Teachers    | **80%**  | 17%      | **40%**  | 15%      | 33%      |
+-------------+----------+----------+----------+----------+----------+
| High        | 3%       | **40%**  | 7%       | 6%       | 12%      |
| sch         |          |          |          |          |          |
| ool/college |          |          |          |          |          |
| students    |          |          |          |          |          |
+-------------+----------+----------+----------+----------+----------+
| Parent      | 1%       | 2%       | 2%       | 1%       | 10%      |
| s/community |          |          |          |          |          |
| members     |          |          |          |          |          |
+-------------+----------+----------+----------+----------+----------+
| Youth       | 3%       | 7%       | 4%       | **69%**  | 3%       |
| development |          |          |          |          |          |
| workers     |          |          |          |          |          |
+-------------+----------+----------+----------+----------+----------+
| Other       | 8%       | 7%       | **43%**  | 3%       | 19%      |
| nonteaching |          |          |          |          |          |
| school day  |          |          |          |          |          |
| staff       |          |          |          |          |          |
+-------------+----------+----------+----------+----------+----------+
| Other staff | 5%       | **27%**  | 4%       | 6%       | 5%       |
| with some   |          |          |          |          |          |
| or no       |          |          |          |          |          |
| college     |          |          |          |          |          |
+-------------+----------+----------+----------+----------+----------+

*Note.* Based on 8,462 centers reporting data in relation to the
2005--06 school year (91 percent of all centers in APR)

With the intent of eventually providing data about program models that
can strongly influence how professional development and program
improvement efforts should be focused, the relationship between program
type (e.g., mostly tutoring, mostly academic enrichment) and staffing
patterns was examined. Figure 21 shows the staffing patterns for the
five different program types, using the individual activities data
obtained from the 2005--06 reporting period. Note that the percentage of
paid school-year staff made up of teachers is highest among centers that
fall within the "mostly tutoring" (60 percent of paid staff on average)
and "mostly homework help" (53 percent of paid staff on average)
clusters and then falls steadily across the remaining clusters, with
teachers representing only 41 percent of paid staff on average among
centers represented in the "mostly recreation" cluster. In addition,
centers represented in the "mostly recreation" cluster are likely to
have a higher percentage of their paid staff fall within the high school
or college student and youth development worker categories than centers
represented in other clusters.

Such results may further bolster the case that quality assessment
approaches and associated quality improvement supports may need to be
differentiated based on the program and staffing models being employed.

**Figure 21. Mean Percentage of Paid School Year Staff Falling Within a
Given Category by Activity Cluster**

![](media/image4.wmf){width="6.500694444444444in"
height="4.053472222222222in"}*Note.* Based on 2,030 centers reporting
individual activities and staffing data in relation to the 2005--06
school year (22 percent of all centers in APR)

**Attendance**

Attendance is an intermediate outcome indicator that reflects the
potential breadth and depth of exposure to afterschool programming.
Grantees completing the APR for the 2005--06 reporting period were asked
to identify the total number of students who participated in the
center's programming over the course of the year and the number of
students meeting the definition of "regular attendee" by participating
30 days or more in center activities during the 2005--06 reporting
period. The former number can be utilized as a measure of the breadth of
a center's reach, whereas the latter can be construed as a measure of
how successful the center was in retaining students in center-provided
services and activities. The design of the APR system and its
performance indicators was predicated on the assumption that regular
attendees participated in afterschool activities at a minimal "critical
mass" for the program to have a reasonable chance of making an impact on
academic or behavioral outcomes.

In a similar fashion, the total number of attendees served each year is
a measure of the size of an individual 21st CCLC program. Data obtained
in the 2005--06 APRs indicate that the median number of total attendees
at a given center was 122, and the median number of regular attendees
was 72 (The median is employed here as a measure of central tendency
given that attendance data reported by centers during the 2005--06
reporting period are characterized by a great deal of variation in the
total number of attendees resulting in a positively skewed
distribution). As demonstrated in Figure 22, these attendance levels
were fairly consistent across both Cohort 1 and Cohort 2 centers,
whereas centers classified as Cohort 3 were likely to serve a lower
median number of total attendees.

**Figure 22. Median Number of Total and Regular Attendees\
for the 2005--06 APR by Cohort Status**

![](media/image19.wmf){width="6.501388888888889in"
height="3.6381944444444443in"}

*Note.* Based on 8,877 centers reporting attendance data (95 percent of
all centers in APR)

In light of the findings highlighted in the section on activities in
terms of the identification of a series of primary program clusters
defined on the relative emphasis given to one or more category of
activity by centers represented in the cluster in question, it seemed
appropriate to compare the average rate of regular attendance across
each of these primary program clusters. (Again, only programs that
operated exclusively during the course of the 2005--06 school year have
been included in the analyses in question.) As outlined in Figure 23,
the average rate of regular attendance is fairly equivalent across
programs that provide "mostly homework help," "mostly enrichment," and a
"variety" of activities, but there is a noticeable drop in the average
rate of regular attendance among programs that provide "mostly
recreational" activities and especially among programs that focus
primarily on the provision of tutoring services. These results are
interesting, yet it should be noted that the domain of centers examined
to support these analyses is relatively small given the focus on only
those centers that exclusively operated during the 2005--06 school year
and reported individual activities data.

**Figure 23. Average Rate of Regular Attendance Among Centers by\
School Year Activity Cluster**

![](media/image20.wmf){width="6.495138888888889in"
height="3.986111111111111in"}

*Note.* Based on 967 centers operating only during the 2005--06 school
providing attendance and individual activities data (10 percent of all
centers in APR)

In Figure 24, a similar set of analyses has been undertaken, but in this
case, the average rate of regular attendance has been calculated both by
activity cluster and cohort (defined by the first year a center provided
APR data). Note that for centers falling within the "mostly homework
help," "mostly enrichment," and "variety" clusters, there is a clear
trend that suggests that more mature programs have a higher rate of
average regular attendance than centers that are relatively new. This
trend is especially pronounced among centers represented in the "mostly
enrichment" cluster in which Cohort 1 centers have an average rate of
regular attendance of 71 percent whereas the rate for Cohort 3 centers
is 48 percent.

**Figure 24. Average Rate of Regular Attendance Among Centers by\
School Year Activity Cluster and Cohort**

![](media/image5.wmf){width="6.502083333333333in"
height="3.5097222222222224in"}

*Note.* Based on 967 centers operating only during the 2005--06 school
providing attendance and individual activities data (10 percent of all
centers in APR)

**Summary of Characteristics of 21st CCLC Programs**

From the perspective of further informing how quality assessment systems
should be constructed and implemented to help programs better meet the
purposes of the 21st CCLC program, the data highlighted in the previous
sections suggest that consideration potentially should be given to
exploring how these efforts should be differentiated in light of the
staffing and activity models employed at a given center. For example,
the conversations related to improving the quality of offerings among
centers providing mostly tutoring services and that largely employ
school-day teachers are likely to be qualitatively different than those
in centers focusing mostly on recreation and that employ a larger
proportion of youth development workers and students drawn from area
high schools and colleges as staff.

Results outlined in this section clearly highlight differences among
centers based on their relative maturity. Over time, there is some
evidence to suggest that centers will increasingly move toward an
academic enrichment model in the delivery of programming and be less
dependent primarily on recreational activities to fill their programming
slate. In addition, there is also some evidence to suggest that programs
get better at retaining students in programming over time, especially
for those centers falling within the academic enrichment, homework help,
and variety clusters. Thinking about how this program evolution should
influence the development of quality assessment systems also would seem
to be a viable and worthwhile undertaking in terms of helping programs
get to a more ideal level of functioning and to get there more quickly.

**Section 2: Student Achievement and Academic\
Behavioral Outcomes**

In addition to collecting information on the operational characteristics
of 21st CCLC programs, one of the primary purposes of the APR is to
collect data to inform how well the program is meeting the GPRA
indicators established for the program. The GPRA indicators associated
with the 21st CCLC program are the primary tools by which the U.S.
Department of Education and other agencies of the federal government
evaluate the effectiveness of 21st CCLCs operating nationwide. The
indicators were established to gauge the extent to which 21st CCLC
programs could demonstrate changes in student academic behaviors and
achievement, as well as the delivery of statutorily required academic
enrichment activities. The metrics associated with the GPRA indicators
fall within three primary categories:

1.  Changes in Student Academic Achievement

    a.  Percentage of regular attendees whose mathematics and
        reading/language arts grades improved from fall to spring.

    b.  Percentage of regular attendees whose achievement test scores
        improved from *not proficient* to *proficient or above* on the
        mathematics and reading/language arts portions of their state's
        assessment system.

2.  Changes in Student Academic Behaviors

    a.  Percentage of regular attendees with teacher-reported
        improvement in\
        homework completion and class participation.

    b.  Percentage of regular attendees with teacher-reported
        improvement in\
        student behavior.

```{=html}
<!-- -->
```
3.  Program Delivery and Content

    a.  Percentage of 21st CCLCs reporting emphasis in at least one core
        academic area.

    b.  Percentage of 21st CCLCs offering enrichment and support
        activities in technology.

    c.  Percentage of 21st CCLCs offering enrichment and support
        activities\
        in other areas.

An overall summary of how the 21st CCLC programs that reported GPRA
indicator data for the 2005−06 reporting period is outlined in Table 5,
along with results from the previous two reporting periods. Generally,
the reported percentage of regular attendees with improvements in grades
was higher than the percentages for the 2004−05 reporting period,
particularly in relation to mathematics grades, whereas teacher-reported
improvements in student behaviors were generally lower than last year's
levels. There were also fairly dramatic drops in the percentage of
regular attendees who attained proficiency on state assessments,
although this change should not be overinterpreted because there was a
dramatic growth in the percentage of states reporting assessment data
for regularly participating students, increasing from 9 percent in
2004--05 to 38 percent of states in 2005--06. This development should be
kept in mind when comparing the most recent state assessment results
with those obtained from earlier reporting periods.

It is also important to emphasize at this juncture that states have some
degree of discretion in terms of reporting data in PPICS related to
changes in student grades, assessment results, and teacher-reported
behaviors. States are required to supply data for at least one of these
categories as part of the annual reporting process but could also opt to
report any combination of these three categories. In completing their
2005−06 APRs, 68 percent of states opted to report teacher survey data,
57 percent reported information on grades, and 38 percent opted to
report state assessment results.

Table 5. Status of the GPRA Indicators Associated With the 21st CCLC
Program\
2003−04, 2004−05, and 2005−06 Reporting Periods

+-----------------------+-----------+-----------+-----------+---------+
| **GPRA Performance    | **Pe      | *         | *         | **      |
| Indicator**           | rformance | *2003--04 | *2004--05 | 2005−06 |
|                       | Targets** | Reporting | Reporting | Re      |
|                       |           | Period**  | Period**  | porting |
|                       |           |           |           | P       |
|                       |           |           |           | eriod** |
+-----------------------+-----------+-----------+-----------+---------+
| > Regular attendees   | 45%       | 44.55%    | 41.47%    | 42.52%  |
| > demonstrating       |           |           |           |         |
| > improved grades in  |           |           |           |         |
| > reading/language    |           |           |           |         |
| > arts                |           |           |           |         |
+-----------------------+-----------+-----------+-----------+---------+
| > Regular attendees   | 45%       | 40.84%    | 38.82%    | 42.49%  |
| > demonstrating       |           |           |           |         |
| > improved grades in  |           |           |           |         |
| > mathematics         |           |           |           |         |
+-----------------------+-----------+-----------+-----------+---------+
| > Regular attendees   | N/A       | Not       | 27.90%    | 20.63%  |
| > demonstrating       |           | reported  |           |         |
| > improved state      |           |           |           |         |
| > assessment results  |           |           |           |         |
| > in reading/language |           |           |           |         |
| > arts                |           |           |           |         |
+-----------------------+-----------+-----------+-----------+---------+
| > Regular attendees   | N/A       | Not       | 29.77%    | 20.82%  |
| > demonstrating       |           | reported  |           |         |
| > improved state      |           |           |           |         |
| > assessment results  |           |           |           |         |
| > in mathematics      |           |           |           |         |
+-----------------------+-----------+-----------+-----------+---------+
| > Regular attendees   | 75%       | 68.72%    | 74.98%\*  | 72.56%  |
| > demonstrating       |           |           |           |         |
| > improved homework   |           |           |           |         |
| > completion and      |           |           |           |         |
| > class participation |           |           |           |         |
+-----------------------+-----------+-----------+-----------+---------+
| > Regular attendees   | 75%       | 64.04%    | 71.08%\*  | 67.94%  |
| > demonstrating       |           |           |           |         |
| > improved student    |           |           |           |         |
| > behavior            |           |           |           |         |
+-----------------------+-----------+-----------+-----------+---------+
| > Centers emphasizing | 85%       | 97.73%    | 95.06%    | 95.49%  |
| > at least one core   |           |           |           |         |
| > academic area       |           |           |           |         |
+-----------------------+-----------+-----------+-----------+---------+
| > Centers offering    | 85%       | 65.61%    | 65.75%    | 64.32%  |
| > enrichment and      |           |           |           |         |
| > support activities  |           |           |           |         |
| > in technology       |           |           |           |         |
+-----------------------+-----------+-----------+-----------+---------+
| > Centers offering    | 85%       | 92.51%    | 94.09%    | 94.91%  |
| > enrichment and      |           |           |           |         |
| > support activities  |           |           |           |         |
| > in other areas      |           |           |           |         |
+-----------------------+-----------+-----------+-----------+---------+

\* The survey instrument was changed this year to allow teachers to
select "did not need improvement." This option was not present in the
2003--04 survey. Efforts to validate the teacher survey demonstrated
that the items functioned differently depending upon the grade level of
the student in question.

**Evaluating the Efficacy of the Current Indicator System**

In recent months, questions have been raised increasingly at meetings
sponsored by the U.S. Department of Education about both the validity of
the data supporting the results highlighted in Table 5 as well as the
utility of these metrics as the primary mechanisms by which the efficacy
of the 21st CCLC program is evaluated. In particular, two design
elements of the current PPICS application may complicate efforts to
obtain more meaningful data on grantee performance:

-   The vast majority of the data collected in PPICS that relate to
    student improvements in academic behaviors and achievement are
    reported directly by the grantees without consistent procedures in
    place to independently verify the accuracy of the data being
    supplied.

-   Information that is supplied to support the calculation of the
    student achievement and behavioral change performance indicators
    associated with the program is collected at the center level, as
    opposed to the individual student level. For example, when
    calculating the metrics associated with improvements in students'
    mathematics grades, fields are referenced in PPICS that contain data
    supplied by a grantee or center-level respondent in which the total
    number of regular attendees witnessing an improvement in mathematics
    grades has been reported. To do this, center officials (or perhaps
    their evaluator) collect and aggregate the student report card
    information and then report the aggregate results when completing
    the grades page in PPICS.

The accuracy of the information being supplied cannot be independently
verified at the level of precision that would be associated with a more
finely controlled data-collection effort because the data reported in
PPICS used to support GPRA indicator calculations is based on grantee
self-reports and is obtained through aggregated figures reported at the
center-level. In some respects, these attributes of PPICS grantee-level
data collection represent the compromises that seemed reasonable three
years ago when PPICS was being designed given concerns raised by some
state 21st CCLC coordinators relative to the collection of individual
student-level data and the inherent challenges in obtaining data from
close to 10,000 21st CCLCs in operation nationwide.

For example, the influence of these earlier design compromises can be
seen by looking at the GPRA indicator that examines the extent to which
students participating in 21st CCLC programming witnessed improvements
in teacher-reported academic behaviors. The data used to support the
calculation of these indicators are obtained from a survey that is taken
by a school-day teacher associated with a student enrolled in the
afterschool program. The survey asks the teacher to reflect on the
degree to which the student needed to improve in a given area\
(e.g., completing homework on time, attending class regularly) and the
extent to which improvement was witnessed during the course of the
school year. Like other data elements obtained in PPICS, teacher survey
results are collected in the aggregate for a given center as part of the
APR process (e.g., the total number of students attending the 21st CCLC
witnessing significant improvement in turning homework in on time). The
fact that aggregate teacher-survey results are collected in PPICS as
opposed to individual survey responses prevents Learning Point
Associates staff from employing the full domain of scoring and scaling
techniques that would yield more psychometrically valid results.

In addition, such results would help determine whether performance
thresholds should be differentially applied depending on other important
factors, such as the grade level of the students served by a given
center. This is especially important in the last instance in which
validation studies performed by Learning Point Associates to date
suggests that the current teacher survey instrument performs much
differently depending upon the grade level of the student for which the
survey is being completed. Finally, there are some lingering concerns
around the method of obtaining data on student academic behaviors
directly from school-day teachers, given the possibility that teachers'
responses may be influenced by a perception that what they endorse may
also be interpreted as a reflection of their instruction and ability to
influence student outcomes. Although this method is not quite as
problematic as the data collection process associated with the
teacher-survey supported indicators, similar types of issues also have
been found to be associated with the reporting of grades and state
assessment information.

In addition to issues of data accuracy, concerns have also been
expressed by grantees, state 21st CCLC coordinators, and members of the
21st CCLC PPICS Evaluation Task Force (ETF) regarding the utility of the
metrics associated with the current domain of GPRA indicators as the
primary mechanisms by which the efficacy of the 21st CCLC program is
evaluated. Generally, these concerns have echoed one or more of the
following themes:

-   The measures associated with the indicators are fairly blunt and, in
    some cases, seem to assume that programs typically can have dramatic
    effects on student achievement. This especially seems to be true in
    relation to the indicator related to state assessment achievement
    outcomes, in which improvements on the part of students are only
    counted when they cross the threshold from below proficiency to
    proficiency or above.

-   There are no comparisons with outcomes obtained by students not
    attending these programs, nor is there any provision to establish
    whether there is evidence of a relation between center attendance
    and changes in student behaviors, grades, and/or assessment scores.

-   Quality of service provision is in no way controlled for when
    reporting indicator information.

-   Many shorter term outcomes that are likely to precede more
    significant changes in student achievement outcomes are not
    adequately represented in the current domain of indicators.

Taken collectively, these potential limitations of the current domain of
performance indicators may be hampering both federal and state efforts
to assess the full impact of these programs and to support decision
making regarding the delivery of training and technical assistance
relative to programs that are performing below expectations. From an SEA
perspective, states have had to rely on a wide variety of additional
data-collection efforts to support key decisions regarding which
programs are eligible for continued funding and which ones should be
allowed to lapse based on provision of programming falling well below a
desired level of quality.

Recognizing that it may be appropriate to reassess the viability and
appropriateness of the current domain of performance indicators for the
21st CCLC program, the U.S. Department of Education has asked Learning
Point Associates to do the following:

-   Undertake a process to obtain systematic input and advice from
    members of the 21st CCLC PPICS ETF and other experts in the field of
    afterschool research and evaluation on how the current domain of
    performance indicators associated with the program should be revised
    and modified.

-   Determine how these suggested changes to the indicators will impact
    data-collection approaches embedded in PPICS.

Meetings held to date for this purpose have yielded the following
recommendations from state 21st CCLC coordinators and afterschool
research and evaluation experts represented on the PPICS ETF:

-   *Indicators should be developed that demonstrate grantee success in
    helping students remain at an acceptable level of academic
    performance*. One of the flaws associated with the indicators
    related to assessing changes in academic behaviors and performance
    is that they fail to capture information about the extent to which
    students performing at an acceptable level remain at such a level,
    even in the face of increasingly difficult and challenging academic
    content. This flaw is probably most evident in relation to grades
    reporting in which students, for example, who receive a B+ in
    mathematics both at the end of the first and last marking periods of
    a given school year are not positively reflected in the indicator
    calculations for a given center. In this regard, a new set of
    indicators should be developed to capture instances in which a
    regular attendee has maintained performance at an acceptable level
    of functioning.

-   *Account for grade-level differences in the indicators that are
    assessed for performance measurement purposes.* There is emergent
    data both in the teacher survey validation studies performed by
    Learning Point Associates and in the program characteristic and
    performance data housed in PPICS that suggest that distinctions can
    be made across programs serving varying grade levels of students. In
    particular, it appears that 21st CCLCs serving high school students
    are meaningfully distinct from those programs serving students
    enrolled in elementary or middle school grades. In terms of program
    attributes, based on data housed in PPICS, 21st CCLCs exclusively
    serving high school students are more apt to serve a larger number
    of total students per year while demonstrating the lowest rate of
    regular attendance. Of some interest, however, is that high school
    programs on the whole perform better on some of the academic
    achievement indicators as compared to programs exclusively serving
    middle school students.

```{=html}
<!-- -->
```
-   Such results offer some interesting areas for further exploration,
    especially in terms of the afterschool programming that may be
    appropriate at the high school level, taking into consideration both
    the specific developmental needs and motivation of such students,
    given the plethora of other options they have in terms of how they
    spend their afterschool hours. In addition, in terms of behavioral
    change, some outcomes are certainly more relevant to high school
    populations as compared to programs serving students from other
    grade levels. An improvement in school-day attendance, for example,
    is significantly more relevant to high school students than either
    middle or elementary students. Similarly, given that most state
    assessment systems are less apt to test high school students
    annually, state assessment results are largely unavailable for this
    population, leaving a gap in terms of assessing how such programs
    may have positively impacted student achievement among high school
    attendees.

```{=html}
<!-- -->
```
-   *Include only student attendees who need improvement in terms of
    academic behaviors and achievement in indicator calculations*.
    Although this approach is already taken in relation to assessing
    regular attendee improvements in state assessment results and
    teacher-reported behaviors, the indicator calculation related to
    assessing changes in student grades has typically not factored out
    students who could not improve because they had attained the highest
    grade possible on the grading scale being employed at the end of the
    first marking period in the fall. Revised indicator calculations
    related to grades should remove such students from the denominator
    associated with these calculations.

-   *Base academic achievement indicators on a net change calculation.*
    At present, when the indicators related to improvements in student
    grades and self-assessment results are calculated, consideration is
    given only to those students who witnessed an improvement relative
    to the total regular attendee population with data reported in that
    area; however, without considering how many students also witnessed
    a decline in grades or state assessment results in calculating these
    respective indicators, the percentage of regular attendees
    witnessing an increase is overstated and less indicative of what the
    program's impact may have been on student achievement.

-   *Revise and expand the current domain of indicators associated with
    program content and quality.* Although there is widespread agreement
    that the current domain of indicators related to program content and
    delivery provide little in the way of useful information about the
    programming being provided by 21st CCLCs nationwide or the quality
    of such offerings, there has been much less consensus on what should
    replace them. Some suggestions in this area advise focusing such
    indicators on assessing center success in meeting more immediate and
    shorter term outcomes (e.g., increasing the acquisition of study
    skills among participants, improving school-day attendance, lowering
    disciplinary referrals). Others have suggested collecting additional
    information in PPICS regarding how states and grantees are engaging
    in quality assessment and program improvement processes that could
    serve as a foundation for a new series of metrics to assess how
    involved and vested states and grantees are in continuous
    improvement efforts.

Although many of these recommendations will require additional study and
discussion to make them more concrete and operationally viable, the
feasibility of several of the recommendations can be assessed relatively
easily by using the existing PPICS dataset to recalculate the indicators
related to teacher-reported changes in student behavior, grades, and
state assessment results.

To that end, the final sections of this report will address the
following three purposes:

-   In a more detailed fashion, describe the status of the GPRA
    indicators associated with the 21st CCLC program based on data
    collected as part of the 2005−06 reporting period, and compare these
    results to prior years.

-   Compare and contrast GPRA-related results based on the present
    method of calculating the indicators with results that would have
    been obtained employing the revised approaches as a way to determine
    the ramifications of adopting the changes recommended by the ETF.

-   Highlight similarities and differences among programs serving
    different grade levels, with a special emphasis on exploring the
    types of outcomes that seem to distinguish programs exclusively
    serving high school students.

This section concludes with an overview of the next steps that would be
required to further explore the viability of making revisions to the
current domain of 21st CCLC indicators and what these next steps may
mean for SEAs and 21st CCLC grantees in future reporting periods.

## 

## Outcome Indicators: Definitions, Limitations, and Proposed Changes

The primary goal of the 21st CCLC program is to improve student academic
behaviors and achievement through the provision of afterschool academic
enrichment and other youth development and support activities. Since the
program's inception, efforts to measure program success relative to
these goals have focused on collecting performance data on grades,
assessment results, and teacher-reported changes in student academic
behaviors. In the sections that follow, additional information is
provided about how each of these impact categories is presently
calculated for GPRA reporting purposes, what limitations may be
associated with these approaches, and how suggested changes in these
calculations may enhance the meaningfulness of the data in question.

### 

### Grades

Currently, the indicator of grades improvement is based on the
percentage of regular attendees whose mathematics and reading/language
arts grades improve by a half-grade or more from the first marking
period in the fall to the last marking period in the spring. For
example, if an A--F scale is being used, a half-grade change is defined
as any decrease or increase in the letter grade (e.g., A to A- or C+ to
B-). Similar guidelines and instructions exist for evaluating whether or
not a given regular attendee witnessed a half-grade or more improvement
in their spring grade for other widely used grading scales (e.g.,
100-point scales, E-S-U scales).

The primary limitation associated with the present indicator is that it
fails to provide more detailed information about how many students at
different levels of academic achievement\
(e.g., below average, average, and above average students) witnessed a
change in mathematics and reading/language arts grades across the school
year. Existing data-collection forms in PPICS ask only for the number of
regular attendees in each of three categories for grade outcomes
(namely, improved a half grade or more, obtained the same grade, or
declined a half grade or more when comparing fall and spring grades).
This limitation is especially problematic when one considers that the
difficulty of academic content is only likely to increase during the
course of the school year, and one could argue that helping students
maintain an acceptable grade in the face of more difficult content is an
achievement worth capturing. For these reasons, the current approach to
collecting grades data and calculating the grade-related indicators are
not of use for assessing program impact for students who vary in their
baseline level of achievement.

A second limitation is that the indicator of grade improvement as
currently calculated reflects the proportion of students whose grades
improved without adjusting for the number of students whose grades
declined. For this reason, it could be argued that the percentage of
regular attendees whose grades increased was overstated and tended to
inflate the program's true impact on student achievement. To address
this limitation, members of the 21st CCLC ETF proposed that the
indicator be adjusted by calculating the net number of students with an
increase in grades, as specified in the following general formula:

  -----------------------------------------------------------------------
  (# of regular of attendees with an increase in grades -- \# of regular
  attendees with a decrease in grades)

  Total number of regular attendees with grades data reported
  -----------------------------------------------------------------------

By the same token, the grades indicator as currently calculated does not
adjust for those students who had achieved the highest possible grade at
the end of the first marking period in the fall and who would therefore
be unable to demonstrate any improvement. Including these students in
the denominator of the indicator biased the indicator to underrepresent
the impact of the program. The data-collection form in PPICS for grades
was modified for the 2005−06 APR to ask respondents to report the number
of regular attendees with grades data reported that fell within this
category. Assessing the impact of removing these students from the
denominator will be explored in the following sections.

### 

### State Assessment

The indicator related to changes in state assessment results currently
is based on the percentage of regular attendees whose achievement test
scores improved from *below proficient* to *proficient or above* on the
mathematics and reading/language arts portions of their state's
assessment system. In this regard, a regular attendee would be counted
in the numerator associated with the calculation if he or she scored in
a category below proficiency on the assessment taken in the prior school
year and scored at *proficient or above* on the assessment taken during
the school year associated with the reporting period. For the 2005−06
reporting period, state assessment results from tests administered
during the 2004−05 school year would be compared with results from
assessments given during the course of the 2005−06 school year. The
denominator associated with the calculation is all regular attendees
with state assessment data reported who scored below proficiency on the
assessment taken during the prior school year. At present, no steps are
taken to account for when a state conducts testing during the course of
the school year (e.g., fall versus spring). When reporting state
assessment results, respondents are presented with the proficiency
categories employed in their state's assessment system, which range from
a minimum of three to a maximum of five, the latter of which are
typically characterized by more than one level representing a
nonproficient status. Results are then aggregated into the standard
federal categories: basic, proficient, and advanced.

The present emphasis on moving participating youth from *below
proficient* to *proficient or above* has two consequences: (1) it fails
to recognize the gains that students make toward proficiency, even
though they continue to score within the below basic or basic
categories, and (2) it limits the domain of students under consideration
to those that are on the cusp of crossing into proficiency. These
characteristics especially reduce the utility of the information from a
continuous improvement standpoint and in supporting states as they go
through the decision-making process relative to which grantees warrant
continued funding based on past and present performance on impacting
student achievement.

In light of these limitations, it seems worthwhile to report how many
students are witnessing an increase in their scores, even if they are
not yet at the *proficient* level. This approach would provide a better
perspective on the rate of improvement in state assessment results. Most
of the recommendations made by the ETF, therefore, involve determining
the percentage of regular attendees whose mathematics and
reading/language arts proficiency levels increased from one year to the
next, irrespective of whether or not proficiency was actually achieved
in such a movement from one category to another. As mentioned earlier,
some states employ more than one level to indicate *below proficient*,
so although students may increase their proficiency levels from *well
below proficient* to *below proficient*, they would not be included in
the numerator in the previously discussed set of calculations, even
though they are making progress. Examining regular attendees who are
improving---even if they are not yet proficient---can still provide
valuable information about the effectiveness of states' programming.

Another limitation of the current approach to calculating the indicator
is that it does not account for those regular attendees whose
proficiency levels decline between the reporting years. The percentage
of students who witnessed an increase in state assessment scores,
therefore, may not accurately reflect the impact of a program on student
achievement. To address this limitation---similar to the grades
indicator calculations---the state assessment indicator can be adjusted
to account for those students whose scores decline from one year to the
next, resulting in a net change formula:

  -----------------------------------------------------------------------
  (# of regular attendees with an increase in proficiency level -- \# of
  regular attendees with a decrease in proficiency level)

  Total number of regular attendees scoring below proficiency last year
  -----------------------------------------------------------------------

Note that, in employing that formula, states that have only three
proficiency categories will not have any regular attendees moving to a
lower proficiency category between assessment administrations, given
that the concern is only with regular attendees scoring below
proficiency to begin with. In this regard, the movement to a net change
calculation as represented in the previous formula will have no bearing
on the results for these states whose assessment system has only three
proficiency categories.

### 

In the analyses of state assessment results presented later in the
report, the current manner in which the indicator is calculated is
compared with the revised approach described above.

### Teacher Surveys

Two indicators associated with the current GPRA reporting process are
predicated on data obtained from a modified version of the teacher
survey employed as part of the APR associated with the federal
discretionary grant program developed by Learning Point Associates:

-   The percentage of regular attendees with teacher-reported
    improvement in homework completion and class participation.

```{=html}
<!-- -->
```
-   The percentage of regular attendees with teacher-reported
    improvement in student behavior.

The teacher survey is meant to be taken by a school-day teacher
providing mathematics or reading/language arts instruction to a given
student who met the definition of a regular attendee (30 days or more)
during the reporting period. (The teacher survey can be accessed at
ppics.learningpt.org/ppics/survey.asp.) The teacher taking the survey on
behalf of a specific student is asked to specify whether the student
needed to improve in a given area and if so, to what extent the teacher
witnessed a change in student behavior in that area during the school
year. Ten questions appear on the survey, each addressing a specific
academic-related behavior. The teacher survey items and the data labels
for each item are as follows:

-   *THW:* Turning in homework on time

-   *CHW:* Completing homework to your satisfaction

-   *PIC:* Participating in class

-   *VOL:* Volunteering in class

-   *ATT:* Attending class regularly

-   *BAC:* Being attentive in class

-   *BEH:* Behaving in class

-   *ACP:* Academic performance

-   *MOT:* Coming to school motivated to learn

-   *ALN:* Getting along well with others

The indicator based on the percentage of regular attendees with
teacher-reported improvement in homework completion and class
participation is calculated from the *turning in homework on time*
(THW), *completing homework to your satisfaction* (CHW), and
*participating in class* (PIC) items that appear on the survey. All ten
items appearing on the survey are used to calculate the percentage of
regular attendees with teacher-reported improvement in student behavior.
These indicators are all predictors of overall improved academic
behavior. For example, students who turn in their homework on time,
participate in class, attend class regularly, and come to school
motivated to learn are more likely to demonstrate academic success.

Currently, several psychometric limitations exist in relation to the
teacher survey. Validation studies performed by Learning Point
Associates in early 2006 suggested that the present teacher survey
performs differently depending upon the grade level of the student
represented in the survey. The survey performed best in relation to high
school students, reasonably well for elementary students, and mediocre
at best in relation to students enrolled in middle school. Additionally,
the fact that aggregate teacher survey results are collected in PPICS
prevents Learning Point Associates staff from employing the full domain
of scoring and scaling techniques that would yield more psychometrically
valid results and help determine what performance threshold would be
most reasonable for students at a given grade level.

Given the findings from the teacher survey validation study, one of the
goals of these analyses is to identify differences among grade levels in
terms of changes in academic-related behaviors to determine how the
survey could be revised to increase its validity across grade levels.
For example, elementary students may not have as much homework to
complete. Attendance levels at the elementary grades are also typically
higher than in middle or high school; consequently, the data obtained
from lower grade levels may not be as relevant or meaningful in terms of
assessing program impact.

In addition, results from the validation study make it clear that an
assessment of changes in student academic behaviors would best be
performed by employing individual teacher survey results. Individually
entered scores would allow Learning Point Associates to employ the full
range of psychometric techniques when validating and analyzing survey
results. The output from adopting such processes will likely be a series
of scale scores that outline where students fall relative to witnessing
improvements in certain behaviors. The manner in which the teacher
survey-based performance indicators are calculated is likely to be based
on the percentage of regular attendees that exceeded a certain cut-off
score relative to improved behaviors. Given that we do not have
widespread access to teacher survey results from states employing this
reporting option during the 2005−06 reporting period, we are unable to
forecast what indicator results would look like when moving to the
collection and analysis of individual teacher survey results. Analyses
related to teacher survey results, therefore, focus on differences
between various subgroups.

To facilitate interpreting the charts in the teacher survey section,
items have been grouped as follows (See Table 6). Group 1 includes the
items most closely associated with the academic nature of the
classrooms, including turning homework in on time, completing homework
to the teacher's satisfaction, academic performance, and students'
motivation to learn. Group 2 is based on items related to students'
participation in class. These survey items include how much improvement
the student has made in participating in class, volunteering in class,
attending class more regularly, and being attentive in class. The third
category involves behaviors relating to classroom behavioral norms,
including behaving better in class and getting along well with others.
Each of these items measures student behaviors that are likely to lead
to academic progress.

Table 6. Item Groupings

  ---------------------- -------------------------- ----------------------
  **Group 1 (Academic)** **Group 2                  **Group 3 (Behavior)**
                         (Participation)**          

  THW                    PIC                        BEH

  CHW                    VOL                        ALN

  ACP                    ATT                        

  MOT                    BAC                        
  ---------------------- -------------------------- ----------------------

In the survey, levels of change were given as "slight," "moderate," and
"significant." These levels were collapsed into categories of "increase"
and "decrease" for the majority of the analyses. In addition, the
denominator associated with indicator calculations excludes any student
who did not need to improve on a given behavior as reported by the
teacher in question, unless otherwise noted. Analyzing the number of
students who did not need to improve provides insight on academic needs
of students in the program. The 21st CCLC program was established to
target students who are most in need of academic assistance. If a large
percentage of students do not need to improve, then questions regarding
recruitment of those students need to be addressed.

## Analysis of Grades, State Assessment, and Teacher Survey Results 

The following sections analyze grades, state assessment, and teacher
survey results through several different lenses. First, results from the
2005--06, 2004--05, and occasionally, the 2003--04 reporting periods are
compared to determine whether a higher percentage of students made
progress between last year and previous reporting cycles. In addition,
results obtained from the 2005--06 APR are reported in multiple formats
to address the following questions that also have been raised during
discussions with the ETF on what elements of the current indicator
system may warrant revision, some of which have been touched on already
in prior sections:

-   *Considering only regular participants, is there evidence that
    students with higher attendance levels during the reporting period
    show a greater net change in grades, state assessment results, and
    teacher-reported behaviors than students with lower attendance
    levels?* Historically, data related to changes in student behaviors
    and achievement have only been collected on those students that
    attended a given center for 30 days or more during the reporting
    period (i.e., regular attendees). There has been discussion since
    the earliest days of PPICS regarding whether or not this definition
    of regular attendance is the most meaningful and feasible way to
    determine the threshold at which program attendance is likely to
    yield the types of outcomes being sought by the program.

> During the 2005−06 APR reporting period, states were given the option
> to require that their grantees submit APR grades, state assessment,
> and teacher survey data separately for three subgroups of regular
> attendees: (1) those attending 30 to 59 days during the reporting
> period, (2) those attending 60 to 89 days, and (3) those attending 90
> days or more. This gradation provides several additional analysis
> opportunities. Research indicates that the more often students attend
> a program, the more likely they are to improve in terms of academic
> behaviors and achievement. Differentiating attendance levels makes it
> possible to assess whether there is a relation between higher levels
> of attendance and positive changes in the outcomes of interest
> represented in the current domain of indicators. Moreover, analyzing
> three attendance levels can help determine whether a threshold exists
> that is associated with large increases in the percentage of students
> showing improvements in academic behaviors and achievement.

-   *Do the GPRA impact measures vary consistently among programs
    exclusively serving elementary, middle, and high school students?*
    During the course of the past three years, only limited effort has
    been undertaken to explore how the reporting of grades, state
    assessment, and teacher survey-related information among centers
    that exclusively serve high school students in particular may differ
    from overall trends across each of these impact categories.

Most of the charts and figures presented are based on a comparison
between the method currently used for calculating the indicator and the
new formulas recommended by the ETF. These comparisons are especially
relevant to changes in grades and state assessment results for which
there are very concrete recommendations relative to moving to a net
change approach by subtracting the number of students witnessing a
decline in performance from those witnessing an improvement.

## Changes in Student Behavior and Academic Performance Across Years

### Grades

The percentage of regular attendees with improved grades is presented in
Figure 25. The previous approach to calculating this percentage is
presented in the left side of the chart, whereas the net gain approach
(i.e., subtracting students with declining grades from the numerator) is
presented on the right side. It is evident that the net gain approach
results in a much lower percentage of students improving their grades.
Moreover, using the net gain approach, the percentage of students
improving their mathematics grades in 2005--06 stands out more clearly
as having increased compared to previous years. Regarding the percentage
of students whose reading grades increased, there were no differences of
apparent importance across years of the APR. The 27 percent of students
whose grades in reading increased in 2005--06 appears slightly less than
in 2003--04 (28 percent) and slightly more than in 2004--05 (24 percent)
(*Note:* Where only a single year is designated, 2004 refers to the
2003--04 period, 2005 refers to the 2004--05 period, and 2006 refers to
the 2005--06 period).

Figure 25. Percentage of Regular Attendees Improving Grades by Program
Year

![](media/image21.wmf){width="6.500694444444444in"
height="3.1555555555555554in"}

*Note.* Based on 1,553 centers providing data for 2003−04 (43 percent of
all centers in the APR); 7,602 centers providing data for 2004−05 (97
percent of all centers in the APR); and 5,284 centers providing data for
2005−06 (56 percent of all centers in the APR).

The 2005--06 APR was the first to identify the number of regular
attendees earning the highest possible fall grade, so the impact of
adjusting the indicator for these students can be estimated for this
year only. Figure 26 demonstrates the impact of this adjustment (i.e.,
subtracting students with the highest possible grade from the
denominator), in combination with calculating or not calculating net
change. As expected, excluding students with the highest possible grades
increased the proportion of students who witnessed an increase in their
grades. It appears, however, that the impact of this change to the
indicator is somewhat attenuated in combination with the net change
approach.

Figure 26. Comparison of Methods for Calculating Grades Improvement

![](media/image22.wmf){width="6.501388888888889in"
height="2.966666666666667in"}

*Note.* The "High Grade" indicator subtracts students with the highest
grade from the denominator. The "net" indicators subtract students whose
grade decreased from the numerator. Based on 5,284 centers providing
data for 2005−06 (56 percent of all centers in the APR).

### 

### State Assessment

The following series of analyses were conducted to determine the
percentage of regular attendees attaining proficiency (i.e., those
students who went from below proficiency to proficiency or above) on the
state assessment exam in reading/language arts and mathematics. As
illustrated in Figure 27, it is clear that a smaller percentage of
students improved to proficiency this year as compared to last year.
Note that a higher number of states opted to report state assessment
data in the 2005--06 APR, thereby substantially increasing the number of
centers for which assessment data was reported. Accordingly, only
looking at the percentage of students who attained proficiency across
years is misleading when attempting to assess trends related to the
effectiveness of 21st CCLC programs.

Figure 27. Percentage of Regular Attendees Attaining Proficiency by
Program Year

![](media/image23.wmf){width="6.502083333333333in"
height="2.9451388888888888in"}

*Note*. Based on 440 centers in five states providing state assessment
outcome data for 2004−05 (six percent of all centers in the APR) and
3,711 centers in 20 states providing data in 2005−06 (40 percent of all
centers in the APR)

As illustrated in Figure 28, if only the five states that reported data
for both the 2005 and 2006 reporting periods are included in the
analysis, the difference in the percentage of students who attained
proficiency by program year reverses, such that regular attendees
attained proficiency at a higher rate in 2006 than in 2005 in both
reading (28 percent in 2005 and 32 percent in 2006) and mathematics (30
percent in 2005 and 33 percent in 2006). One possible explanation for
this trend is that the five states that reported proficiency level data
for both reporting periods were more effective in increasing student
scores this year compared to last; however, there may also be several
other factors influencing this pattern. These analyses are based on data
from only five states, and only account for about 6 percent of centers
in the APR for both years.

Figure 28. Percentage of Regular Attendees Attaining Proficiency by
Program Year\
for States Reporting Both Years

![](media/image24.wmf){width="6.502083333333333in"
height="2.9319444444444445in"}

*Note*. Based on 440 centers in five states providing data in 2004−05 (6
percent of all centers in the APR) and 457 centers in 2005−06 (5 percent
of all centers in the APR)

Then, the percentage of regular attendees who increased their
proficiency level from one year to the next was examined. The previous
analysis determined the percentage of regular attendees who moved from
below proficient to proficient or above, whereas the following analyses
focus on those students who demonstrated an increase in their
proficiency level regardless of whether they actually attained
proficiency. Again, these analyses reflect a recommendation made by the
ETF to better account for any improvement in the proficiency level
obtained by a given student in the numerator of the calculations
employed to assess the proportion of regular attendees who improved on
state assessments. Such improvements, even if not associated with the
attainment of proficiency, are significant in and of themselves and
warrant representation in indicator calculations.

Also note that the following analyses incorporate a net change in
improvement among centers by examining the total number of regular
attendees who improved their proficiency level minus the number who
declined, divided by the total number of regular attendees scoring below
proficiency. As discussed earlier, the rationale for making such a
change as articulated by members of the ETF is to avoid overrepresenting
the achievement levels obtained by the 21st CCLCs operating during a
given reporting period. As shown in Figure 29, centers reported an
increase from 2003−04 to 2004−05 in the percentage of regular attendees
improving their proficiency level; however, the percentage of regular
attendees who increased their proficiency levels declined from 2004−05
to 2005−06 (although the percentage in 2005−06 was generally still
higher than in 2003−04). Here again, it is important to note that the
number of states reporting state assessment data increased from 5 to 20.
This considerable increase in the number of centers included in the
analysis for 2005−06 should be taken into account when interpreting
these data.

Figure 29. Percentage of Regular Attendees Improving Proficiency\
Level, by Program Year

![](media/image25.wmf){width="6.5in" height="2.834722222222222in"}

*Note*. Based on 159 centers in three states providing data for 2003−04
(4 percent of all centers in the APR); 440 centers in five states
providing data for 2004−05 (6 percent of all centers in the APR); and
3,711 centers in 20 states providing data for 2005−06 (40 percent of all
centers in the APR).

Figure 30 depicts both the current method and proposed revised method
(net change) of calculating students' proficiency level improvements for
only those three states that reported state assessment data across all
three years. A similar pattern for the net change analysis is seen
compared to the analysis that included all reporting states each year,
such that with the revised approach, states reported a higher rate of
improvement in 2005 than in the other two years.

Figure 30. Percentage of Regular Attendees Improving Proficiency Level,\
by Program Year, for States Reporting All Three Years

![](media/image26.wmf){width="6.5in" height="2.90625in"}

*Note*. Based on 159 centers in three states providing data for 2003−04
(4 percent of all centers in the APR); 126 centers in three states
providing data for 2004−05 (2 percent of all centers in the APR); and
350 centers in three states providing data for 2005−06 (4 percent of all
centers in the APR).

In summary, the charts in this section show how the assessment
improvement indicator has been calculated in the past and then
recalculated using the proposed methodologies recommended by the ETF.
One proposed revision counts any improvement regular attendees witness
in their proficiency level, regardless of whether or not they actually
attained proficiency. Another revision subtracts the number of regular
attendees who may have declined in their proficiency level from the
number of those who improved their proficiency level, thereby
calculating a net change in improvement among centers and perhaps more
accurately reflecting the impact of programming on state assessment
performance. As such, with these revisions, a smaller percentage of
regular attendees appear to have attained proficiency or improved their
proficiency level compared to prior years; however, the substantial
increase in the number of states providing state assessment data for the
2005−06 reporting period warrant interpreting these data cautiously.

If the goals of revising the manner in which both grades and state
assessment-related indicator calculations are performed are to ensure
that only students who improve are included in indicator calculations
and students whose performance declines are also accounted for, then the
new denominator/net approach highlighted in Figures 26, 29, and 30
appear to address both of these concerns. From a practical perspective,
however, the movement to this approach offers limited insight into the
relation between program attendance, baseline academic functioning,
changes in academic behaviors, and the degree of improvement on state
assessments by program participants. This limitation is largely due to
the aggregate center-level data on student attendance and performance
collected in PPICS. If understanding more fully the nature of such
relations is an important consideration, then other data-collection
options warrant exploration, including the collection of student-level
attendance, grades, and related achievement data.

### Teacher Surveys

The following figures show the percentage of regular attendees with
teacher-reported improvements in various academic behaviors in both 2005
and 2006. The response rates across the years were similar, with 76
percent of the given teacher surveys being returned in 2005 and 71
percent in 2006. These charts reflect the actual percentage of students
demonstrating an improvement.

The academic items shown in Figure 31 demonstrate a decline in the
percentage of students who showed an improvement on the teacher survey
items associated with academic behaviors. The percentage of students
demonstrating an improvement is above 65 percent on each item. There was
a slight dip between 2005 and 2006 on each item with the exception of
*academic performance* (ACP). In both 2005 and 2006, teachers reported
76 percent of 21st CCLC regular attendees improved their academic
performance.

Figure 31. Percentage of Regular Attendees Who Showed\
Improvement on Academic Items by Year

![](media/image27.wmf){width="6.5in" height="3.165277777777778in"}

*Note.* Based on 4,602 centers providing data in 2005 (59 percent of all
centers in the APR) and 5,079 centers providing data in 2006 (54 percent
of all centers in the APR).

In a similar fashion, teacher survey items concerned with participation,
shown in Figure 32, demonstrated a higher percentage of students showing
improvement in 2005 than in 2006. *Participating in class* (PIC) was the
only item that reached over 70 percent in both years. It is interesting
to note that the percentage of students demonstrating an improvement in
2006 for *attending class regularly* (ATT) only reached 58 percent.

Figure 32. Percentage of Regular Attendees Who Showed\
Improvement on Participation Items by Year

![](media/image28.wmf){width="6.5in" height="3.0652777777777778in"}

*Note.* Based on 4,602 centers providing data in 2005 (59 percent of all
centers in the APR) and 5,079 centers providing data in 2006 (54 percent
of all centers in the APR).

Figure 33 analyzes the items most closely associated with behaviors
relating to conforming to classroom behavioral norms, including
*behaving better in class* (BEH) and *getting along well with others*
(ALN). Similar to the other items, the percentage of students showing
improvements was higher in 2005 than in 2006. Teachers reported that 64
percent of regular attendees showed improvements in 2006 on *behaving in
class* and 65 percent on *getting along well with others*. In 2005, the
percentages were 66 percent and 68 percent, respectively.

Figure 33. Percentage of Regular Attendees Who Showed\
Improvement on Behavior Items by Year

![](media/image29.wmf){width="6.5in" height="2.9444444444444446in"}

*Note.* Based on 4,602 centers providing data in 2005 (59 percent of all
centers in the APR) and 5,079 centers providing data in 2006 (54 percent
of all centers in the APR)

Although the percentage of students who showed improvement declined
between 2005 and 2006, the percentage of students that teachers
indicated did not need improvement increased across all items (The "Did
Not Need to Improve" option became available in 2005, but some teachers
continued to use the former survey that did not allow this option, which
may have influenced the results). For example, the percentage of
students who did not need improvement on *turning homework in on time*
(THW) was 18 percent in 2005 and 27 percent in 2006, as shown in Figure
34.

Figure 34. Percentage of Regular Attendees Who Did Not Need\
Improvement on Academic Items by Year

![](media/image30.wmf){width="6.502083333333333in"
height="2.9451388888888888in"}

*Note.* Based on 4,602 centers providing data in 2005 (59 percent of all
centers in the APR) and 5,079 centers providing data in 2006 (54 percent
of all centers in the APR)

Previously it was noted that the percentage of students that
demonstrated an increase on the *attending class regularly* variable was
less than 60 percent. As demonstrated in Figure 35, a large percentage
of the regular attendees did not need improvement in this category: 30
percent in 2005 and 46 percent in 2006.

Figure 35. Percentage of Regular Attendees Who Did Not Need\
Improvement on Participation Items by Year

![](media/image31.wmf){width="6.502083333333333in"
height="3.029861111111111in"}

*Note.* Based on 4,602 centers providing data in 2005 (59 percent of all
centers in the APR) and 5,079 centers providing data in 2006 (54 percent
of all centers in the APR)

A similar pattern can be found in the behavior items. Figure 36 shows
that in 2006, approximately one third of the students did not need
improvement in *behaving in class* or *getting along with others* (28
percent and 35 percent, respectively). This compares with 20 percent of
the students who did not need behavior improvement and 23 percent who
did not need improvement in getting along with others in 2005.

Figure 36. Percentage of Regular Attendees Who Did Not Need\
Improvement on Behavior Items by Year

![](media/image32.wmf){width="6.502083333333333in"
height="3.1708333333333334in"}

*Note.* Based on 4,602 centers providing data in 2005 (59 percent of all
centers in the APR) and 5,079 centers providing data in 2006 (54 percent
of all centers in the APR)

In addition, given that teacher survey data are not collected at the
individual student level across multiple years of potential program
participation, it is unknown how long-term program participation over
time affects the extent of improvement witnessed by students. For
example, the results highlighted in the previous charts show that 46
percent of regular attendees did not need to improve their school-day
attendance. It is unknown, though, whether these students had been in
21st CCLC for multiple years and what cumulative effect programming may
have had on this particular academic-related behavior. In this regard,
students who had been in the program longer may have developed social
and behavioral skills that resulted in teacher reports of not needing
improvement on one or more of the items appearing on the teacher survey.

## Changes in Student Behavior and Academic Performance by Improvement Category

## 

The percentage of students who showed a decrease in functioning on a
given behavior is extremely low on all items in the teacher survey. To
examine these results more closely, the following figures analyze each
item by significant improvement, moderate improvement, slight
improvement, no change, and total decline on each item.

Figure 37 shows that the percentage of students who demonstrated a
decrease on the academic items is below 10 percent on all items.
Although *academic performance* had the highest overall increase (76
percent), only 23 percent showed significant improvement compared to 27
and 26 percent showing moderate and slight improvement, respectively.
The data associated with the *motivated to learn* (MOT) question are
also of some interest given the strong linear pattern they follow as the
degree of improvement lessens. There is a gradual increase from
significant to slight improvement, but the number of students who did
not show a change is the highest at 25 percent, whereas a slight 7
percent declined. On the other items, the percentage of students who did
not show a change was lower than the percentage of students who showed
an improvement in any of the categories.

Figure 37. Percentage of Regular Attendees Overall Academic\
Item Results by Improvement Category

![](media/image33.wmf){width="6.502083333333333in"
height="2.9319444444444445in"}

*Note.* Based on 5,079 centers providing data in 2006 (54 percent of all
centers in the APR)

With the participation items, the percentage of students who
demonstrated a decline was again under 10 percent. The percentage of
students who did not show a change, however, increased dramatically,
particularly on the *volunteering in class* (VOL; 39 percent) and
*attending* *class regularly* (35 percent) items as displayed in Figure
38. *Attending class regularly*, however, had the highest percentage of
students showing a significant improvement (23 percent).

Figure 38. Percentage of Regular Attendees Overall Participation\
Item Results by Improvement Category

![](media/image34.wmf){width="6.5in" height="2.859027777777778in"}

*Note.* Based on 5,079 centers providing data in 2006 (54 percent of all
centers in the APR)

## 

Figure 39 displays the overall improvement, no change, and decline
results for behavior items. The largest percentage of students showing a
decline, 10 percent, can be found on the *behaving* *in class* (BEH)
item. Teachers reported over 25 percent of students demonstrating no
change on both *behaving in class* (BEH) and *getting along well with
others* (ALN). As the degree of improvement represented by a given
response category lessened, the percentage of students who showed
improvement increased.

Figure 39. Percentage of Regular Attendees Overall on Behavior\
Items by Improvement Category

![](media/image35.wmf){width="6.5in" height="3.089583333333333in"}

*Note.* Based on 5,079 centers providing data in 2006 (54 percent of all
centers in the APR)

These results indicate that teachers perceive a low percentage of 21st
CCLC participants as declining across the survey items. In general, the
trend is a higher percentage of students showing moderate or slight
improvement across the indicators.

On five of the items---*motivated to learn*, *volunteering in class*,
*attending class regularly*, *behaving in class*, and *getting along
well with others*---the percentage of students who showed no change is
the highest response category. This response category is especially high
on the *volunteering* *in class* item (39 percent) and *attending class
regularly* item (35 percent).

Another item warranting further discussion is *academic performance*.
Only 17 percent of the participants showed no change in their academic
performance, whereas 23 percent made significant improvements. Teachers
responded that over 25 percent made both moderate and slight
improvements to their academic performance. This would seem to suggest
that teachers perceive academic progress among 21st CCLC participants
and that only a small percentage of 21st CCLC attendees have shown no
improvement. In examining these results, it is important to reiterate
that there have been concerns regarding the method of obtaining data on
student academic behaviors directly from school-day teachers, given the
possibility that teachers' responses may be influenced by a perception
that what they endorse may also be interpreted as a reflection of their
instruction and ability to influence student outcomes.

## Changes in Student Behavior and Academic Performance by Student Attendance

As in previous years, the 2006 APR defined a regular attendee as a
student who attended the center for 30 or more days during the reporting
period. There has been some concern expressed by both grantees and SEA
staff that this definition may be too low of a threshold of interaction
with center programming to produce the expected outcomes in terms of
grades, student achievement, and behavior. To further explore the
relationship between levels of program attendance and student behavioral
change and academic outcomes, states were afforded the option for the
first time in 2005--06 to require that their grantees submit APR grades
data separately for three subgroups of regular attendees: (1) those
attending 30 to 59 days during the reporting period, (2) those attending
60 to 89 days, and (3) those attending 90 days or more. An analysis of
this information can provide insight into the relationship between
program attendance and behavioral and achievement outcomes and
contribute to the discussion about what an appropriate threshold may be
when considering how to define a regular attendee.

### Grades

Figure 40 displays the percent of students with improving grades
separately for the three different categories of regular attendee. The
indicator displayed in the chart was calculated using the net increase
in students' grades and excluding students with the highest possible
initial grade. The figure indicates a difference between the 30- to
59-day threshold for defining a regular attendee (on the one hand) and
the 60- to 89- and 90-day or more thresholds (on the other). Whereas the
net increase for students attending for 30 to 59 days was 18 percent,
the increase for the next two attendance thresholds was 26 and 27
percent, respectively.

The proportion of students improving their mathematics grade (net
change) also increased in association with increased program attendance;
however, this trend exhibited a more linear pattern. Although the
proportion of students with a net change in improvement at the 30- to
59-day threshold (13 percent) was once again lower than the 60- to
89-day threshold (18 percent), the 90-day or more threshold was higher
still (21 percent). Looking across both types of reading and mathematics
grades, however, there appears to be a clear distinction between the
lowest threshold and the middle and highest thresholds. This finding may
suggest that raising the threshold for the definition of regular
attendee to 60 or more days when evaluating changes in grades
performance would make the outcomes of the program more demonstrable in
this regard.

Figure 40. Percentage of Regular Attendees Improving Grades by
Attendance Gradation

![](media/image36.wmf){width="6.501388888888889in"
height="2.8243055555555556in"}

*Note.* Based on 3,023 centers providing attendance gradation data (32
percent of all centers in the APR)

### State Assessment

Figure 41 displays the percentage of regular attendees attaining
proficiency disaggregated by the three different categories of
attendance gradation. Although there does not appear to be a significant
difference between the lower two attendance categories (31 percent and
27 percent, respectively for reading, and 24 percent and 24 percent,
respectively for mathematics), it appears that students who attended the
program for 90 or more days during the 2005−06 reporting period had a
higher rate of proficiency attainment in reading (33 percent) and
especially in mathematics (31 percent). Only a subset of states (12 out
of the 20 states who reported state assessment data) employed the
gradation option when reporting state assessment results.

Figure 41. Percentage of Regular Attendees Attaining Proficiency\
by Attendance Gradation

![](media/image37.wmf){width="6.502083333333333in"
height="2.861111111111111in"}

*Note*. Based on 1,676 centers in 12 states providing attendance
gradation data (18 percent of all centers in the APR)

Figure 42 demonstrates a similar pattern of student achievement by
attendance gradation, with students who attended 90 days or more showing
a greater degree of proficiency-level improvement than students who
attended less than 90 days. This is evident even when accounting for the
net change in student improvement (taking into account those students
who declined in their proficiency level from one year to the next),
especially in relation to improvement in mathematics achievement.

Figure 42. Percentage of Regular Attendees Improving Proficiency Level\
by Attendance Gradation

![](media/image38.wmf){width="6.502083333333333in"
height="2.9090277777777778in"}

*Note*. Based on 1,676 centers in 12 states providing attendance
gradation data (18 percent of all centers in the APR)

Although the findings from the grades analyses highlight a difference
between those students who attend 30 to 59 days versus those who attend
60 or more days (60 to 89 and 90 or more days), the state assessment
findings highlighted in relation to Figures 41 and 42 may suggest that
raising the threshold for the definition of regular attendee to 90 or
more days when evaluating changes in student performance on state
assessments would make the outcomes of the program in this regard more
demonstrable.

### Teacher Survey

As would be expected, and as seen with the grades and state assessment
data, the percentage of students who showed teacher-reported improvement
increased with level of attendance. This was the trend found in all
academic categories, as seen in Figure 43. *Academic performance* (ACP)
was again the item that showed the most improvement at 74 percent, 77
percent, and 80 percent for 30 to 59 days, 60 to 89 days, and 90 days or
more categories, respectively. This finding supports the argument that
the more time students spend in a 21st CCLC program, the more likely
they are to improve their academic performance.

Figure 43. Percentage of Regular Attendees Who Showed Improvement on\
Academic Items by Attendance Gradation

![](media/image39.wmf){width="6.502083333333333in"
height="3.0548611111111112in"}

*Note.* Based on 2,819 centers reporting attendance gradation data (30
percent of all centers in 2006)

The percentage of students who showed an increase on participation
variables also tended to increase by attendance gradation as shown in
Figure 44. *Participating in class* (PIC) had the highest percentage for
all attendance gradation levels. For students who attended 30 to 59
days, teachers reported 71 percent of the students demonstrating
improvement; for 60 to 89 days, 74 percent improved; and for students
attending 90 days or more of programming, 76 percent improved their
participation.

Figure 44. Percentage of Regular Attendees Who Showed Improvement on
Participation Variables by Attendance Gradation

![](media/image40.wmf){width="6.5in" height="2.9194444444444443in"}

*Note.* Based on 2,819 centers reporting attendance gradation data (30
percent of all centers in the APR)

[]{.mark}

Figure 45, which highlights the behavior items, shows a similar pattern.
With each gradation threshold, the percentage of students showing
improvement also increased. Students who attended for 90 days or more
during the reporting period had the highest percentage of improvement
for both *behaving in class* (BEH) at 68 percent and *getting along with
others* (ALN) at 69 percent.

Figure 45. Percentage of Regular Attendees Who Showed Improvement on\
Behavioral Variables by Attendance Gradation

![](media/image41.wmf){width="6.497916666666667in"
height="3.2847222222222223in"}

*Note.* Based on 2,819 centers reporting attendance gradation data (30
percent of all centers in the APR).

It is also interesting to note the percentage of students who did not
need improvement relative to attendance gradations. As shown in Figure
46, the students who had attended the program for 90 days or more were
most likely to be perceived as not needing improvement by their
teachers. On the *turning in homework on time* (THW) item, for example,
the students who had been in the program for 90 or more days had the
highest percentage of students who did not need improvement, 29 percent.
Based on teacher perceptions, students at all attendance gradation
levels needed the least improvement on *motivated to learn* (MOT).

Figure 46. Percentage of Regular Attendees Who Did Not Need Improvement
on\
Academic Items by Attendance Gradation

![](media/image42.wmf){width="6.5in" height="3.0180555555555557in"}

*Note.* Based on 2,819 centers reporting attendance gradation data (30
percent of all centers in the APR)

Figure 47 shows the percentage of students who did not need improvement
on the participation variables by gradation. Note that on the *attending
class regularly* (ATT) item in this group, nearly half (49 percent) of
the students who attended 90 days or more did not need improvement, and
45 percent of the students who had attended 30 to 59 days did not need
improvement, according to teachers. Again, this finding may suggest that
students who are more likely to come to the program more regularly may
also be more likely to attend school more often.

Figure 47. Percentage of Regular Attendees Who Did Not Need Improvement
on Participation Variables by Attendance Gradation

![](media/image43.wmf){width="6.502083333333333in"
height="2.717361111111111in"}

*Note.* Based on 2,819 centers reporting attendance gradation data (30
percent of all centers in the APR)

The behavior items showed a higher percentage of students who did not
need improvement in the 30 to 59 days and 90 days or more categories on
*behaving in class* (BEH), as shown in Figure 48.

Figure 48. Percentage of Regular Attendees Who Did Not Need Improvement
on\
Behavior Variables by Attendance Gradation

![](media/image44.wmf){width="6.500694444444444in" height="2.78125in"}

*Note.* Based on 2,819 centers reporting attendance gradation data (30
percent of all centers in the APR)

Generally, teacher survey results across the three attendance gradation
subgroups suggest that students who attended the program more frequently
showed the most positive growth across all items. Each item showed
incremental increases across gradation categories. These results may
suggest that the current approach to collecting behavioral change data
by employing teacher-completed surveys has more value when coupled with
the gradations reporting option, as compared to demonstrating the
overall percentage of regular attendees showing an increase.
Consistently, higher levels of attendance appear to be associated with
greater levels of improvement.

The variability in the number of students who did not need to improve
among the different levels of attendance should also be highlighted.
Students who had been in the program for 90 days or more were most
likely to be rated by their teachers as not needing improvement on all
variables. Without student-level data, however, it is impossible to tell
whether these results can be attributed to students being enrolled in
the program over time.

In addition, as discussed earlier, the teacher survey has shown higher
validity at the high school grade levels. Given that the results
highlighted in this section suggest that teacher surveys may demonstrate
their greatest utility when exploring results predicated on attendance
gradation reporting, further analysis was conducted on teacher survey
gradation data separately for centers that exclusively serve elementary
as opposed to middle/high school students.

Figure 49 shows the academic items from the teacher survey by grade
level and gradation. The general trend shows that as the number of days
increases, the higher the percentage of students showing improvement.
This is particularly true for middle/high school students across all
variables. In fact, on *academic performance* (ACP), 85 percent of the
regular attendees in middle/high school grades attending 90 days or more
showed an improvement---more than any category of attendee based on
grade level.

Figure 49. Percentage of Regular Attendees Who Showed Improvement on
Academic Items by Attendance Gradation and Grade Level

![](media/image45.wmf){width="6.497222222222222in"
height="2.6375in"}*Note.* Based on 1,577 elementary schools (17 percent
of all centers in the APR) and 754 middle/high schools reporting data (8
percent of all centers in the APR)

The teacher survey items relating to participation show a similar
pattern. The percentage of students showing an improvement at the
middle/high school level increases as the days of attendance increase.
For students who have attended 90 days or more, the percentages of
middle/high school students who showed improvement on *attending class
more regularly* (ATT) and *being attentive in class* (BAC) were 75
percent and 80 percent, respectively, as shown in Figure 50.

Figure 50. Percentage of Regular Attendees Who Showed Improvement on
Participation Variables by Attendance Gradation and Grade Levels

![](media/image46.wmf){width="6.4944444444444445in"
height="2.5527777777777776in"}*Note.* Based on 1,577 elementary schools
(17 percent of all centers in the APR) and 754 middle/high schools
reporting data (8 percent of all centers in the APR)

As outlined in Figure 51, both elementary and middle/high school
students showed an increase in the percentage of students demonstrating
an improvement in *behaving in class* (BEH) and *getting along well with
others* (ALN), with increased levels of performance noted as attendance
levels increased. As Figure 51 shows, however, middle/high school
students showed a higher increase in both areas between the 60 to 89
days and 90 days or more gradations as compared to elementary students.
On the *behaving in class* variable, 73 percent of the middle/high
school students who attended 90 days or more showed an improvement, 8
percent more than students attending 60 to 89 days. Similarly, the
*getting along well with others* variable demonstrated a 10 percent
increase from 64 percent for the 60- to 89-day group to 74 percent for
students represented in the 90 plus days category.

Figure 51. Percentage of Regular Attendees Who Showed Improvement\
on Behavior Items by Attendance Gradation and Grade Level

![](media/image47.wmf){width="6.502083333333333in"
height="2.8958333333333335in"}

*Note.* Based on 1,577 elementary schools (17 percent of all centers in
the APR) and 754 middle/high schools reporting data (8 percent of all
centers in the APR)

## Changes in Student Behavior and Academic Performance by Grade Level

The 2006 APR data was disaggregated by grade level to explore whether
findings seemed to be different based on the grade level of the
participants. Analyses were conducted on the grades, state assessment,
and teacher survey indicators by disaggregating results into the
following grade level categories:

-   "Elementary Only," defined as those centers serving only elementary
    school students through Grade 6

-   "Elementary/Middle," defined as those centers serving students in
    elementary and middle school up to Grade 8

-   "Middle Only," defined as centers serving students in Grades 5 to 8

-   "Middle/High," defined as centers serving students in Grades 5 to 12

-   "High School Only" centers serving only high school students in
    Grades 9 to 12

A sixth "Other" category includes centers serving students across grades
or those centers that do not fit one of the other categories. The "High
School Only" category is especially important to analyze because
afterschool programming for older children often looks considerably
different from an elementary or middle school program. High school
students have different needs from younger students, and they often have
other afternoon obligations, such as jobs or extracurricular activities.
In addition, in some instances, the ETF made specific recommendations
regarding which grade levels should be included in a given indicator
calculation. To the extent possible, calculations were performed to
highlight the ramification of limiting indicator calculations to
specific grade-level bands.

### Grades 

### 

The grades indicator was disaggregated to capture the differences
between centers serving Grades 2 through 8 versus those serving Grades 9
through 12. In light of recommendations from members of the ETF and in
the interest of demonstrating the effects of adopting such
recommendations, data from centers serving students in first grade or
lower have been excluded for some of the following analyses, based on
the conclusion reached by the ETF that grades of such students are
typically not as diagnostic of academic achievement. Also, in light of
interest on the part of the ETF in exploring how the 21st CCLC indicator
system should be modified specifically for programs exclusively serving
high school students, revised grades indicator data are reported
separately for high school programs (i.e., Grades 9--12) because such
programs tend to have a different emphasis than programs serving the
lower grades.

Figure 52 displays the ramifications of redefining the population of
centers included in grades indicator calculations in this manner. The
first series, labeled "2 to 8 Only," reflects data from centers serving
students exclusively in Grades 2 through 8 (*N* = 2,000). The second
series, labeled "2 to 8 Overlap," reflects centers that serve any part
of the grade span of 2 to 8\
(*N* = 2,254). For example, centers serving students in Grades 6 to 9
would fall into this category. The third series of bars, labeled "High
School," reflects data from centers serving Grades 9 through 12 (*N* =
298).

Figure 52. Percentage of Regular Attendees Improving Grades by Grade
Level

![](media/image48.wmf){width="6.498611111111111in"
height="2.9520833333333334in"}

*Note.* Based on 2,000 "2 to 8 Only" centers (21 percent of all centers
in the APR); 2,254 "2 to 8 Overlap" centers (24 percent of all centers
in 2006); and 298 "High School" centers (3 percent of all centers in
2006)

The high school centers report slightly greater proportions of students
increasing their reading and mathematics grades (25 and 24 percent,
respectively) than those of centers exclusively serving Grades 2 to 8
(23 and 22 percent, respectively). Centers that served a population that
overlapped partially with Grades 2 through 8, however, had markedly
higher proportions of students increasing reading and mathematics grades
(31 and 28 percent, respectively). It is difficult to interpret the
higher percentages in the "overlap" category because this category by
definition includes several different types of school levels. For this
reason, it was necessary to revert to the approach of classifying
schools that were employed in the analyses of previous APR years.

Figure 53 displays the percentage of regular attendees increasing their
grade based on the six grade-level categories described earlier. The
overall findings indicate a U-shaped trend that is biased toward the
elementary level as one moves in ascending order through the various
grade- level categories through high school. Specifically, the
proportion of regular attendees increasing their grades decreases
steadily from the elementary category through the middle/high school
category. This trend abruptly reverses at the high school level, where
the percentage of mathematics grade improvement is equivalent to the
elementary/middle level, and the percentage of reading grade improvement
is nearly equivalent.

The reasons for this pattern are a matter of speculation. One
possibility is that the high school students were a more academically
motivated group, as they chose to participate in an academic activity
rather than a plethora of other nonacademic activities available to high
school students after school (e.g., work, athletics, or other student
clubs).

Figure 53. Percent of Regular Attendees Increasing Grade by Grade-Level
Breakdown

![](media/image49.wmf){width="6.501388888888889in"
height="2.8243055555555556in"}

*Note.* Based on 492 "Elementary Only" centers (5 percent of all centers
in 2006); 2,401 "Elementary/Middle" centers (26 percent of all centers
in 2006); 298 "Middle Only" centers (3 percent of all centers in 2006);
175 "Middle/High" centers (2 percent of all centers in 2006); 1,073
"High School" centers (11 percent of all centers in 2006); and 153
"Other" centers (2 percent of all centers in 2006).

### State Assessment 

The current method of analyzing state assessment data is predicated on
reporting on all grade levels served by programs, with separate figures
reported on programs that only serve elementary students and those that
only serve middle and/or high school students. A proposed revision to
this method made by the ETF members is to separately examine rates of
proficiency attainment and improvement for students in Grades 4 to 8.
With this approach, proficiency levels for students below Grade 4 and in
Grades 9 to 12 would not be reported. Proposing to limit the domain of
grade levels examined to Grades 4 to 8 seems appropriate given current
NCLB assessment requirements. Assessment data for students below Grade 4
are generally not focused on for NCLB requirements (though Grade 3
assessment results serve as the pretest for grade 4 GPRA analysis), and
although assessment scores for students in Grades 9 to 12 are of greater
consequence for students and schools, again it is believed that the
characteristics of high school programming for 21st CCLCs are different
enough from elementary and middle school programs to warrant isolating
this group of centers in this analysis.

Figure 54 demonstrates the result of only including data for centers
serving students in Grades 4 to 8. The first bar in each grouping,
labeled "4 to 8," represents the centers serving students exclusively in
the proposed span of Grades 4 to 8 (*N* = 760). The second bar in each
group, labeled "4 to 8 Overlap," represents the centers that serve
students in any part of the grade span of 4 to 8 (*N* = 2,391). For
example, centers serving students in Grades K to 5 would fall into this
category. We believe that these data more accurately reflect students'
state assessment scores and trends in Grades 4 to 8, as centers are not
excluded simply because they serve students in Grades K to 3, who are
unlikely to have prior state assessment data reported for them anyway.
The third bar in each grouping reflects data from centers serving high
school students in Grades 9 to 12\
(*N* = 145). The fourth bar in each grouping reflects data from centers
serving all grade levels\
(*N* = 3,296).

As demonstrated in Figure 54, including centers that also serve students
in any part of the Grades 4 to 8 span results in a slight drop in
percentage of students attaining proficiency in reading (17 percent
versus 16 percent) and an increase in percentage of students attaining
proficiency in mathematics (14 percent versus 19 percent). Centers
serving only high school students had the lowest percentage of
proficiency attainment for both reading and mathematics (15 percent and
11 percent, respectively).

[]{.mark}

Figure 54. Comparison of Centers Attaining Proficiency by Grade Level

![](media/image50.wmf){width="6.498611111111111in"
height="2.7222222222222223in"}

*Note*. Based on 760 "4 to 8" centers (8 percent of all centers in
2006); 2,391 "4 to 8 Overlap" centers (26 percent of all centers in
2006); 145 "High School" centers (2 percent of all centers in 2006); and
3,479 centers providing programming for all students across any grade
(37 percent of all centers in 2006)

When examining students in centers who witnessed an improvement in
proficiency level (and not just those who improved to *proficiency or
above*), the percentage of students in centers serving only Grades 4 to
8 who witnessed an improvement was lower than for students in all
grades, but centers serving any part of Grades 4 to 8 more closely match
the improvement levels for all grades. Again, those centers serving only
high school students had the lowest percentage of regular attendees
improving their proficiency level in reading and mathematics. Figure 55
illustrates these findings for both the current method of reporting
proficiency improvements, as well as the revised method that examines
the net change in improvement.

Generally, of the findings represented in Figure 55, it is believed that
the results associated with the "4 to 8 Overlap" grouping employing the
net calculation approach is best representative of what the impact would
be in adopting the ETF recommendation to limit state assessments
performance indicator calculations to those students enrolled in Grades
4 to 8, given that very few centers would have the ability to report
state assessment results for students enrolled in Grades K to 3.

Figure 55. Comparison of Centers Improving Proficiency Level by Grade
Level

![](media/image51.wmf){width="6.498611111111111in"
height="2.817361111111111in"}

*Note*. Based on 760 "4 to 8" centers (8 percent of all centers in
2006); 2,391 "4 to 8 Overlap" centers (26 percent of all centers in
2006); 145 "High School" centers (2 percent of all centers in 2006); and
3,479 centers providing programming for all students across any grade
(37 percent of all centers in 2006)

In addition, given a general interest in further exploring how
performance indicator results may vary by the grade level of students
served by a given center, additional analyses were conducted that
further considered state assessment results among centers serving
students at various grade levels. As mentioned previously, there were
five grade-level categories used in the grade breakdown analysis,
consisting of "Elementary Only," "Elementary/Middle," "Middle Only,"
"Middle/High," and "High Only," along with a sixth "Other category" for
centers that do not fit in one of the above categories.

As shown in Figure 56, centers exclusively serving high school students
witnessed the lowest rate of proficiency attainment (15 percent for
reading and 11 percent for mathematics), compared to centers serving
other grade-level categories. It is interesting to note the decline in
the percentage of students attaining proficiency in mathematics as
students reach the higher grade levels. This same decline is not
necessarily duplicated in reading; specifically, the gap between reading
and mathematics performance declines appears to emerge in middle school.

Figure 56. Percentage of Regular Attendees Attaining\
Proficiency by Grade-Level Breakdown

![](media/image52.wmf){width="6.502083333333333in"
height="3.004861111111111in"}

*Note*. Based on 2,249 "Elementary Only" centers (24 percent of all
centers in 2006); 282 "Elementary/Middle" centers (3 percent of all
centers in 2006); 616 "Middle Only" centers (7 percent of all centers in
2006); 89 "Middle/High" centers (1 percent of all centers in 2006); 145
"High Only" centers (2 percent of all centers in 2006); and 98 "Other"
centers (1 percent of all centers in 2006)

When using this same approach to examine students in centers who
witnessed an improvement in their proficiency level, a similar pattern
emerges for centers serving high school students only, especially when
looking at the net change in improvement. Figure 57 demonstrates that
centers serving only high school students actually had more students
decline in proficiency level than increase their proficiency in
mathematics, as illustrated by the -5 percent net improvement.

Figure 57. Percentage of Regular Attendees Improving\
Proficiency Level by Grade-Level Breakdown

![](media/image53.wmf){width="6.499305555555556in" height="2.76875in"}

*Note*. Based on 2,249 "Elementary Only" centers (24 percent of all
centers in 2006); 282 "Elementary/Middle" centers (3 percent of all
centers in 2006); 616 "Middle Only" centers (7 percent of all centers in
2006); 89 "Middle/High" centers (1 percent of all centers in 2006); 145
"High Only" centers (2 percent of all centers in 2006); and 98 "Other"
centers (1 percent of all centers in 2006).

### Teacher Survey

As we learned in the validity study mentioned earlier, the teacher
survey performed best for the high school level, fairly well for
elementary, and only mediocre at the middle school level. Since students
in different grade levels have various and specific developmental
outcomes associated with their age range, the development of an
additional teacher survey to address the needs of various grade levels
may be warranted. The following section begins that exploration by first
analyzing the results, in the same manner from previous years, through
elementary school and middle/high school. Further analyses are conducted
on the teacher survey items based on the five different grade-level
categories described earlier.

Figure 58 shows the difference between elementary centers and
middle/high school centers on the academic items. The "Elementary Only"
centers tended to have slightly higher percentages of students
demonstrating an improvement as reported by teachers across all
variables. The highest rate of improvement came from students who
demonstrated an increase in a*cademic performance* (78 percent at the
elementary centers and 73 percent at the middle/high schools).

Figure 58. Percentage of Regular Attendees Who Showed\
Improvement on Academic Items by Grade Level

![](media/image54.wmf){width="6.497222222222222in"
height="2.9763888888888888in"}

*Note.* Based on 2,412 "Elementary" centers (26 percent of all centers
in 2006), 1,577 "Middle/High School" centers (17 percent of all centers
in 2006), and 400 "Other" centers (4 percent of all centers in 2006).

On the participation items in Figure 59, the middle/high school centers
showed slightly higher percentages of students improving on *attending
class more regularly (ATT)*. Given the particular struggles high school
programs often have with levels of attendance, this finding indicates
that high school students who attend 21st CCLC programs may also attend
school more often, especially in light of other afterschool activities
that often draw high school students.

Figure 59. Percentage of Regular Attendees Who Showed\
Improvement on Participation Items by Grade Level

![](media/image55.wmf){width="6.497222222222222in"
height="2.763888888888889in"}

*Note.* Based on 2,412 "Elementary" centers (26 percent of all centers
in 2006), 1,577 "Middle/High School" centers (17 percent of all centers
in 2006), and 400 "Other" centers (4 percent of all centers in 2006).

Figure 60 shows that teachers reported students at the elementary and
middle/high school levels had almost equal percentages of students
showing an increase on the *behaving in class* (BEH) and *getting along
well with others* (ALN) variables.

Figure 60. Percentage of Regular Attendees Who Showed Improvement on\
Behavior Items by Grade Level

![](media/image56.wmf){width="6.497222222222222in"
height="2.9763888888888888in"}

*Note.* Based on 2,412 "Elementary" centers (26 percent of all centers
in 2006), 1,577 "Middle/High" centers (17 percent of all centers in
2006), and 400 "Other" centers (4 percent of all centers in 2006).

In order to probe the data further to determine how the teacher survey
worked at more discrete grade levels, particularly high school, grade
levels were further disaggregated into six categories: (1) Elementary
Only, (2) Elementary/Middle, (3) Middle Only, (4) Middle/High, (5) High
School Only, and (6) Other.

Figure 61 shows the percentage of students who improved on the teacher
survey's academic items disaggregated by the new grade levels. The
elementary and elementary/middle centers had a higher percentage across
all items. The middle only and high only centers tended to have the
lowest percentage on all items. All grade levels showed higher
percentages of improvement on the academic performance variable with
elementary only and elementary/middle centers exceeding or meeting the
75 percent targeted threshold. *Getting homework in on time* and
*completing homework to the teacher's satisfaction* had the highest
increases at the "Elementary Only" level, although all grade levels
except "High School Only" reached 70 percent or higher.

Figure 61. Percentage of Regular Attendees Who Showed Improvement on\
Academic Items by New Grade Level

![](media/image57.wmf){width="6.499305555555556in"
height="3.203472222222222in"}

*Note.* Based on 2,415 "Elementary Only" centers (26 percent of all
centers in 2006); 683 "Elementary/Middle" centers (7 percent of all
centers in 2006); 999 "Middle Only" centers (11 percent of all centers
in 2006); 240 "Middle/High" centers (3 percent of all centers in 2006);
338 "High Only" centers (4 percent of all centers in 2006); and 400
"Other" centers (4 percent of all centers in 2006).

The variables concerning participation demonstrated a different pattern.
Elementary only and elementary/middle had the highest percentage on
*participating in class* (74 percent), but elementary only had the
lowest percentage of students showing improvement on the remaining
variables, as shown in Figure 62. "Elementary/Middle," "Middle/High,"
and "High Only" centers demonstrated the highest percentage of students
showing improvement on the other variables. "Middle/High" and "High
Only" centers, in particular, had the highest percentage of students
*attending class more regularly*.

Figure 62. Percentage of Regular Attendees Who Showed\
Improvement on Participation Items by New Grade Level

![](media/image58.wmf){width="6.497222222222222in"
height="2.763888888888889in"}

*Note.* Based on 2,415 "Elementary Only" centers (26 percent of all
centers in 2006); 683 "Elementary/Middle" centers (7 percent of all
centers in 2006); 999 "Middle Only" centers (11 percent of all centers
in 2006); 240 "Middle/High" centers (3 percent of all centers in 2006);
338 "High Only" centers (4 percent of all centers in 2006); and 400
"Other" centers (4 percent of all centers in 2006).

There is little difference among grade levels on the behavioral items,
as seen in Figure 63. The Elementary/Middle and Middle/High centers tend
to be slightly higher.

Figure 63. Percentage of Regular Attendees Who Showed\
Improvement on Behavior Items by New Grade Level

![](media/image59.wmf){width="6.497222222222222in"
height="2.9763888888888888in"}

*Note.* Based on 2,415 "Elementary Only" centers (26 percent of all
centers in 2006); 683 "Elementary/Middle" centers (7 percent of all
centers in 2006); 999 "Middle Only" centers (11 percent of all centers
in 2006); 240 "Middle/High" centers (3 percent of all centers in 2006);
338 "High Only" centers (4 percent of all centers in 2006); and 400
"Other" centers (4 percent of all centers in 2006).

Overall, by performing the teacher survey item analysis separately for
centers falling either in the "Elementary Only" or "Middle/High"
categories, the elementary students show slightly higher increases
across five of the items (*completing homework on time*, *completing
homework to the teacher's satisfaction*, *academic performance*,
*motivated to learn*, and *participating in class*). The only items in
which the middle/high school centers had a higher percentage of students
showing an increase were *attending class more regularly* and *being
attentive in class*.

The further breakdown of grades into "Elementary Only,"
"Elementary/Middle," "Middle Only," "Middle/High," and "High Only"
provides a slightly different summary of the variables. On the academic
items, the "Elementary Only" and "Elementary/Middle" centers tended to
have a higher percentage of students showing improvement, whereas the
"Middle Only" and "High Only" centers had the lowest percentage of
students showing an improvement. Interestingly, the "High Only" centers
had a higher percentage than the "Middle Only" group on *motivated to
learn*. The participation variables tell a similar story. With these
items, the "Elementary/Middle," "Middle Only," and "High School Only"
centers tend to have the highest number of participants showing
improvement.

These findings indicate that in order to further discriminate the
differences among grade levels and ask more appropriate developmental
questions, a separate survey should be developed for high school
students. Further research needs to be conducted to determine which
items and constructs are most appropriate for various grade levels.

## Next Steps

### Grades

The revised approach to calculating the indicator related to grades
improvement addresses some of the limitations associated with this
indicator, including a failure to remove students from the denominator
that could not improve further given that they had achieved the highest
grade possible during the course of the first fall marking period. This
modification to the manner in which the indicator is calculated,
however, does not address the concern that grades may naturally have a
downward trajectory during the course of a given school year, given the
increasing difficulty of the academic content being addressed in the
classroom. As such, it may be appropriate to consider developing a new
indicator that measures the extent to which students demonstrating an
acceptable level of grades performance at baseline remain at that level
when post-data are examined. This would reflect that center's role in
helping students remain at an acceptable level of functioning during the
course of the school year in question.

### State Assessment

The changes to the manner in which the state assessment-related
indicator is calculated yield results that are better aligned to what
21st CCLCs can realistically achieve in terms of impacting state
assessment results; however, the movement toward this approach still
offers limited insight into the relation between program attendance,
baseline academic functioning, changes in academic behaviors, and the
degree of improvement in state assessment results by program
participants. If understanding more fully the nature of such relations
is an important consideration with respect to the utility and relevance
of PPICS-collected data, then other data collection options warrant
exploration, including the collection of student-level attendance,
grades, and related achievement data.

### Teacher Survey

The current teacher survey has several psychometric limitations. Data
are generated but not collected at the individual student level, so it
is not possible to employ the full domain of scoring and scaling
techniques that would yield more psychometrically valid results and help
determine what performance threshold would be most reasonable for
students at a given grade level. To date, the performance threshold has
been set at 75 percent of the regular attendees demonstrating
improvement on each survey item; however, these results demonstrate that
the only item for which students have met the threshold is academic
performance.

Much research has been conducted on the impacts of afterschool programs
since the initial implementation of the teacher survey. Revising the
domain of constructs that can be accurately measured and assessed
through a teacher survey may deserve future investigation. Taking into
consideration that the impact of 21st CCLC programs will be different
for different grade levels, determining appropriate indicators for
various grade levels may be the next step in advancing how we report on
teacher perceptions of students making gains in 21st CCLC programs.

Consideration should also be given to moving the teacher survey to a
pre- and postadministration cycle. The burden of doubling the number of
times the survey is administered during the school year could be
partially alleviated by asking centers to collect survey data on only a
random sample of their attendee population.

**Summary of Key Findings and Next Steps Recommended by Learning Point
Associates**

As noted in the introduction to Section 2, there has been a significant
amount of recent discussion to identify limitations associated with the
current domain of 21st CCLC performance indicators. These limitations
have proven to constrain both federal and state efforts to assess the
full impact of 21st CCLC programs and support decision making regarding
the delivery of training and technical assistance relative to programs
that are performing below expectations. On the positive side, these
discussions have also led to the proposal of alternative methods for
collecting and analyzing indicator data that can increase their validity
and overall utility. To build on recent efforts to re-assess the
viability and appropriateness of the current domain of performance
indicators for the 21st CCLC program, the results highlighted in this
report represent an attempt to utilize the existing data housed in PPICS
to assess what the ramifications would be of adopting a series of
recommendations made by U.S. Department of Education staff staff, state
21st CCLC coordinators, and representatives of the afterschool research
and evaluation community. The goal of all these analyses is to support a
performance indicator reporting process that is as meaningful, valid,
and useful as possible.

Based on the reanalysis of PPICS data consistent with the
recommendations offered by the ETF, some of the more interesting and
meaningful findings are presented below:

-   Revising the denominator associated with the improvement in grades
    indicator by excluding regular attendees that had already achieved
    the highest grade possible at the end of the first fall marking
    period had only a small impact on results associated with the net
    change approach to calculating the indicator. Although this result
    suggests that on the whole, 21st CCLC programs are not serving large
    percentages of straight-A students, it does not reveal the extent to
    which programs are serving students that generally are performing at
    an acceptable level of functioning or the role programs are playing
    in helping students remain in good academic standing during the
    course of the school year. Members of the ETF have suggested that
    consideration should be given to adding an indicator to the
    performance measurement system that captures the role programs play
    in helping students maintain good grades.

-   Once students reach the 60-day attendance threshold, there appears
    to be a noticeable increase in the percentage of students improving
    their grades. A similar finding was witnessed in relation to student
    gains on state assessments, although in this case the jump in the
    percentage of regular attendees witnessing an increase occurs at the
    90-day or more attendance threshold. Such results seem to raise a
    series of questions around both the appropriateness of varying the
    definition of a regular attendee, depending upon the type of outcome
    being evaluated, and the relation between higher levels of
    attendance and the achievement of certain types of youth outcomes.

-   Centers that exclusively served high school students demonstrate by
    far the lowest net percentage of regular attendees moving to a
    higher proficiency category between state assessments. Such results,
    however, may be more reflective of the fact that many states do not
    annually test high school students, resulting in a fairly small
    sample of students upon which these findings are based. In light of
    this, additional discussion seems warranted on excluding high school
    students from indicators predicated on an analysis of state
    assessment results.

-   In contrast to state assessment results, centers that exclusively
    served high school students demonstrated higher levels of
    performance on the grades indicator as compared to centers that only
    served middle school students or some combination of middle and high
    school students. This was the case even though centers exclusively
    serving high school students demonstrated a lower rate of regular
    attendance as compared to centers serving youth enrolled in middle
    school. Such results offer some tantalizing areas for further
    exploration, especially in terms of the afterschool programming that
    may be most appropriate at the high school level.

-   There appears to be a noticeable linear increase in the percentage
    of regular attendees witnessing an improvement in teacher-reported
    behaviors across each of the 10 constructs under consideration when
    examining results by attendance gradation. In this regard, although
    there are a number of concerns regarding the meaningfulness of the
    overall percentage of regular attendees with teacher-reported
    improvements in academic-related behaviors, the linear increase in
    improvement across ascending levels of program attendance seems to
    warrant attention and adds weight to the benefits that could be
    obtained from a program assessment standpoint if student-level
    teacher survey and attendance data could be obtained. This would
    allow for the further examination of the relationship between levels
    of program participation and teacher-reported changes in student
    behavior.

-   In terms of teacher-reported improvements in homework completion and
    homework quality, participating in class, and overall academic
    performance, high school students generally demonstrate the lowest
    levels of improvement as compared to their peers in other grade
    levels. High school students, however, typically witnessed higher
    levels of improvement in terms of attending class regularly than
    students in other grade levels. Such results may suggest which
    elements warrant attention when developing different teacher surveys
    by grade level.

It is clear that many of the recommendations made in the past few months
on how the performance indicators could be revised will require
additional study and discussion to make them more concrete and
operationally viable. In addition, in thinking about how these
recommendations could be incorporated into a revised set of indicators
for the 21st CCLC program and the federal reporting requirements housed
in PPICS, discussions held to date on this topic have increasingly
focused on the utility of collecting individual student-level attendance
and impact category data in PPICS. Generally, collecting data at the
individual level has the following benefits:

-   Alleviates the need for grantees to perform complex data aggregation
    tasks that can be easily misunderstood or subject to errors in
    compilation.

-   Allows for greater precision in exploring the relationship among
    program attendance, grade level, and student behavioral and academic
    outcomes.

-   Allows for the application of a broader array of psychometric
    techniques when analyzing teacher survey data.

Given the benefits that can be gained by collecting student-level
attendance and impact category data in PPICS, there have been
preliminary discussions around the possibility of sponsoring a pilot
during the course of the 2007--08 reporting period that would afford
states and grantees the option of reporting student-level attendance
data in PPICS. Data resulting from this pilot should further inform what
constitutes reasonable measures of student achievement and academic
behavioral change as they relate to the provision of the afterschool
activities and services provided as part of the 21st CCLC program and
how these metrics should be differentiated to account for various types
of programs.

Finally, revisions to the performance indicators should also be viewed
within the context of supporting program improvement efforts, both at
the state and grantee levels. This may mean that consideration should be
given to adopting indicators that speak to the extent to which states
and programs are making strides in developing and participating in
quality assessment systems that lead to concrete efforts to improve
program quality, especially at the point of service delivery.

**References**

Durlak, J. A., & Weissberg, R. P. (2007). *The impact of after-school
programs that promote personal and social skills*. Chicago:
Collaborative for Academic, Social, and Emotional Learning. Retrieved
November 28, 2007, http://www.casel.org/downloads/ASP-Full.pdf

Eccles, J., & Gootman, J. A. (2002). Features of positive developmental
settings.

> In J. Eccles & J. A. Gootman (Eds.), *Community programs to promote
> youth* *development* (pp. 86--118). Washington, DC: National Academy
> Press.

Granger, R., Durlak, J. A., Yohalem, N., & Reisner, E. (April, 2007).
*Improving after-*

*school program quality.* New York: William T. Grant Foundation.

James-Burdumy, S., Dynarski, M., Moore, M., Deke, J., Mansfield, W., &
Pistorino, C. (2004). *When schools stay open late: The national
evaluation of the 21st Century Community Learning Centers program: Final
report*. Washington, DC: U.S. Department of Education, National Center
for Education Evaluation and Regional Assistance.

Jolly, E., Campbell, P., & Perlman, L. (2004). *Engagement, capacity and
continuity: A trilogy*

> *for student success.* Groton, MA: Campbell-Kibler Associates.

Little, P. (2007). *The quality of school-age care in afterschool
settings.* New York, NY and Ann

Arbor, MI: National Center for Children in Poverty, Columbia University
and Institute

for Social Research, University of Michigan.

Pechman, E., & Fiester, L. (2002). *Leadership, program quality, and
sustainability*. Washington,

> DC: Policy Studies Associates. Retrieved November 28, 2007, from
> http://www.policystudies.com/studies/youth/FINAL%20Issue%20Brief%20Nov2002

WEB.pdf

Rosenthal, R., & Vandell, D. L. (1996). Quality of school-aged child
care programs: Regulatable

> features, observed experiences, child perspectives, and parent
> perspectives. *Child* *Development, 67,* 2434--2445.

Southwest Educational Development Laboratory. (2007). *Afterschool
training toolkit: Building quality enrichment activities* \[Website\].
Retrieved November 28, 2007, from
http://www.sedl.org/afterschool/toolkits/

Vandell, D. L., Reisner, E. R., Brown, B. B., Pierce, K. M., Dadisman,
K., & Pechman, E. M.

(2004). *The study of promising after-school programs: Descriptive
report of*

> *the promising programs*. Washington, DC: Policy Studies Associates.
> Retrieved November 28, 2007, from
> http://www.wcer.wisc.edu/childcare/pdf/pp/study_of_after_school_activities_descriptive_report_year1.pdf

Wilson-Ahlstrom, A., & Yohalem, N. (with Pittman, K.). (2007). *Building
quality*

*improvement systems: Lessons from three emerging efforts in the
youth-serving*

> *sector*. Washington, DC: The Forum for Youth Investment, Impact
> Strategies Inc.

Yohalem, N., Wilson-Ahlstrom, A. (with Fischer, S., & Shinn, M.).
(2007). *Measuring youth program quality: A guide to assessment tools*.
Washington,

DC: The Forum for Youth Investment, Impact Strategies Inc.

# 

# 

# 

# 

# 

# 

# 

# 

# 

# 

# 

# 

# 

# 

# 

# 

# 

# 

# 

# 

# 

# 

# 

# 

# 

# 

# 

# 

# 

# 

# 

# 

# Appendix A

**State Tables**

**Table A1. Competition Overview (2006 Competitions)**

  ---------------------------------------------------------------------------------------------------------------------------------
  **State**   **Compe-tition   **Applicants**   **Awards**   **% of       **\$ Requested**       **\$ Awarded**         **% of \$
              Records**                                      Applicants                                                 Requested
                                                             Funded**                                                   Awarded**
  ----------- ---------------- ---------------- ------------ ------------ ---------------------- ---------------------- -----------
  AK                                                                                                                    

  AL          1                53               26           49%          \$7,500,000.00         \$3,961,779.00         53%

  AR                                                                                                                    

  AZ          1                33               17           52%          \$5,896,280.50         \$3,228,307.50         55%

  BIA                                                                                                                   

  CA                                                                                                                    

  CO                                                                                                                    

  CT          1                27               6            22%          \$5,400,000.00         \$971,892.00           18%

  DC          1                14               3            21%          \$3,710,722.20         \$800,000.00           22%

  DE                                                                                                                    

  FL          1                80               14           18%          \$30,804,837.00        \$4,732,620.00         15%

  GA          1                53               30           57%          \$57,393,190.00        \$10,650,759.00        19%

  HI          1                6                2            33%          \$2,686,899.00         \$1,004,500.00         37%

  IA                                                                                                                    

  ID                                                                                                                    

  IL          1                73               16           22%          \$26,335,998.00        \$6,100,000.00         23%

  IN                                                                                                                    

  KS          1                11               6            55%          \$1,085,686.00         \$521,754.00           48%

  KY          1                78               4            5%           \$11,302,294.41        \$1,923,194.00         17%

  LA          1                38               19           50%          \$38,600,785.00        \$20,597,819.00        53%

  MA                                                                                                                    

  MD          1                36               16           44%          \$9,962,236.00         \$4,328,786.00         43%

  ME                                                                                                                    

  MI                                                                                                                    

  MN                                                                                                                    

  MO                                                                                                                    

  MS                                                                                                                    

  MT                                                                                                                    

  NC          1                40               7            18%          \$8,550,368.00         \$1,375,462.00         16%

  ND                                                                                                                    

  NE          1                9                8            89%          \$2,635,007.00         \$2,106,080.00         80%

  NH          1                10               5            50%          \$1,303,751.00         \$625,000.00           48%

  NJ                                                                                                                    

  NM                                                                                                                    

  NV                                                                                                                    

  NY                                                                                                                    

  OH          1                104              6            6%           \$28,182,413.27        \$1,750,642.00         6%

  OK          1                39               9            23%          \$9,379,224.06         \$1,372,000.00         15%

  OR          2                54               17           31%          \$14,773,300.00        \$5,317,238.00         36%

  PA          1                94               12           13%          \$44,945,982.00        \$3,790,976.00         8%

  PR                                                                                                                    

  RI                                                                                                                    

  SC          1                94               17           18%          \$18,361,670.00        \$2,740,052.00         15%

  SD          1                20               12           60%          \$1,817,906.79         \$1,027,497.00         57%

  TN                                                                                                                    

  TX          1                166              23           14%          \$73,781,698.00        \$14,036,893.00        19%

  UT          1                9                4            44%          \$478,986.00           \$425,130.00           89%

  VA          2                106              46           43%          \$16,580,142.00        \$7,584,119.00         46%

  VT                                                                                                                    

  WA                                                                                                                    

  WI                                                                                                                    

  WV                                                                                                                    

  WY                                                                                                                    

  **Total**   **26**           **1,247**        **325**      **26%**      **\$421,469,376.23**   **\$100,972,499.50**   **24%**
  ---------------------------------------------------------------------------------------------------------------------------------

**Table A1a. Competition Overview (January 2004--December 2006)**

  -----------------------------------------------------------------------------------------------------------------------------------
  **State**   **Compe-tition   **Applicants**   **Awards**   **% of       **\$ Requested**         **\$ Awarded**         **% of \$
              Records**                                      Applicants                                                   Requested
                                                             Funded**                                                     Awarded**
  ----------- ---------------- ---------------- ------------ ------------ ------------------------ ---------------------- -----------
  AK          1                22               6            27%          \$7,701,361.00           \$2,027,328.00         26%

  AL          4                211              96           45%          \$25,438,400.00          \$13,177,249.00        52%

  AR          1                63               27           43%          \$9,162,202.00           \$3,785,618.00         41%

  AZ          3                106              57           54%          \$26,137,098.40          \$14,182,795.12        54%

  BIA                                                                                                                     

  CA          4                409              191          47%          \$178,172,847.00         \$91,176,463.00        51%

  CO          2                59               34           58%          \$15,056,936.11          \$7,610,831.00         51%

  CT          2                65               16           25%          \$17,096,861.00          \$3,264,490.00         19%

  DC          4                107              24           22%          \$25,562,656.70          \$5,540,280.80         22%

  DE          1                12               10           83%          \$2,511,448.00           \$1,907,000.00         76%

  FL          2                154              49           32%          \$85,762,310.00          \$29,126,038.00        34%

  GA          2                148              44           30%          \$101,006,856.00         \$18,933,131.00        19%

  HI          2                16               6            38%          \$7,390,289.00           \$3,048,658.00         41%

  IA          1                25               5            20%          \$7,056,913.00           \$1,533,038.00         22%

  ID          2                32               24           75%          \$4,288,300.00           \$3,075,939.00         72%

  IL          2                132              61           46%          \$51,435,998.00          \$22,900,000.00        45%

  IN          1                47               18           38%          \$17,745,481.63          \$5,015,594.72         28%

  KS          2                55               15           27%          \$12,417,219.00          \$3,044,103.00         25%

  KY          2                158              41           26%          \$23,350,938.41          \$7,252,962.00         31%

  LA          2                81               37           46%          \$66,988,167.72          \$28,863,711.00        43%

  MA          1                40               28           70%          \$11,965,976.00          \$8,000,000.00         67%

  MD          3                119              39           33%          \$37,191,496.00          \$11,585,135.00        31%

  ME          1                29               19           66%          \$3,639,499.54           \$2,718,416.00         75%

  MI          1                73               15           21%          \$71,380,394.00          \$30,159,081.00        42%

  MN          1                57               19           33%          \$13,621,180.00          \$5,129,836.00         38%

  MO          2                99               34           34%          \$25,552,702.00          \$8,565,092.00         34%

  MS          1                83               34           41%          \$6,133,683.00           \$5,520,314.39         90%

  MT          1                33               16           48%          \$3,462,666.00           \$1,855,000.00         54%

  NC          3                147              69           47%          \$41,542,769.58          \$19,706,568.42        47%

  ND          2                17               16           94%          \$6,035,651.00           \$4,115,000.00         68%

  NE          3                29               22           76%          \$6,854,054.00           \$4,596,567.00         67%

  NH          2                22               10           45%          \$4,773,001.00           \$2,483,716.00         52%

  NJ          2                50               33           66%          \$6,433,016.00           \$6,381,488.00         99%

  NM          1                36               10           28%          \$12,474,295.40          \$2,929,498.00         23%

  NV          2                30               28           93%          \$3,620,943.00           \$2,856,917.00         79%

  NY          2                540              177          33%          \$160,382,018.00         \$61,467,789.00        38%

  OH          3                386              61           16%          \$99,806,706.25          \$16,670,416.65        17%

  OK          3                175              31           18%          \$43,216,133.06          \$6,438,186.00         15%

  OR          4                117              29           25%          \$30,677,407.16          \$8,315,642.27         27%

  PA          3                336              90           27%          \$156,604,355.00         \$28,466,977.00        18%

  PR          2                179              114          64%          \$119,679,996.37         \$41,675,887.00        35%

  RI          2                35               15           43%          \$7,694,387.00           \$3,800,313.00         49%

  SC          4                351              72           21%          \$58,487,333.00          \$11,298,933.00        19%

  SD          3                66               44           67%          \$5,949,364.79           \$4,088,811.00         69%

  TN          1                89               51           57%          \$27,039,385.29          \$7,615,000.00         28%

  TX          3                620              112          18%          \$323,642,571.00         \$74,782,134.00        23%

  UT          3                36               13           36%          \$47,122,041.00          \$2,904,343.00         6%

  VA          5                193              96           50%          \$31,268,177.87          \$15,467,274.00        49%

  VT          2                23               18           78%          \$3,979,122.00           \$2,524,505.00         63%

  WA          1                51               12           24%          \$20,364,729.00          \$4,927,004.00         24%

  WI          2                106              20           19%          \$20,488,474.00          \$3,000,000.00         15%

  WV          1                32               12           38%          \$7,925,074.00           \$2,916,347.00         37%

  WY          2                49               38           78%          \$4,902,526.00           \$3,202,009.00         65%

  **Total**   **112**          **6,150**        **2,158**    **35%**      **\$2,108,193,410.28**   **\$681,629,429.37**   **32%**
  -----------------------------------------------------------------------------------------------------------------------------------

**Table A2. Grantee Profile Basic Information\
(for Grants Awarded Through December 2006)**

  ----------------------------------------------------------------------------------------------------------------
  **State**   **Grantees**   **Centers**   **Feeder     **Partners**   **Anticipated   **Anticipated   **Average
                                           Schools**                   Students**      Adults**        Hours per
                                                                                                       Week, SY**
  ----------- -------------- ------------- ------------ -------------- --------------- --------------- -----------
  AK          15             54            73           171            5,627           3,194           11.7

  AL          140            204           345          1,094          20,354          12,790          14.6

  AR          82             82            82           320            12,745          5,700           20.6

  AZ          73             154           189          807            31,143          9,584           13.1

  BI          31             52            93           250            8,373           3,483           14.6

  CA          161            861           1,427        1,555          116,324         33,287          17.4

  CO          53             100           123          414            19,161          7,502           16.1

  CT          34             82            422          316            14,465          3,503           16.4

  DC          28             49            134          128            4,724           1,563           15.2

  DE          21             46            89           72             4,014           1,507           22.2

  FL          92             299           1,113        995            42,111          16,613          17.1

  GA          77             267           567          1,158          25,997          16,440          15.7

  HI          9              54            82           139            10,736          1,768           10.7

  IA          16             40            92           265            3,378           2,153           16.6

  ID          33             68            141          415            4,445           1,425           8.2

  IL          112            363           651          760            44,658          18,310          14.5

  IN          45             146           294          392            17,882          4,758           14.1

  KS          34             80            103          398            10,951          4,001           11.6

  KY          100            153           359          803            24,294          9,219           17.9

  LA          32             84            244          335            10,569          3,464           12.2

  MA          39             193           219          233            12,306          1,517           13.2

  MD          37             132           404          289            10,045          4,226           9.1

  ME          37             121           149          122            9,320           1,866           12.1

  MI          52             187           281          658            19,989          4,854           13.6

  MN          38             120           294          411            18,776          4,547           16.4

  MO          54             129           289          408            15,564          12,045          17.4

  MS          63             164           303          240            15,968          4,620           10.5

  MT          38             78            198          523            9,975           1,137           7.3

  NC          99             306           639          807            23,271          14,078          14.6

  ND          14             93            138          293            8,289           2,807           15.8

  NE          29             79            118          315            5,534           2,670           16.1

  NH          20             48            64           199            8,239           2,206           14.0

  NJ          46             124           278          380            13,426          6,425           18.1

  NM          29             91            128          228            10,081          3,873           12.0

  NV          49             57            333          353            6,941           2,790           18.9

  NY          243            686           1,721        1,551          110,950         38,330          16.0

  OH          108            273           599          927            32,711          13,884          14.5

  OK          63             103           141          327            13,603          6,426           14.4

  OR          41             108           141          280            13,610          6,738           14.7

  PA          107            387           693          1,083          34,795          15,322          14.5

  PR          132            952           1,291        751            107,780         20,468          9.7

  RI          21             37            70           253            7,078           2,365           21.1

  SC          92             179           240          571            14,220          5,521           13.8

  SD          53             107           307          309            16,534          4,830           14.5

  TN          79             248           464          755            25,009          10,419          13.0

  TX          145            573           890          2,779          116,380         47,732          17.9

  UT          22             66            100          273            15,121          6,842           22.1

  VA          149            242           338          947            28,079          13,912          12.6

  VT          39             107           167          313            10,700          979             10.9

  WA          29             159           209          339            17,121          12,799          11.4

  WI          61             121           466          756            23,496          8,637           15.6

  WV          32             160           358          429            14,786          7,379           10.1

  WY          61             156           680          384            17,852          3,123           17.7

  **Total**   **3,309**      **9,824**     **19,333**   **29,273**     **1,239,500**   **455,631**     **15**
  ----------------------------------------------------------------------------------------------------------------

**Table A3. APR Basic Information**

  -------------------------------------------------------------------------------------------------------------------------------------------------------------
  **State**   **Grantees**   **Centers**   **Feeder     **Total           **Regular       **Total Adult    **School Year **School     **Summer     **Summer
                                           Schools**    Students**        Attendees**     Participants**   Paid Staff**  Year         Paid Staff** Volunteer
                                                                                                                         Volunteer                 Staff**
                                                                                                                         Staff**                   
  ----------- -------------- ------------- ------------ ----------------- --------------- ---------------- ------------- ------------ ------------ ------------
  AK          15             54            65           5,574             3,551           855              862           228          158          12

  AL          92             120           183          14,394            9,591           6,198            1,383         805          1,314        668

  AR          82             82            82           12,058            5,815           3                1,040         529          284          42

  AZ          58             129           137          40,654            16,563          9,008            2,802         512          1,614        187

  BI          31             50            82           8,641             4,354           2,880            873           149          725          96

  CA          251            1,303         1,508        290,608           177,558         9,650            15,748        6,448        3,272        944

  CO          53             106           120          18,757            8,391           3,555            2,337         651          761          189

  CT          34             81            184          14,825            7,240           605              994           72           338          7

  DC          25             48            105          4,929             3,486           679              539           332          300          100

  DE          21             47            86           3,786             2,829           933              452           197          355          208

  FL          78             300           1,114        64,982            40,241          5,200            5,389         287          2,156        54

  GA          47             198           356          28,761            13,558          4,404            2,731         139          1,293        42

  HI          9              54            75           9,181             2,603           1,195            725           119          347          39

  IA          16             39            82           4,543             2,962           2,417            841           494          352          27

  ID          32             66            109          6,159             3,962           109              521           584          147          82

  IL          96             316           494          40,387            24,851          7,560            4,487         1,321        2,002        512

  IN          45             148           289          22,043            8,548           4,606            1,068         55           420          4

  KS          28             70            92           12,107            6,368           5,258            1,093         459          658          71

  KY          91             141           320          25,466            8,559           3,883            2,180         322          708          133

  LA          21             59            170          13,249            7,220           1,129            1,034         117          826          83

  MA          39             177           190          22,721            18,294          0                3,157         722          2,023        200

  MD          35             128           275          11,167            6,425           2,652            1,569         590          676          329

  ME          37             110           127          10,129            2,920           678              1,043         648          709          298

  MI          52             188           241          25,912            10,993          281              1,709         217          917          60

  MN          37             114           246          20,409            11,886          4,176            1,667         1,144        869          545

  MO          53             118           224          18,076            9,119           2,684            2,016         1,054        436          141

  MS          51             161           260          17,806            9,660           1,867            1,726         841          1,157        505

  MT          37             75            145          10,122            4,528           20               608           821          341          243

  NC          92             288           505          22,639            14,343          10,492           2,702         1,516        626          186

  ND          14             92            128          8,478             5,388           1,274            741           480          278          307

  NE          21             63            83           6,073             4,002           1,134            687           496          361          135

  NH          17             44            52           6,844             3,316           1,754            819           503          339          52

  NJ          47             127           232          13,455            12,951          948              1,536         0            298          1

  NM          29             91            128          11,386            5,661           1,948            1,270         221          791          157

  NV          49             54            222          11,507            5,228           622              887           91           53           16

  NY          229            663           1,487        129,278           62,316          15,800           12,385        2,524        2,873        483

  OH          104            252           499          28,830            16,641          4,688            3,388         2,079        1,596        601

  OK          54             88            117          13,556            7,712           2,045            1,604         454          1,067        201

  OR          24             72            95           10,357            6,279           3,337            965           426          373          191

  PA          86             293           527          30,065            15,941          5,529            4,863         1,598        1,879        426

  PR          75             860           762          53,900            49,190          2,935            4,269         2,271        733          779

  RI          21             38            59           6,969             3,045           1,953            764           291          443          73

  SC          90             192           233          16,647            13,291          3,390            2,408         1,331        636          266

  SD          46             91            261          14,580            4,951           2,515            759           231          493          192

  TN          79             240           409          31,778            17,976          5,114            3,129         1,513        1,838        931

  TX          122            498           751          129,086           68,915          20,609           12,854        2,779        5,893        876

  UT          19             57            80           14,862            6,490           7,603            1,047         786          402          167

  VA          89             146           185          22,105            13,444          4,546            2,269         895          1,010        349

  VT          36             101           134          11,539            3,518           314              1,403         345          381          71

  WA          29             158           204          23,210            9,708           8,841            1,569         1,023        574          250

  WI          61             122           392          34,651            15,662          9,437            2,104         862          847          154

  WV          29             142           274          15,420            5,298           3,027            1,369         952          477          490

  WY          46             101           385          11,786            3,850           763              702           308          406          205

  **Total**   **2,974**      **9,355**     **15,565**   **1,456,447\***   **807,191\***   **203,103\***    **127,087**   **43,832**   **49,825**   **13,380**
  -------------------------------------------------------------------------------------------------------------------------------------------------------------

\*Note: Attendance totals represent the sum of unduplicated student and
regular attendees reported by individual centers; however, no mechanism
exists for accounting for students who may have attended more than one
center during the reporting period. In this regard, student and regular
attendee totals are not meant to show the unduplicated number of
students served by an individual state or the nation as a whole.

# Appendix B

**Methods**

**Grade-Level Assignment**

When assigning grade-level categories (e.g., elementary, middle school,
high school), we first specify what grade-level classification a given
center should receive based on the number of students from each grade
level who attended the center during the reporting period. So, for
example, a center would receive an *Elementary Only* classification if
the minimum grade level of students served by the center was equal to or
less fifth grade and the maximum grade level served by the center was
less than or equal to sixth grade. However, centers that only served
sixth-grade students during the reporting period would receive a *Middle
School Only* designation as would a center where the minimum grade level
of students served by the center was equal to or greater than fifth
grade and the maximum grade level served by the center was greater than
sixth grade but less than or equal to eighth grade. Overall, we use the
following logic to classify centers by grade level:

IF (min_grade \>= 0 & min_grade \<=5 & max_grade \<= 6) strGradeLevel =
\'ElemOnly\' .

IF (min_grade \>= 0 & min_grade \< 5 & max_grade \> 6 & max_grade \<= 8)
strGradeLevel = \'ElemMid\' .

IF (min_grade = 6 & max_grade = 6) strGradeLevel = \'MidOnly\' .

IF (min_grade \>= 5 & max_grade \<= 8 & max_grade \> 6) strGradeLevel =
\'MidOnly\' .

IF (min_grade \>= 5 & min_grade \<= 8 & max_grade \>= 9 & max_grade \<=
12) strGradeLevel = \'MidHigh\' .

IF (min_grade \>= 9 & max_grade \<= 12) strGradeLevel = \'HighOnly\' .

IF (strGradeLevel = \' \') strGradeLevel = \'Other\' .

So, in this regard, the term *elementary* as we use it in the report
always includes students served in Grades PreK to 4 and may include
students served in Grades 5 and 6 as well, depending upon the minimum
and maximum grade levels associated with students served by the center
in question during the reporting period.

When reporting Grade Performance and Results Act (GPRA) results by grade
level, only centers that have a pure grade-level classification are
included in the calculations associated with a given grade level. In
this regard, performance indicator results reported separately for
centers serving elementary and middle/high school students exclude those
centers that (1) serve both elementary students and middle school
students and (2) serve students enrolled across elementary, middle, and
high school grade levels. The category *Elementary* in Table 1 refers to
centers that exclusively serve elementary students while the
*Middle/High School* category refers specifically to those centers that
only serve middle and/or high school students. Generally, centers that
fail to fall either in the *Elementary* or the *Middle/High School*
categories as defined in Table 1 are more likely to demonstrate higher
levels of performance relative to the grades and state assessment
performance indicators. This explains why for the grades and state
assessment performance indicators that performance by centers classified
as serving *Elementary* and *Middle/High School* students only
demonstrate performance below the overall indicator level reported in
Table 1.

**Characteristics of the Individual Activities Dataset and\
Data Cleansing Approaches**

The initial individual activities dataset extracted from the Profile and
Performance Information Collection System (PPICS) contained a total of
58,155 discrete, individual activity records reported by 3,974 centers
providing afterschool and related services during the 2005--06 reporting
period.

The first step in preparing the dataset for analysis was to eliminate
those centers and their associated activities from the dataset if the
center in question failed to meet a series of validation parameters
enforced by the system in terms of completeness and consistency of
reporting. For example, if a center reported providing activities during
the school year via the individual activities page in PPICS but failed
to specify information about school year operations in another component
of the APR or reported having no school year participants when reporting
attendance outcomes, the system would not show a given activity record
associated with the center as being complete. Removing activity records
associated with centers that failed to meet one or more validation
parameters resulted in a culled dataset with 55,619 activity records
spanning 3,894 centers.

At this point in the process, in addition to containing activity records
that targeted student participants during the 2005--06 school year, the
dataset also contained records relative to activities provided at a
given center that were provided to youth during summer 2006 and
activities that targeted the adult family members of students
participating in the program. Given the interest in classifying programs
based on the relative emphasis placed on delivering activities targeting
youth during the school year, the dataset was further culled by removing
summer activities and activities that targeted adult family members,
resulting in a modified dataset consisting of 31,953 activities spanning
a total 3,525 centers. At this stage in the process, activity records
also were removed if the activity in question was offered at the site
less frequently than weekly (i.e., the activity was offered once or
twice a month).

One of the characteristics associated with most of the data housed in
PPICS that often needs to be addressed in preparing data extracted from
the system for analysis is that many data elements have a tendency to be
characterized by positively skewed distributions. This is largely due to
the wide annual funding range associated with the 21st CCLC grantee
population, for which the vast majority of 21st CCLC grants fall within
a range of \$50,000 to \$300,000 per year while a small number of
programs located in mostly large urban areas receive significantly
larger grants of \$1 million or more per year to support programming.
Obviously, a program funded by a \$1 million grant can provide a much
larger domain of services to a larger student population than a program
annually receiving \$50,000 in grant funds.

In addition, with each wave of data collection, there is a small number
of respondents that misinterpret the instructions housed in the system
and mistakenly provide incorrect data that has a tendency to inflate the
mean of certain data elements housed in the system, further aggravating
the problems associated with positively skewed distributions. To address
the problem of overly skewed distributions, a series of SPSS syntax
routines have been developed relying upon historical data that identify
the values associated with the 75th and 25th percentile of a given
distribution, calculate the inter-quartile range, and multiply this
value by three, and then add this product to the value associated with
the 75th percentile to create a threshold that if exceeded represents an
extreme outlier for the distribution in question. Applying these
techniques resulted in a further culling of the dataset by removing
records exceeding one or more extreme outlier target, resulting in a
modified dataset consisting of 25,807 activity records spanning 3,047
centers.

One of the approaches taken with the activities dataset under
consideration to evaluate the extent to which the problem of positively
skewed distributions may be present was to compare the total number of
activity hours accounted for in the culled activities file (i.e., the
total number of weeks an activity was provided multiplied by the days
per week multiplied by the total number of hours per day summed across
activities provided during the 2005--06 school year) with the total
hours of operation the center reported being open during the course of
the 2005--06 school year as identified in a different section of the
APR. A low ratio would suggest that programs likely underreported their
activities information while a ratio greater than one would suggest that
a center offered a number of activities concurrently on the same day
serving different groups of students. When examining the distribution of
the ratios resulting from these calculations, the skewness value for
this distribution was found to be 5.770 while the value for kurtosis was
57.025. Given these results, it seemed appropriate to eliminate some of
the more extreme outliers associated with the distribution in question.
Using the typical approach employed in addressing positively skewed
distributions in PPICS, centers with a ratio that exceeded 3 x IQR plus
the value associated with the 75th percentile were tagged as extreme
outliers (i.e., in this case records whose ratio exceeded 6.7648) and
removed from the dataset. In a similar fashion, centers with extremely
low ratios are likely to have drastically underreported the domain of
activities provided during the reporting period. Ultimately, centers
with a ratio below .8900 (the value associated with the 25th percentile
associated with the distribution) also were removed from the dataset.
Once these modifications were made, the domain of centers included in
the dataset dropped to 2,142. After the dataset was culled in the manner
just described, the values for skewness and kurtosis dropped to 1.496
and 1.1813, respectively.

**Summary of Cluster Analysis Methods**

After taking the steps necessary to prepare the dataset for analysis,
final variables representing the relative emphasis a given center gave
to providing a given category of activity were calculated for each of
the 10 activity categories referenced previously in the report by
dividing the total number of hours that category of activity was
provided during the 2005--06 school year by the total activity hours for
the center in question. In this regard, variables were created that
represented the percentage of total hours accounted for by a given
activity category.

K-means cluster analysis was then employed using these 10 variables in
an initial attempt to identify primary program clusters. Several
different cluster configurations were run, including 20, 15, 10, and 5
cluster solutions. In an effort to further draw comparisons among the
products resulting from these analyses, the mean percentage of hours
within a cluster were graphed simultaneously with separate lines
representing the 20, 15, 10, and 5 cluster solutions for the four
categories of activities theorized to have the greatest discriminatory
power in segmenting the domain of centers under consideration: (1)
academic enrichment activities, (2) tutoring,\
(3) homework help, and (4) recreational activities. Generally, these
areas appear to represent the most common categories of activities
provided by centers, and it was expected that as a result there would be
a fair degree of variation from one center to the next in terms of the
relative emphasis given to programming in each of these areas. In Figure
B1, the mean percentage of hours dedicated to academic enrichment
activities is graphed for each cluster solution, starting with the
cluster associated with the cluster solution in question that
demonstrated the highest mean percentage of hours dedicated to academic
enrichment activities and then moving to the cluster with the second
highest percentage, etc. Of some interest in Figure B1 is that the
smaller the number of clusters specified in the analysis the steeper the
slope, at least initially, which would seem to suggest that approaches
employing a smaller number of clusters allows for a more dichotomous
classification of centers into high- and low-academic enrichment groups.
Slopes associated with the 15- and 20-cluster solutions are noticeably
less steep, creating clusters of centers that have moderate levels of
academic enrichment associated with their program, which may be helpful
if there is a need to explore the interaction of moderate levels of
enrichment with other categories of activities.

**Figure B1. Mean Percentage of Activity Hours Dedicated to\
Academic Enrichment by Cluster Solution**

![](media/image60.wmf){width="6.502083333333333in"
height="3.678472222222222in"}

In Figure B2, a similar approach was taken to graphing multiple cluster
solutions for tutoring. Here, the initial slopes across each of the
cluster solutions employed are quite steep in comparison to those
highlighted in Figure B1. This may suggest that little benefit is
derived from expanding the number of clusters beyond the initial
five-cluster solution. This trend across cluster solutions seems to
suggest that programs either place a high relative emphasis on tutoring
activities or that they relegate such activities to minority status in
terms of what they offer in the way of programming to students attending
their sites.

**Figure B2. Mean Percentage of Activity Hours Dedicated to Tutoring by
Cluster Solution**

![](media/image61.wmf){width="6.500694444444444in"
height="3.786111111111111in"}

When the mean percentage of activity hours within a cluster are graphed
in relation to homework help activities across each of the cluster
solutions, the results were found to be quite similar to the graph
represented in Figure B2 for tutoring activities. As shown in Figure B3,
the initial slopes associated with each cluster solution are quite
steep, demonstrating that there are few, if any, clusters with a
moderate relative emphasis on homework help activities. Of some interest
in Figure B3 is the significant gap across cluster solutions in terms of
the cluster with the highest mean percentage of total hours dedicated to
homework help activities. For example, for the\
20-cluster solution, the cluster with the highest mean percentage of
hours dedicated to homework help (83 percent) is 34 percentage points
higher than the cluster with the highest mean for the five-cluster
solution (49 percent). A similar result can also be observed in Figure
B2 in relation to tutoring activities, although here the gap between the
20- and five-cluster solutions is only 19 percentage points. In each of
these cases, the 15- and 20-cluster solutions seem to be revealing three
bands or breaks in the clustering of centers: those with rather high
concentrations of tutoring or homework help activities, those with a
more moderate emphasis, and those with a relative low emphasis. The
existence of these moderate bands seems to be masked in the five- and
10-cluster solutions in both Figures B2 and B3.

**Figure B3. Mean Percentage of Activity Hours Dedicated to Homework
Help\
by Cluster Solution**

![](media/image62.wmf){width="6.499305555555556in"
height="3.682638888888889in"}

In terms of the relative emphasis centers place upon the provision of
recreation activities, as shown in Figure B4, the slopes associated with
most of the cluster solutions are more level than what was witnessed for
tutoring and homework help in Figures B2 and B3 but not quite as smooth
as those associated with academic enrichment found in Figure B1. Of some
interest in Figure B4 is the way in which there is an initial, fairly
steep drop across the initial two to three clusters in a given cluster
solution but eventually the change in slope becomes more gradual around
the 20 percent mark, which also is true in Figures B2 and B3; however,
in Figure B4 the change after the 20 percent mark is not nearly as flat
as is witnessed for tutoring and homework help. This would seem suggest
that there are less likely to be these hard "breaks" or "bands" when the
relative emphasis upon recreation activities falls below 20 percent in a
given cluster, meaning there is likely to be more variation among
clusters on this lower end of the spectrum than is the case with
tutoring or homework help.

**Figure B4. Mean Percentage of Activity Hours Dedicated to Recreational
Activities\
by Cluster Solution**

![](media/image63.wmf){width="6.5in" height="3.8819444444444446in"}

So, the primary question we faced was, what cluster solution seemed to
be the most useful in identifying a domain of primary clusters of
centers based on the relative emphasis they give to providing certain
types of activities that will be helpful to the field in exploring how
key quality constructs should be differentially applied to the unique
programmatic context represented by a given cluster? It seems that
although the 20- and 15-cluster solutions provide a great degree of
insight into both the degree of variation in the relative emphasis of
activities across clusters and the existence of key breaks or bands in
the distribution of activity types, these approaches seem too complex to
be viable in an applied setting. One can even argue that the 10-cluster
solution falls in this camp as well for similar reasons. It would seem
then that the five-cluster approach is the most practical in terms of
facilitating further discussions around the extent to which
differentiating program quality indicators may be warranted. As outlined
in Table 3 in the body of the full report, the five-cluster approach
results in a cluster made up mostly of centers that focus mostly on
homework help, one that focuses mostly on tutoring, one that focuses
mostly on recreation, one that focuses mostly on academic enrichment,
and one consisting of centers that provide a wide variety of activities
across multiple categories. Having four fairly homogenous clusters (in
terms of the relative emphasis given to one activity type) with a fifth
cluster containing those centers (with more varied implementation of
multiple categories of activities) is a more straightforward product
that lends itself to use more easily than a greater number of clusters
characterized by a wide variety of program hybrids in terms of the
categories of activities they provide.

**Staffing Cluster Analysis**

Basically, in constructing clusters based on the staffing model employed
by centers represented in the 2005--06 APR, the exact domain of steps
associated with the development of program clusters based individual
activities data were followed as well and are not described in
significant detail here. One substantial difference between the two
analyses was that a much lower proportion of centers was culled from the
data cleansing procedures associated with prepping the staffing data as
compared to the domain of centers removed from the activities-based
analyses. This result, along with the fact that staffing data across all
states could be employed in the cluster analyses in question, make the
results from the staffing-related cluster analyses likely more robust
than those associated with clusters formed from individual activities
data.

# Appendix C

**APR Reporting Options Afforded to States**

Prior to activating their APR module in PPICS, states were afforded
options that allowed them to customize the APR for their state to some
extent based on their information needs and priorities. When submitting
APR data for the 2004 and 2005 reporting periods, states were afforded
the option to select the impact categories (i.e., grades, state
assessment, and/or teacher survey results) that their grantees would be
reporting on as part of the APR completion process. For the 2006
reporting period, states still were afforded a series of options related
to impact category reporting, but the domain of choices that states
could choose from in order to be in compliance with federal reporting
requirements was narrowed. Some impact categories that in previous APR
years could be used to meet federal reporting requirements could no
longer be used for this purpose, although they were still available to
states to employ on an optional basis if they had found these options to
provide information useful for internal monitoring and evaluation
efforts.

In addition, given difficulties associated with the previous method of
obtaining APR activities information, the approach to collecting APR
activities data was significantly modified for the 2006 APR, consisting
of two options that states needed to choose from when completing the
State Decision pages in PPICS. States could have opted to continue to
report aggregated activity information through a revised version of the
APR Activities page found in PPICS in previous APR years or they could
have opted to implement a newly developed APR Activities page that
allowed for data to be collected at the level of an individual activity
offered during the reporting period.

Finally, for the 2006 APR, functionality was added to PPICS that allowed
states the option either of expanding or decreasing the domain of data
submission requirements to the APR in order to obtain more detailed
information about the activities undertaken by grantees during the 2006
reporting period and the outcomes they were able to attain. More
specifically, states were given the option to (1) make the Objectives
section of the APR optional or mandatory for their grantees or (2)
require grantees to report impact category data for regular attendees
separately for each of the following attendance categories: 30 to 59
days, 60 to 89 days, or 90 days or more.

In summary, states needed to make the following decisions before the APR
for the 2005--06 reporting period could be activated in their state:

-   What approach the state would be adopting for the reporting period
    in regards to the provision of APR Activities data.

-   Whether the state would require grantees to report APR Objectives
    data for the reporting period.

-   What impact categories the state would be making mandatory for
    grantees to report on as part of the APR process (i.e., Grades,
    State Assessment Cross Year Disaggregated, and/or Teacher Survey
    results).

-   What, if any, optional impact categories the state would require
    grantees to submit data about as part of the APR process (i.e.,
    State Assessment Current Year, State Assessment Cross Year Standard,
    and/or State Teacher Survey).

-   Whether the state would require grantees to report impact category
    data based on subgroups of regular attendees based on how they fell
    within three gradations of attendance: 30 to 59 days, 60 to 89 days,
    90 days or more.

A summary of which reporting options were selected by states in outlined
in Table C1.

Table C1. Number and Percentage of States Selecting Different Reporting
Options

+-----------------------------------------+------------+--------------+
| **APR Reporting Option**                | **\# of    | **% of       |
|                                         | States**   | States**     |
+-----------------------------------------+------------+--------------+
| Opted to Report APR Objectives          | 39         | 73.6%        |
+-----------------------------------------+------------+--------------+
| Activities                              |            |              |
+-----------------------------------------+------------+--------------+
| > Aggregated Activities                 | 31         | 58.5%        |
+-----------------------------------------+------------+--------------+
| > Individual Activities                 | 22         | 41.5%        |
+-----------------------------------------+------------+--------------+
| Mandatory Impact Categories             |            | 15.6%        |
+-----------------------------------------+------------+--------------+
| > Grades                                | 30         | 56.6%        |
+-----------------------------------------+------------+--------------+
| > State Assessment Cross Year           | 20         | 37.7%        |
| > Disaggregated                         |            |              |
+-----------------------------------------+------------+--------------+
| > Federal Teacher Survey                | 36         | 67.9%        |
+-----------------------------------------+------------+--------------+
| Optional Impact Categories              |            | 15.6%        |
+-----------------------------------------+------------+--------------+
| > State Assessment Current Year         | 23         | 43.4%        |
+-----------------------------------------+------------+--------------+
| > State Assessment Cross Year Standard  | 0          | 0.0%         |
+-----------------------------------------+------------+--------------+
| > State Teacher Survey                  | 1          | 1.9%         |
+-----------------------------------------+------------+--------------+
| Opted to Report Impact Data by          | 25         | 47.2%        |
| Attendance Gradation                    |            |              |
+-----------------------------------------+------------+--------------+

*Note.* Based on 53 states providing data (100 percent of all states in
the APR).

# Appendix D 

# Glossary

+-------------------+--------------------------------------------------+
| **21st CCLC       | From the U.S. Department of Education website    |
| Program**         | (www.ed.gov/programs/21stcclc/index.html):       |
|                   |                                                  |
|                   | > "The 21st Century Community Learning Centers   |
|                   | > Program is a key component of ... \[the\] No   |
|                   | > Child Left Behind Act \[of 2001\]. It is an    |
|                   | > opportunity for students and their families to |
|                   | > continue to learn new skills and discover new  |
|                   | > abilities after the school day has ended....\  |
|                   | > \                                              |
|                   | > "The focus of this program, re-authorized      |
|                   | > under Title IV, Part B, of the No Child Left   |
|                   | > Behind Act of 2001, is to provide expanded     |
|                   | > academic enrichment opportunities for children |
|                   | > attending low performing schools. Tutorial     |
|                   | > services and academic enrichment activities    |
|                   | > are designed to help students meet local and   |
|                   | > state academic standards in subjects such as   |
|                   | > reading and math. In addition 21st CCLC        |
|                   | > programs provide youth development activities, |
|                   | > drug and violence prevention programs,         |
|                   | > technology education programs, art, music and  |
|                   | > recreation programs, counseling and character  |
|                   | > education to enhance the academic component of |
|                   | > the program."                                  |
|                   |                                                  |
|                   | As part of the reauthorization under NCLB, the   |
|                   | program is now administered through state        |
|                   | education agencies rather than directly by the   |
|                   | U.S. Department of Education.                    |
+-------------------+--------------------------------------------------+
| **21st Century    | A community learning center offers academic,     |
| Community         | artistic, and cultural enrichment opportunities  |
| Learning Center   | to students and their families during nonschool  |
| (CCLC)**          | hours (before or after school) or periods when   |
|                   | school is not in session (including holidays,    |
|                   | weekends, and summer recess). A center supported |
|                   | with 21st CCLC funds is considered to be the     |
|                   | physical location where grant-funded services    |
|                   | and activities are provided to participating     |
|                   | students and adults. A center is characterized   |
|                   | by defined hours of operation; a dedicated staff |
|                   | that plans, facilitates, and supervises program  |
|                   | activities; and an administrative structure that |
|                   | may include a position akin to a center          |
|                   | coordinator. A 21st CCLC grant must fund at      |
|                   | least one 21st CCLC center. If the same          |
|                   | participants attending a program participate in  |
|                   | activities at multiple sites, only one of these  |
|                   | locations should be selected as the primary      |
|                   | center serving that group of participants.       |
+-------------------+--------------------------------------------------+
| **Academic        | Enrichment activities expand on students'        |
| enrichment        | learning in ways that differ from the methods    |
| learning          | used during the school day. They often are       |
| programs**        | interactive and project-focused. They enhance a  |
|                   | student's education by bringing new concepts to  |
|                   | light or by using old concepts in new ways.      |
|                   | These activities are fun for the student, but    |
|                   | they also impart knowledge. They allow the       |
|                   | participants to apply knowledge and skills       |
|                   | stressed in school to real-life experiences.     |
+-------------------+--------------------------------------------------+
| **Activities**    | Statutorily authorized events or undertakings at |
|                   | a CCLC that involve one or more program          |
|                   | participants.                                    |
+-------------------+--------------------------------------------------+
| **Annual          | All grantees active across the span of a given   |
| Performance       | reporting period are required to provide         |
| Report (APR)**    | information required as part of the Annual       |
|                   | Performance Report process. The purposes of the  |
|                   | APR are (1) to collect data from 21st CCLC       |
|                   | grantees on progress made during the preceding   |
|                   | year in meeting their project objectives, (2) to |
|                   | collect data on the elements that characterized  |
|                   | center operation during the reporting period,    |
|                   | including the student and adult populations      |
|                   | served, and (3) to collect data that address the |
|                   | GPRA performance indicators for the 21st CCLC    |
|                   | program.                                         |
+-------------------+--------------------------------------------------+
| **Career/job      | These activities may target youths and/or adults |
| training**        | participating in the 21st CCLC program. They are |
|                   | designed to support the development of a defined |
|                   | skill set that is directly transferable to a     |
|                   | specific vocation, industry, or career. For      |
|                   | youths participating in center programming, this |
|                   | category includes activities that are designed   |
|                   | to provide exposure to various types of careers  |
|                   | and that help inform youths of the skills needed |
|                   | to obtain a given career.                        |
+-------------------+--------------------------------------------------+
| **Center**        | The physical location where grant-funded         |
|                   | services and activities are provided to          |
|                   | participating students and adults. (See also:    |
|                   | **21st Century Community Learning Center**.)     |
+-------------------+--------------------------------------------------+
| **Center          | Staff members whose primary role is coordinating |
| administrators/\  | the center's activities.                         |
| coordinators**    |                                                  |
+-------------------+--------------------------------------------------+
| **Cohort**        | A grouping of grantees or centers determined by  |
|                   | whether a given grantee/center first reported    |
|                   | APR information for the 2003--04 reporting       |
|                   | period (Cohort 1), for the 2004--05 reporting    |
|                   | period (Cohort 2), or for the 2005--06 reporting |
|                   | period (Cohort 3). This definition is different  |
|                   | from the definition of *cohort* that may be used |
|                   | in individual states according to when the       |
|                   | grantee received its award.                      |
+-------------------+--------------------------------------------------+
| **Community-based | An entity organized and operated exclusively for |
| organ             | one or more of the purposes set forth in         |
| ization/nonprofit | Internal Revenue Code Section 501(c)(3). For the |
| agency**          | purposes of completing the APR, in order to be   |
|                   | identified as a community-based                  |
|                   | organization/nonprofit agency, an organization   |
|                   | should not be classifiable as a nationally       |
|                   | affiliated nonprofit agency or a faith-based     |
|                   | organization.                                    |
+-------------------+--------------------------------------------------+
| **Community       | An organization other than the grantee that      |
| partner**         | actively contributes to the 21st CCLC-funded     |
|                   | project.                                         |
+-------------------+--------------------------------------------------+
| **Community       | These activities are characterized by defined    |
| service/\         | service tasks performed by students that address |
| service learning  | a given community need and that provide for      |
| programs**        | structured opportunities that link tasks to the  |
|                   | acquisition of values, skills, or knowledge by   |
|                   | participating youths.                            |
+-------------------+--------------------------------------------------+
| **Drug and        | These activities are designed to (1) prevent     |
| violence          | youths from engaging in high-risk behaviors,     |
| prevention,       | including the use of drugs and alcohol, and (2)  |
| counseling, and   | promote the amelioration of the causal factors   |
| character         | that may lead youths to participate in such      |
| education         | activities through counseling and support,       |
| programs**        | and/or the cultivation of core ethical values    |
|                   | such as caring, honesty, fairness,               |
|                   | responsibility, and respect for self and others  |
|                   | that are likely to contribute to prevention      |
|                   | efforts.                                         |
+-------------------+--------------------------------------------------+
| **Expanded        | 21st CCLC funds are used specifically to expand  |
| library hours**   | the normal operating hours of a library.         |
+-------------------+--------------------------------------------------+
| **Faith-based     | An entity whose primary program area can be      |
| organization**    | defined as being religion-related. A faith-based |
|                   | organization could be a religious congregation   |
|                   | or an organization that primarily undertakes     |
|                   | activities that are of a religious nature. Note  |
|                   | that YMCAs/YWCAs are not considered to be        |
|                   | faith-based organizations.                       |
+-------------------+--------------------------------------------------+
| **Federal         | State proficiency levels have been matched to    |
| proficiency       | one of three federal proficiency levels: basic,  |
| level**           | proficient, and advanced.                        |
+-------------------+--------------------------------------------------+
| **Government      | > From the U.S. Department of Agriculture        |
| Performance and   | > website                                        |
| Results Act       | > (www.rma.usda.gov/news/2004/05/gpra.pdf):\     |
| (GPRA)**          | > "The Government Performance and Results Act of |
|                   | > 1993 (GPRA) is a straightforward statute that  |
|                   | > requires all federal agencies to manage their  |
|                   | > activities with attention to the consequences  |
|                   | > of those activities. Each agency is to clearly |
|                   | > state what it intends to accomplish, identify  |
|                   | > the resources required, and periodically       |
|                   | > report their progress to the Congress. In so   |
|                   | > doing, it is expected that the GPRA will       |
|                   | > contribute to improvements in accountability   |
|                   | > for the expenditures of public funds, improve  |
|                   | > congressional decision-making through more     |
|                   | > objective information on the effectiveness of  |
|                   | > federal programs, and promote a new government |
|                   | > focus on results, service delivery, and        |
|                   | > customer satisfaction."                        |
+-------------------+--------------------------------------------------+
| **Grantee**       | The entity serving as the fiduciary agent for a  |
|                   | given 21st CCLC grant.                           |
+-------------------+--------------------------------------------------+
| **Homework help** | Homework help refers to program time that is     |
|                   | dedicated to assisting students work             |
|                   | independently on homework, with or without       |
|                   | assistance from staff, volunteers, or older      |
|                   | peers.                                           |
+-------------------+--------------------------------------------------+
| **Mentoring**     | Mentoring activities primarily are characterized |
|                   | by matching students one-on-one with one or more |
|                   | adult role models, often from business or the    |
|                   | community, for guidance and support.             |
+-------------------+--------------------------------------------------+
| **Nationally      | A nonprofit entity that is associated with a     |
| affiliated        | national organization. For example, local YMCAs, |
| nonprofit         | YWCAs, the Girl Scouts, the Boy Scouts, Big      |
| agency**          | Brothers/Big Sisters, and Boys and Girls Clubs   |
|                   | all are considered to be nationally affiliated   |
|                   | nonprofit agencies.                              |
+-------------------+--------------------------------------------------+
| **Partner**       | See **Community Partner**.                       |
+-------------------+--------------------------------------------------+
| **Performance     | A measure intended to determine the              |
| indicator**       | effectiveness of the program in achieving one of |
|                   | its goals                                        |
+-------------------+--------------------------------------------------+
| **Profile and     | PPICS is a Web-based data collection system      |
| Performance       | developed to capture information regarding 21st  |
| Information       | Century Community Learning Center (21st CCLC)    |
| Collection System | programs.                                        |
| (PPICS)**         |                                                  |
+-------------------+--------------------------------------------------+
| **Recreational    | These activities are not academic in nature, but |
| activities**      | rather allow students time to relax or play.     |
|                   | Sports, games, and clubs fall into this          |
|                   | category. Occasional academic aspects of         |
|                   | recreation activities can be pointed out, but    |
|                   | the primary lessons learned in recreational      |
|                   | activities are in the areas of social skills,    |
|                   | teamwork, leadership, competition, and           |
|                   | discipline.                                      |
+-------------------+--------------------------------------------------+
| **Regular         | Refers to students who have attended a 21st CCLC |
| attendee**        | program for at least 30 days (which do not have  |
|                   | to be consecutive) during the reporting period.  |
+-------------------+--------------------------------------------------+
| **Reporting       | The reporting period for the Annual Performance  |
| period**          | Report coincides with the school year and        |
|                   | includes the summer prior to the school year.    |
+-------------------+--------------------------------------------------+
| **Service         | See **Community service/service learning         |
| learning**        | programs**.                                      |
+-------------------+--------------------------------------------------+
| **State           | The assessment(s) administered by a given state  |
| assessment**      | relied upon by the state education agency (SEA)  |
|                   | to meet consolidated reporting requirements      |
|                   | under the NCLB Act of 2001.                      |
+-------------------+--------------------------------------------------+
| **Subcontractor** | An organization that receives 21st CCLC grant    |
|                   | funds under contract with the grantee to provide |
|                   | 21st CCLC grant-funded activities or services.   |
|                   | For APR purposes, a subcontractor is considered  |
|                   | to be a type of partner.                         |
+-------------------+--------------------------------------------------+
| **Supplemental    | Supplemental educational services are a          |
| educational       | component of Title I of the Elementary and       |
| services**        | Secondary Education Act (ESEA), as reauthorized  |
|                   | by the No Child Left Behind Act (NCLB) Act.      |
|                   | These services are meant to provide extra        |
|                   | academic assistance to increase the academic     |
|                   | achievement of eligible students in schools that |
|                   | have not met state targets for increasing        |
|                   | student achievement (adequate yearly progress).  |
|                   | These services may include tutoring and          |
|                   | afterschool services. They may be offered        |
|                   | through public- or private-sector providers that |
|                   | are approved by the state, such as public        |
|                   | schools, public charter schools, local education |
|                   | agencies, educational service agencies, and      |
|                   | faith-based organizations. Students from         |
|                   | low-income families who remain in Title I        |
|                   | schools that fail to meet state standards for at |
|                   | least three years are eligible to these          |
|                   | services.                                        |
+-------------------+--------------------------------------------------+
| **Teacher         | This survey is administered at the end of the    |
| survey**          | year. The survey asks school-day teachers to     |
|                   | report if the behavior of regular attendees      |
|                   | improved or did not improve in certain areas.    |
|                   | Teacher selection: For every student identified  |
|                   | as a regular attendee (30 days or more), one of  |
|                   | his or her regular school-day teachers should    |
|                   | have been selected to complete the teacher       |
|                   | survey. For elementary school students, the      |
|                   | teacher should be the regular classroom teacher. |
|                   | For middle and high school students, a           |
|                   | mathematics or English teacher should be         |
|                   | surveyed. Although teachers who also are serving |
|                   | as 21st CCLC program staff may be included, it   |
|                   | is preferred that programs survey teachers who   |
|                   | are not also program staff. Only one teacher     |
|                   | survey should be filled out for every student    |
|                   | identified as a regular attendee.                |
+-------------------+--------------------------------------------------+
| **Tutoring**      | These activities involve the direct provision of |
|                   | assistance to students in order to facilitate    |
|                   | the acquisition of skills and knowledge related  |
|                   | to concepts addressed during the school day.     |
|                   | Tutors or teachers directly work with students   |
|                   | individually and/or in small groups to complete  |
|                   | their homework, prepare for tests, and work      |
|                   | specifically on developing an understanding and  |
|                   | mastery of concepts covered during the school    |
|                   | day. Please note that tutoring services directly |
|                   | supported through Supplemental Educational       |
|                   | Services provided under the auspices of Title I  |
|                   | of the Elementary and Secondary Education Act    |
|                   | (ESEA) should be counted in the Supplemental     |
|                   | Educational Services activity category.          |
+-------------------+--------------------------------------------------+
| **Youth           | A youth development worker is any paid staff or  |
| development       | volunteer staff member who is not certified as a |
| worker**          | school-day teacher, is not employed during the   |
|                   | school day in some other capacity (e.g.,         |
|                   | librarian, school counselor) by one or more of   |
|                   | the feeder schools and/or districts associated   |
|                   | with the 21st CCLC, and has a non-teaching-based |
|                   | college degree or higher.                        |
+-------------------+--------------------------------------------------+
| **Youth           | These activities intentionally promote youth     |
| leadership        | leadership through skill development and the     |
| activities**      | provision of formal leadership opportunities     |
|                   | that are designed to foster and inspire          |
|                   | leadership aptitude in participating youth.      |
+-------------------+--------------------------------------------------+

[^1]: Here, and in the sections that follow, bar charts will be used to
    convey much of the descriptive data highlighted in this report, and
    many of the findings identified will be predicated on a visual
    inspection of subgroup differences depicted in the charts in
    question. In this regard, inferential statistics have not been
    employed to test for statistical differences across subgroups.