# Archived Information

# III. Impacts Of Middle School Centers

Estimating the impacts of centers was a major goal of the national
evaluation. This chapter presents estimates of impacts of middle school
centers on a range of student outcomes. These outcomes include
activities after school, grades, test scores, behavior in and out of
school, and perceptions of safety after school. We also present
estimates for the impacts of different types of centers and of centers
for different types of students.

Results from the analysis suggest that middle school centers were
associated with small increases in school attendance, classroom effort,
and math grades, as well as with greater parental involvement in
school-related events. On the other hand, centers had no effects on
student classroom performance, student disciplinary problems, grades in
English, science, and history and students' social development. These
were all about the same for center participants and comparison students.
Participants were also somewhat more likely to engage in negative
behavior outside the classroom and to have been victimized in some way.

This pattern of mixed results has been observed in other studies of
after-school programs for middle school students. For example, an
evaluation of the Big Brothers and Big Sisters Program found that
participants earned slightly higher grades and had slightly better
attendance than control group students but that there were no
differences between the groups on homework completion (Tierney et al.
1995). An evaluation of the LA's BEST after-school program (Brooks et
al. 1995) found that the program improved grades but that its effects on
test scores depended on the method used to estimate effects (Brooks et
al. 1995). An evaluation of a 21st-Century program in San Francisco
found no effects on test scores or grades, although the program did
affect some types of students (Trousdale 2000).

## A. Characteristics of Participant and Comparison Students Were Similar 

As noted in Chapter I, to estimate impacts of middle school centers, a
random sample of grantees serving middle school students was selected,
and for each grantee, comparison students were identified using
propensity score matching (Rosenbaum and Rubin 1983). Propensity score
matching selects comparison group students from a pool of potential
students based on how closely their characteristics resemble those of
participants. Potential comparison group members are matched to
participants with similar propensity scores.

The score matching procedure ensures that the overall group of
comparison students resembles participants on most observed
characteristics, but some differences can arise for two reasons. First,
the matching process is inexact and can yield participant and comparison
groups that differ on some characteristics. Second, parents had to give
their consent to be in the evaluation after matches were identified, and
whether or not parents gave their consent could be related to the
students' characteristics. We used regression modeling techniques to
adjust for differences between participants and comparison students on
characteristics that could be observed. Appendix B provides more details
about the matching process and the regression techniques used to
estimate impacts.

The success of the matching method is suggested by the small differences
of the participants and comparison groups on a variety of
characteristics (Table III.1). For example, 47.1 percent of the
participant group and 46.5 percent of the comparison group were male,
12.3 percent of the participant group and 11.9 percent of the comparison
group were Hispanic, and 52.0 percent of the participant group and 58.0
percent of the comparison group lived in a two-parent household.

On the other hand, because information from parent questionnaires was
available only after matching was completed, the design could not use
information about parents to match students.

Table III.I

Characteristics of Center Participants and Comparison Group Students:

Middle School Centers

+-----------------------------+------------+------------+------------+
| Characteristic              | Percentage | Percentage | p-Value^a^ |
|                             | of Program | of         |            |
|                             | Pa         | Comparison |            |
|                             | rticipants | Group      |            |
|                             |            | Members    |            |
+=============================+============+============+============+
| Gender                      |            |            |            |
+-----------------------------+------------+------------+------------+
| > Male                      | 47.1       | 46.5       | 0.70       |
+-----------------------------+------------+------------+------------+
| > Female                    | 52.9       | 53.5       | 0.70       |
+-----------------------------+------------+------------+------------+
| Race/Ethnicity              |            |            |            |
+-----------------------------+------------+------------+------------+
| > White (non-Hispanic)      | 38.2       | 40.6       | 0.15       |
+-----------------------------+------------+------------+------------+
| > Black (non-Hispanic)      | 27.8       | 24.8       | 0.05\*     |
+-----------------------------+------------+------------+------------+
| > Hispanic                  | 12.3       | 11.9       | 0.73       |
+-----------------------------+------------+------------+------------+
| > Other                     | 15.5       | 15.8       | 0.79       |
+-----------------------------+------------+------------+------------+
| > Mixed race                | 6.3        | 6.8        | 0.53       |
+-----------------------------+------------+------------+------------+
| Grade Level                 |            |            |            |
+-----------------------------+------------+------------+------------+
| > 6                         | 20.5       | 21.3       | 0.63       |
+-----------------------------+------------+------------+------------+
| > 7                         | 37.6       | 38.2       | 0.62       |
+-----------------------------+------------+------------+------------+
| > 8                         | 34.3       | 34.4       | 0.82       |
+-----------------------------+------------+------------+------------+
| > Other or ungraded         | 7.7        | 6.0        | 0.03\*\*   |
+-----------------------------+------------+------------+------------+
| Primary Language at Home Is | 17.7       | 18.8       | 0.38       |
| Not English                 |            |            |            |
+-----------------------------+------------+------------+------------+
| Household Structure         |            |            |            |
+-----------------------------+------------+------------+------------+
| > Two parents               | 52.0       | 58.0       | 0.44       |
+-----------------------------+------------+------------+------------+
| > Single mother             | 28.5       | 24.3       | 0.09\*     |
+-----------------------------+------------+------------+------------+
| > Single father             | 2.9        | 2.7        | 0.87       |
+-----------------------------+------------+------------+------------+
| > Other                     | 16.6       | 15.0       | 0.16       |
+-----------------------------+------------+------------+------------+
| Parental Education          |            |            |            |
+-----------------------------+------------+------------+------------+
| > Mothers have at least a   | 79.9       | 78.5       | 0.30       |
| > high school degree/GED    |            |            |            |
+-----------------------------+------------+------------+------------+
| > Mothers have a four-year  | 10.8       | 14.4       | 0.00\*\*\* |
| > college degree            |            |            |            |
+-----------------------------+------------+------------+------------+
| > Fathers have at least a   | 76.9       | 76.7       | 0.90       |
| > high school degree/GED    |            |            |            |
+-----------------------------+------------+------------+------------+
| > Fathers have a four-year  | 11.7       | 16.0       | 0.00\*\*\* |
| > college degree            |            |            |            |
+-----------------------------+------------+------------+------------+
| Employment Status           |            |            |            |
+-----------------------------+------------+------------+------------+
| > Both parents with a       | 26.6       | 31.0       | 0.00\*\*\* |
| > full-time job             |            |            |            |
+-----------------------------+------------+------------+------------+
| > Mothers with a full-time  | 59.7       | 60.5       | 0.57       |
| > job                       |            |            |            |
+-----------------------------+------------+------------+------------+
| > Mothers with a part-time  | 14.3       | 15.8       | 0.21       |
| > job                       |            |            |            |
+-----------------------------+------------+------------+------------+
| > Fathers with a full-time  | 81.1       | 84.6       | 0.01\*\*\* |
| > job                       |            |            |            |
+-----------------------------+------------+------------+------------+
| > Fathers with a part-time  | 4.5        | 3.6        | 0.53       |
| > job                       |            |            |            |
+-----------------------------+------------+------------+------------+
| Household Income            |            |            |            |
+-----------------------------+------------+------------+------------+
| > Less than \$10,999        | 16.1       | 15.0       | 0.35       |
+-----------------------------+------------+------------+------------+
| > \$11,000-\$24,999         | 27.5       | 24.8       | 0.07\*     |
+-----------------------------+------------+------------+------------+
| > \$25,000-\$39,999         | 24.7       | 21.0       | 0.01\*\*\* |
+-----------------------------+------------+------------+------------+
| > \$40,000-\$59,999         | 17.5       | 18.4       | 0.49       |
+-----------------------------+------------+------------+------------+
| > More than \$60,000        | 14.1       | 20.7       | 0.00\*\*\* |
+-----------------------------+------------+------------+------------+
| Receipt of Government       |            |            |            |
| Assistance                  |            |            |            |
+-----------------------------+------------+------------+------------+
| > Food stamps               | 18.3       | 17.3       | 0.44       |
+-----------------------------+------------+------------+------------+
| > TANF                      | 7.4        | 6.8        | 0.51       |
+-----------------------------+------------+------------+------------+
|                             |            |            |            |
+-----------------------------+------------+------------+------------+
| Student-Reported Baseline   |            |            |            |
| Grades                      |            |            |            |
+-----------------------------+------------+------------+------------+
| > Mostly A's                | 30.3       | 34.1       | 0.02\*\*   |
+-----------------------------+------------+------------+------------+
| > Mostly B's                | 35.7       | 36.4       | 0.31       |
+-----------------------------+------------+------------+------------+
| > Mostly C's                | 23.3       | 21.2       | 0.03\*\*   |
+-----------------------------+------------+------------+------------+
| > Mostly D's or below       | 8.8        | 7.6        | 0.34       |
+-----------------------------+------------+------------+------------+
| > Not graded                | 1.8        | 0.7        | 0.00\*\*\* |
+-----------------------------+------------+------------+------------+
| **Sample Size^b^**          | **1,752**  | **2,437**  |            |
+-----------------------------+------------+------------+------------+

[Source]{.smallcaps}: Student survey, parent survey, school records.

^a^The p-value is the smallest level of significance at which the null
hypothesis that the difference in means between program participants and
comparison group members equals zero can be rejected. If the p-value is
less than .01, the difference is significant at the 1 percent level; if
the p-value is less than .05, the difference is significant at the 5
percent level, and so on.

^b^Sample sizes for some characteristics differ because of missing
values.

\*Significantly different from zero at the .10 level, two-tailed test.

\*\*Significantly different from zero at the .05 level, two-tailed test.

\*\*\*Significantly different from zero at the .01 level, two-tailed
test.

Data gathered from parents after matching found that a larger proportion
of participants lived in households with low annual incomes, and
participants were more likely to be from a single-parent household and
to have a mother or father with less than a four-year college degree.

The differences between the participant and comparison groups underscore
the importance of adjusting for student characteristics in measuring
impacts. The regression models (discussed in Appendix B) used to
estimate impacts adjust for observed participant-comparison differences
in characteristics such as those shown in Table III.1.[^1]

## B. How Did Middle School Centers Affect Students?

The conceptual framework presented in Chapter I links participation in
centers with a range of outcomes. For example, participation might
immediately influence students' after-school activities, location, and
supervision. These factors could in turn influence other outcomes, such
as school attendance, participation in class, and completion of
homework. They could also extend to long-term outcomes such as course
grades and test scores. Similarly, participation could influence student
behavior and personal development through the activities and services
centers provide.[^2]

### 1. Centers Increased Adult Supervision but Did Not Affect Self-Care 

Centers decreased how often parents or older siblings cared for students
and increased how often adults who were not their parents (for example,
program staff members) cared for these students. Table III.2 shows that
a parent cared for 53.2 percent of participants at least three days
during a typical week, compared with 59.2 percent of comparison
students.[^3] It also shows that an older sibling cared for 4.6 percent
of participants, compared with 7.2 percent of comparison group students.
On the other hand, 20.2 percent of participants and 11.7 percent of
comparison group students were being cared for by an adult who was not
their parent. However, centers did not affect whether students were in
self-care (defined as students being by themselves, with friends, or
with younger siblings after school, and not being cared for by an
adult). About 17 percent of both participants and comparison group
students were in self-care at the time of the follow-up survey. The net
effect was to increase the proportion of students being cared for by an
adult (either a parent or a non-parent adult) by reducing the proportion
being cared for by older siblings.[^4]

Not surprisingly, centers also increased the time students spent at
school during the after-school hours. Participants were more likely to
remain at school after the regular school day ended, with 30 percent
spending three or more days in a typical week at school, compared with
18 percent of comparison students (Table III.2). During the average
week, participants spent twice as many days at school as comparison
group members (1.2 versus 0.6 days, on average, an effect size of 10
percent). Participants also were less likely than comparison students to
be in the

Table III.2

Center Impacts on Location, Supervision, and Student Activities After
School:

Middle School Centers

+------------------------------------+--------+-------+-------+-------+
|                                    | Stud   |       |       |       |
|                                    | ent-Re |       |       |       |
|                                    | ported |       |       |       |
|                                    | Super  |       |       |       |
|                                    | vision |       |       |       |
|                                    | and    |       |       |       |
|                                    | Lo     |       |       |       |
|                                    | cation |       |       |       |
+====================================+========+=======+=======+=======+
| Outcome                            | Center | Compa | Esti  | p-Val |
|                                    | Partic | rison | mated | ue^a^ |
|                                    | ipants | Group | I     |       |
|                                    |        |       | mpact |       |
|                                    |        |       | of    |       |
|                                    |        |       | Par   |       |
|                                    |        |       | ticip |       |
|                                    |        |       | ation |       |
+------------------------------------+--------+-------+-------+-------+
| Percentage of Students in the      |        |       |       |       |
| Following Types of Supervision at  |        |       |       |       |
| Least Three Days after School in a |        |       |       |       |
| Typical Week:                      |        |       |       |       |
+------------------------------------+--------+-------+-------+-------+
| > Self-care                        | 17.5   | 17.3  | 0.1   | .92   |
+------------------------------------+--------+-------+-------+-------+
| > Parent care                      | 53.2   | 59.2  | -     | .00   |
|                                    |        |       | -6.0\ |       |
|                                    |        |       | *\*\* |       |
+------------------------------------+--------+-------+-------+-------+
| > Nonparent adult care             | 20.2   | 11.7  | 8.5\  | .00   |
|                                    |        |       | *\*\* |       |
+------------------------------------+--------+-------+-------+-------+
| > Sibling care                     | 4.6    | 7.2   | -     | .00   |
|                                    |        |       | -2.6\ |       |
|                                    |        |       | *\*\* |       |
+------------------------------------+--------+-------+-------+-------+
| > Mixed care (not in any one       | 4.6    | 4.6   | 0.0   | .94   |
| > category for at least three      |        |       |       |       |
| > days)                            |        |       |       |       |
+------------------------------------+--------+-------+-------+-------+
| Percentage of Students in the      |        |       |       |       |
| Following Locations after School   |        |       |       |       |
| at Least Three Days in a Typical   |        |       |       |       |
| Week:                              |        |       |       |       |
+------------------------------------+--------+-------+-------+-------+
| > Own home                         | 67.5   | 78.4  | --    | .00   |
|                                    |        |       | 10.9\ |       |
|                                    |        |       | *\*\* |       |
+------------------------------------+--------+-------+-------+-------+
| > Someone else's home              | 8.9    | 13.3  | -     | .00   |
|                                    |        |       | -4.4\ |       |
|                                    |        |       | *\*\* |       |
+------------------------------------+--------+-------+-------+-------+
| > School or other place for        | 30.0   | 18.0  | 12.0\ | .00   |
| > activities                       |        |       | *\*\* |       |
+------------------------------------+--------+-------+-------+-------+
| > Somewhere to "hang out"          | 9.0    | 10.3  | --1.3 | .24   |
+------------------------------------+--------+-------+-------+-------+
| > Mixed location (not in one       | 8.7    | 7.2   | 1.4   | .16   |
| > location for at least three      |        |       |       |       |
| > days)                            |        |       |       |       |
+------------------------------------+--------+-------+-------+-------+
| Mean Number of Days Stayed after   | 1.2    | 0.6   | 0.6\  | .00   |
| School for Activities in Typical   |        |       | *\*\* |       |
| Week                               |        |       |       |       |
+------------------------------------+--------+-------+-------+-------+
| Mean Number of Days Students       |        |       |       |       |
| Participate in Activity after      |        |       |       |       |
| School:                            |        |       |       |       |
+------------------------------------+--------+-------+-------+-------+
| > Homework                         | 3.2    | 3.1   | 0.1   | .12   |
+------------------------------------+--------+-------+-------+-------+
| Tutoring                           | 0.6    | 0.3   | 0.3\  | .00   |
|                                    |        |       | *\*\* |       |
+------------------------------------+--------+-------+-------+-------+
| > Non-homework reading, writing,   | 1.4    | 1.2   | 0.    | .01   |
| > or science activities            |        |       | 2\*\* |       |
+------------------------------------+--------+-------+-------+-------+
| > School activities (band, drama,  | 1.0    | 0.6   | 0.4\  | .00   |
| > etc.)                            |        |       | *\*\* |       |
+------------------------------------+--------+-------+-------+-------+
| > Lessons (music, art, dance,      | 0.6    | 0.5   | 0.1   | .19   |
| > etc.)                            |        |       |       |       |
+------------------------------------+--------+-------+-------+-------+
| > Organized sports                 | 1.4    | 1.2   | 0.2\  | .00   |
|                                    |        |       | *\*\* |       |
+------------------------------------+--------+-------+-------+-------+
| > Clubs (Boy and Girl Scouts, Boys | 0.4    | 0.2   | 0.1\  | .00   |
| > and Girls Club, etc.)            |        |       | *\*\* |       |
+------------------------------------+--------+-------+-------+-------+
| > Activities at church, temple,    | 0.5    | 0.5   | 0.0   | .83   |
| > mosque                           |        |       |       |       |
+------------------------------------+--------+-------+-------+-------+
| > Watched TV or videos             | 3.5    | 3.5   | 0.0   | .91   |
+------------------------------------+--------+-------+-------+-------+
| > Surfed the Internet or did other | 2.0    | 1.8   | 0.2\  | .01   |
| > things on a computer             |        |       | *\*\* |       |
+------------------------------------+--------+-------+-------+-------+
| > "Hung out" with friends          | 2.7    | 2.4   | 0.3\  | .00   |
|                                    |        |       | *\*\* |       |
+------------------------------------+--------+-------+-------+-------+
| > Volunteered or did community     | 0.4    | 0.3   | 0.1   | .11   |
| > service                          |        |       |       |       |
+------------------------------------+--------+-------+-------+-------+
| > Worked at a job                  | 0.5    | 0.4   | 0.0   | .51   |
+------------------------------------+--------+-------+-------+-------+
| > Did chores around the house      | 3.0    | 3.1   | -0.1  | .17   |
+------------------------------------+--------+-------+-------+-------+
| > Took care of a brother or sister | 1.6    | 1.6   | 0.0   | .89   |
+------------------------------------+--------+-------+-------+-------+
| **Sample Size^b^**                 | **1    | **2,  |       |       |
|                                    | ,750** | 437** |       |       |
+------------------------------------+--------+-------+-------+-------+

[Source:]{.smallcaps} Student survey.

[Note:]{.smallcaps} The percentages and mean values of outcomes for
participants and comparison group members have been regression-adjusted
for baseline differences between the groups. The control variables in
the regression included 26 different student and household
characteristics, such as indicators of students' demographic
characteristics, household socioeconomic status, and students' baseline
test scores, attendance, disciplinary problems, and self-reported
grades. Due to rounding, estimated impacts shown in the table do not
always equal the difference between center participants and the
comparison group.

\*Significantly different from zero at the .10 level, two-tailed test.

\*\*Significantly different from zero at the .05 level, two-tailed test.

\*\*\*Significantly different from zero at the .01 level, two-tailed
test.

^a^The p-value is the smallest level of significance at which the null
hypothesis that the difference in means between program participants and
control group members equals zero can be rejected. If the p-value is
less than .01, the difference is significant at the 1 percent level. If
the p-value is less than .05, the difference is significant at the 5
percent level, and so on.

^b^Sample sizes differ for some outcomes due to missing values.

homes of others after school. However, centers did not affect the amount
of time students spent "hanging out" away from school or
home---participants were as likely as comparison students to report
going somewhere after school to "hang out."[^5]

### 2. Centers Affected After-School Activities

Center participants were more likely to report doing academic activities
after school. Students reported on whether they had engaged in up to 15
different activities after school during a recent typical week,
including doing homework, being tutored, watching television, and doing
chores around the house. Participants were significantly more likely
than comparison students to have engaged in seven of these activities
(Table III.2), and, in particular, participants were more likely to say
that they received tutoring after school (with the average participant
being tutored twice as often as the average comparison student during a
typical week, an effect size of almost 30 percent). Participants also
were more likely to say that they participated in some reading, writing,
or science activity not related to homework.[^6]

Centers also affected nonacademic activities. During a typical week,
participants spent an average of 1.0 days participating in school
activities such as band or drama, compared with 0.6 days for comparison
students, an effect size of 27 percent (Table III.2). Participants also
were more likely to engage in organized sports or activities at clubs,
possibly because some after-school programs linked with clubs to provide
activities during program hours. In addition, while the two groups spent
the same amount of time watching television after school, participants
spent more time doing things on a computer, such as surfing the Internet
(2.0 versus 1.8 days, effect size is 10 percent).[^7]

The location and care information in Table III.2 provides some
information about how students spend after-school time that may have a
bearing on impacts. Interestingly, most students in the comparison group
reported that they were at home for at least part of the after-school
time, and many also reported that they were with a parent. Additional
analyses not reported here found that the majority (53 percent) of
comparison-group students reported being at home *and* being with a
parent after school. Centers and center staff can have impacts by being
a possibly more academically oriented form of care, but may not be a
better form of care in other dimensions.

Also, differences in location and care arrangements (shown in Table
III.2) were related to differences in participation in after-school
activities, but the largest differences were for activities that are
more likely to take place in school (for example, band and drama, and
sports). Being cared for by other adults and being at school were
associated with higher levels of these kinds of after-school activities
for all students. However, activities that could occur either at school
or elsewhere, such as music lessons, often were similar for students in
various locations or types of care. For example, 24 percent of
comparison students in the care of other adults reported participating
in lessons, compared with 23 percent of comparison students being cared
for by a parent and 22 percent of comparison students in self-care. And
participation in homework generally was high for all care and location
categories.[^8] The data are not detailed enough to describe
after-school activities of students who attend other types of formal
after-school programs such as church-based programs or Boys and Girls
Clubs.

### 3. Centers Improved Some Academic Outcomes

**a. Centers Improved Attendance and Classroom Effort but Not Homework
Completion**

Centers were associated with slightly higher school attendance. School
records indicated that center participants were absent an average of 9.0
days during the 2000-2001 school year and comparison students an average
of 10.1, an effect size of 11 percent (Table III.3). Similarly,
participants were less likely to be late to school (5.0 days late,
versus 6.2 days late, effect size 11 percent).[^9]

Centers did not increase homework completion, with 83.4 percent of both
groups reporting that they "often" or "always" did the homework their
teachers assigned (Table III.3). (Teachers reported that about half of
participants and comparison students completed their homework.)
Participants, however, were more likely to complete assignments to the
teacher's satisfaction (58.0 versus 53.3 percent, effect size 9
percent).[^10] Another indication of greater classroom effort among
participants is that a composite measure of student effort as reported
by teachers was

Table III.3

Impacts on Homework Completion and Level of Effort and Behavior in the
Classroom:

Middle School Centers

  -------------------------------- -------------- ------------ --------------- ------------
  Outcome                          Center         Comparison   Estimated       p-Value^a^
                                   Participants   Group        Impact of       
                                                               Participation   

  Percentage of Students Who       83.4           83.4         0.1             .97
  Report That They "Often" or                                                  
  "Always" Do the Homework                                                     
  Teachers Assign                                                              

  Percentage of Students Whose     50.3           49.6         0.7             .69
  Teachers Report That They                                                    
  "Often" Complete Their Homework                                              

  Percentage of Students Whose     58.0           53.3         4.8\*\*\*       .01
  Teachers "Agree" or "Strongly                                                
  Agree" That They Complete                                                    
  Assignments to the Teacher's                                                 
  Satisfaction                                                                 

  Percentage of Students Whose     51.3           48.4         2.9             .12
  Teachers Report That They                                                    
  "Usually Try Hard" in English                                                
  Class                                                                        

  Teacher-Reported Level of Effort 3.6            3.5          0.1\*\*\*       .00
  Composite^b^ (Mean)                                                          

  Percentage of Students Whose     75.1           75.3         -0.2            .91
  Parents "Agree" or "Strongly                                                 
  Agree" That Their Child Works                                                
  Hard at School                                                               

  Student-Based Discipline Problem 1.4            1.4          0.0             .91
  Composite^c^ (Mean)                                                          

  Teacher-Based Discipline Problem 1.4            1.4          0.0             .26
  Composite^d^ (Mean)                                                          

  Mean Number of Days Student Was:                                             

  Absent from class                9.0            10.1         -1.1\*\*\*      .00

  Late for class                   5.0            6.2          -1.1\*\*\*      .00

  **Sample Size^e^**               **1,752**      **2,437**                    
  -------------------------------- -------------- ------------ --------------- ------------

[Source:]{.smallcaps} Student survey; teacher survey.

[Note:]{.smallcaps} The percentages and mean values of outcomes for
program participants and comparison group members have been
regression-adjusted for baseline differences between the groups. The
regression's control variables included 26 different student and
household characteristics, such as indicators of students' demographic
characteristics, household socioeconomic status, and students' baseline
test scores, attendance, disciplinary problems, and self-reported
grades. Due to rounding, estimated impacts shown in the table do not
always equal the difference between center participants and the
comparison group.

^a^The p-value is the smallest level of significance at which the null
hypothesis that the difference in means between program participants and
control group members equals zero can be rejected. If the p-value is
less than .01, the difference is significant at the 1 percent level. If
the p-value is less than .05, the difference is significant at the 5
percent level, and so on.

^b^The level of effort composite is based on five items reported by
teachers: whether the student (1) usually tries hard, (2) often performs
at or above his or her ability level, (3) is attentive in class, (4)
participates in class, and (5) volunteers in class. The composite is
equal to the mean of the five variables. Values on these items range
from 1 to 5; a value of 1 on the composite indicates a low level of
effort, and a value of 5 indicates a high level of effort.

^c^The student-based discipline problem composite is based on four
items: the extent to which students report (1) skipping school or class,
(2) getting sent to the office for doing something wrong, (3) getting
detention, and (4) having their parents called to school about a problem
they are having. The composite is equal to the mean of the four
variables. A value of 1 on the composite indicates infrequent discipline
problems, while a value of 4 indicates frequent discipline problems.

^d^The teacher-based discipline problem composite is based on four
items: the extent to which the teacher reports that the student is
(1) skipping school or class, (2) getting sent to the office for doing
something wrong, (3) getting detention, and (4) having their parents
called to school about a problem they are having. The composite is equal
to the mean of the four variables. A value of 1 on the composite
indicates infrequent discipline problems, while a value of 4 indicates
frequent discipline problems.

^e^Sample sizes may differ for some outcomes due to nonresponse.

\*Significantly different from zero at the .10 level, two-tailed test.

\*\*Significantly different from zero at the .05 level, two-tailed test.

\*\*\*Significantly different from zero at the .01 level, two-tailed
test.

higher for participants, although this evidence is tempered by two
factors.[^11] First, as with the impact on attendance, the impact on
classroom effort was small. For example, one of the items contributing
to the effort composite is whether the student "usually tries hard" in
English class. Teachers were more likely to report that participants
usually tried hard, but the difference is only 3 percentage points (51.3
percent among participants, versus 48.4 percent among comparison
students) and the effect size is only 7 percent. Second, according to
their parents, participants were no more likely to "work hard in
school": about three-fourths of the parents of both groups reported that
their child works hard.

**b. Centers Did Not Affect Classroom Behavior**

According to students and their teachers, participants were equally
likely to skip school, be sent to the office, get a detention for
misbehaving, or have their parents called because of a behavior problem.
Table III.3 shows that participants and comparison group students had
the same value of a composite measure of disciplinary problems.

**c. Centers Increased Grades in Math but Not in Other Subjects**

Centers improved grades in math but not in other subjects (Table III.4).
Participants and comparison students had about the same English,
science, and social studies or history grades (in each case, the average
grade was about 81). In math, however, participants had a marginally
higher grade (80.3 points, compared to 79.5 points). This effect was
statistically significant at the 10 percent level, but the effect size
is only 6 percent.[^12]

Despite reporting slightly higher levels of classroom effort among
participants, teachers were no more likely to report that program
participants performed better than comparison students academically.
Teachers reported that about one-third of each group achieved at an
"above-average" or "very high" level (Table III.4), and 52 percent of
teachers reported that the two groups "get good grades on tests."

### Centers Did Not Improve Behavioral and Youth Development Outcomes

**a. Centers Did Not Improve Social and Personal Development**

In general, centers did not improve developmental outcomes measured by
the evaluation (Table III.5). For example, participants and comparison
students had about the same values of a social engagement composite
variable based on a set of variables that reflect how the students get
along with others and how easily they can make and keep friends
(however, the high value of the variable, 3.5 on a scale of 4, may have
made impacts difficult to generate). In addition, no effect was found on
a peer interaction and empathy composite, which reflects the extent to
which students work well with others, have empathy for others, and
believe the best about others. Centers had a small positive effect on
educational expectations, with 83 percent of participants versus 80
percent of comparison students reporting that they expect to graduate
from college, an effect size of 9 percent.[^13]

**b. Centers Increased Parent Involvement**

Parents of participants were more likely to report that they regularly
participated in school events and school-related activities. For
example, 27 percent of the parents of participants

Table III.4

Impacts on Teacher-Reported Achievement and Grades:

Middle School Centers

+----------------------------+---------+---------+----------+---------+
| Outcome                    | Center  | Com     | E        | p-V     |
|                            | Parti   | parison | stimated | alue^a^ |
|                            | cipants | Group   | Impact   |         |
|                            |         | Members | of       |         |
|                            |         |         | Parti    |         |
|                            |         |         | cipation |         |
+============================+=========+=========+==========+=========+
| Percentage of Students     | 33.6    | 32.9    | 0.7      | .67     |
| Whose Teachers Report That |         |         |          |         |
| They Achieve at an "Above- |         |         |          |         |
| Average" or "Very High"    |         |         |          |         |
| Level                      |         |         |          |         |
+----------------------------+---------+---------+----------+---------+
| Percentage of Students     | 52.2    | 52.2    | 0.0      | .99     |
| Whose Teachers "Agree" or  |         |         |          |         |
| "Strongly Agree" That They |         |         |          |         |
| Get Good Grades on Tests   |         |         |          |         |
+----------------------------+---------+---------+----------+---------+
| Teacher-Reported           | 3.4     | 3.4     | 0.0      | .58     |
| Achievement Composite      |         |         |          |         |
| (Mean)^b^                  |         |         |          |         |
+----------------------------+---------+---------+----------+---------+
| Mean Grade                 |         |         |          |         |
+----------------------------+---------+---------+----------+---------+
| > Math                     | 80.3    | 79.5    | 0.7\*    | .06     |
+----------------------------+---------+---------+----------+---------+
| > English                  | 80.9    | 80.9    | 0.1      | .87     |
+----------------------------+---------+---------+----------+---------+
| > Science                  | 81.3    | 81.1    | 0.1      | .81     |
+----------------------------+---------+---------+----------+---------+
| > Social studies/history   | 81.0    | 80.5    | 0.4      | .33     |
+----------------------------+---------+---------+----------+---------+
| **Sample Size**            | **      | **      |          |         |
|                            | 1,752** | 2,437** |          |         |
+----------------------------+---------+---------+----------+---------+

[Source:]{.smallcaps} Teacher survey; school records.

[Note:]{.smallcaps} The percentages and mean values of outcomes for
program participants and comparison group members have been
regression-adjusted for baseline differences between the groups. The
control variables in the regression included 26 different student and
household characteristics, such as indicators of students' demographic
characteristics, household socioeconomic status, and students' baseline
test scores, attendance, disciplinary problems, and self-reported
grades.

^a^The p-value is the smallest level of significance at which the null
hypothesis that the difference in means between program participants and
control group members equals zero can be rejected. If the p-value is
less than .01, the difference is significant at the 1 percent level. If
the p-value is less than .05, the difference is significant at the 5
percent level, and so on.

^b^The teacher-reported achievement composite is based on teacher
responses to five questions: (1) At what level is this student
performing in reading? (2) Does this student get good grades on tests?
(3) Does this student complete assignments to my satisfaction? (4) Does
this student have good communication skills? (5) Is this student a
proficient reader? Values on these items range from 1 to 5; a value of 1
on the composite indicates low achievement, and a value of 5 indicates
high achievement.

\*Significantly different from zero at the .10 level, two-tailed test.

\*\*Significantly different from zero at the .05 level, two-tailed test.

\*\*\*Significantly different from zero at the .01 level, two-tailed
test.

Table III.5

Impacts on Social Engagement, Educational Expectations, and Parental
Involvement:

Middle School Centers

+---------------------------+---------+---------+---------+---------+
| Outcome                   | Center  | Com     | Es      | p-V     |
|                           | Parti   | parison | timated | alue^a^ |
|                           | cipants | Group   | Impact  |         |
|                           |         | Members | of      |         |
|                           |         |         | Partic  |         |
|                           |         |         | ipation |         |
+---------------------------+---------+---------+---------+---------+
| Student-Reported Social   | 3.5     | 3.5     | 0.0     | .43     |
| Engagement Composite^b^   |         |         |         |         |
| (Mean)                    |         |         |         |         |
+---------------------------+---------+---------+---------+---------+
| Peer Interaction/Empathy  | 3.0     | 3.1     | 0.0     | .12     |
| Composite^c^ (Mean)       |         |         |         |         |
+---------------------------+---------+---------+---------+---------+
| Percentage of Students    | 60.7    | 65.0    | --      | .01     |
| Who Rate Themselves as    |         |         | 4.2\*\* |         |
| "Good" or "Excellent" at  |         |         |         |         |
| Working Out Conflicts     |         |         |         |         |
| with Others               |         |         |         |         |
+---------------------------+---------+---------+---------+---------+
| Percentage of Students    | 36.8    | 34.4    | 2.3     | .18     |
| Who Rate Themselves as    |         |         |         |         |
| "Good" or "Excellent" on  |         |         |         |         |
| Using a Computer to Look  |         |         |         |         |
| Up Information            |         |         |         |         |
+---------------------------+---------+---------+---------+---------+
| Percentage of Students    |         |         |         |         |
| Who Think They Will:      |         |         |         |         |
+---------------------------+---------+---------+---------+---------+
| > Graduate from college   | 82.9    | 79.7    | 3.2\*\* | .01     |
+---------------------------+---------+---------+---------+---------+
| > Graduate from high      | 15.3    | 18.0    | --      | .03     |
| > school but not college  |         |         | 2.7\*\* |         |
+---------------------------+---------+---------+---------+---------+
| > Attend high school but  | 1.9     | 2.3     | --0.5   | .34     |
| > not graduate            |         |         |         |         |
+---------------------------+---------+---------+---------+---------+
| Percentage of Students    |         |         |         |         |
| Whose Parents Did the     |         |         |         |         |
| Following at Least Three  |         |         |         |         |
| Times Last Year           |         |         |         |         |
+---------------------------+---------+---------+---------+---------+
| > Attended an open house  | 27.4    | 19.1    | 8.      | .00     |
| > at the school           |         |         | 4\*\*\* |         |
+---------------------------+---------+---------+---------+---------+
| > Attended parent-teacher | 33.8    | 27.6    | 6.      | .00     |
| > organization meetings   |         |         | 1\*\*\* |         |
+---------------------------+---------+---------+---------+---------+
| > Attended an             | 47.0    | 40.2    | 6.      | .00     |
| > after-school event      |         |         | 8\*\*\* |         |
+---------------------------+---------+---------+---------+---------+
| > Volunteered to help out | 17.8    | 14.5    | 3.3\*\* | .02     |
| > at school               |         |         |         |         |
+---------------------------+---------+---------+---------+---------+
| **Sample Size^d^**        | **      | **      |         |         |
|                           | 1,752** | 2,437** |         |         |
+---------------------------+---------+---------+---------+---------+

[Source:]{.smallcaps} Student survey; teacher survey; parent survey.

[Note:]{.smallcaps} The percentages and mean values of outcomes for
program participants and comparison group members have been
regression-adjusted for baseline differences between the groups. The
control variables in the regression included 26 different student and
household characteristics, such as indicators of students' demographic
characteristics, household socioeconomic status, and students' baseline
test scores, attendance, disciplinary problems, and self-reported
grades. Due to rounding, estimated impacts shown in the table do not
always equal the difference between center participants and the
comparison group.

^a^The p-value is the smallest level of significance at which the null
hypothesis that the difference in means between program participants and
control group members equals zero can be rejected. If the p-value is
less than .01, the difference is significant at the 1 percent level. If
the p-value is less than .05, the difference is significant at the 5
percent level, and so on.

^b^The social engagement composite is based on five items: the extent to
which students report that they (1) have friends to "hang out with," 2)
are never lonely, (3) get along with others their age, (4) find it easy
to make new friends, and (5) never feel left out of things. The
composite is equal to the mean of the five variables. Values on these
items range from 1 to 4; a value of 1 on the composite indicates a low
level of social engagement, and a value of 4 indicates a high level of
engagement.

^c^The peer interaction/empathy composite is based on three items:
students' rating of their ability to (1) work with others in a team or
group, (2) feel bad for other people who are having difficulties, and
(3) believe the best about other people. Values on these items range
from 1 to 4; a value of 1 on the composite indicates poor peer
interactions, while a value of 4 indicates excellent peer interactions.

^d^Sample sizes may differ for some outcomes due to nonresponse.

\*Significantly different from zero at the .10 level, two-tailed test.

\*\*Significantly different from zero at the .05 level, two-tailed test.

\*\*\*Significantly different from zero at the .01 level, two-tailed
test.

attended an open house at the school three or more times during the
year, compared with 19 percent of comparison parents, an effect size of
21 percent (Table III.5). Parents of participants were more likely to
attend parent-teacher organization meetings (34 versus 28 percent,
effect size 14 percent), to attend after-school events (47 versus 40
percent, effect size 14 percent), and to volunteer to help out at school
(18 versus 15 percent, effect size 8 percent).

**c. Centers Did Not Affect Feelings of Safety and Increased Some
Negative Behaviors**

Centers did not affect feelings of safety after school. More than 60
percent of students reported that they felt "very safe" after school,
and only 3 percent reported that they feel "not at all safe" (Table
III.6). The rest of the students reported feeling "somewhat safe."
However, center participants were somewhat more likely to report having
had their property (such as clothing or books) damaged on purpose, with
17 percent of participants and 14 percent off comparison students
reporting having been victimized in this way, an effect size of 7
percent (Table III.6).

Students were asked about different types of negative behavior,
including the extent to which they break things on purpose, punch or hit
others, argue with or lie to their parents, give a teacher a "hard
time," steal from a store, sell illegal drugs, or get arrested by
police.[^14] Participants were more likely than comparison students to
report selling illegal drugs (Table III.6), with 3 percent of
participants and 2 percent of comparison students reporting that they
did this "some" or "a lot" (an effect size of 6 percent). Participants
also had a higher value of the negative behavior composite variable,
which reflects the frequency with which students reported engaging in
these behaviors.

Table III.6

Impact on Student Safety, Negative Behavior, and Victimization: Middle
School Centers

  ------------------------------- -------------- ------------ --------------- ------------
  Outcome                         Center         Comparison   Estimated       p-Value^a^
                                  Participants   Group        Impact of       
                                                              Participation   

  Percentage of Students Who                                                  
  Report Feeling the Following                                                
  Levels of Safety After School                                               
  up Until 6:00                                                               
  [p.m]{.smallcaps}.:                                                         

  Very Safe                       60.5           62.1         --1.6           .35

  Somewhat safe                   36.2           34.7         1.6             .37

  Not at all safe                 3.2            3.2          0.1             .92

  Percentage of Students Who                                                  
  Report That They Do the                                                     
  Following "Some" or "A Lot":                                                

  Break something on purpose      8.8            7.8          1.0             .31

  Punch or hit someone            20.9           18.9         2.1             .15

  Steal from a store              4.8            3.7          1.2             .11

  Sell illegal drugs              3.3            1.8          1.5\*\*\*       .01

  Get arrested or detained by     3.8            3.3          0.5             .48
  police                                                                      

  Negative Behavior Composite^b^  1.53           1.49         0.04\*\*\*      .01
  (Mean)                                                                      

  Percentage of Students Who                                                  
  Report the Following Happened                                               
  to Them "Some" or "A Lot":                                                  

  Been offered, sold, or given an 15.1           13.2         1.9             .13
  illegal drug                                                                

  Been "picked on" after school   32.4           30.7         1.7             .31

  Been threatened or hurt with a  7.7            6.3          1.4             .15
  weapon                                                                      

  Been threatened by a gang or    8.2            8.1          0.1             .88
  gang member                                                                 

  Had property damaged on purpose 16.9           14.1         2.8\*\*         .03

  Percentage of Students Who                                                  
  Report That They Do the                                                     
  Following "Some" or "A Lot":                                                

  Smoke cigarettes                2.6            2.3          0.3             .90

  Have at least one alcoholic     6.6            6.7          --0.1           .90
  drink                                                                       

  Smoke marijuana                 3.7            2.7          1.0\*           .10

  Student-Reported Tobacco,       1.1            1.1          0.0             .68
  Alcohol, and Drug Use                                                       
  Composite^c^ (Mean)                                                         

  **Sample Size^d^**              **1,752**      **2,437**                    
  ------------------------------- -------------- ------------ --------------- ------------

[Source:]{.smallcaps} Student survey; parent survey.

[Note:]{.smallcaps} The percentages and mean values of outcomes for
program participants and comparison group members have been
regression-adjusted for baseline differences between the groups. The
control variables in the regression included 26 different student and
household characteristics, such as indicators of students' demographic
characteristics, household socioeconomic status, and students' baseline
test scores, attendance, disciplinary problems, and self-reported
grades. Due to rounding, estimated impacts shown in the table do not
always equal the difference between center participants and the
comparison group.

^a^The p-value is the smallest level of significance at which the null
hypothesis that the difference in means between program participants and
control group members equals zero can be rejected. If the p-value is
less than .01, the difference is significant at the 1 percent level. If
the p-value is less than .05, the difference is significant at the 5
percent level, and so on.

^b^The negative behavior composite is based on student responses to
eight questions regarding how frequently they do the following:
(1) break something on purpose, (2) punch or hit someone, (3) argue with
their parents, (4) lie to their parents, (5) steal from a store, (6)
give a teacher a hard time, (7) sell illegal drugs, and (8) get arrested
or detained by police. Values on these items range from 1 to 4; a value
of 1 on the composite indicates a low level of negative behavior, while
a value of 4 indicates a high level of negative behavior.

^c^The tobacco, alcohol, and drug use composite is based on seven items:
the extent to which students (1) smoke cigarettes, (2) use smokeless
tobacco, (3) have at least one drink of alcohol, (4) have five or more
drinks of alcohol in a row, (5) smoke marijuana, (6) use inhalants, and
(7) use any other illegal drug. Values on these items range from 1 to 4;
a value of 1 on the composite indicates no substance abuse, while a
value of 4 indicates frequent substance abuse.

^d^Sample sizes may differ for some outcomes due to nonresponse.

\*Significantly different from zero at the .10 level, two-tailed test.

\*\*Significantly different from zero at the .05 level, two-tailed test.

\*\*\*Significantly different from zero at the .01 level, two-tailed
test.

## Center Participation Affected Some Students More than Others

Center participation may affect some students more than others. Effects
were estimated and compared for six subgroups defined by (1) grade
level, (2) race or ethnicity, (3) gender, (4) high versus low baseline
grades, (5) high versus low baseline disciplinary problems, and (6)
single-parent versus two-parent households.[^15]

Findings from the subgroup analysis indicate that participation improved
a range of student outcomes for black students and for students who had
had fewer disciplinary problems in the prior year. Among black students,
for example, centers increased effort in the classroom, reduced lateness
for school, and increased math grades (Table III.7A).[^16] Impacts on
math grades and being on time to class were evident for Hispanic
students. None of these impacts were evident for white students. For
students with fewer behavioral problems (in the baseline year), centers
increased effort in the classroom and math and social studies grades
(Table III.7B). None of these impacts were evident among students that
had had more disciplinary problems. Participation also increased the
extent to which female students were victimized either by being "picked
on" after school or by having their property damaged (Tables III.8A and
III.8B).[^17] Among males, participation did not significantly affect
either of these outcomes. And, although centers increased parental
involvement in school-related activities for nearly all groups of
students, increases in involvement for parents in two-parent families
were larger than for parents in single-parent families. For example,
participation led to a 14 percentage point increase in parents from two-
parent households attending open houses but only a 6 percentage point
increase for single parents.

Table III.7A

Impacts on Homework Completion, Level of Effort, and Classroom Behavior,
by Subgroup:

Middle School Centers

+--------------------------+-----+-----+-----+-----+-----+-----+-----+
|                          | Est |     |     |     |     |     |     |
|                          | ima |     |     |     |     |     |     |
|                          | ted |     |     |     |     |     |     |
|                          | Imp |     |     |     |     |     |     |
|                          | act |     |     |     |     |     |     |
+==========================+=====+=====+=====+=====+=====+=====+=====+
|                          | Gr  |     | Ra  |     |     | Gen |     |
|                          | ade |     | ce/ |     |     | der |     |
|                          | Le  |     | Eth |     |     |     |     |
|                          | vel |     | nic |     |     |     |     |
|                          |     |     | ity |     |     |     |     |
+--------------------------+-----+-----+-----+-----+-----+-----+-----+
| Outcome                  | 5-6 | 7-8 | Wh  | Bl  | Hi  | M   | Fem |
|                          |     |     | ite | ack | spa | ale | ale |
|                          |     |     |     |     | nic |     |     |
+--------------------------+-----+-----+-----+-----+-----+-----+-----+
| Percentage of Students   |     |     |     |     |     |     |     |
| Whose Teachers "Agree"   |     |     |     |     |     |     |     |
| or "Strongly Agree"      |     |     |     |     |     |     |     |
| That:                    |     |     |     |     |     |     |     |
+--------------------------+-----+-----+-----+-----+-----+-----+-----+
| > Student completes      | 2.5 | 5   | 4.  | 7   | 2.5 | 3.9 | 5   |
| > assignments to my      |     | .3\ | 6\* | .3\ |     |     | .0\ |
| > satisfaction           |     | *\* |     | *\* |     |     | *\* |
+--------------------------+-----+-----+-----+-----+-----+-----+-----+
| > Student comes prepared | 1.6 | 5   | 3.5 | 6.  | 5.4 | 5   | 2.9 |
| > and ready to learn     |     | .2\ |     | 2\* |     | .6\ |     |
|                          |     | *\* |     |     |     | *\* |     |
+--------------------------+-----+-----+-----+-----+-----+-----+-----+
| Percentage of Students   |     |     |     |     |     |     |     |
| Whose Teachers "Agree"   |     |     |     |     |     |     |     |
| or "Strongly Agree"      |     |     |     |     |     |     |     |
| That:                    |     |     |     |     |     |     |     |
+--------------------------+-----+-----+-----+-----+-----+-----+-----+
| > The student is         | 0.9 | 3.  | *   | **  | *   | 4.2 | 1.8 |
| > attentive in class     |     | 8\* | *1. | 11. | *1. |     |     |
|                          |     |     | 3** | 0\* | 6** |     |     |
|                          |     |     |     | \*\ |     |     |     |
|                          |     |     |     | *** |     |     |     |
+--------------------------+-----+-----+-----+-----+-----+-----+-----+
| > The student            | 4.8 | 4   | 2.4 | 7   | 5.0 | 4.  | 4.  |
| > participates in class  |     | .5\ |     | .4\ |     | 5\* | 4\* |
|                          |     | *\* |     | *\* |     |     |     |
+--------------------------+-----+-----+-----+-----+-----+-----+-----+
| Teacher-Reported Level   | 0.0 | 0.1 | *   | *   | *   | 0.2 | 0.1 |
| of Effort Composite      |     | \*\ | *0. | *0. | *0. | \*\ |     |
| (Mean)                   |     | *\* | 0** | 2\* | 1** | *\* |     |
|                          |     |     |     | \*\ |     |     |     |
|                          |     |     |     | *** |     |     |     |
+--------------------------+-----+-----+-----+-----+-----+-----+-----+
| Teacher-Reported         | **0 | **  | 0.0 | --  | --  | --  | 0.0 |
| Disciplinary Problems    | .1\ | --0 |     | 0.1 | 0.0 | 0.0 |     |
| Composite (Mean)         | *** | .1\ |     |     |     |     |     |
|                          |     | *** |     |     |     |     |     |
+--------------------------+-----+-----+-----+-----+-----+-----+-----+
| Number of Days of School |     |     |     |     |     |     |     |
| Student Was:             |     |     |     |     |     |     |     |
+--------------------------+-----+-----+-----+-----+-----+-----+-----+
| > Absent from class      | **- | *   | --  | --1 | --1 | --0 | --  |
|                          | -2. | *-- | 0.7 | .6\ | .3\ | .9\ | 1.3 |
|                          | 0\* | 0.8 |     | *\* | *\* | *\* | \*\ |
|                          | \*\ | \*\ |     |     |     |     | *\* |
|                          | *** | *** |     |     |     |     |     |
+--------------------------+-----+-----+-----+-----+-----+-----+-----+
| > Late for class         | --  | --  | *   | **- | *   | **- | **- |
|                          | 0.7 | 1.3 | *0. | -1. | *-- | -1. | -0. |
|                          |     | \*\ | 1** | 8\* | 1.6 | 8\* | 6** |
|                          |     | *\* |     | \*\ | \*\ | \*\ |     |
|                          |     |     |     | *** | *** | *** |     |
+--------------------------+-----+-----+-----+-----+-----+-----+-----+
| Percentage of Students   | --  | 1.7 | --  | 2.1 | 2.6 | **4 | **- |
| Whose Teachers Report    | 3.4 |     | 2.4 |     |     | .4\ | -3. |
| That They Achieve at an  |     |     |     |     |     | *** | 1** |
| "Above-Average" or "Very |     |     |     |     |     |     |     |
| High" Level              |     |     |     |     |     |     |     |
+--------------------------+-----+-----+-----+-----+-----+-----+-----+
| Mean Grades              |     |     |     |     |     |     |     |
+--------------------------+-----+-----+-----+-----+-----+-----+-----+
| > Math                   | 1.  | 1.6 | **- | **  | **1 | 0.6 | 0.  |
|                          | 2\* |     | -0. | 1.7 | .5\ |     | 9\* |
|                          |     |     | 3** | \*\ | *** |     |     |
|                          |     |     |     | *** |     |     |     |
+--------------------------+-----+-----+-----+-----+-----+-----+-----+
| > English                | 0.7 | 0.3 | --  | 0.8 | 1.0 | 0.0 | 0.4 |
|                          |     |     | 0.2 |     |     |     |     |
+--------------------------+-----+-----+-----+-----+-----+-----+-----+
| > Science                | 1.0 | 0.0 | 0.2 | 0.4 | 0.4 | --  | 0.8 |
|                          |     |     |     |     |     | 0.5 |     |
+--------------------------+-----+-----+-----+-----+-----+-----+-----+
| > Social studies         | *   | *   | --  | 0.7 | 1.0 | 0.2 | 0.6 |
|                          | *2. | *0. | 0.3 |     |     |     |     |
|                          | 2\* | 1** |     |     |     |     |     |
|                          | \*\ |     |     |     |     |     |     |
|                          | *** |     |     |     |     |     |     |
+--------------------------+-----+-----+-----+-----+-----+-----+-----+
| Sample Size              |     |     |     |     |     |     |     |
+--------------------------+-----+-----+-----+-----+-----+-----+-----+
| > Student-reported       | 1,  | 2,  | 1,  | 940 | 1,  | 1,  | 2,  |
| > outcomes               | 138 | 893 | 391 |     | 091 | 868 | 156 |
+--------------------------+-----+-----+-----+-----+-----+-----+-----+
| > Parent-reported        | 987 | 2,  | 1,  | 819 | 937 | 1,  | 1,  |
| > outcomes               |     | 588 | 304 |     |     | 657 | 916 |
+--------------------------+-----+-----+-----+-----+-----+-----+-----+
| > Teacher-reported       | 939 | 2,  | 1,  | 721 | 884 | 1,  | 1,  |
| > outcomes               |     | 238 | 152 |     |     | 494 | 679 |
+--------------------------+-----+-----+-----+-----+-----+-----+-----+
| > School records         | 1,  | 2,  | 1,  | 939 | 1,  | 1,  | 2,  |
| > outcomes (attendance)  | 095 | 818 | 345 |     | 067 | 834 | 078 |
+--------------------------+-----+-----+-----+-----+-----+-----+-----+
| > School records         | 889 | 2,  | 1,  | 890 | 935 | 1,  | 1,  |
| > outcomes (grades)      |     | 685 | 214 |     |     | 702 | 871 |
+--------------------------+-----+-----+-----+-----+-----+-----+-----+

[Source:]{.smallcaps} Parent survey; student survey; teacher survey;
school records.

[Note:]{.smallcaps} Subgroup impacts in bold indicate that the estimated
impact for one subgroup differed significantly from the estimated
subgroup impact for the other subgroup(s) at the .10 level or higher.

\*Significantly different from zero at the .10 level, two-tailed test.

\*\*Significantly different from zero at the .05 level, two-tailed test.

\*\*\*Significantly different from zero at the .01 level, two-tailed
test.

# Table III.7B

Impacts on Homework Completion, Level of Effort, and Classroom Behavior,
by Subgroup:

Middle School Centers

+--------------------------+------+------+------+------+------+------+
|                          | E    |      |      |      |      |      |
|                          | stim |      |      |      |      |      |
|                          | ated |      |      |      |      |      |
|                          | Im   |      |      |      |      |      |
|                          | pact |      |      |      |      |      |
+==========================+======+======+======+======+======+======+
|                          | Base |      | Base |      | Nu   |      |
|                          | line |      | line |      | mber |      |
|                          | Gr   |      | Disc |      | of   |      |
|                          | ades |      | ipli |      | Par  |      |
|                          |      |      | nary |      | ents |      |
|                          |      |      | Prob |      | in   |      |
|                          |      |      | lems |      | H    |      |
|                          |      |      | C    |      | ouse |      |
|                          |      |      | ompo |      | hold |      |
|                          |      |      | site |      |      |      |
+--------------------------+------+------+------+------+------+------+
| Outcome                  | Low  | High | Low  | High | Two  | One  |
+--------------------------+------+------+------+------+------+------+
| Percentage of Students   |      |      |      |      |      |      |
| Whose Teachers "Agree"   |      |      |      |      |      |      |
| or "Strongly Agree"      |      |      |      |      |      |      |
| That:                    |      |      |      |      |      |      |
+--------------------------+------+------+------+------+------+------+
| > Student completes      | 6.1  | 4.5  | *    | *    | 6.1  | 5.1  |
| > assignments to my      | \*\* | \*\* | *7.1 | *--0 | \*\* | \*\* |
| > satisfaction           |      |      | \*\* | .2** |      |      |
|                          |      |      | \*** |      |      |      |
+--------------------------+------+------+------+------+------+------+
| > Student comes prepared | 4.1  | 4.6  | *    | *    | 3.8  | 4    |
| > and ready to learn     |      | \*\* | *7.6 | *--2 |      | .0\* |
|                          |      |      | \*\* | .7** |      |      |
|                          |      |      | \*** |      |      |      |
+--------------------------+------+------+------+------+------+------+
| Percentage of Students   |      |      |      |      |      |      |
| Whose Teachers "Agree"   |      |      |      |      |      |      |
| or "Strongly Agree"      |      |      |      |      |      |      |
| That:                    |      |      |      |      |      |      |
+--------------------------+------+------+------+------+------+------+
| > The student is         | 6    | 2.3  | 4.9  | -    | 4.5  | 3.3  |
| > attentive in class     | .0\* |      | \*\* | -0.2 |      |      |
+--------------------------+------+------+------+------+------+------+
| > The student            | 8.3  | 3    | 5    | 1.4  | 3.9  | 6    |
| > participates in class  | \*\* | .5\* | .9\* |      |      | .9\* |
|                          |      |      | \*\* |      |      | \*\* |
+--------------------------+------+------+------+------+------+------+
| Teacher-Reported Level   | 0    | 0    | 0    | 0.0  | 0    | 0.1  |
| of Effort Composite      | .2\* | .1\* | .1\* |      | .1\* | \*\* |
| (Mean)                   | \*\* |      | \*\* |      |      |      |
+--------------------------+------+------+------+------+------+------+
| Teacher-Reported         | 0.0  | 0.0  | 0.0  | 0.0  | 0.0  | 0.0  |
| Disciplinary Problems    |      |      |      |      |      |      |
| Composite (Mean)         |      |      |      |      |      |      |
+--------------------------+------+------+------+------+------+------+
| Mean Number of Days      |      |      |      |      |      |      |
| Student Was:             |      |      |      |      |      |      |
+--------------------------+------+------+------+------+------+------+
| > Absent from class      | --1  | --1  | --0  | -    | --0  | -    |
|                          | .1\* | .1\* | .9\* | -1.3 | .9\* | -0.4 |
|                          |      | \*\* | \*\* | \*\* |      |      |
+--------------------------+------+------+------+------+------+------+
| > Late for class         | -    | -    | --0  | -    | -    | -    |
|                          | -1.6 | -0.9 | .9\* | -1.5 | -0.3 | -1.1 |
|                          | \*\* |      | \*\* | \*\* |      |      |
+--------------------------+------+------+------+------+------+------+
| Percentage of Students   | *    | *    | 0.7  | 0.8  | 0.9  | 0.37 |
| Whose Teachers Report    | *6.1 | *--1 |      |      |      |      |
| That They Achieve at an  | \*\* | .4** |      |      |      |      |
| "Above-Average" or "Very | \*** |      |      |      |      |      |
| High" Level              |      |      |      |      |      |      |
+--------------------------+------+------+------+------+------+------+
| Mean Grades              |      |      |      |      |      |      |
+--------------------------+------+------+------+------+------+------+
| > Math                   | 0.1  | 1.0  | *    | *    | 1.0  | 0.4  |
|                          |      | \*\* | *1.4 | *--0 |      |      |
|                          |      |      | \*\* | .3** |      |      |
|                          |      |      | \*** |      |      |      |
+--------------------------+------+------+------+------+------+------+
| > English                | 0.5  | 0.3  | 0.5  | -    | **1  | *    |
|                          |      |      |      | -0.4 | .6\* | *--0 |
|                          |      |      |      |      | \*** | .3** |
+--------------------------+------+------+------+------+------+------+
| > Science                | **-  | **1  | **1  | *    | 1.1  | -    |
|                          | -1.5 | .0** | .0** | *--1 |      | -0.5 |
|                          | \*** |      |      | .1** |      |      |
+--------------------------+------+------+------+------+------+------+
| > Social studies         | 0.4  | 0.6  | 0    | -    | 1.0  | -    |
|                          |      |      | .9\* | -0.4 |      | -0.1 |
+--------------------------+------+------+------+------+------+------+
| Sample Size              |      |      |      |      |      |      |
+--------------------------+------+------+------+------+------+------+
| > Student-reported       | 1    | 2    | 2    | 1    | 1    | 2    |
| > outcomes               | ,219 | ,729 | ,613 | ,307 | ,248 | ,045 |
+--------------------------+------+------+------+------+------+------+
| > Parent-reported        | 1    | 2    | 2    | 1    | 1    | 2    |
| > outcomes               | ,066 | ,427 | ,343 | ,137 | ,470 | ,297 |
+--------------------------+------+------+------+------+------+------+
| > Teacher-reported       | 943  | 2    | 2    | 1    | 1    | 1    |
| > outcomes               |      | ,154 | ,090 | ,010 | ,044 | ,664 |
+--------------------------+------+------+------+------+------+------+
| > School records         | 1    | 2    | 2    | 1    | 1    | 2    |
| > outcomes (attendance)  | ,184 | ,641 | ,508 | ,301 | ,266 | ,040 |
+--------------------------+------+------+------+------+------+------+
| > School records         | 1    | 2    | 2    | 1    | 1    | 1    |
| > outcomes (grades)      | ,110 | ,382 | ,276 | ,205 | ,162 | ,858 |
+--------------------------+------+------+------+------+------+------+

[Source:]{.smallcaps} Parent survey; student followup survey; teacher
survey; school records.

[Note:]{.smallcaps} Subgroup impacts reported in bold indicate that the
estimated impact for one subgroup differed significantly from the
estimated subgroup impact for the other related subgroup(s) at the .10
level or higher.

\*Significantly different from zero at the .10 level, two-tailed test.

\*\*Significantly different from zero at the .05 level, two-tailed test.

\*\*\*Significantly different from zero at the .01 level, two-tailed
test.

# Table III.8A

Impacts on Other Student and Parent Outcomes, by Subgroup:

Middle School Centers

+--------------------------+-----+-----+-----+-----+-----+-----+-----+
|                          | Est |     |     |     |     |     |     |
|                          | ima |     |     |     |     |     |     |
|                          | ted |     |     |     |     |     |     |
|                          | Imp |     |     |     |     |     |     |
|                          | act |     |     |     |     |     |     |
+==========================+=====+=====+=====+=====+=====+=====+=====+
|                          | Gr  |     | Ra  |     |     | Gen |     |
|                          | ade |     | ce/ |     |     | der |     |
|                          | Le  |     | Eth |     |     |     |     |
|                          | vel |     | nic |     |     |     |     |
|                          |     |     | ity |     |     |     |     |
+--------------------------+-----+-----+-----+-----+-----+-----+-----+
| Outcome                  | 5-6 | 7-8 | Wh  | Bl  | Hi  | M   | Fem |
|                          |     |     | ite | ack | spa | ale | ale |
|                          |     |     |     |     | nic |     |     |
+--------------------------+-----+-----+-----+-----+-----+-----+-----+
| Student-Reported         | 0   | 0.  | 0   | 0.  | 0   | 0.  | 0.0 |
| Delinquent Behavior      | .04 | 04\ | .06 | 07\ | .00 | 05\ | 3\* |
| Composite (Mean)         |     | *\* | \*\ | *\* |     | *\* |     |
|                          |     |     | *\* |     |     |     |     |
+--------------------------+-----+-----+-----+-----+-----+-----+-----+
| Percentage of Students   |     |     |     |     |     |     |     |
| Whose Parents Did the    |     |     |     |     |     |     |     |
| Following at Least Three |     |     |     |     |     |     |     |
| Times Last Year:         |     |     |     |     |     |     |     |
+--------------------------+-----+-----+-----+-----+-----+-----+-----+
| > Attended an open house | 7.9 | 8.8 | 8.2 | 1   | 7   | 8.6 | 8.6 |
| > at school              | \*\ | \*\ | \*\ | 1.1 | .7\ | \*\ | \*\ |
|                          | *\* | *\* | *\* | \*\ | *\* | *\* | *\* |
|                          |     |     |     | *\* |     |     |     |
+--------------------------+-----+-----+-----+-----+-----+-----+-----+
| > Attended a             | 4.6 | 6.5 | 4.6 | 3.4 | 9   | 6.8 | 5   |
| > parent-teacher         |     | \*\ |     |     | .1\ | \*\ | .5\ |
| > organization meeting   |     | *\* |     |     | *\* | *\* | *\* |
+--------------------------+-----+-----+-----+-----+-----+-----+-----+
| > Attended an            | 8   | 6.8 | 5.  | 7.  | 6   | 7.8 | 6   |
| > after-school event     | .0\ | \*\ | 9\* | 3\* | .7\ | \*\ | .4\ |
|                          | *\* | *\* |     |     | *\* | *\* | *\* |
+--------------------------+-----+-----+-----+-----+-----+-----+-----+
| > Volunteered to help    | --  | 4.4 | 3.2 | 2.0 | 3.0 | 2.1 | 4   |
| > out at school          | 0.7 | \*\ |     |     |     |     | .3\ |
|                          |     | *\* |     |     |     |     | *\* |
+--------------------------+-----+-----+-----+-----+-----+-----+-----+
| Percentage of Students   |     |     |     |     |     |     |     |
| Who Report the Following |     |     |     |     |     |     |     |
|                          |     |     |     |     |     |     |     |
| Happened to Them "Some"  |     |     |     |     |     |     |     |
| or "A Lot"               |     |     |     |     |     |     |     |
+--------------------------+-----+-----+-----+-----+-----+-----+-----+
| > Been offered, sold, or | 0.9 | 1.9 | *   | **  | **- | 1.4 | 1.9 |
| > given an illegal drug  |     |     | *1. | 5.8 | -0. |     |     |
|                          |     |     | 4** | \*\ | 7** |     |     |
|                          |     |     |     | *** |     |     |     |
+--------------------------+-----+-----+-----+-----+-----+-----+-----+
| > Been "picked on" after | --  | 2.5 | 3.3 | 5.1 | 0.1 | **- | **  |
| > school                 | 0.2 |     |     |     |     | -1. | 4.7 |
|                          |     |     |     |     |     | 2** | \*\ |
|                          |     |     |     |     |     |     | *** |
+--------------------------+-----+-----+-----+-----+-----+-----+-----+
| > Been threatened or     | 1.0 | 1.3 | 1.6 | 0.6 | --  | 1.1 | 1.4 |
| > hurt with a weapon     |     |     |     |     | 0.2 |     |     |
+--------------------------+-----+-----+-----+-----+-----+-----+-----+
| > Been threatened by a   | --  | 0.7 | 1.9 | --  | 0.2 | 0.7 | --  |
| > gang member            | 2.3 |     |     | 1.0 |     |     | 0.6 |
+--------------------------+-----+-----+-----+-----+-----+-----+-----+
| > Had property damaged   | 0.4 | 3   | 3.  | 3.9 | 0.5 | **- | *   |
| > on purpose             |     | .1\ | 8\* |     |     | -0. | *4. |
|                          |     | *\* |     |     |     | 2** | 9\* |
|                          |     |     |     |     |     |     | \*\ |
|                          |     |     |     |     |     |     | *** |
+--------------------------+-----+-----+-----+-----+-----+-----+-----+
| Student-Reported         | --0 | 0   | **  | **  | *   | 0   | 0   |
| Tobacco, Alcohol, Drug   | .01 | .01 | 0.0 | 0.0 | *0. | .00 | .01 |
| Use Composite (Mean)     |     |     | 2** | 1** | 0** |     |     |
+--------------------------+-----+-----+-----+-----+-----+-----+-----+
| Sample Size              |     |     |     |     |     |     |     |
+--------------------------+-----+-----+-----+-----+-----+-----+-----+
| > Student-reported       | 1,  | 2,  | 1,  | 940 | 1,  | 1,  | 2,  |
| > outcomes               | 138 | 893 | 391 |     | 091 | 868 | 156 |
+--------------------------+-----+-----+-----+-----+-----+-----+-----+
| > Parent-reported        | 987 | 2,  | 1,  | 819 | 937 | 1,  | 1,  |
| > outcomes               |     | 588 | 304 |     |     | 657 | 916 |
+--------------------------+-----+-----+-----+-----+-----+-----+-----+

[Source:]{.smallcaps} Parent survey; student followup survey.

[Note:]{.smallcaps} Subgroup impacts reported in bold indicate that the
estimated impact for one subgroup differed significantly from the
estimated subgroup impact for the other related subgroup(s) at the .10
level or higher.

\*Significantly different from zero at the .10 level, two-tailed test.

\*\*Significantly different from zero at the .05 level, two-tailed test.

\*\*\*Significantly different from zero at the .01 level, two-tailed
test.

# Table III.8B

Impacts on Other Student and Parent Outcomes, by Subgroup:

Middle School Centers

+--------------------------+------+------+------+------+------+------+
|                          | E    |      |      |      |      |      |
|                          | stim |      |      |      |      |      |
|                          | ated |      |      |      |      |      |
|                          | Im   |      |      |      |      |      |
|                          | pact |      |      |      |      |      |
+==========================+======+======+======+======+======+======+
|                          | Base |      | Base |      | Nu   |      |
|                          | line |      | line |      | mber |      |
|                          | Gr   |      | Disc |      | of   |      |
|                          | ades |      | ipli |      | Par  |      |
|                          |      |      | nary |      | ents |      |
|                          |      |      | Prob |      | in   |      |
|                          |      |      | lems |      | H    |      |
|                          |      |      | C    |      | ouse |      |
|                          |      |      | ompo |      | hold |      |
|                          |      |      | site |      |      |      |
+--------------------------+------+------+------+------+------+------+
| Outcome                  | Low  | High | Low  | High | Two  | One  |
+--------------------------+------+------+------+------+------+------+
| Student-Reported         | 0.04 | 0.04 | 0.03 | 0.   | 0.   | 0.03 |
| Delinquent Behavior      |      | \*\* | \*\* | 06\* | 05\* |      |
| Composite (Mean)         |      |      |      |      |      |      |
+--------------------------+------+------+------+------+------+------+
| Percentage of Students   |      |      |      |      |      |      |
| Whose Parents Did the    |      |      |      |      |      |      |
| Following at Least Three |      |      |      |      |      |      |
| Times Last Year:         |      |      |      |      |      |      |
+--------------------------+------+------+------+------+------+------+
| > Attended an open house | 8    | 8    | 8    | 8    | **   | *    |
| > at school              | .9\* | .5\* | .9\* | .5\* | 13.5 | *5.5 |
|                          | \*\* | \*\* | \*\* | \*\* | \*\* | \*\* |
|                          |      |      |      |      | \*** | \*** |
+--------------------------+------+------+------+------+------+------+
| > Attended a             | 7.6  | 5    | *    | **0  | 7    | 5.0  |
| > parent-teacher         | \*\* | .6\* | *7.4 | .8** | .5\* | \*\* |
| > organization meeting   |      | \*\* | \*\* |      | \*\* |      |
|                          |      |      | \*** |      |      |      |
+--------------------------+------+------+------+------+------+------+
| > Attended an            | 9    | 5.1  | 5.6  | 9    | 10   | 4.5  |
| > after-school event     | .6\* | \*\* | \*\* | .2\* | .2\* | \*\* |
|                          | \*\* |      |      | \*\* | \*\* |      |
+--------------------------+------+------+------+------+------+------+
| > Volunteered to help    | 6    | 1.9  | 3    | 3.4  | 1.7  | 4.4  |
| > out at school          | .7\* |      | .2\* |      |      | \*\* |
|                          | \*\* |      |      |      |      |      |
+--------------------------+------+------+------+------+------+------+
| Percentage of Students   |      |      |      |      |      |      |
| Who Report the Following |      |      |      |      |      |      |
|                          |      |      |      |      |      |      |
| Happened to Them "Some"  |      |      |      |      |      |      |
| or "A Lot":              |      |      |      |      |      |      |
+--------------------------+------+------+------+------+------+------+
| > Been offered, sold, or | 1.9  | 1.0  | *    | **5  | 1.2  | 0.7  |
| > given an illegal drug  |      |      | *--0 | .6\* |      |      |
|                          |      |      | .6** | \*** |      |      |
+--------------------------+------+------+------+------+------+------+
| > Been "picked on" after | 1.7  | 1.7  | 1.7  | 0.8  | 4.0  | -    |
| > school                 |      |      |      |      |      | -0.4 |
+--------------------------+------+------+------+------+------+------+
| > Been threatened or     | **-  | *    | 0.8  | 0.6  | 1.1  | 0.9  |
| > hurt with a weapon     | -3.1 | *3.2 |      |      |      |      |
|                          | \*** | \*\* |      |      |      |      |
|                          |      | \*** |      |      |      |      |
+--------------------------+------+------+------+------+------+------+
| > Been threatened by a   | -    | -    | -    | 0.8  | -    | -    |
| > gang member            | -1.1 | -0.1 | -1.4 |      | -1.7 | -0.4 |
+--------------------------+------+------+------+------+------+------+
| > Had property damaged   | 1.6  | 2    | 1.6  | 2.4  | 2.0  | 0.7  |
| > on purpose             |      | .6\* |      |      |      |      |
+--------------------------+------+------+------+------+------+------+
| Student-Reported         | **   | **0. | 0.00 | 0.01 | 0.00 | 0.00 |
| Tobacco, Alcohol, Drug   | --0. | 01** |      |      |      |      |
| Use Composite (Mean)     | 02** |      |      |      |      |      |
+--------------------------+------+------+------+------+------+------+
| Sample Size              |      |      |      |      |      |      |
+--------------------------+------+------+------+------+------+------+
| > Student-reported       | 1    | 2    | 2    | 1    | 1    | 2    |
| > outcomes               | ,219 | ,729 | ,613 | ,307 | ,248 | ,045 |
+--------------------------+------+------+------+------+------+------+
| > Parent-reported        | 1    | 2    | 2    | 1    | 1    | 2    |
| > outcomes               | ,066 | ,427 | ,343 | ,137 | ,470 | ,297 |
+--------------------------+------+------+------+------+------+------+

[Source:]{.smallcaps} Parent survey; student followup survey.

[Note:]{.smallcaps} Subgroup impacts reported in bold indicate that the
estimated impact for one subgroup differed significantly from the
estimated subgroup impact for the other related subgroup(s) at the .10
level or higher.

\*Significantly different from zero at the .10 level, two--tailed test.

\*\*Significantly different from zero at the .05 level, two--tailed
test.

\*\*\*Significantly different from zero at the .01 level, two--tailed
test.

## D. Programs with More Academic Focus Did Not Have Larger Effects

We estimated impacts separately for grantees based on a variety of
characteristics, including proxies for their academic focus, the extent
to which project directors felt that getting support from school day
teachers was a challenge, the extent to which project directors felt
retaining staff was a challenge, the extent to which they felt that
getting support from the community was a challenge, whether grantees
were from the first, second, or third cohort, and whether comparison
group students were drawn from the school where the centers operated or
from other schools. Few patterns emerged from the analysis.

In particular, we estimated impacts separately by grantee to assess
whether more academic programs had larger effects. A center's academic
focus was assessed in two ways. First, program directors reported
whether academic enrichment was a major objective of the program.
Second, centers were considered to have an academic focus according to
the extent to which they offered math classes to students.[^18] These
are rough proxies for academic focus, but more detailed data about
program academic activities are not available. No patterns emerged
between the two measures of academic emphasis and impacts (Table III.9).
Only two impacts were statistically significant at the 10 percent level,
and the differences indicate that English and science grades increased
more when grantees did not have academic improvement as a major
objective. However, that finding is not supported when grantees'
academic focus is assessed by whether they provide math classes. The
small number of significant differences and the

# Table III.9

Impacts by Grantee Characteristics:

Middle School Grantees

+--------------------------------+-------+-------+---+--------+--------+
|                                | Gr    |       |   |        |        |
|                                | antee |       |   |        |        |
|                                | Char  |       |   |        |        |
|                                | acter |       |   |        |        |
|                                | istic |       |   |        |        |
+================================+=======+=======+===+========+========+
|                                | Gran  |       |   | W      |        |
|                                | tee's |       |   | hether |        |
|                                | Asses |       |   | Math   |        |
|                                | sment |       |   | C      |        |
|                                | of    |       |   | lasses |        |
|                                | Wh    |       |   | Were   |        |
|                                | ether |       |   | O      |        |
|                                | Aca   |       |   | ffered |        |
|                                | demic |       |   | Freq   |        |
|                                | Enric |       |   | uently |        |
|                                | hment |       |   |        |        |
|                                | Is    |       |   |        |        |
|                                | Major |       |   |        |        |
|                                | Obje  |       |   |        |        |
|                                | ctive |       |   |        |        |
+--------------------------------+-------+-------+---+--------+--------+
| Outcome                        | Major | Not   |   | Fr     | Infr   |
|                                | Obje  | Major |   | equent | equent |
|                                | ctive | Obje  |   | Math   | Math   |
|                                |       | ctive |   | C      | C      |
|                                |       |       |   | lasses | lasses |
+--------------------------------+-------+-------+---+--------+--------+
| Teacher-Reported Level of      | 0.1   | 0.1   |   | 0.1    | 0.1    |
| Effort Composite (Mean)        |       |       |   |        |        |
+--------------------------------+-------+-------+---+--------+--------+
| Mean Class Grade               |       |       |   |        |        |
+--------------------------------+-------+-------+---+--------+--------+
| > Math                         | --0.2 | 0.9   |   | 0.9    | --0.3  |
+--------------------------------+-------+-------+---+--------+--------+
| > English                      | --0.4 | 1.    |   | 0.4    | --0.1  |
|                                |       | 1\*\* |   |        |        |
+--------------------------------+-------+-------+---+--------+--------+
| > Science                      | --0.5 | 1.1\* |   | 0.8    | --0.5  |
+--------------------------------+-------+-------+---+--------+--------+
| > Social studies               | --0.0 | 1.2   |   | 1.1    | --0.1  |
+--------------------------------+-------+-------+---+--------+--------+
| Percentage of Students Whose   | 5.0   | 6.0   |   | 9.0    | 3.0    |
| Parents Attended a             |       |       |   |        |        |
| Parent-Teacher Organization    |       |       |   |        |        |
| Meeting at Least Three Times   |       |       |   |        |        |
| Last Year                      |       |       |   |        |        |
+--------------------------------+-------+-------+---+--------+--------+
| Student-Reported Negative      | 0.02  | 0.05  |   | 0.01   | 0.05   |
| Behavior Composite (Mean)      |       |       |   |        |        |
+--------------------------------+-------+-------+---+--------+--------+
| **Number of Grantees^a^**      | *     | *     |   | **13** | **19** |
|                                | *19** | *13** |   |        |        |
+--------------------------------+-------+-------+---+--------+--------+

[Source:]{.smallcaps} Parent survey; student followup survey; teacher
survey; school records; site visitor assessments.

^a^Not all outcomes were available for all grantees, so sample sizes
vary.

\* The difference is statistically significant at the .10 level,
two-tailed test.

\*\* The difference is statistically significant at the .05 level,
two-tailed test.

inconsistency of the differences suggests that any relationship between
the academic proxies and impacts is weak or nonexistent.

## E. Attendance Was Not Related to Most Outcomes

Many students attended centers sporadically but some attended more
regularly. If impacts of centers are related to attendance, it may be
possible to use attendance data to estimate a relationship between
impacts and attendance. This relationship would provide insights into
the kinds of students most likely to benefit from center participation
and the potential gains from efforts to improve attendance. We looked at
the relationship in two ways, first at the level of the program and then
at the level of the individual student.

At the program level, we divided grantees into three attendance
categories to assess whether higher-attendance programs showed larger
effects. The average participant attended fewer than 20 days during the
school year for grantees that were considered "low-attendance," 20 to 40
days for medium-attendance grantees, and more than 40 days for
high-attendance grantees. No relationship is evident between average
attendance and impacts (Table III.10). The only impact that attendance
appeared to influence was parental involvement. High-attendance grantees
had a larger impact (14.6 percentage points) than low- and
medium-attendance grantees (0 to 4 points) on attendance at
parent-teacher organization meetings.

Measurement issues are more complex at the individual student level. The
relationship between impacts and student attendance (the "dosage"
effect) is difficult to measure correctly because students and parents
can choose their dosage and the factors affecting their choice of dosage
(such as whether a student likes to be at school) may also affect
outcomes. An ideal scheme for measuring the "dosage" effect would be to
assign participants randomly to various dosages (for example, short,
medium, and long), which would ensure that the groups experiencing the
different dosages were statistically equivalent at the outset. However,
for

# Table III.10

Impacts by Average Attendance:

Grantee-Level School

+--------------------------+-------------+-------------+---------------+
|                          | Estimated   |             |               |
|                          | Impact      |             |               |
+==========================+=============+=============+===============+
|                          | Average     |             |               |
|                          | Grantee     |             |               |
|                          | Attendance  |             |               |
+--------------------------+-------------+-------------+---------------+
| Outcome                  | Low         | Medium      | High          |
|                          |             |             |               |
|                          | (Less than  | (20 to 40   | (More than 40 |
|                          | 20 days)    | days)       | days)         |
+--------------------------+-------------+-------------+---------------+
| Teacher--Reported Level  | 0.13        | 0.06        | 0.18          |
| of Effort Composite      |             |             |               |
| (Mean)                   |             |             |               |
+--------------------------+-------------+-------------+---------------+
| Mean Class Grade         |             |             |               |
+--------------------------+-------------+-------------+---------------+
| > Math                   | --0.1       | 0.2         | 0.1           |
+--------------------------+-------------+-------------+---------------+
| > English                | 1.5         | 0.3         | --0.9         |
+--------------------------+-------------+-------------+---------------+
| > Science                | 0.1         | 0.0         | --0.2         |
+--------------------------+-------------+-------------+---------------+
| > Social Studies         | 0.8         | 0.3         | 0.1           |
+--------------------------+-------------+-------------+---------------+
| Percentage of Students   | 4.3\*\*     | --0.4\*\*   | 14.6\*\*      |
| Whose Parents Attended a |             |             |               |
| Parent-Teacher           |             |             |               |
| Organization Meeting At  |             |             |               |
| Least Three Times Last   |             |             |               |
| Year                     |             |             |               |
+--------------------------+-------------+-------------+---------------+
| Student-Reported         | 0.01        | 0.05        | 0.03          |
| Delinquent Behavior      |             |             |               |
| Composite (Mean)         |             |             |               |
+--------------------------+-------------+-------------+---------------+
| **Number of Grantees**   | **12**      | **11**      | **9**         |
+--------------------------+-------------+-------------+---------------+

[Source:]{.smallcaps} Parent survey, student followup survey, teacher
survey, school records, program attendance records.

\*\*The differences are jointly statistically significant at the .05
level, two-tailed test.

practical reasons it is not possible to assign participants to center
dosages. Instead, participants (implicitly) choose their own dosage, and
those choosing large dosages could differ systematically from those
choosing small ones. Statistical methods can adjust for observed
characteristics that differ between the groups, but unobserved
characteristics (such as motivation) also may affect outcomes. Outcome
differences between low- and high-dose participants therefore do not
have a strict causal interpretation as the effect of the dosage
difference.[^19]

Comparisons of the characteristics of participants who attended centers
frequently (in the top third of days attended) and participants who
attended them infrequently (in the bottom third of days attended)
confirm that the groups differed (Table III.11).[^20] Participants who
attended frequently were more often black (37 percent, compared to 20
percent) and living in single-parent households (33 percent, compared to
27 percent). They also had lower average household incomes and higher
rates of public assistance receipt. However, mothers of frequent
participants were less likely to have dropped out of high school.

Under the assumption that students who are more motivated attend centers
more often, outcome differences can provide some information about
dosage effects. For example, under the assumption that those who
frequently attend are more motivated, negligible outcome differences
suggest that centers are having no dosage effect on outcomes (or
affecting them negatively), and negative outcome differences suggest
that dosage may be reducing outcomes. Positive outcome

# Table III.11

Baseline Differences between Frequent and Infrequent Participants:

Middle School Centers

  -------------------------------------------------------------------------------------------
  Baseline Characteristic          Infrequent     Frequent       Difference^a^   p-Value^b^
                                   Participants   Participants                   
                                   (Bottom Third) (Top Third)                    
  -------------------------------- -------------- -------------- --------------- ------------
  Percent of Hispanic Students     30.8           24.0           6.7\*\*         0.01

  Percent of Black Students        20.0           37.3           --17.3\*\*\*    0.00

  Percent of Students Whose        30.1           36.0           --5.9\*\*       0.02
  Parents Received Public                                                        
  Assistance                                                                     

  Percent Whose Mother Dropped Out 23.2           16.3           6.9\*\*\*       0.00
  of High School                                                                 

  Household Income (in Thousands   35.8           33.5           2.3\*           0.07
  of Dollars)                                                                    

  Percent of Students in a         27.1           33.4           --6.3\*\*       0.01
  Single-Parent Household                                                        

  Number of Absences Last Year     8.4            7.1            1.3\*\*\*       0.00

  **Sample Size**                  **548**        **602**                        
  -------------------------------------------------------------------------------------------

[Source:]{.smallcaps} Student baseline survey; parent survey; school
records; program attendance records.

^a^Due to rounding, estimated impacts shown in the table do not always
equal the difference between center infrequent participants and the
frequent participants.

^b^The p-value is the smallest level of significance at which the null
hypothesis that the impact equals zero can be rejected. If the p-value
is less than .01, an impact is significant at the 1 percent level; if
the p-value is less than .05, the impact is significant at the 5 percent
level, and so on.

\*Significantly different from zero at the .10 level, two-tailed test.

\*\*Significantly different from zero at the .05 level, two-tailed test.

\*\*\*Significantly different from zero at the .01 level, two-tailed
test.

differences may arise because of the assumed motivation difference and
are not interpreted here as evidence that more frequent attendance would
improve the outcome.

Turning to the outcome differences, frequent participants spend more
time at school after the regular school day ends (Table III.12).[^21]
They also spend less time with parents and more time in the care of
other adults. There are some indications that frequent participants
behave better in school than less frequent participants. Frequent
participants were less likely to say that they give their teacher a
"hard time" "some" or "a lot" of the time (Table III.12). Teachers also
report that frequent participants were less likely to be disruptive, and
frequent participants were absent 1.5 fewer days than infrequent
participants. However, academic achievement of frequent participants is
about the same as for less frequent participants. Grades were not
statistically different, and teachers reported no differences in overall
academic achievement. Frequent participants appear to be somewhat less
able to interact socially---they were more likely to feel lonely, more
likely to feel "picked on," and less likely to find it easy to make new
friends (Table III.12). Furthermore, teachers were less likely to say
that frequent participants were good at getting along with others.

The picture that emerges from this analysis suggests that frequent
participants are more likely to be from disadvantaged households and to
want to improve in school, as their better behavior in school and more
frequent attendance indicate. However, the analysis does not suggest
that higher levels of center attendance lead to improved outcomes.

# Table III.12

> Outcome Differences by Attendance:
>
> Middle School Centers

+---------------------------------------+---------+---------+---------+
| Outcome                               | Median  | F       | p-Value |
|                                       | Partic  | requent | ^a^     |
|                                       | ipation | Partic  |         |
|                                       |         | ipation |         |
|                                       | (44     |         |         |
|                                       | Days)   | (104    |         |
|                                       |         | Days)   |         |
+---------------------------------------+---------+---------+---------+
| Percentage of Students in the         |         |         |         |
| Following Locations after School at   |         |         |         |
| Least Three Days in a Typical Week:   |         |         |         |
+---------------------------------------+---------+---------+---------+
| > Own home                            | 71.7    | 53.2    | 0.0     |
|                                       |         |         | 0\*\*\* |
+---------------------------------------+---------+---------+---------+
| > Someone else's home                 | 9.7     | 4.3     | 0.0     |
|                                       |         |         | 0\*\*\* |
+---------------------------------------+---------+---------+---------+
| > School or other place for           | 27.3    | 48.2    | 0.0     |
| > activities                          |         |         | 0\*\*\* |
+---------------------------------------+---------+---------+---------+
| Percentage of Students in Following   |         |         |         |
| Types of Supervision at Least Three   |         |         |         |
| Days After School in a Typical Week:  |         |         |         |
+---------------------------------------+---------+---------+---------+
| > Self-care                           | 18.3    | 13.2    | 0.0     |
|                                       |         |         | 1\*\*\* |
+---------------------------------------+---------+---------+---------+
| > Parent care                         | 55.5    | 43.5    | 0.0     |
|                                       |         |         | 0\*\*\* |
+---------------------------------------+---------+---------+---------+
| > Nonparent adult care                | 17.2    | 36.3    | 0.0     |
|                                       |         |         | 0\*\*\* |
+---------------------------------------+---------+---------+---------+
| Percentage of Students Who Report     | 18.9    | 15.1    | 0       |
| That They Give a Teacher a "Hard      |         |         | .04\*\* |
| Time" "Some" or "A Lot"               |         |         |         |
+---------------------------------------+---------+---------+---------+
| Percentage of Students Whose Teachers | 8.4     | 5.3     | 0       |
| Report That They Are "Often"          |         |         | .05\*\* |
| Disruptive                            |         |         |         |
+---------------------------------------+---------+---------+---------+
| Teacher-Reported Level of Effort      | 3.6     | 3.6     | 0.51    |
| Composite                             |         |         |         |
+---------------------------------------+---------+---------+---------+
| Teacher-Reported Academic Achievement | 3.4     | 3.3     | 0.36    |
| Composite                             |         |         |         |
+---------------------------------------+---------+---------+---------+
| Delinquent Behavior Composite         | 1.5     | 1.5     | 0.40    |
+---------------------------------------+---------+---------+---------+
| Tobacco, Alcohol, Drug Use Composite  | 1.1     | 1.1     | 0.64    |
+---------------------------------------+---------+---------+---------+
| Percentage of Students Who Report     |         |         |         |
| That They Do the Following "Some" or  |         |         |         |
| "A Lot":                              |         |         |         |
+---------------------------------------+---------+---------+---------+
| > Sell illegal drugs                  | 3.0     | 1.9     | 0.16    |
+---------------------------------------+---------+---------+---------+
| > Get arrested or detained by police  | 3.6     | 1.4     | 0       |
|                                       |         |         | .02\*\* |
+---------------------------------------+---------+---------+---------+
| Percentage of Students Who Report the |         |         |         |
| Following Happened to Them "Some" or  |         |         |         |
| "A Lot":                              |         |         |         |
+---------------------------------------+---------+---------+---------+
| > Been offered, sold, or given an     | 15.4    | 13.4    | 0.27    |
| > illegal drug                        |         |         |         |
+---------------------------------------+---------+---------+---------+
| > Been picked on after school         | 31.5    | 37.3    | 0       |
|                                       |         |         | .02\*\* |
+---------------------------------------+---------+---------+---------+
| Mean Number of Days Student Was:      |         |         |         |
+---------------------------------------+---------+---------+---------+
| > Absent from class                   | 9.6     | 8.1     | 0.0     |
|                                       |         |         | 0\*\*\* |
+---------------------------------------+---------+---------+---------+
| > Late for class                      | 5.4     | 5.3     | 0.91    |
+---------------------------------------+---------+---------+---------+
| Mean Grade in English                 | 80.7    | 79.5    | 0       |
|                                       |         |         | .05\*\* |
+---------------------------------------+---------+---------+---------+
| Percentage of Students Who Report the |         |         |         |
| Following Are "Somewhat True" or      |         |         |         |
| "Very True":                          |         |         |         |
+---------------------------------------+---------+---------+---------+
| > They are lonely                     | 12.2    | 15.5    | 0.09\*  |
+---------------------------------------+---------+---------+---------+
| > They find it easy to make new       | 89.2    | 86.0    | 0.07\*  |
| > friends                             |         |         |         |
+---------------------------------------+---------+---------+---------+
| Percentage of Students Whose Teachers | 78.6    | 74.3    | 0.08\*  |
| "Agree" or "Strongly Agree" That They |         |         |         |
| Get Along Well with Others            |         |         |         |
+---------------------------------------+---------+---------+---------+

[Source:]{.smallcaps} Student survey; parent survey; school records;
program attendance records.

[Note:]{.smallcaps} The percentages and mean values of outcomes have
been regression-adjusted to account for baseline differences between the
groups. The control variables in the regression included 26 different
student and household characteristics such as indicators of students'
demographic characteristics, household socioeconomic status, and
students' baseline test scores, attendance, disciplinary problems, and
self-reported grades. Estimated impacts shown in the table do not always
equal the difference between center participants and the comparison
group due to rounding.

^a^The p-value is for the significance of the coefficient on attendance
in the regression model. The p-value is the smallest level of
significance at which the null hypothesis that the impact equals zero
can be rejected. If the p-value is less than .01, an impact is
significant at the 1 percent level; if the p-value is less than .05, the
impact is significant at the 5 percent level, and so on.

\*Significantly different from zero at the .10 level, two-tailed test.

\*\*Significantly different from zero at the .05 level, two-tailed test.

\*\*\*Significantly different from zero at the .01 level, two-tailed
test.

[^1]: We tested the extent to which regression models were able to
    reduce baseline differences by estimating models in which the
    baseline outcome was regressed on student characteristics. Results
    from these models indicate that the baseline differences were
    substantially reduced by the adjustment. In practice, the regression
    adjustment approach is stronger than what is found in the test
    because the baseline value of the outcome variable is included in
    the model.

[^2]: Because impacts can represent different units and the absolute
    magnitude of the impacts is not always informative, the text
    sometimes refers to an impact's "effect size," which is the impact
    expressed as a percentage of the outcome's standard deviation.
    Effect sizes of 10 to 20 percent are common in program evaluation
    and effect sizes of 30 percent or more are considered large and are
    relatively uncommon.

[^3]: The results reported in Table III.2 are based on student reports
    of their after-school location and care arrangements. We also
    examined impacts based on parent reports of location and care
    arrangements. The patterns of impacts were similar, though parents
    were more likely to report their child was cared for by a parent.

[^4]: Defining self-care in other ways yielded similar results. There
    was no significant difference between treatment and comparison
    groups in whether students were home alone for three or more days a
    week, or for one or more days a week.

[^5]: The term "hanging out" is used in the questionnaires completed by
    students and they were free to interpret the expression in their own
    way.

[^6]: As with location and care, we also examined student and parent
    reports about after-school activities. There were some differences
    between students and parents in reported activities after school,
    but student reports are presented here based on the assumption that
    student reports of their after school activities are more accurate.

[^7]: Participants possibly considered time at the center as time spent
    "hanging out with friends." In this case, even if they reported that
    one of their activities was "hanging out with friends," they may
    have reported their location as being at school.

[^8]: Across location and care categories, the lowest rate of doing
    homework (77 percent) was for students who said they went somewhere
    to "hang out" after school. The highest rate of doing homework (88
    percent) was for students in parent care. Students in parent care
    were more likely to do homework (88 percent) than students in
    self-care (81 percent), but the difference is not as large as might
    be expected.

[^9]: We also examined impacts on attendance and tardiness based on
    teachers reports. Attendance impacts were consistent with impacts
    based on records data. Tardiness impacts based on teacher reports
    were small and statistically insignificant.

[^10]: The difference between teacher evaluations of whether students
    completed homework and whether they completed assignments to
    teachers' satisfaction may arise because the more general term
    "assignments" includes work done in class as well as work done at
    home.

[^11]: The five items on the effort composite are whether the student
    (1) usually tries hard, (2) often performs at or above ability
    level, (3) is attentive in class, (4) participates in class, and (5)
    volunteers in class.

[^12]: Impacts on test scores also were estimated but not reported here,
    as the magnitude and direction of the estimated impacts was
    sensitive to the method used to impute missing baseline test scores.
    Only a fourth of the sample had a follow-up score available and only
    a tenth of the sample had both a baseline and a followup score
    available.

[^13]: We also estimated insignificant impacts on student confidence in
    their reading ability, whether students had a positive attitude
    toward learning according to their teacher, whether students go
    along well with others, students' ability to plan and solve
    problems, and the extent to which students helped their parents.

[^14]: Giving teachers a "hard time" was the expression used in the
    questionnaire completed by students and they were free to interpret
    the expression in their own way.

[^15]: In addition, we estimated subgroup impacts based on the mother's
    education, whether a student's teacher worked in an after-school
    program, and whether a student had attended an after-school program
    in the previous year. There were almost no statistically significant
    impacts for these subgroups.

[^16]: As with score results for the full sample, subgroup score impacts
    were estimated but found to be sensitive to alternative estimation
    methods and not reported.

[^17]: Being "picked on" was the expression used in the questionnaire
    completed by students and they were free to interpret the expression
    in their own way.

[^18]: Whether centers offered other types of academic classes was
    highly correlated with math offerings---centers that frequently
    offered math classes also tended to frequently offer other academic
    classes.

[^19]: Instrumental variables techniques can be used to adjust for
    unobserved differences if a reasonable instrumental variable or set
    of such variables can be identified. In this context, an instrument
    is a variable that would be correlated with attendance but not with
    the outcomes. Several variables were tried as instruments but failed
    statistical tests as a result of their correlation with outcomes.

[^20]: To ensure that only unobserved characteristics affect outcome
    differences between frequent and infrequent attenders, the
    comparisons reported here have all been adjusted for observed
    characteristics using regression models. The adjustment variables
    are the same as those in the impact regression models used earlier
    in the chapter.

[^21]: Appendix B provides details about the calculations of outcome
    differences for frequent and infrequent participants.