Supplemental Educational Services and Student Achievement in Waiver
Districts:

Anchorage and Hillsborough

Submitted to:

U.S. Department of Education\
Office of Planning, Evaluation and Policy Development\
Policy and Program Studies Service

Prepared by:

Miguel Socias\
Juliette‑Marie deSousa\
Kerstin Carlson Le Floch

American Institutes for Research\
1000 Thomas Jefferson Street NW\
Washington, DC 20007

and

RTI International\
3040 Cornwallis Road\
Research Triangle Park, NC 27709‑2194

January 2009

This report was prepared for the U.S. Department of Education under
Contract Number ED‑04‑CO‑0036/002/02 with RTI International. Erica Lee
served as the contracting officer's representative. The views expressed
herein do not necessarily represent the positions or policies of the
Department of Education. No official endorsement by the U.S. Department
of Education is intended or should be inferred.

**U.S. Department of Education**\
Margaret Spellings\
*Secretary*

**Office of Planning, Evaluation and Policy Development**\
Bill Evers\
*Assistant Secretary*

**Policy and Program Studies Service**\
Alan Ginsburg\
*Director*

**Program and Analytic Studies Division**\
David Goodwin\
*Director*

January 2009

This report is in the public domain. Authorization to reproduce it in
whole or in part is granted. Although permission to reprint this
publication is not necessary, the citation should be: U.S. Department of
Education, Office of Planning, Evaluation and Policy Development, Policy
and Program Studies Service, *Supplemental Educational Services and
Student Achievement in Waiver Districts: Anchorage and Hillsborough*,
Washington, D.C., 2009.

This report is also available on the Department's Web site at:
http://www.ed.gov/about/offices/list/opepd/ppss/index.html.

On request, this publication is available in alternative formats, such
as Braille, large print, or computer diskette. For more information,
please contact the Department's Alternate Format Center at (202)
260-0852 or (202) 260-0818.

# Contents {#contents .unnumbered}

List of Exhibits v

**Acknowledgments [ix](#acknowledgments)**

Statement from the U.S. Department of Education
[xi](#statement-from-the-u.s.-department-of-education)

Executive Summary [xiii](#executive-summary)

Key Findings [xiv](#key-findings)

I. Introduction [1](#introduction)

Evaluation Questions [1](#evaluation-questions)

Overview of the Supplemental Educational Services Provisions of *NCLB*
[2](#overview-of-the-supplemental-educational-services-provisions-of-nclb)

Data Sources [3](#data-sources)

II\. Findings: Eligibility for and Participation in Supplemental
Educational Services
[7](#findings-eligibility-for-and-participation-in-supplemental-educational-services)

Student Eligibility for Supplemental Educational Services
[7](#student-eligibility-for-supplemental-educational-services)

Participation in Supplemental Educational Services
[8](#participation-in-supplemental-educational-services)

Characteristics of Eligible and Participating Students
[14](#characteristics-of-eligible-and-participating-students)

Hours and Subject of Tutoring [16](#hours-and-subject-of-tutoring)

III\. Findings: Student Achievement [19](#findings-student-achievement)

Analytic Approach [19](#analytic-approach)

Prior Achievement Levels by Student Eligibility for Supplemental
Educational Services
[22](#prior-achievement-levels-by-student-eligibility-for-supplemental-educational-services)

Prior Achievement Levels by Student Participation in Supplemental
Educational Services
[24](#prior-achievement-levels-by-student-participation-in-supplemental-educational-services)

Prior Achievement Levels by Provider Type
[26](#prior-achievement-levels-by-provider-type)

Supplemental Educational Services Participation and Student Achievement
[28](#supplemental-educational-services-participation-and-student-achievement)

Gains in Student Achievement by Provider
[30](#gains-in-student-achievement-by-provider)

Effect of Number of Hours of Services on Student Achievement
[36](#effect-of-number-of-hours-of-services-on-student-achievement)

Effect of Subject of Services on Achievement
[40](#effect-of-subject-of-services-on-achievement)

Effect of Participation by Student Characteristics
[41](#effect-of-participation-by-student-characteristics)

IV\. District Communication With Parents
[43](#district-communication-with-parents-2)

V. Conclusions [47](#conclusions)

Appendix A: Description of District Data
[49](#appendix-a-description-of-district-data)

Anchorage [49](#anchorage)

Hillsborough [49](#hillsborough)

Appendix B: Supplemental Exhibits
[51](#appendix-b-supplemental-exhibits-1)

Appendix C: Standard Error Exhibits
[57](#appendix-c-standard-error-exhibits-1)

#  {#section .unnumbered}

# Exhibits {#exhibits .unnumbered}

Executive Summary

[Exhibit S.1 Number of Supplemental Educational Services Participants in
Grades K--12, by Provider Type, in Anchorage, 2004--05 Through 2006--07
xv](#__RefHeading___Toc220485718)

[Exhibit S.2 Number of Supplemental Educational Services Participants in
Grades K--12, by Provider Type, in Hillsborough, 2004--05 Through
2006--07 xv](#__RefHeading___Toc220485719)

[Exhibit S.3 Relative Achievement Gains for Supplemental Educational
Services Participants, by Provider Type, in Anchorage and Hillsborough,
2002--03 Through 2006--07 xvi](#__RefHeading___Toc220485720)

**I. Introduction**

[Exhibit 1 Data Available From Anchorage and Hillsborough
4](#__RefHeading___Toc220485721)

**II. Findings: Eligibility for and Participation in Supplemental
Education Services**

[Exhibit 2 Number and Percentage of Eligible Students in Grades K--12
Who Participated in Supplemental Educational Services, in Anchorage and
Hillsborough, 2004--05 Through 2006--07 8](#__RefHeading___Toc220485722)

[Exhibit 3 Distribution of Supplemental Educational Services
Participants, by Grade Level, in Anchorage and Hillsborough, 2005--06
and 2006--07 9](#__RefHeading___Toc220485723)

[Exhibit 4 Number of Supplemental Educational Services Participants in
Grades K--12, by Provider Type, in Anchorage, 2004--05 Through 2006--07
10](#__RefHeading___Toc220485724)

[Exhibit 5 Number and Percentage of Supplemental Educational Services
Participants in Grades K--12, by Provider, in Anchorage, 2005--06 and
2006--07 11](#__RefHeading___Toc220485725)

[Exhibit 6 Number of Supplemental Educational Services Participants in
Grades K--12, by Provider Type, in Hillsborough, 2004--05 Through
2006--07 12](#__RefHeading___Toc220485726)

[Exhibit 7 Number and Percentage of Supplemental Educational Services
Participants in Grades K--12, by Provider, in Hillsborough, 2005--06 and
2006--07 13](#__RefHeading___Toc220485727)

[Exhibit 8 Demographic Characteristics of Supplemental Educational
Services Participants in Grades K--12, by Provider Type, in Anchorage,
2006--07 14](#__RefHeading___Toc220485728)

[Exhibit 9 Demographic Characteristics of Supplemental Educational
Services Participants in Grades K--12, by Provider Type, in
Hillsborough, 2006--07 15](#__RefHeading___Toc220485729)

[Exhibit 10 Average Number of Hours of Supplemental Educational Services
Received by Participants in Grades K--12, by Provider Type, in Anchorage
and Hillsborough, 2006--07 16](#__RefHeading___Toc220485730)

**III. Findings: Student Achievement**

[Exhibit 11 Prior Achievement Levels (in 2005--06) on the SBA of
Students in Grades 3--10 by Eligibility for Supplemental Educational
Services in 2006--07, in Anchorage 22](#__RefHeading___Toc220485731)

[Exhibit 12 Prior Achievement Levels (in 2005--06) on the FCAT of
Students in Grades 1--12 by Eligibility for Supplemental Educational
Services in 2006--07, in Hillsborough 23](#__RefHeading___Toc220485732)

[Exhibit 13 Prior Achievement Levels (in 2005--06) on the Benchmark of
Students in Grades 3--10 by Participation in Supplemental Educational
Services in 2006--07, in Anchorage 24](#__RefHeading___Toc220485733)

[Exhibit 14 Prior Achievement Levels (in 2005--06) on the FCAT of
Students in Grades 1--12 by Participation in Supplemental Educational
Services in 2006--07, in Hillsborough 25](#__RefHeading___Toc220485734)

[Exhibit 15 Prior Achievement Levels (in 2005--06) on the Benchmark of
Students in Grades 3--10, by Provider Type in 2006--07, in Anchorage
26](#__RefHeading___Toc220485735)

[Exhibit 16 Prior Achievement Levels (in 2005--06) on the FCAT of
Students in Grades 1--12, by Provider Type in 2006--07, in Hillsborough
27](#__RefHeading___Toc220485736)

[Exhibit 17 Achievement Gains on the Benchmark and SBA of Students in
Grades 3--10 by Participation in Supplemental Educational Services, in
Anchorage, 2002--03 Through 2006--07 28](#__RefHeading___Toc220485737)

[Exhibit 18 Achievement Gains on the FCAT of Students in Grades 1--12 by
Participation in Supplemental Educational Services, in Hillsborough,
2002--03 Through 2006--07 29](#__RefHeading___Toc220485738)

[Exhibit 19 Achievement Gains on the Benchmark and SBA of Students in
Grades 3--10 by Provider Type, in Anchorage, 2002--03 Through 2006--07
30](#__RefHeading___Toc220485739)

[Exhibit 20 Mathematics Achievement Gains on the Benchmark and SBA of
Students in Grades 3--10 by Eligibility for and Participation in
Supplemental Educational Services, by Provider, in Anchorage, 2002--03
Through 2006--07 31](#__RefHeading___Toc220485740)

[Exhibit 21 Reading Achievement Gains on the Benchmark and SBA of
Students in Grades 3--10 by Eligibility for and Participation in
Supplemental Educational Services, by Provider, in Anchorage, 2002--03
Through 2006--07 32](#__RefHeading___Toc220485741)

[Exhibit 22 Achievement Gains on the FCAT of Students in Grades 1--12 by
Provider Type, in Hillsborough, 2002--03 through 2006--07
33](#__RefHeading___Toc220485742)

[Exhibit 23 Mathematics Achievement Gains on the FCAT of Students in
Grades 1--12 by Eligibility for and Participation in Supplemental
Educational Services, by Provider, in Hillsborough, 2002--03 through
2006--07 34](#__RefHeading___Toc220485743)

[Exhibit 24 Reading Achievement Gains on the FCAT of Students in Grades
1--12 by Eligibility for and Participation in Supplemental Educational
Services, by Provider, in Hillsborough, 2002--03 Through 2006--07
35](#__RefHeading___Toc220485744)

[Exhibit 25 Distribution of Supplemental Educational Services
Participants by Number of Hours of Services Received, by Provider Type,
in Anchorage and Hillsborough, 2004--05 Through 2006--07
36](#__RefHeading___Toc220485745)

[Exhibit 26 Achievement Gains on the Benchmark and SBA of Students in
Grades 3--10 by Provider Type and Hours of Services Received, in
Anchorage, 2002--03 Through 2006--07 38](#__RefHeading___Toc220485746)

[Exhibit 27 Achievement Gains on the FCAT of Students in Grades 1--12 by
Provider Type and Hours of Services Received, in Hillsborough, 2002--03
Through 2006--07 39](#__RefHeading___Toc220485747)

[Exhibit 28 Achievement Gains on the Benchmark and SBA of Students in
Grades 3--10, by Subject, in Anchorage, 2002--03 Through 2006--07
40](#__RefHeading___Toc220485748)

[Exhibit 29 Achievement Gains on the FCAT of Students in Grades 1--12 by
Subject, in Hillsborough, 2002--03 Through 2006--07
41](#__RefHeading___Toc220485749)

[Exhibit 30 Achievement Gains on the Benchmark and SBA of Students in
Grades 3--10 for Supplemental Educational Services Participants, by
Student Characteristics, in Anchorage, 2002--03 Through 2006--07
42](#__RefHeading___Toc220485750)

[Exhibit 31 Achievement Gains on the FCAT of Students in Grades 1--12
for Supplemental Educational Services Participants, by Student
Characteristics, in Hillsborough, 2002--03 Through 2006--07
42](#__RefHeading___Toc220485751)

####  {#section-1 .unnumbered}

#### Appendix B: Supplemental Exhibits {#appendix-b-supplemental-exhibits .unnumbered}

[Exhibit B.1 Participation Rates for Supplemental Educational Services
in Grades K--12, by Race‑Ethnicity, in Anchorage, 2006--07
51](#__RefHeading___Toc220485752)

[Exhibit B.2 Participation Rates for Supplemental Educational Services
in Grades K--12, by Race‑Ethnicity, in Hillsborough, 2006--07
51](#__RefHeading___Toc220485753)

[Exhibit B.3 Number and Percentage of Supplemental Educational Services
Participants in Grades K--12, by Subject, in Anchorage, 2006--07
52](#__RefHeading___Toc220485754)

[Exhibit B.4 Number and Percentage of Supplemental Educational Services
Participants in Grades K--12 by Subject, in Hillsborough, 2006--07
53](#__RefHeading___Toc220485755)

[Exhibit B.5 Subject of Supplemental Services in Grades K--12, by
Provider Type, in Anchorage, 2006--07 54](#__RefHeading___Toc220485756)

[Exhibit B.6 Subject of Supplemental Services in Grades K--12, by
Provider Type, in Hillsborough, 2006--07
54](#__RefHeading___Toc220485757)

[Exhibit B.7 Average Number of Hours of Supplemental Educational
Services Received by Participants in Grades K--12, by Provider, in
Anchorage and Hillsborough 55](#__RefHeading___Toc220485758)

#### Appendix C: Standard Error Exhibits {#appendix-c-standard-error-exhibits .unnumbered}

[Exhibit C.1 Achievement Gains on the Benchmark and SBA of Students in
Grades 3--10 by Participation in Supplemental Educational Services, in
Anchorage, 2002--03 Through 2006--07 57](#__RefHeading___Toc220485759)

[Exhibit C.2 Achievement Gains on the FCAT of Students in Grades 1--12
by Participation in Supplemental Educational Services, in Hillsborough,
2002--03 Through 2006--07 57](#__RefHeading___Toc220485760)

[Exhibit C.3 Achievement Gains on the Benchmark and SBA of Students in
Grades 3--10 by Provider Type, in Anchorage, 2002--03 Through 2006--07
58](#__RefHeading___Toc220485761)

[Exhibit C.4 Mathematics Achievement Gains on the Benchmark and SBA of
Students in Grades 3--10 by Eligibility for and Participation in
Supplemental Educational Services, by Provider, in Anchorage, 2002--03
Through 2006--07 58](#__RefHeading___Toc220485762)

[Exhibit C.5 Reading Achievement Gains on the Benchmark and SBA of
Students in Grades 3--10 by Eligibility for and Participation in
Supplemental Educational Services, by Provider, in Anchorage, 2002--03
Through 2006--07 59](#__RefHeading___Toc220485763)

[Exhibit C.6 Achievement Gains on the FCAT of Students in Grades 1--12
by Provider Type, in Hillsborough, 2002--03 Through 2006--07
59](#__RefHeading___Toc220485764)

[Exhibit C.7 Mathematics Achievement Gains on the FCAT of Students in
Grades 1--12 by Eligibility for and Participation in Supplemental
Educational Services, by Provider, in Hillsborough, 2002--03 Through
2006--07 60](#__RefHeading___Toc220485765)

[Exhibit C.8 Reading Achievement Gains on the FCAT of Students in Grades
1--12 by Eligibility for and Participation in Supplemental Educational
Services, by Provider, in Hillsborough, 2002--03 Through 2006--07
61](#__RefHeading___Toc220485766)

[Exhibit C.9 Achievement Gains on the Benchmark and SBA of Students in
Grades 3--10 by Provider Type and Hours of Services Received, in
Anchorage, 2002--03 Through 2006--07 62](#__RefHeading___Toc220485767)

[Exhibit C.10 Achievement Gains on the FCAT of Students in Grades 1--12
by Provider Type and Hours of Services Received, in Hillsborough,
2002--03 Through 2006--07 62](#__RefHeading___Toc220485768)

[Exhibit C.11 Achievement Gains on the Benchmark and SBA of Students in
Grades 3--10 by Subject, in Anchorage, 2002--03 Through 2006--07
63](#__RefHeading___Toc220485769)

[Exhibit C.12 Achievement Gains on the FCAT of Students in Grades 1--12
by Subject, in Hillsborough, 2002--03 Through 2006--07
63](#__RefHeading___Toc220485770)

[Exhibit C.13 Achievement Gains on the Benchmark and SBA of Students in
Grades 3--10 for Supplemental Educational Services Participants, by
Student Characteristics, in Anchorage, 2002--03 Through 2006--07
64](#__RefHeading___Toc220485771)

[Exhibit C.14 Achievement Gains on the FCAT of Students in Grades 1--12
for Supplemental Educational Services Participants, by Student
Characteristics, in Hillsborough, 2002--03 Through 2006--07
64](#__RefHeading___Toc220485772)

# Acknowledgments {#acknowledgments .unnumbered}

We wish to thank several individuals who contributed to the completion
of this study.

First, we are grateful for the efforts of Vernon Campbell from Anchorage
Public Schools and Jessica Swere from Hillsborough County Public Schools
for their work to provide the data files used for this study.
Additionally, we would like to acknowledge the guidance and support of
the U.S. Department of Education. In particular, we thank Erica Lee,
Collette Roney, Stephanie Stullich, and Margie Yeager of the Policy and
Program Studies Service, who served as our project officers on this
report. We also thank Stacy Kreppel from the Office of Innovation and
Improvement for assisting with the data collection. We are grateful to
the staff of the American Institutes for Research for their assistance
in producing this report. In particular, we thank Mike Garet for his
guidance on data and analysis issues. Finally, we would like to
acknowledge Ron Zimmer and Brian Gill of RAND for their advice on the
analytic approach and data issues.

Many Department staff reviewed drafts of this report and provided useful
comments and suggestions. We would like to acknowledge the assistance of
David Goodwin, director of program and analytic studies in PPSS and
Daphne Kaplan, PPSS team leader, as well as William Cordes, Thomas
Corwin, Anne Dudro, Amanda Farris, Gregory Jackson, Julia Keleher, Stacy
Kreppel, Audrey Pendleton, Kelly Rhoads, Kay Rigling, Phil Rosenfelt,
Wendy Tada, and Ze'ev Wurman.

While we appreciate the assistance and support of all of the above
individuals, any errors in judgment or fact are of course the
responsibility of the authors.

#   {#section-2 .unnumbered}

# Statement from the U.S. Department of Education {#statement-from-the-u.s.-department-of-education .unnumbered}

Data were collected from the first year of the SES pilot (2005-06) in
Chicago and Boston, but they are not included in this evaluation report
or issued as part of a separate evaluation report because there were
some issues with and some gaps in the data collected. Thus, this
evaluation data was used for internal purposes only.

Subsequently, the Department took steps to improve data collection with
participating districts. Specifically, the Department (a) clarified the
data requirements with the participating districts; (b) instituted a
process to contact the districts earlier in the school year for the
background data; and (c) collected only the current year achievement
data later in the year. Thus, the data collection process with the
second group of waiver grantees that is described in this report was
improved. A future evaluation and evaluation report will examine the
implementation of supplemental educational services in all five waiver
districts (Anchorage, Boston, Charlotte-Mecklenburg, Chicago, and
Hillsborough) and will review all relevant data, but will focus on
results from the 2006-07 school year through the 2008-09 school year.

#   {#section-3 .unnumbered}

# Executive Summary {#executive-summary .unnumbered}

Under the *Elementary and Secondary Education Act of 1965 (ESEA)*, as
amended by the *No Child Left Behind Act of 2001* (*NCLB*), Title I
schools that do not make adequate yearly progress (AYP) for two
consecutive years are identified for school improvement. If an
identified Title I school does not make AYP while in that status and
thus enters the second year of school improvement status, its district
must offer students from low-income families in the school the
opportunity to receive free supplemental educational services, such as
tutoring, remediation or other academic enrichment that are provided in
addition to instruction provided during the school day. The district
must also offer supplemental educational services to students from
low-income families in schools that continue not to make AYP and are
identified for corrective action or restructuring. Parents of eligible
students may choose a provider from the state‑approved list and the
school district pays for the services. Many types of organizations are
eligible to apply to become state‑approved supplemental educational
service providers, including for‑profit agencies, not‑for‑profit groups,
faith‑based and community-based organizations, private schools, and
public schools and districts.

Under federal regulations, school districts that have been identified
for improvement or corrective action are not eligible to be supplemental
educational service providers. However, the U.S. Department of Education
has granted waivers to five such districts, on a pilot basis, to allow
those districts to provide supplemental educational services. Boston,
Massachusetts and Chicago, Illinois received waivers beginning in the
2005--06 school year; Hillsborough County, Florida and Anchorage, Alaska
received waivers beginning in the 2006--07 school year; and
Charlotte-Mecklenburg, North Carolina received a waiver beginning in the
2008--09 school year. These pilot districts are eligible to serve as
providers in exchange for expanding students' access to supplemental
educational services and providing student achievement data for an
evaluation to examine program effectiveness. The goals of this pilot
program are to help ensure that more eligible students receive
supplemental educational service, and to provide more accurate and
comprehensive information to the U.S. Department of Education on the
effectiveness of supplemental educational services in improving student
academic achievement.

This report examines the implementation of supplemental educational
services in two of the waiver districts, Anchorage and Hillsborough, in
2006--07, including issues relating to student participation,
achievement gains, and communication with parents. A future report will
examine the implementation of supplemental educational services in all
five waiver districts (Anchorage, Boston, Charlotte-Mecklenburg,
Chicago, and Hillsborough) through the 2008--09 school year.

The central evaluation question for this report is whether students
served by the Anchorage and Hillsborough County district providers show
achievement gains that are at least comparable to those of students
served by non‑district providers. This report also examines the effects
of supplemental educational services more generally by evaluating
whether, and to what extent, supplemental educational services
participants in these two districts show achievement gains relative to
students who do not participate. Finally, the report examines whether
student achievement gains vary by individual provider or by student
characteristics, such as race/ethnicity, limited English proficiency,
and special education status.

With respect to the analysis of student achievement, the analyses
presented here apply to a subset of students in each district---those in
tested grades only. In Anchorage, students in grades 3--9 took the
Alaska Standards Based Assessment (SBA) in 2004--05. In subsequent
years, 10th graders were tested as well. Prior to 2004--05 only students
in grades 3, 6, and 8 participated in standardized tests, and the test
administered was the Alaska Benchmark Exam[^1], not the SBA. In
Hillsborough, students in grades 1--12 participated in Florida's
Comprehensive Assessment Test (FCAT) from 2002--03 through 2004--05. In
2005--06 and 2006--07 this test was administered to students in grades
2--12 and 3--12, respectively. Given these data, the achievement
analyses presented in this report cover 71 and 84 percent of all
students receiving supplemental educational services in Anchorage and
Hillsborough, respectively.

## Key Findings {#key-findings .unnumbered}

### Eligibility for and Participation in Supplemental Educational Services {#eligibility-for-and-participation-in-supplemental-educational-services .unnumbered}

-   In Anchorage, 6 percent of all students in the district were
    eligible for supplemental educational services in 2006--07. In
    Hillsborough about 20 percent of all students in the district were
    eligible for these services in 2006--07.

-   In both districts, the percentage of eligible students participating
    in supplemental educational services increased between 2004--05 and
    2006--07, reaching 11 percent (from 4 percent) in Anchorage and
    14 percent (from 2 percent) in Hillsborough in 2006--07.

-   The Anchorage school district served 138 students through its own
    supplemental educational services program (45 percent of students
    served) in 2006--07. The district did not offer supplemental
    educational services during the previous school years (see Exhibit
    S.1).

-   The Hillsborough school district served 971 students through its own
    supplemental educational services program (16 percent of students
    served) in 2006--07. The district did not offer supplemental
    educational services during the previous school years (see Exhibit
    S.2).

-   Participants in supplemental educational services received an
    average of 28 and 22 hours of tutoring in Anchorage and Hillsborough
    during 2006--07, respectively.

-   In both Anchorage and Hillsborough, students who were eligible for
    supplemental educational services had significantly lower levels of
    prior achievement in mathematics and reading than students who were
    not eligible.

-   In both Anchorage and Hillsborough, students who participated in
    supplemental educational services in 2006--07 had significantly
    lower levels of prior achievement (in 2005--06) than eligible
    students who chose not to participate.

-   In Hillsborough, participants served by the school district in
    2006--07 had higher prior achievement levels in 2005--06 than
    students served by non-district providers. In Anchorage, there was
    no clear pattern of prior achievement levels in 2005--06 between
    students served by the district and those served by non‑district
    providers in 2006--07.

+--------------------------------+---+---------------------------------+
| > [                            |   | []{#__RefHeading___Toc220485719 |
| ]{#__RefHeading___Toc220485718 |   | .anchor}Exhibit S.2\            |
| > .anchor}Exhibit S.1\         |   | Number of Supplemental          |
| > Number of Supplemental       |   | Educational Services            |
| > Educational Services         |   | Participants in Grades K--12,   |
| > Participants in Grades       |   | by Provider Type, in            |
| > K--12, by Provider Type, in  |   | Hillsborough, 2004--05 Through  |
| > Anchorage, 2004--05 Through  |   | 2006--07                        |
| > 2006--07                     |   |                                 |
+--------------------------------+---+---------------------------------+
| ![](media                      |   | ![](media/image2.p              |
| /image1.png){width="2.39375in" |   | ng){width="2.479861111111111in" |
| height="2.6215277777777777in"} |   | height="2.6215277777777777in"}  |
+--------------------------------+---+---------------------------------+
| **Exhibit reads:** In          |   | **Exhibit reads:** In           |
| Anchorage in\                  |   | Hillsborough in\                |
| 2006--07, 138 participants, or |   | 2006--07, 971 participants, or  |
| 45 percent of all              |   | 16 percent of all participants, |
| participants, received         |   | received supplemental           |
| supplemental educational       |   | educational services from the   |
| services from the Anchorage    |   | Hillsborough school district;   |
| Public Schools; 170 students   |   | 5,201 students were served by   |
| were served by non‑district    |   | non‑district providers.         |
| providers.                     |   |                                 |
|                                |   | Source: Hillsborough School     |
| Source: Anchorage School       |   | District Administrative Data,   |
| District Administrative Data,  |   | 2002--03 through 2006--07.      |
| 2002--03 through 2006--07.     |   |                                 |
+--------------------------------+---+---------------------------------+

### Student Achievement Gains {#student-achievement-gains .unnumbered}

-   In Anchorage, there was not a statistically significant difference
    in the academic achievement gains between participants and eligible
    non‑participants (see Exhibit S.3). However, the participant group
    was extremely small which makes it difficult to detect significant
    effects.

-   In Hillsborough, students receiving supplemental educational
    services experienced larger academic gains in mathematics---but not
    in reading---than eligible non‑participants. Participants improved
    their academic achievement by 0.05 standard deviations more than
    eligible non‑participants in mathematics.

-   In both districts, students served by non‑district providers showed,
    on average, larger gains than eligible non-participants in
    mathematics (but not in reading). Students served by the two
    district providers (Anchorage and Hillsborough) did not show
    achievement gains in either subject that were statistically
    distinguishable from those of eligible non-participants.

+-----------------------+----------+----------+-----------+-----------+
| []{#__RefH            |          |          |           |           |
| eading___Toc220485720 |          |          |           |           |
| .anchor}Exhibit S.3\  |          |          |           |           |
| Relative Achievement  |          |          |           |           |
| Gains for             |          |          |           |           |
| Supplemental          |          |          |           |           |
| Educational Services  |          |          |           |           |
| Participants, by      |          |          |           |           |
| Provider Type, in     |          |          |           |           |
| Anchorage and         |          |          |           |           |
| Hillsborough,         |          |          |           |           |
| 2002--03 Through      |          |          |           |           |
| 2006--07              |          |          |           |           |
+-----------------------+----------+----------+-----------+-----------+
|                       | A        |          | Hil       |           |
|                       | nchorage |          | lsborough |           |
+-----------------------+----------+----------+-----------+-----------+
|                       | Mat      | Reading  | Ma        | Reading   |
|                       | hematics |          | thematics |           |
+-----------------------+----------+----------+-----------+-----------+
|                       |          |          |           |           |
+-----------------------+----------+----------+-----------+-----------+
| Overall effect        | --0.05   | 0.01     | 0.05\*    | --0.001   |
+-----------------------+----------+----------+-----------+-----------+
| District provider     | --0.17   | --0.11   | 0.05      | --0.02    |
+-----------------------+----------+----------+-----------+-----------+
| Non-district provider | 0.18\*   | 0.12     | 0.08\*    | --0.01    |
+-----------------------+----------+----------+-----------+-----------+
| **Exhibit reads:** In |          |          |           |           |
| Anchorage, students   |          |          |           |           |
| who participated in   |          |          |           |           |
| supplemental          |          |          |           |           |
| educational services  |          |          |           |           |
| in mathematics showed |          |          |           |           |
| no significant        |          |          |           |           |
| differences in        |          |          |           |           |
| mathematics           |          |          |           |           |
| achievement gains     |          |          |           |           |
| from those of         |          |          |           |           |
| eligible              |          |          |           |           |
| non-participants.     |          |          |           |           |
|                       |          |          |           |           |
| Notes: Scores         |          |          |           |           |
| reported are          |          |          |           |           |
| rank-based z‑scores.  |          |          |           |           |
| \* indicates gains    |          |          |           |           |
| that were             |          |          |           |           |
| statistically         |          |          |           |           |
| different form the    |          |          |           |           |
| gains of eligible     |          |          |           |           |
| non-participants at   |          |          |           |           |
| the .05 level.        |          |          |           |           |
|                       |          |          |           |           |
| Source:               |          |          |           |           |
| Administrative data   |          |          |           |           |
| provided by the       |          |          |           |           |
| Anchorage and         |          |          |           |           |
| Hillsborough school   |          |          |           |           |
| districts, 2002--03   |          |          |           |           |
| through 2006--07.     |          |          |           |           |
+-----------------------+----------+----------+-----------+-----------+

-   Looking at individual non-district providers, only one such provider
    in Hillsborough showed a larger achievement gain (in mathematics)
    relative to the gains of eligible non-participants. No individual
    provider in Anchorage showed any significant gains relative to
    eligible non-participants.

Overall, there was not a consistent relationship between the amount of
services received and achievement gains in either district.

-   In both districts, most student subgroups did not show significant
    achievement gains.

### District Communication with Parents {#district-communication-with-parents .unnumbered}

-   Despite the use of phrases such as "free tutoring," both Anchorage
    and Hillsborough sent families notification materials that were
    difficult to understand due to use of jargon (such as quoting the
    law) and the high grade level at which the documents were written.

-   Both Anchorage and Hillsborough translated some materials into
    languages other than English.

-   Both Anchorage and Hillsborough provided unbiased materials
    describing all supplemental educational service providers.

# Introduction

Under the *Elementary and Secondary Education Act of 1965 (ESEA),* as
amended by the *No Child Left Behind Act of 2001* (*NCLB*), Title I
schools that do not make adequate yearly progress (AYP) for two
consecutive years are identified for school improvement. If an
identified Title I school does not make AYP while in that status, and
thus enters the second year of school improvement status, its district
must offer students from low-income families in the school the
opportunity to receive free supplemental educational services, such as
tutoring, remediation or other academic instruction provided outside the
regular school day. The district must also offer supplemental
educational services to students from low-income families in schools
that continue not to make AYP and are identified for corrective action
or restructuring. Parents of eligible students may choose a provider
from the state‑approved list and the school district pays for the
services. Many types of organizations are eligible to apply to become
state‑approved supplemental service providers, including for‑profit
agencies, not for‑profit groups, faith‑based and community-based
organizations, private schools, and public schools and districts. Under
federal regulations, school districts that have been identified for
improvement or corrective action are not eligible to be supplemental
educational service providers. However, the U.S. Department of Education
has granted waivers to five such districts, on a pilot basis, to allow
those districts to provide supplemental educational services. Boston,
Massachusetts and Chicago, Illinois received waivers beginning in the
2005--06 school year; Hillsborough County, Florida and Anchorage, Alaska
received waivers beginning in the 2006--07 school year; and
Charlotte-Mecklenburg, North Carolina received a waiver beginning in the
2008--09 school year.

This report examines the implementation of supplemental educational
services in two of the waiver districts, Anchorage and Hillsborough, in
2006--07, including issues relating to student participation,
achievement gains, and communication with parents. A future report will
examine the implementation of supplemental educational services in all
five waiver districts (Anchorage, Boston, Charlotte-Mecklenburg,
Chicago, and Hillsborough) through the 2008--09 school year.

The central evaluation question for this report is whether students
served by the Anchorage and Hillsborough County district providers show
achievement gains that are at least comparable to those of students
served by non‑district providers. This report also examines the effects
of supplemental educational services more generally by evaluating
whether, and to what extent, supplemental educational services
participants in these two districts show achievement gains relative to
students who do not participate. Finally, the report examines whether
achievement gains vary by individual provider, amount of services
received, or by student characteristics, such as race/ethnicity, limited
English proficiency, and special education status.

## Evaluation Questions {#evaluation-questions .unnumbered}

These analyses rely on student‑level achievement data from the 2002--03
through 2006--07 school years plus student‑level data on participation
in supplemental educational services in 2004--05 through\
2006--07 to address the following sets of evaluation questions:

### Eligibility for and Participation in Supplemental Educational Services {#eligibility-for-and-participation-in-supplemental-educational-services-1 .unnumbered}

1.  Did the percent of eligible students participating in supplemental
    educational services during the pilot year increase from the prior
    year(s)?

2.  How were participating students distributed among providers? How do
    students served by the districts' supplemental educational services
    programs compare to students served by non‑district providers?

3.  Were there significant demographic or academic differences between
    students served by different providers, specifically those served by
    the districts' programs compared to those served by non‑district
    providers?

### Student Achievement {#student-achievement .unnumbered}

1.  Are gains in achievement for students receiving supplemental
    educational services from district providers comparable to gains for
    students served by non‑district providers?

```{=html}
<!-- -->
```
4.  How do levels of achievement vary based on student eligibility for,
    and participation in, supplemental educational services?

5.  Controlling for amount of exposure to supplemental educational
    services, are reading and mathematics gains for students served by
    district providers comparable to students served by non‑district
    providers?

### District Communication with Parents {#district-communication-with-parents-1 .unnumbered}

1.  Are districts communicating with parents about the availability of
    supplemental educational services in a manner that is clear, user
    friendly, and accessible?

```{=html}
<!-- -->
```
6.  Is the nature of district communication unbiased with respect to
    particular providers, specifically non‑district providers versus the
    district‑provided services?

## Overview of the Supplemental Educational Services Provisions of *NCLB* {#overview-of-the-supplemental-educational-services-provisions-of-nclb .unnumbered}

A key objective of Title I of the *ESEA*, as amended by *NCLB*, is to
provide greater parental choice to families with children attending
Title I schools identified for improvement, corrective action, or
restructuring. One option is for parents to enroll children in
supplemental educational services, such as tutoring, remediation, or
other supplemental academic enrichment that are in addition to
instruction provided during the school day. Students from low‑income
families are eligible for supplemental educational services if they
attend a Title I school that is in the second year of school
improvement, in corrective action, or in restructuring. Parents of
eligible students may choose a provider from the state‑approved list,
which may include for‑profit agencies, not‑for‑profit groups,
faith‑based and community-based organizations, private schools, and
public schools and districts. Districts that are required to offer
supplemental educational services must pay for these services and spend,
subject to demand, an amount equal to at least 20 percent of their Title
I, Part A allocation on supplemental education services and
transportation for public school choice.[^2] School districts must
consult with providers and parents to establish achievement goals for
students, and providers are required to measure progress towards these
goals and communicate with parents regarding their child's progress.
States are responsible for monitoring and evaluating provider
performance, and may use monitoring and evaluation results to remove
from the state‑approved list a provider that fails, for two consecutive
years, to contribute to increasing students' academic proficiency.

Few studies to date have examined the effect of supplemental educational
services on student achievement. One study by the U.S. Department of
Education found that, on average, across seven large, urban districts,
participation in supplemental educational services had a statistically
significant effect on students' achievement in mathematics (a gain of
0.09 standard deviations) and reading (a gain of 0.08 standard
deviations) relative to eligible students who did not participate.[^3]
Gains by students served by district versus non-district providers
varied, but showed no clear pattern.

Under federal regulations, school districts that have been identified
for improvement or corrective action are not eligible to be supplemental
educational service providers. However, the U.S. Department of Education
granted waivers to five such districts, on a pilot basis, to allow those
districts to provide supplemental educational services. These pilot
districts are: Boston, Massachusetts; Chicago, Illinois; Hillsborough
County, Florida; Anchorage, Alaska, and Charlotte-Mecklenburg, North
Carolina. These pilot districts are eligible to serve as providers in
exchange for expanding students' ability to access supplemental
educational services and providing student achievement data for an
evaluation to examine program effectiveness. As part of the waiver,
districts agreed to provide:

-   Early notification to parents of their children's eligibility to
    participate in supplemental educational services;

-   Extended enrollment periods so that parents can make the best choice
    for their child;

-   The use of district facilities by non‑district providers for a
    reasonable fee; and

-   Academic data to an independent third party for evaluation of the
    effectiveness of supplemental educational services.

The goals of the pilot are to help ensure that more eligible students
are receiving supplemental educational services, and to provide more
accurate and comprehensive information on the effectiveness of
supplemental educational services provided by districts in improvement
in improving student academic achievement.

This report on the pilot waivers examines the implementation of
supplemental educational services in the Anchorage and Hillsborough
County school districts, including issues relating to student
participation, achievement gains, and communication with parents.

## Data Sources {#data-sources .unnumbered}

Analyses of supplemental educational services pose several challenges,
including: obtaining student‑level achievement data that are
longitudinally linked and enable examination of student achievement
gains over time, acquiring achievement data for students prior to their
receiving supplemental educational services, and determining an
appropriate comparison group for participants. Further, it may be
difficult for districts to provide complete and accurate data on student
eligibility for supplemental educational services. A student is eligible
for supplemental educational services if the student attends a Title I
school that is in the second year of school improvement, in corrective
action, or in restructuring, and if the student comes from a low‑income
family. To determine whether a student is from a low-income family, the
district must use the same poverty measure that it uses to rank its
Title I schools. In most districts, student eligibility for free or
reduced‑price lunches is used to determine family income levels.
However, district datafiles on student eligibility for free or
reduced‑price lunch are frequently missing data for some students, in
part because not all eligible students are enrolled in this program. .

Anchorage and Hillsborough each provided data on student eligibility,
participation, the provider from whom each student received services,
and the number of hours of tutoring participants received in 2004--05
through 2006--07 (see Exhibit 1).

In Anchorage, student‑level data on limited English proficiency and
special education status were not available. Therefore, no separate
analyses for these subgroups were conducted for Anchorage.

### Determining Eligibility and Participation {#determining-eligibility-and-participation .unnumbered}

For the purposes of these analyses, the research team established
parameters to determine student eligibility for supplemental educational
services. First, any student in a school not required to offer
supplemental educational services (based on district designations of
school eligibility) was considered ineligible. In addition, if student
demographic data indicated that students were not eligible for services
by virtue of not being eligible for free or reduced‑price lunch, they
were considered ineligible for the purpose of the analysis, even if they
were, in fact, receiving services (the number of students removed from
the analysis for this reason was very small).

+--------------------+-----------------------+------------------------+
| []{#__RefHead      |                       |                        |
| ing___Toc220485721 |                       |                        |
| .anchor}Exhibit 1\ |                       |                        |
| Data Available     |                       |                        |
| From Anchorage and |                       |                        |
| Hillsborough       |                       |                        |
+--------------------+-----------------------+------------------------+
|                    | Anchorage             | Hillsborough           |
+--------------------+-----------------------+------------------------+
| Eligibility and    |                       |                        |
| Participation Data |                       |                        |
+--------------------+-----------------------+------------------------+
| Eligibility        | 2004--05, 2005--06,   | 2002--03, 2003--04,    |
|                    | 2006--07              | 2004--05, 2005--06,    |
|                    |                       | 2006--07               |
+--------------------+-----------------------+------------------------+
| Participation      | 2004--05, 2005--06,   | 2004--05, 2005--06,    |
|                    | 2006--07              | 2006--07               |
+--------------------+-----------------------+------------------------+
| Grades with data   | K--12                 | K--12                  |
| available          |                       |                        |
+--------------------+-----------------------+------------------------+
| Participation Data |                       |                        |
+--------------------+-----------------------+------------------------+
| Provider name and  | 2004--05, 2005--06,   | 2004--05, 2005--06,    |
| type               | 2006--07              | 2006--07               |
+--------------------+-----------------------+------------------------+
|  Hours of services | 2004--05, 2005--06,   | 2004--05, 2005--06,    |
| received           | 2006--07              | 2006--07               |
+--------------------+-----------------------+------------------------+
| Subject of         | 2004--05, 2005--06,   | 2004--05, 2005--06,    |
| tutoring           | 2006--07              | 2006--07               |
+--------------------+-----------------------+------------------------+
| Grades with data   | K--12                 | K--12                  |
| available          |                       |                        |
+--------------------+-----------------------+------------------------+
| Student            |                       |                        |
| Achievement Data   |                       |                        |
+--------------------+-----------------------+------------------------+
| Test (years)       | Benchmark (2002--03   | FCAT (2002--03 to      |
|                    | to 2003--04);         | 2006--07)              |
|                    |                       |                        |
|                    | SBA (2004--05 to      |                        |
|                    | 2006--07)             |                        |
+--------------------+-----------------------+------------------------+
| Tested grades      | 3,6, and 8 (2002--03  | 1--12 (2002--03 to     |
|                    | to 2003--04)\         | 2004--05)              |
|                    | 3--9 (2004--05)\      |                        |
|                    | 3--10 (2005--06 to    | 2--12 (2005--06)       |
|                    | 2006--07)             |                        |
|                    |                       | 3--12 (2006--07)       |
+--------------------+-----------------------+------------------------+

###   {#section-4 .unnumbered}

### Measuring Achievement {#measuring-achievement .unnumbered}

Each district provided longitudinal student achievement data from their
state assessment. In Anchorage, all students in grades 3--9 in 2004--05,
and grades 3--10 in 2005--06 and 2006--07 took the Alaska Standards
Based Assessments (SBAs). Before that, achievement data was only
available for students in grades 3, 6, and 8, on the previous state
assessment, the Alaska Benchmark Exam. Hillsborough provided reading and
mathematics scale scores on Florida's Comprehensive Assessment Test
(FCAT) for all students in grades 1--12 from 2002--03 through 2004--05,
grades 2--12 in 2005--06, and grades 3--12 in 2006--07. More information
on the data received from each district is provided in Appendix A.

The analytic models in this study estimate student achievement gains
over time, comparing students who do and do not participate in
supplemental educational services. The analyses use a fixed‑effects
approach, which operates by taking the within‑student difference of the
characteristics under study, and comparing the pre‑treatment and
post‑treatment characteristics of each individual student. By examining
the performance of students before and after receiving services, we are
able to gain a "value‑added" estimate of the gains associated with the
services.

### Communication {#communication .unnumbered}

Each district provided information about their communication efforts
with parents. The research team examined examples of submitted documents
and reviewed the methods of distribution as reported by the districts.

# Findings: Eligibility for and Participation in Supplemental Educational Services

Although data on Anchorage and Hillsborough are presented together in
this analysis when possible, it is important to note that distinct
features of each district, such as administration of supplemental
educational services and supply of providers, pose challenges to a
side-by-side comparison of the individual district's results. While it
is beyond the scope of this analysis to explore the reasons for possible
variations in results for the districts, explanations for differences in
district results might include:

-   **Context:** District size, demographic characteristics, prior
    achievement levels, and annual fluctuations in the number of
    students eligible for and participating in supplemental educational
    services might each influence the effect of supplemental educational
    services on student achievement.

-   **Implementation:** District capacity for implementing and
    monitoring supplemental educational services likely varies across
    districts. Differences in district implementation, such as
    notification practices and enrollment opportunities, and differences
    in the duration, nature, and quality of the various supplemental
    educational services programs offered in each district, including
    the hours and location of services and tutor‑student ratios, may
    contribute to differences in results.

Throughout the rest of the report, results for Anchorage and
Hillsborough are shown together when the data available for each
district are similar. However, with respect to achievement results, we
cannot make direct comparisons between the two districts because the
achievement measures and the grades tested in each district differ.

## Student Eligibility for Supplemental Educational Services {#student-eligibility-for-supplemental-educational-services .unnumbered}

In Anchorage, 6 percent of students were eligible for supplemental
educational services during 2006--07. In Hillsborough, about 20 percent
of students were eligible for these services in 2006--07.

The percentages of students eligible for supplemental educational
services increased from 2004--05 to 2006--07 in both districts. In
Hillsborough, the percentage of students eligible for supplemental
educational services increased from 1 percent to 20 percent, while in
Anchorage, the eligibility rate increased from 1 percent to 6 percent
(see Exhibit 2).

+-----------------------------------------------------------------------+
| []{#__RefHeading___Toc220485722 .anchor}Exhibit 2\                    |
| Number and Percentage of Eligible Students in Grades K--12 Who        |
| Participated in Supplemental Educational Services, in Anchorage and   |
| Hillsborough,\                                                        |
| 2004--05 Through 2006--07                                             |
+-----------------------------------------------------------------------+
|   ----                                                                |
| --------- --------------- ------------- -------------- -------------- |
|                                                                       |
|            Number Eligible Percent       Number of      Percentage of |
|                                 Eligible      Participants   Eligible |
|                                                                       |
|                                                          Students who |
|                                                                       |
|                                                          Participated |
|                                                                       |
|                 Anchorage,                                            |
|                 Alaska                                                |
|                                                                       |
|   2004--05      709             1%            30             4%       |
|                                                                       |
|   2005--06      2,093           4%            100            5%       |
|                                                                       |
|   2006--07      2,890           6%            308            11%      |
|                                                                       |
|                 Hillsborough,                                         |
|                 Florida                                               |
|                                                                       |
|   2004--05      1,931           1%            398            2%       |
|                                                                       |
|   2005--06      38,184          18%           3,771          10%      |
|                                                                       |
|   2006--07      43,276          20%           6,172          14%      |
|   ----                                                                |
| --------- --------------- ------------- -------------- -------------- |
+-----------------------------------------------------------------------+
| **Exhibit reads:** In 2004--05, 709 students were eligible for        |
| supplemental educational services in Anchorage, Alaska.               |
|                                                                       |
| Source: Anchorage School District Administrative Data, 2002--03       |
| through 2006--07; Hillsborough School District Administrative Data,   |
| 2002--03 through 2006--07.                                            |
+-----------------------------------------------------------------------+

## Participation in Supplemental Educational Services {#participation-in-supplemental-educational-services .unnumbered}

In both districts, the percentage of eligible students participating in
supplemental educational services increased between 2004--05 and
2006--07, reaching 11 percent in Anchorage and 14 percent Hillsborough
in 2006--07.

The total number of students receiving supplemental educational services
in the Anchorage school district has been notably low.[^4] In 2004--05
only 30 students received services, which increased to 308 during
2006--07 (see Exhibit 2). These low numbers are due in part to the low
number of students eligible for these services in Anchorage and also to
a low ratio of eligible students actually participating. During the
2006--07 school year, 11 percent of students eligible for services
received them, an increase from the 4 percent who participated in
supplemental educational services in 2004--05.

In Hillsborough, the total number of students receiving supplemental
educational services was considerably larger than in Anchorage. Though
there were just under 400 participants in 2004--05, there were about
3,800 and 6,200 participants in 2005--06 and 2006--07, respectively (see
Exhibit 2). This larger number of participants was a result of a much
larger pool of eligible students. The participation rates of eligible
students in Hillsborough were higher than in Anchorage (i.e., 10 versus
5 percent in 2005--06, and 14 versus 11 percent in 2006--07).

In Anchorage, the majority of supplemental educational services
participants were enrolled in elementary and middle school grades, while
in Hillsborough almost all of them were in elementary school grades.

In Anchorage, the distribution of supplemental educational services
participants was fairly evenly distributed between elementary and middle
school grades. During the 2006--07 school year, 63 and 35 percent of
participants were in grades 1--5 and 6--8, respectively (see Exhibit 3).

The students who participated in supplemental educational services in
Hillsborough were mainly enrolled in elementary school grades
(Kindergarten though grade 5): 89 and 10 percent of all participants
were in elementary and middle school grades, respectively, in 2006--07
(see Exhibit 3).

+-----------------------------------------------------------------------+
| []{#__RefHeading___Toc220485723 .anchor}Exhibit 3\                    |
| Distribution of Supplemental Educational Services Participants, by    |
| Grade Level, in Anchorage and Hillsborough, 2005--06 and 2006--07     |
+-----------------------------------------------------------------------+
| ![](media/image3.png){width="3.3625in" height="2.998611111111111in"}  |
+-----------------------------------------------------------------------+
| **Exhibit reads:** Among supplemental educational services            |
| participants in Anchorage in 2005--06, 55 percent of participants     |
| were in elementary school grades (K--5), and 45 percent in middle     |
| school grades (6--8). There were no participants in high school       |
| grades (9--12).                                                       |
|                                                                       |
| Source: Anchorage School District Administrative Data, 2002--03       |
| through 2006--07; Hillsborough School District Administrative Data,   |
| 2002--03 through 2006--07.                                            |
+-----------------------------------------------------------------------+

The Anchorage school district served 45 percent of the students
receiving supplemental educational services in 2006--07. In previous
years, all participants were served by non‑district providers.

As mentioned above, the Anchorage school district began serving as a
supplemental educational service provider in 2006--07 after it received
its waiver from the U.S. Department of Education. During that year, the
Anchorage school district served 138 students in the program, or 45
percent of all 308 participants. In the previous years, non‑district
providers served all students who received supplemental educational
services (see Exhibit 4).

+-----------------------------------------------------------------------+
| []{#__RefHeading___Toc220485724 .anchor}Exhibit 4\                    |
| Number of Supplemental Educational Services Participants in Grades    |
| K--12, by Provider Type, in Anchorage, 2004--05 Through 2006--07      |
+-----------------------------------------------------------------------+
| ![](media/image1.png){width="2.39375in"                               |
| height="2.6215277777777777in"}                                        |
+-----------------------------------------------------------------------+
| **Exhibit reads:** In Anchorage in 2006--07, 138 participants, or 45  |
| percent of all participants, received supplemental educational        |
| services from the district provider, and 170 students were served by  |
| non‑district providers.                                               |
|                                                                       |
| Source: Anchorage School District Administrative Data, 2002--03       |
| through 2006--07.                                                     |
+-----------------------------------------------------------------------+

In Anchorage, five providers served students in 2005--06 and in
2006--07. Two providers served students in both years, so there were a
total of eight providers Anchorage serving students across the two
years. Three providers in 2005--06 and 2006--07 had fewer than 20
participants (see Exhibit 5).[^5]

+-----------------------+----------+-----------+----------+-----------+
| []{#__RefH            |          |           |          |           |
| eading___Toc220485725 |          |           |          |           |
| .anchor}Exhibit 5\    |          |           |          |           |
| Number and Percentage |          |           |          |           |
| of Supplemental       |          |           |          |           |
| Educational Services  |          |           |          |           |
| Participants in       |          |           |          |           |
| Grades K--12, by      |          |           |          |           |
| Provider, in          |          |           |          |           |
| Anchorage, 2005--06   |          |           |          |           |
| and 2006--07          |          |           |          |           |
+-----------------------+----------+-----------+----------+-----------+
|                       | 2005--06 |           | 2006--07 |           |
+-----------------------+----------+-----------+----------+-----------+
|                       | \#       | \%        | \#       | \%        |
|                       | Part     | Par       | Part     | Par       |
|                       | icipants | ticipants | icipants | ticipants |
+-----------------------+----------+-----------+----------+-----------+
| Anchorage School      | NA       | --        | 138      | 45%       |
| District              |          |           |          |           |
+-----------------------+----------+-----------+----------+-----------+
| Private Provider A    | NA       | --        | 128      | 42%       |
+-----------------------+----------+-----------+----------+-----------+
| Private Provider B    | 28       | 28%       | 19       | 6%        |
+-----------------------+----------+-----------+----------+-----------+
| Private Provider C    | NA       | --        | 16       | 5%        |
+-----------------------+----------+-----------+----------+-----------+
| Private Provider D    | 8        | 8%        | 7        | 2%        |
+-----------------------+----------+-----------+----------+-----------+
| Private Provider E    | 61       | 61%       | NA       | --        |
+-----------------------+----------+-----------+----------+-----------+
| Private Provider F    | 2        | 2%        | NA       | --        |
+-----------------------+----------+-----------+----------+-----------+
| Private Provider G    | 1        | 1%        | NA       | --        |
+-----------------------+----------+-----------+----------+-----------+
| **All Participants**  | 100      | 100%      | 308      | 100%      |
+-----------------------+----------+-----------+----------+-----------+
| **Exhibit reads:** In |          |           |          |           |
| Anchorage in          |          |           |          |           |
| 2006--07, 138         |          |           |          |           |
| supplemental          |          |           |          |           |
| educational services  |          |           |          |           |
| participants were     |          |           |          |           |
| served by the         |          |           |          |           |
| Anchorage School      |          |           |          |           |
| District,             |          |           |          |           |
| representing 45       |          |           |          |           |
| percent of all        |          |           |          |           |
| participants.         |          |           |          |           |
|                       |          |           |          |           |
| Notes: NA indicates   |          |           |          |           |
| that the provider did |          |           |          |           |
| not provide services  |          |           |          |           |
| during that year.     |          |           |          |           |
|                       |          |           |          |           |
| Source: Anchorage     |          |           |          |           |
| School District       |          |           |          |           |
| Administrative Data,  |          |           |          |           |
| 2002--03 through      |          |           |          |           |
| 2006--07.             |          |           |          |           |
+-----------------------+----------+-----------+----------+-----------+

The Hillsborough school district served 971 participants (16 percent of
the total) in 2006--07. It did not offer supplemental educational
services during the previous school years.

As in Anchorage, the Hillsborough school district started providing
supplemental educational services in 2006--07 after it received its
waiver from the U.S. Department of Education, and served 16 percent of
all participants that school year. The other 5,201 students were served
by non‑district providers, a figure that increased by about 40 percent
from the 3,771 students served by these non‑district providers the year
before (see Exhibit 6).

+-----------------------------------------------------------------------+
| []{#__RefHeading___Toc220485726 .anchor}Exhibit 6\                    |
| Number of Supplemental Educational Services Participants in Grades    |
| K--12, by Provider Type, in Hillsborough, 2004--05 Through 2006--07   |
+-----------------------------------------------------------------------+
| ![](media/image2.png){width="2.479861111111111in"                     |
| height="2.6215277777777777in"}                                        |
+-----------------------------------------------------------------------+
| **Exhibit reads:** In Hillsborough in 2006--07, 971 participants, or  |
| 16 percent of all participants, received supplemental educational     |
| services from the Hillsborough school district and 5,201, or 84       |
| percent of participants, received supplemental educational services   |
| from non-district providers                                           |
|                                                                       |
| Source: Hillsborough School District Administrative Data, 2002--03    |
| through 2006--07.                                                     |
+-----------------------------------------------------------------------+

The number of supplemental educational service providers was
substantially higher in Hillsborough than in Anchorage. There were 23
service providers in Hillsborough in 2005--06 and 24 in 2006--07,
respectively (see Exhibit 7). In Hillsborough only one provider served
fewer than 20 students in 2005--06 and only three providers served fewer
than 20 students in 2006--07.

It is important to note that, in Hillsborough in 2005--06 and 2006--07,
some participants received services from more than one provider.[^6]

+------------------------------------+--------+-------+-------+-------+
| []{#__RefHeading___Toc220485727    |        |       |       |       |
| .anchor}Exhibit 7\                 |        |       |       |       |
| Number and Percentage of           |        |       |       |       |
| Supplemental Educational Services  |        |       |       |       |
| Participants in Grades K--12, by   |        |       |       |       |
| Provider, in Hillsborough,         |        |       |       |       |
| 2005--06 and 2006--07              |        |       |       |       |
+------------------------------------+--------+-------+-------+-------+
|                                    | 20     |       | 200   |       |
|                                    | 05--06 |       | 6--07 |       |
+------------------------------------+--------+-------+-------+-------+
|                                    | N      | \%    | N     | \%    |
+------------------------------------+--------+-------+-------+-------+
| Hillsborough County Public         | NA     | --    | 971   | 16%   |
| Schools---Academy For Success      |        |       |       |       |
| (District provider)                |        |       |       |       |
+------------------------------------+--------+-------+-------+-------+
| Private Provider H                 | 637    | 17%   | 1,226 | 20%   |
+------------------------------------+--------+-------+-------+-------+
| Private Provider I                 | 739    | 19%   | 820   | 13%   |
+------------------------------------+--------+-------+-------+-------+
| Private Provider J                 | 121    | 3%    | 787   | 13%   |
+------------------------------------+--------+-------+-------+-------+
| Private Provider K                 | 309    | 8%    | 332   | 5%    |
+------------------------------------+--------+-------+-------+-------+
| Private Provider L                 | 328    | 9%    | 312   | 5%    |
+------------------------------------+--------+-------+-------+-------+
| Private Provider M                 | NA     | --    | 298   | 5%    |
+------------------------------------+--------+-------+-------+-------+
| Private Provider B                 | NA     | --    | 276   | 0%    |
+------------------------------------+--------+-------+-------+-------+
| Private Provider N                 | 336    | 9%    | 214   | 3%    |
+------------------------------------+--------+-------+-------+-------+
| Private Provider O                 | 183    | 5%    | 150   | 2%    |
+------------------------------------+--------+-------+-------+-------+
| Private Provider P                 | 72     | 2%    | 148   | 2%    |
+------------------------------------+--------+-------+-------+-------+
| Private Provider Q                 | 40     | 1%    | 121   | 2%    |
+------------------------------------+--------+-------+-------+-------+
| Private Provider R                 | 308    | 8%    | 106   | 2%    |
+------------------------------------+--------+-------+-------+-------+
| Private Provider S                 | NA     | --    | 84    | 1%    |
+------------------------------------+--------+-------+-------+-------+
| Private Provider T                 | 66     | 2%    | 72    | 1%    |
+------------------------------------+--------+-------+-------+-------+
| Private Provider A                 | NA     | --    | 71    | 1%    |
+------------------------------------+--------+-------+-------+-------+
| Private Provider U                 | 75     | 2%    | 67    | 1%    |
+------------------------------------+--------+-------+-------+-------+
| Private Provider V                 | 28     | \<1%  | 61    | 1%    |
+------------------------------------+--------+-------+-------+-------+
| Private Provider W                 | NA     | --    | 54    | \<1%  |
+------------------------------------+--------+-------+-------+-------+
| Private Provider Y                 | 97     | 3%    | 36    | \<1%  |
+------------------------------------+--------+-------+-------+-------+
| Private Provider Z                 | NA     | --    | 29    | \<1%  |
+------------------------------------+--------+-------+-------+-------+
| Private Provider AA                | NA     | --    | 5     | \<1%  |
+------------------------------------+--------+-------+-------+-------+
| Private Provider AB                | NA     | --    | 4     | \<1%  |
+------------------------------------+--------+-------+-------+-------+
| Private Provider AC                | NA     | --    | 2     | \<1%  |
+------------------------------------+--------+-------+-------+-------+
| Private Provider AD                | 135    | 4%    | NA    | --    |
+------------------------------------+--------+-------+-------+-------+
| Private Provider AE                | 80     | 2%    | NA    | --    |
+------------------------------------+--------+-------+-------+-------+
| Private Provider AF                | 56     | 2%    | NA    | --    |
+------------------------------------+--------+-------+-------+-------+
| Private Provider AG                | 54     | 1%    | NA    | --    |
+------------------------------------+--------+-------+-------+-------+
| Private Provider AH                | 54     | 1%    | NA    | --    |
+------------------------------------+--------+-------+-------+-------+
| Private Provider AI                | 43     | 1%    | NA    | --    |
+------------------------------------+--------+-------+-------+-------+
| Private Provider AJ                | 40     | 1%    | NA    | --    |
+------------------------------------+--------+-------+-------+-------+
| Private Provider AK                | 32     | \<1%  | NA    | --    |
+------------------------------------+--------+-------+-------+-------+
| Private Provider AL                | 17     | \<1%  | NA    | --    |
+------------------------------------+--------+-------+-------+-------+
| Total                              | **3    | **1   | **6,  | **1   |
|                                    | ,850** | 00%** | 246** | 00%** |
+------------------------------------+--------+-------+-------+-------+
| **Exhibit reads:** In Hillsborough |        |       |       |       |
| in 2006--07, 971 supplemental      |        |       |       |       |
| educational services participants  |        |       |       |       |
| were served by the district,       |        |       |       |       |
| representing 16 percent of all     |        |       |       |       |
| participants.                      |        |       |       |       |
|                                    |        |       |       |       |
| Note: NA indicates that the        |        |       |       |       |
| provider did not provide services  |        |       |       |       |
| during that year.                  |        |       |       |       |
|                                    |        |       |       |       |
| Source: Hillsborough School        |        |       |       |       |
| District Administrative Data,      |        |       |       |       |
| 2002--03 through 2006--07.         |        |       |       |       |
+------------------------------------+--------+-------+-------+-------+

## Characteristics of Eligible and Participating Students {#characteristics-of-eligible-and-participating-students .unnumbered}

African-American and Alaska Native students constituted the largest
ethnic groups of all students receiving supplemental educational
services in Anchorage. Students served by the Anchorage school district
were more likely to be Alaska Native or Hispanic, while students served
by non‑district providers were more likely to be African-American or
Asian.

In Anchorage, African-American and Alaska Native students constituted
the largest ethnic groups of all participants, representing 21 and 20
percent of the total, respectively. Other racial‑ethnic groups that each
comprised over ten percent of participants were white, Asian,
multi‑racial, and Hispanic students (see Exhibit 8).

+------------------+-------------+------------+-----------+----------+
| []{#__RefHeadin  |             |            |           |          |
| g___Toc220485728 |             |            |           |          |
| .a               |             |            |           |          |
| nchor}Exhibit 8\ |             |            |           |          |
| Demographic      |             |            |           |          |
| Characteristics  |             |            |           |          |
| of Supplemental  |             |            |           |          |
| Educational      |             |            |           |          |
| Services         |             |            |           |          |
| Participants in  |             |            |           |          |
| Grades K--12, by |             |            |           |          |
| Provider Type,   |             |            |           |          |
| in Anchorage,    |             |            |           |          |
| 2006--07         |             |            |           |          |
+------------------+-------------+------------+-----------+----------+
|                  | All         | All        | Anchorage | Non-     |
|                  | Eligible    | Pa         | School    | District |
|                  |             | rticipants | District  | P        |
|                  |             |            |           | roviders |
+------------------+-------------+------------+-----------+----------+
| Distribution by  |             |            |           |          |
| Race and         |             |            |           |          |
| Ethnicity        |             |            |           |          |
+------------------+-------------+------------+-----------+----------+
| \%               | 14\*\*      | 21\*\*     | 15\*      | 25\*     |
| African-American |             |            |           |          |
+------------------+-------------+------------+-----------+----------+
| \% Alaska Native | 25\*\*      | 20\*\*     | 22        | 18       |
+------------------+-------------+------------+-----------+----------+
| \% White         | 16          | 17         | 16        | 17       |
+------------------+-------------+------------+-----------+----------+
| \% Asian         | 25\*\*      | 16\*\*     | 11\*      | 19\*     |
+------------------+-------------+------------+-----------+----------+
| \% Multi‑racial  | 11\*\*      | 14\*\*     | 16        | 13       |
+------------------+-------------+------------+-----------+----------+
| \% Hispanic      | 8\*\*       | 12\*\*     | 20\*      | 6\*      |
+------------------+-------------+------------+-----------+----------+
| \% American      | 1           | 1          | 1         | 1        |
| Indian           |             |            |           |          |
+------------------+-------------+------------+-----------+----------+
| Total            | 100%        | 100%       | 100%      | 100%     |
+------------------+-------------+------------+-----------+----------+
| N                | 2,890       | 308        | 138       | 170      |
+------------------+-------------+------------+-----------+----------+
| **Exhibit        |             |            |           |          |
| reads:** In      |             |            |           |          |
| Anchorage in     |             |            |           |          |
| 2006--07, 14     |             |            |           |          |
| percent of       |             |            |           |          |
| students         |             |            |           |          |
| eligible for     |             |            |           |          |
| supplemental     |             |            |           |          |
| educational      |             |            |           |          |
| services were    |             |            |           |          |
| African-American |             |            |           |          |
| and 21 percent   |             |            |           |          |
| of all           |             |            |           |          |
| participants     |             |            |           |          |
| were             |             |            |           |          |
| A                |             |            |           |          |
| frican-American. |             |            |           |          |
| Fifteen percent  |             |            |           |          |
| of supplemental  |             |            |           |          |
| educational      |             |            |           |          |
| services         |             |            |           |          |
| participants     |             |            |           |          |
| served by the    |             |            |           |          |
| district were    |             |            |           |          |
| A                |             |            |           |          |
| frican-American, |             |            |           |          |
| while 25 percent |             |            |           |          |
| of participants  |             |            |           |          |
| served by        |             |            |           |          |
| non‑district     |             |            |           |          |
| providers were   |             |            |           |          |
| A                |             |            |           |          |
| frican-American. |             |            |           |          |
|                  |             |            |           |          |
| Note: \*         |             |            |           |          |
| indicates a      |             |            |           |          |
| significant      |             |            |           |          |
| difference       |             |            |           |          |
| between district |             |            |           |          |
| and non-district |             |            |           |          |
| providers at the |             |            |           |          |
| .05 level. \*\*  |             |            |           |          |
| indicates a      |             |            |           |          |
| significant      |             |            |           |          |
| difference       |             |            |           |          |
| between all      |             |            |           |          |
| eligible         |             |            |           |          |
| students and all |             |            |           |          |
| participants at  |             |            |           |          |
| the .05 level.   |             |            |           |          |
|                  |             |            |           |          |
| Source:          |             |            |           |          |
| Anchorage School |             |            |           |          |
| District         |             |            |           |          |
| Administrative   |             |            |           |          |
| Data, 2002--03   |             |            |           |          |
| through          |             |            |           |          |
| 2006--07.        |             |            |           |          |
+------------------+-------------+------------+-----------+----------+

There were statistically significant differences in the ethnic
background of students served by district and non‑district providers in
Anchorage. The Anchorage school district tended to serve more Alaska
Native and Hispanic students, while non‑district providers were more
likely to serve African-American and Asian students. The differences in
the percentages of white and multi‑racial students across these
providers were less pronounced, though still statistically significant.

Moreover, African-American students---and to a lesser degree white,
multi‑racial and Hispanic students---were more likely to participate in
supplemental educational services than were Alaska Native and Asian
students. Appendix B presents participation rates by racial-ethnic group
in Anchorage.

Data on English proficiency and special education status were not
available for Anchorage.

In Hillsborough, Hispanic and African-American students represented more
than 80 percent of all students receiving supplemental educational
services.

In Hillsborough, Hispanic and African-American students constituted the
vast majority, over 80 percent, of participants. The differences in
student ethnic background across district and non‑district providers
were less pronounced in Hillsborough than in Anchorage. African-American
and white students were more likely to choose the district provider for
services, while Hispanic students were more likely to choose a
non‑district provider. Hispanic students were also more likely to
participate in supplemental educational services than were white
students (see Exhibit 9). Appendix B presents participation rates by
racial-ethnic group in Hillsborough.

About 45 percent of all students receiving supplemental educational
services in Hillsborough were limited English proficient. The pool of
students receiving services from the district had a slightly lower
percentage of limited English proficient students than those served by
non-district providers (42 versus 46 percent, respectively). In
addition, 21 percent of supplemental educational services participants
were special education students. Students served by the Hillsborough
school district were somewhat more likely to be special education
students than those served by non-district providers (see Exhibit 9).

+----------------------+---------+------------+-----------+----------+
| []{#__RefHe          |         |            |           |          |
| ading___Toc220485729 |         |            |           |          |
| .anchor}Exhibit 9\   |         |            |           |          |
| Demographic          |         |            |           |          |
| Characteristics of   |         |            |           |          |
| Supplemental         |         |            |           |          |
| Educational Services |         |            |           |          |
| Participants in      |         |            |           |          |
| Grades K--12, by     |         |            |           |          |
| Provider Type, in    |         |            |           |          |
| Hillsborough,        |         |            |           |          |
| 2006--07             |         |            |           |          |
+----------------------+---------+------------+-----------+----------+
|                      | All     | All        | Hil       | Non-     |
|                      | E       | Pa         | lsborough | District |
|                      | ligible | rticipants | School    | P        |
|                      |         |            | District  | roviders |
+----------------------+---------+------------+-----------+----------+
| Distribution by Race |         |            |           |          |
| and Ethnicity        |         |            |           |          |
+----------------------+---------+------------+-----------+----------+
| \% African-American  | 35\*\*  | 33\*\*     | 35        | 33       |
+----------------------+---------+------------+-----------+----------+
| \% White             | 17\*\*  | 14\*\*     | 15        | 13       |
+----------------------+---------+------------+-----------+----------+
| \% Asian             | 1       | 1          | 1         | 1        |
+----------------------+---------+------------+-----------+----------+
| \% Multi‑racial      | 4       | 4          | 3\*       | 4\*      |
+----------------------+---------+------------+-----------+----------+
| \% Hispanic          | 43\*\*  | 48\*\*     | 45\*      | 49\*     |
+----------------------+---------+------------+-----------+----------+
| \% American Indian   | \<1     | \<1        | \<1       | \<1      |
+----------------------+---------+------------+-----------+----------+
| Students with        |         |            |           |          |
| Special Needs        |         |            |           |          |
+----------------------+---------+------------+-----------+----------+
| \% Limited English   | 38      | 45         | 42\*      | 46\*     |
| proficient           |         |            |           |          |
+----------------------+---------+------------+-----------+----------+
| \% Special education | 18      | 21         | 24\*      | 20\*     |
+----------------------+---------+------------+-----------+----------+
| N                    | 43,276  | 6,172      | 971       | 5,201    |
+----------------------+---------+------------+-----------+----------+
| **Exhibit reads**:   |         |            |           |          |
| In Hillsborough in   |         |            |           |          |
| 2006--07, 35 percent |         |            |           |          |
| of students eligible |         |            |           |          |
| for supplemental     |         |            |           |          |
| educational services |         |            |           |          |
| were                 |         |            |           |          |
| African-American and |         |            |           |          |
| 33 percent of all    |         |            |           |          |
| participants were    |         |            |           |          |
| African-American.    |         |            |           |          |
| Thirty‑five percent  |         |            |           |          |
| of supplemental      |         |            |           |          |
| educational services |         |            |           |          |
| participants served  |         |            |           |          |
| by the district were |         |            |           |          |
| African-American,    |         |            |           |          |
| while 33 percent of  |         |            |           |          |
| participants served  |         |            |           |          |
| by non‑district      |         |            |           |          |
| providers were       |         |            |           |          |
| African-American.    |         |            |           |          |
|                      |         |            |           |          |
| Note: \* indicates a |         |            |           |          |
| significant          |         |            |           |          |
| difference between   |         |            |           |          |
| district and         |         |            |           |          |
| non-district         |         |            |           |          |
| providers at the .05 |         |            |           |          |
| level. \*\*          |         |            |           |          |
| indicates a          |         |            |           |          |
| significant          |         |            |           |          |
| difference between   |         |            |           |          |
| all eligible         |         |            |           |          |
| students and all     |         |            |           |          |
| participants at the  |         |            |           |          |
| .05 level.           |         |            |           |          |
|                      |         |            |           |          |
| Source: Hillsborough |         |            |           |          |
| School District      |         |            |           |          |
| Administrative Data, |         |            |           |          |
| 2002--03 through     |         |            |           |          |
| 2006--07.            |         |            |           |          |
+----------------------+---------+------------+-----------+----------+

## Hours and Subject of Tutoring {#hours-and-subject-of-tutoring .unnumbered}

Participants in the program received an average of 28 and 22 hours of
supplemental educational services in Anchorage and Hillsborough during
2006--07, respectively.

In 2006--07, the average number of hours of supplemental educational
services was 28 hours in Anchorage. Students served by the Anchorage
school district received, on average, six (or 24 percent) more service
hours than those who were served by non‑district providers. In the case
of Hillsborough, the average number of hours of tutoring and the
difference between district and non‑district providers was lower than in
Anchorage. In Hillsborough, participants received an average of 22 hours
of supplemental educational services. Non‑district providers served
students, on average, for about one hour more than the district provider
(see Exhibit 10).

+-----------------------------------------------------------------------+
| []{#__RefHeading___Toc220485730 .anchor}Exhibit 10\                   |
| Average Number of Hours of Supplemental Educational Services Received |
| by Participants in Grades K--12, by Provider Type, in Anchorage and   |
| Hillsborough, 2006--07                                                |
+-----------------------------------------------------------------------+
| ![](media/image4.png){width="2.754166666666667in"                     |
| height="3.1381944444444443in"}                                        |
+-----------------------------------------------------------------------+
| **Exhibit reads:** In Anchorage in 2006--07, the district provider    |
| delivered an average of 31 hours of supplemental educational services |
| per student, while non‑district providers delivered an average of 25  |
| hours of supplemental educational services per student. The average   |
| number of hours for services for all participating students was 28.   |
|                                                                       |
| Note: \* Indicates a statistically significant difference between the |
| district provider and non‑district providers at the .05 level.        |
|                                                                       |
| Source: Anchorage School District Administrative Data, 2002--03       |
| through 2006--07; Hillsborough School District Administrative Data,   |
| 2002--03 through 2006--07.                                            |
+-----------------------------------------------------------------------+

In Anchorage during 2006--07, 44 percent of students (135 students)
received only supplemental educational services in reading, while 17
percent (51 students) received only services in mathematics. Another 23
percent (72 students) received tutoring in both subjects. There are 50
supplemental educational services participants in Anchorage for whom
information on the subject of tutoring is missing (see Appendix C).
Students served by the Anchorage school district were more likely to
receive supplemental educational services in both subjects (39 percent
of district participants), compared to students served by non‑district
providers (11 percent). See Appendix C for further details.

In Hillsborough, about 79 percent of all participants (4,857 students)
received supplemental educational services only in reading during the
2006--07 school year. Twenty‑one percent of students (1,283 students)
received supplemental educational services only in mathematics. This
means that in Hillsborough only 32 students received supplemental
educational services in both subjects (see Appendix C). In terms of the
differences across district and non‑district providers, students served
by the Hillsborough school district were more likely to receive only
mathematics services (31 percent of students served by the district
compared to 19 percent of students served by non‑district providers
received support in mathematics only). See Appendix C for further
details.

# Findings: Student Achievement

## Analytic Approach {#analytic-approach .unnumbered}

The analytic approach follows the methods used for a previous analysis
of SES and student achievement in nine large urban school districts that
was conducted as part of the National Longitudinal Study of *No Child
Left Behind* (Zimmer et al., 2006). For Hillsborough, we use
longitudinal, student-level data for all students in grades 1--12 from
2002--03 through 2004--05, grades 2--12 in 2005--06, and grades 3--12 in
2006--07. The achievement test scores come from Florida's Comprehensive
Assessment Test (FCAT). In Hillsborough, information on participation in
supplemental educational services, type of provider, number of hours of
services, and subject of services was available for 2005--06 and
2006--07.

In Anchorage, we used longitudinal, student level data for all students
in grades 3--9 in 2004--05, and grades 3--10 in 2005--06 and 2006--07.
Unfortunately, before 2004--05 only achievement data for students in
grades 3, 6, and 8 were available. The achievement data in Anchorage
comes from Alaska's Standards Based Assessment (SBAs) for 2004--05
onwards, while previous achievement data come from the Alaska Benchmark
Exam.

The outcomes for both districts are rank‑based z‑scores that indicate a
student's position in the test score distribution for his or her grade
level. For example, a score of 0.5 indicates that the student scored
one half of a standard deviation above the district mean. It is
important to note that gains on these metrics do not indicate a
developmental improvement, such as a five-point gain, rather they
represent a student's movement within the test score distribution
relative to his or her classmates.[^7]

The analytic models estimate student achievement gains over time,
comparing students who do and do not participate in supplemental
educational services. By combining longitudinal data and student level
fixed‑effects, we can isolate the gains associated with participation in
supplemental educational services by parceling out the effect of time
invariant student characteristics, such as parent education, income or
underlying student ability or motivation. The fixed‑effects approach
operates by taking the within‑student difference of the characteristics
under study (e.g., eligibility for supplemental educational services,
participation in services, and provider chosen), and comparing the
pre‑treatment and post‑treatment characteristics of each individual
student. By examining the performance of students before and after
receiving services, we are able to gain a "value‑added" estimate of the
services. Further, by including a term for each individual student in
the analytic model, we further guard against selection bias that could
result if students choosing services were different from others in some
immeasurable way. This student-level fixed‑effect essentially controls
for the characteristics of each student that may otherwise affect their
achievement levels. For example, students who choose a program may be
more motivated than others, or may have other unmeasured traits (e.g.,
higher parental involvement or education) that may make them more likely
to achieve at higher levels even in absence of the program, which can
lead to biased results. Further, students who are more motivated or who
may have greater levels of innate ability (beyond what may be reflected
in a standardized test score) may benefit more from services. It is
possible that selection bias may lead to either overestimation or
underestimation of the true gains associated with the program.

[The gains model is:]{.underline}

\(1\) A~it~ -- A~it‑1~= α Eligible~it~ + λ Participation~it~ + µ~t~ +
θ~gt~ + ε~it~

*Where*:

Ait -- A~it‑1~ is the achievement gain of student i from year t--1 to
year t

Eligible~it~ is the eligibility status of student i in year t

Participation~it~ is the participation status of student i in year t

µ~t~ represents the individual student's fixed effect

θ~gt~ represents the grade by year fixed‑effects

ε~it~ represents the random error

An additional variation on the model includes interactions for
participation and student characteristics, which evaluates whether
certain groups of students may benefit more from supplemental
educational services. These models build on equation 1 by adding an
interaction term for the participation by a student group, such as
limited English proficient (LEP) students. A significant positive
coefficient for LEP students by participation interaction term would
indicate that LEP participants showed significant gains in
achievement.[^8]

In order to evaluate the gains associated with various characteristics
of supplemental educational services, such as provider type, hours,
etc., we estimate a separate model for each characteristic. We estimate
a total of 7 models for each district to evaluate the effect of the
following variables: overall participation; first year of participation
as compared to multiple years of participation; district versus
non‑district provider; individual providers; hours of services received;
and subject of tutoring. These models substitute the full set of options
for the participation variable, such as all providers, and estimate the
gains associated with each separately. For example, the model for
provider type is as follows:

\(2\) A~it~ -- A~it‑1~ = α Eligible~it~ + λ District Provider~it~ + ξ
Non‑District Provider~it~ + µ~t~ + θ~gt~ + ε~it~

The coefficient for the district provider variable indicates the
achievement gain for students served by the district's supplemental
educational services program, while the non‑district coefficient
indicates the gain for participants served by non-district
providers.[^9]

The analyses of district providers versus non-district providers are
restricted to those students who received supplemental educational
services during 2006-07, the year of the waiver. That is, students who
received SES services in 2004-05 or 2005-06 but not in 2006-07 were
excluded from these analyses.

As a result of the analytic approach described above, the results of the
achievement analyses reported in this chapter are based on a subset of
the full student populations in each district. While earlier chapters
included all students for whom we had information on eligibility and
participation information, this chapter includes only students in tested
grades. The analytic approach further requires that students have
achievement scores for at least two time points in either reading or
mathematics in order to construct a gain score. Finally, students who
repeated a grade were not included in the analyses. Given that the
outcome measure is designed to measure a student's progress in relation
to the rest of the students in his or her grade level, including a
student who has repeated a grade inherently leads to comparing an
on‑track student to a student who is one year older and has taken the
same test twice. For example, if we examine a cohort of students in
fifth grade in 2004--05, we look at their outcome as the difference
between their reading score in the prior year as fourth graders and
their reading score in 2004--05 as fifth graders, both of which are
standardized to the full population of test takers of that grade level
in that year. Any fifth graders who repeated fifth grade in that year
may have much larger gains, given they have taken the same test in both
years, and the non‑repeaters may appear to be doing less well.

In addition to describing the caveats to the analytical approach above,
is important to note that because the achievement measures as well as
the grades tested in each district differ, we are limited in our ability
to make direct comparisons between achievement results in Anchorage and
Hillsborough.

## Prior Achievement Levels by Student Eligibility for Supplemental Educational Services {#prior-achievement-levels-by-student-eligibility-for-supplemental-educational-services .unnumbered}

In both Anchorage and Hillsborough, students who were eligible for
supplemental educational services had significantly lower levels of
prior achievement in mathematics and reading than students who were not
eligible.

Comparing the previous year's achievement scores of students who were
eligible for supplemental educational services (students from low‑income
families who attend Title I schools that did not make AYP for the third
year) to the scores of students in the district who were not eligible
shows whether students who were eligible for supplemental educational
services were more likely to have lower levels of prior achievement. As
one would expect, eligible students in both districts had lower prior
achievement scores than ineligible students, in both reading and
mathematics.

As mentioned in the previous section, scale scores have been
standardized within grade (to a mean of zero, and a standard deviation
of one) in order to make comparisons across grades more meaningful. In
Anchorage, students who were eligible for supplemental educational
services in 2006--07 scored 0.57 and 0.69 standard deviations below the
overall district mean on the prior year's mathematics and reading test,
respectively. In contrast, ineligible students were just above the
overall district mean (see Exhibit 11).

+------------------+-------------------------+-------------------------+
| []{#__RefHeadin  |                         |                         |
| g___Toc220485731 |                         |                         |
| .an              |                         |                         |
| chor}Exhibit 11\ |                         |                         |
| Prior            |                         |                         |
| Achievement      |                         |                         |
| Levels (in       |                         |                         |
| 2005--06) on the |                         |                         |
| SBA of Students  |                         |                         |
| in Grades 3--10  |                         |                         |
| by Eligibility   |                         |                         |
| for Supplemental |                         |                         |
| Educational      |                         |                         |
| Services in      |                         |                         |
| 2006--07, in     |                         |                         |
| Anchorage        |                         |                         |
+------------------+-------------------------+-------------------------+
|                  | Mathematics             | Reading                 |
+------------------+-------------------------+-------------------------+
| Ineligible in    | 0.04                    | 0.05                    |
| 2006--07         |                         |                         |
+------------------+-------------------------+-------------------------+
| Eligible in      | --0.57\*                | --0.69\*                |
| 2006--07         |                         |                         |
+------------------+-------------------------+-------------------------+
| n                | 27,430                  | 27,376                  |
+------------------+-------------------------+-------------------------+
| **Exhibit        |                         |                         |
| reads:** In      |                         |                         |
| Anchorage,       |                         |                         |
| students who     |                         |                         |
| were not         |                         |                         |
| eligible for     |                         |                         |
| supplemental     |                         |                         |
| educational      |                         |                         |
| services in      |                         |                         |
| 2006--07 had     |                         |                         |
| levels of prior  |                         |                         |
| mathematics      |                         |                         |
| achievement in   |                         |                         |
| 2005--06 of      |                         |                         |
| about 0.04       |                         |                         |
| standard         |                         |                         |
| deviations above |                         |                         |
| the district     |                         |                         |
| average, while   |                         |                         |
| students who     |                         |                         |
| were eligible    |                         |                         |
| had prior        |                         |                         |
| mathematics      |                         |                         |
| achievement      |                         |                         |
| levels 0.57      |                         |                         |
| standard         |                         |                         |
| deviations below |                         |                         |
| the district     |                         |                         |
| average, a       |                         |                         |
| significant      |                         |                         |
| difference.      |                         |                         |
|                  |                         |                         |
| Notes: Means are |                         |                         |
| for all students |                         |                         |
| with at least    |                         |                         |
| one gain score   |                         |                         |
| and who are      |                         |                         |
| therefore        |                         |                         |
| included in the  |                         |                         |
| regression       |                         |                         |
| analyses. Scores |                         |                         |
| reported are     |                         |                         |
| rank‑based       |                         |                         |
| z‑scores.        |                         |                         |
|                  |                         |                         |
| \* indicates a   |                         |                         |
| significant      |                         |                         |
| difference       |                         |                         |
| between eligible |                         |                         |
| and ineligible   |                         |                         |
| students at the  |                         |                         |
| .05 level.       |                         |                         |
|                  |                         |                         |
| Source:          |                         |                         |
| Anchorage School |                         |                         |
| District         |                         |                         |
| Administrative   |                         |                         |
| Data, 2005--06   |                         |                         |
| through          |                         |                         |
| 2006--07.        |                         |                         |
+------------------+-------------------------+-------------------------+

The prior achievement gap between eligible and ineligible students was
much smaller in Hillsborough than in Anchorage, possibly a result of the
much higher percentage of eligible students in Hillsborough. Still, as
in Anchorage, students eligible for supplemental educational services in
Hillsborough had lower achievement levels than ineligible students.
Students who were eligible for supplemental educational services in
2006--07 scored 0.10 and 0.11 standard deviations below the overall
district mean on the prior year's mathematics and reading test,
respectively. On the other hand, ineligible students scored
0.02 standard deviations above the district mean on the prior year's
mathematics and reading tests (see Exhibit 12).

+------------------+-------------------------+-------------------------+
| []{#__RefHeadin  |                         |                         |
| g___Toc220485732 |                         |                         |
| .an              |                         |                         |
| chor}Exhibit 12\ |                         |                         |
| Prior            |                         |                         |
| Achievement      |                         |                         |
| Levels (in       |                         |                         |
| 2005--06) on the |                         |                         |
| FCAT of Students |                         |                         |
| in Grades 1--12  |                         |                         |
| by Eligibility   |                         |                         |
| for Supplemental |                         |                         |
| Educational      |                         |                         |
| Services in      |                         |                         |
| 2006--07, in     |                         |                         |
| Hillsborough     |                         |                         |
+------------------+-------------------------+-------------------------+
|                  | Mathematics             | Reading                 |
+------------------+-------------------------+-------------------------+
| Ineligible in    | 0.02                    | 0.02                    |
| 2006--07         |                         |                         |
+------------------+-------------------------+-------------------------+
| Eligible in      | --0.10\*                | --0.11\*                |
| 2006--07         |                         |                         |
+------------------+-------------------------+-------------------------+
| n                | 195,432                 | 195,432                 |
+------------------+-------------------------+-------------------------+
| **Exhibit        |                         |                         |
| reads:** In      |                         |                         |
| Hillsborough,    |                         |                         |
| students who     |                         |                         |
| were not         |                         |                         |
| eligible for     |                         |                         |
| supplemental     |                         |                         |
| educational      |                         |                         |
| services in      |                         |                         |
| 2006--07 had     |                         |                         |
| levels of prior  |                         |                         |
| mathematics      |                         |                         |
| achievement in   |                         |                         |
| 2005--06 of      |                         |                         |
| about 0.02       |                         |                         |
| standard         |                         |                         |
| deviations above |                         |                         |
| the district     |                         |                         |
| average, while   |                         |                         |
| students who     |                         |                         |
| were eligible    |                         |                         |
| had prior        |                         |                         |
| mathematics      |                         |                         |
| achievement      |                         |                         |
| levels 0.10      |                         |                         |
| standard         |                         |                         |
| deviations below |                         |                         |
| the district     |                         |                         |
| average, a       |                         |                         |
| significant      |                         |                         |
| difference.      |                         |                         |
|                  |                         |                         |
| Notes: Means are |                         |                         |
| for all students |                         |                         |
| with at least    |                         |                         |
| one gain score   |                         |                         |
| and who are      |                         |                         |
| therefore        |                         |                         |
| included in the  |                         |                         |
| regression       |                         |                         |
| analyses. Scores |                         |                         |
| reported are     |                         |                         |
| rank‑based       |                         |                         |
| z‑scores.        |                         |                         |
|                  |                         |                         |
| \* indicates a   |                         |                         |
| significant      |                         |                         |
| difference       |                         |                         |
| between eligible |                         |                         |
| and ineligible   |                         |                         |
| students at the  |                         |                         |
| .05 level.       |                         |                         |
|                  |                         |                         |
| Source:          |                         |                         |
| Hillsborough     |                         |                         |
| School District  |                         |                         |
| Administrative   |                         |                         |
| Data, 2005--06   |                         |                         |
| through          |                         |                         |
| 2006--07.        |                         |                         |
+------------------+-------------------------+-------------------------+

##  Prior Achievement Levels by Student Participation in Supplemental Educational Services {#prior-achievement-levels-by-student-participation-in-supplemental-educational-services .unnumbered}

Students who participated in supplemental educational services had lower
prior achievement levels than eligible non-participants in Anchorage and
Hillsborough

In Anchorage, students who participated in supplemental educational
services in 2006--07 performed worse in mathematics and reading than
eligible non‑participants in 2005--06. Participating students scored
0.95 and 1.03 standard deviations below the Anchorage district mean in
mathematics and reading, respectively, while eligible non‑participants
scored 0.52 and 0.64 standard deviations below this overall mean (see
Exhibit 13).

+--------------------------+---------------------+---------------------+
| []{#__R                  |                     |                     |
| efHeading___Toc220485733 |                     |                     |
| .anchor}Exhibit 13\      |                     |                     |
| Prior Achievement Levels |                     |                     |
| (in 2005--06) on the     |                     |                     |
| Benchmark of Students in |                     |                     |
| Grades 3--10 by          |                     |                     |
| Participation in         |                     |                     |
| Supplemental Educational |                     |                     |
| Services in 2006--07, in |                     |                     |
| Anchorage                |                     |                     |
+--------------------------+---------------------+---------------------+
|                          | Mathematics         | Reading             |
+--------------------------+---------------------+---------------------+
| Participants in 2006--07 | --0.95\*            | --1.03\*            |
+--------------------------+---------------------+---------------------+
| Eligible                 | --0.52              | --0.64              |
| non‑participants in      |                     |                     |
| 2006--07                 |                     |                     |
+--------------------------+---------------------+---------------------+
| Students who were        | 0.04                | 0.05                |
| ineligible in 2006--07   |                     |                     |
+--------------------------+---------------------+---------------------+
| N                        | 1,474               | 1,460               |
+--------------------------+---------------------+---------------------+
| **Exhibit reads:** In    |                     |                     |
| Anchorage, students who  |                     |                     |
| received supplemental    |                     |                     |
| educational services in  |                     |                     |
| 2006--07 had levels of   |                     |                     |
| prior mathematics        |                     |                     |
| achievement in 2005--06  |                     |                     |
| of about 0.95 standard   |                     |                     |
| deviations below the     |                     |                     |
| district average.        |                     |                     |
|                          |                     |                     |
| Notes: Means are for all |                     |                     |
| students with at least   |                     |                     |
| one gain score and who   |                     |                     |
| are therefore included   |                     |                     |
| in the regression        |                     |                     |
| analyses. Scores         |                     |                     |
| reported are rank‑based  |                     |                     |
| z‑scores.                |                     |                     |
|                          |                     |                     |
| \* indicates a           |                     |                     |
| significant difference   |                     |                     |
| between eligible         |                     |                     |
| non‑participants and     |                     |                     |
| participants at the .05  |                     |                     |
| level.                   |                     |                     |
|                          |                     |                     |
| Source: Anchorage School |                     |                     |
| District Administrative  |                     |                     |
| Data, 2005--06 through   |                     |                     |
| 2006--07.                |                     |                     |
+--------------------------+---------------------+---------------------+

In Hillsborough the pattern was similar, with participants having lower
prior achievement than eligible non‑participants. The academic
achievement of students who participated in supplemental educational
services in 2006--07 was equal to 0.18 standard deviations below the
district mean in 2005--06 across both subjects (see Exhibit 14). During
that year, the performance of eligible non‑participants ranged from 0.09
to 0.10 standard deviations below the district mean in mathematics and
reading, respectively. This indicates that eligible non‑participants
scored between 0.08 to 0.09 standard deviations above participants on
prior achievement tests.

+--------------------------+---------------------+---------------------+
| []{#__R                  |                     |                     |
| efHeading___Toc220485734 |                     |                     |
| .anchor}Exhibit 14\      |                     |                     |
| Prior Achievement Levels |                     |                     |
| (in 2005--06) on the     |                     |                     |
| FCAT of Students in      |                     |                     |
| Grades 1--12 by          |                     |                     |
| Participation in         |                     |                     |
| Supplemental Educational |                     |                     |
| Services in 2006--07, in |                     |                     |
| Hillsborough             |                     |                     |
+--------------------------+---------------------+---------------------+
|                          | Mathematics         | Reading             |
+--------------------------+---------------------+---------------------+
| Participants in 2006--07 | --0.18\*            | --0.18\*            |
+--------------------------+---------------------+---------------------+
| Eligible                 | --0.09              | --0.10              |
| non‑participants in      |                     |                     |
| 2006--07                 |                     |                     |
+--------------------------+---------------------+---------------------+
| Students who were        | 0.02                | 0.02                |
| ineligible in 2006--07   |                     |                     |
+--------------------------+---------------------+---------------------+
| n                        | 31,227              | 31,227              |
+--------------------------+---------------------+---------------------+
| **Exhibit reads:** In    |                     |                     |
| Hillsborough, students   |                     |                     |
| who received             |                     |                     |
| supplemental educational |                     |                     |
| services in 2006--07 had |                     |                     |
| levels of mathematics    |                     |                     |
| achievement in 2005--06  |                     |                     |
| of about 0.18 standard   |                     |                     |
| deviations below the     |                     |                     |
| district average.        |                     |                     |
|                          |                     |                     |
| Notes: Means are for all |                     |                     |
| students with at least   |                     |                     |
| one gain score and who   |                     |                     |
| are therefore included   |                     |                     |
| in the regression        |                     |                     |
| analyses. Scores         |                     |                     |
| reported are rank‑based  |                     |                     |
| z‑scores.                |                     |                     |
|                          |                     |                     |
| \* indicates a           |                     |                     |
| significant difference   |                     |                     |
| between eligible         |                     |                     |
| non‑participants and     |                     |                     |
| participants at the .05  |                     |                     |
| level.                   |                     |                     |
|                          |                     |                     |
| Source: Hillsborough     |                     |                     |
| School District          |                     |                     |
| Administrative Data,     |                     |                     |
| 2005--06 through         |                     |                     |
| 2006--07.                |                     |                     |
+--------------------------+---------------------+---------------------+

##   {#section-5 .unnumbered}

## Prior Achievement Levels by Provider Type {#prior-achievement-levels-by-provider-type .unnumbered}

In Hillsborough, participants served by the school district had higher
prior achievement levels than students served by other providers. In
Anchorage, prior student achievement across district and non-district
providers was statistically indistinguishable.

In Anchorage, there was no clear pattern of prior achievement levels in
2005--06 between students served by the district and those served by
non‑district providers. All of these students scored around 1 standard
deviation below the district mean in 2005--06 in mathematics and reading
(see Exhibit 15).

+------------------------+---------------------+----------------------+
| []{#__Ref              |                     |                      |
| Heading___Toc220485735 |                     |                      |
| .anchor}Exhibit 15\    |                     |                      |
| Prior Achievement      |                     |                      |
| Levels (in 2005--06)   |                     |                      |
| on the Benchmark of    |                     |                      |
| Students in            |                     |                      |
| Grades 3--10,          |                     |                      |
| by Provider Type in    |                     |                      |
| 2006--07, in Anchorage |                     |                      |
+------------------------+---------------------+----------------------+
|                        | Mathematics         | Reading              |
+------------------------+---------------------+----------------------+
| District provider in   | --1.05              | --1.06               |
| 2006--07               |                     |                      |
+------------------------+---------------------+----------------------+
| Non‑district providers | --0.89              | --1.01               |
| in 2006--07            |                     |                      |
+------------------------+---------------------+----------------------+
| n                      | 181                 | 182                  |
+------------------------+---------------------+----------------------+
| **Exhibit reads:** In  |                     |                      |
| Anchorage, students    |                     |                      |
| who were served in     |                     |                      |
| 2006--07 by the        |                     |                      |
| district provider had  |                     |                      |
| an average level of    |                     |                      |
| mathematics            |                     |                      |
| achievement of 1.05    |                     |                      |
| standard deviations    |                     |                      |
| below the district     |                     |                      |
| average in 2005--06,   |                     |                      |
| while students served  |                     |                      |
| by non-district        |                     |                      |
| providers had prior    |                     |                      |
| mathematics            |                     |                      |
| achievement levels     |                     |                      |
| during that year of    |                     |                      |
| 0.89 standard          |                     |                      |
| deviations below the   |                     |                      |
| district average.      |                     |                      |
| Nevertheless, this     |                     |                      |
| difference was not     |                     |                      |
| statistically          |                     |                      |
| significant.           |                     |                      |
|                        |                     |                      |
| Notes: Means are for   |                     |                      |
| all students with at   |                     |                      |
| least one gain score   |                     |                      |
| and who are therefore  |                     |                      |
| included in the        |                     |                      |
| regression analyses.   |                     |                      |
| Scores reported are    |                     |                      |
| rank‑based z‑scores.   |                     |                      |
|                        |                     |                      |
| \* indicates a         |                     |                      |
| significant difference |                     |                      |
| between students       |                     |                      |
| served by district     |                     |                      |
| versus non-district    |                     |                      |
| providers at the .05   |                     |                      |
| level.                 |                     |                      |
|                        |                     |                      |
| Source: Anchorage      |                     |                      |
| School District        |                     |                      |
| Administrative Data,   |                     |                      |
| 2005--06 through       |                     |                      |
| 2006--07.              |                     |                      |
+------------------------+---------------------+----------------------+

In Hillsborough, participants served by the school district in 2006--07
scored slightly higher on prior achievement tests than those served by
non‑district providers. For instance, students served by non-district
providers in 2006--07 scored, on average, 0.19 standard deviations below
the district mean in mathematics and reading in 2005--06, while students
served by the Hillsborough school district scored 0.14 and 0.13 standard
deviations below the district mean in mathematics and reading,
respectively (see Exhibit 16).

+------------------------+---------------------+----------------------+
| []{#__Ref              |                     |                      |
| Heading___Toc220485736 |                     |                      |
| .anchor}Exhibit 16\    |                     |                      |
| Prior Achievement      |                     |                      |
| Levels (in 2005--06)   |                     |                      |
| on the FCAT of         |                     |                      |
| Students in Grades     |                     |                      |
| 1--12, by Provider     |                     |                      |
| Type in 2006--07, in   |                     |                      |
| Hillsborough           |                     |                      |
+------------------------+---------------------+----------------------+
|                        | Mathematics         | Reading              |
+------------------------+---------------------+----------------------+
| District provider in   | --0.14              | --0.13               |
| 2006--07               |                     |                      |
+------------------------+---------------------+----------------------+
| Non‑district providers | --0.19\*            | --0.19\*             |
| in 2006--07            |                     |                      |
+------------------------+---------------------+----------------------+
| n                      | 4,192               | 4,192                |
+------------------------+---------------------+----------------------+
| **Exhibit reads:** In  |                     |                      |
| Hillsborough, students |                     |                      |
| who were served by the |                     |                      |
| district provider in   |                     |                      |
| 2006--07 had an        |                     |                      |
| average level of       |                     |                      |
| mathematics            |                     |                      |
| achievement of 0.19    |                     |                      |
| standard deviations    |                     |                      |
| below the district     |                     |                      |
| average in 2005--06,   |                     |                      |
| while students served  |                     |                      |
| by non-district        |                     |                      |
| providers had prior    |                     |                      |
| mathematics            |                     |                      |
| achievement levels     |                     |                      |
| during that year of    |                     |                      |
| 0.14 standard          |                     |                      |
| deviations below the   |                     |                      |
| district average, a    |                     |                      |
| significant            |                     |                      |
| difference.            |                     |                      |
|                        |                     |                      |
| Notes: Means are for   |                     |                      |
| all students with at   |                     |                      |
| least one gain score   |                     |                      |
| and who are therefore  |                     |                      |
| included in the        |                     |                      |
| regression analyses.   |                     |                      |
| Scores reported are    |                     |                      |
| rank‑based z‑scores.   |                     |                      |
|                        |                     |                      |
| \* indicates a         |                     |                      |
| significant difference |                     |                      |
| between students       |                     |                      |
| served by district     |                     |                      |
| versus non-district    |                     |                      |
| providers at the .05   |                     |                      |
| level.                 |                     |                      |
|                        |                     |                      |
| Source: Hillsborough   |                     |                      |
| School District        |                     |                      |
| Administrative Data,   |                     |                      |
| 2005--06 through       |                     |                      |
| 2006--07.              |                     |                      |
+------------------------+---------------------+----------------------+

##  Supplemental Educational Services Participation and Student Achievement {#supplemental-educational-services-participation-and-student-achievement .unnumbered}

The following results are based on the student‑level fixed‑effects
models described in the analytic approach section. All models include an
indicator for student eligibility for services, plus grade, year, and
grade-by-year fixed‑effects. Note that the coefficients presented in
these regression tables always compare the achievement gains of
participants to the gain of eligible non-participants, using
standardized scale scores (i.e., a z-score metric). This means that each
coefficient indicates whether participants experienced a larger or
smaller achievement gain (measured in standard deviations) than the one
of eligible non-participants, and whether this difference was
statistically significantly different or not.

Of the students in this study, only those who received supplemental
educational services in mathematics in Hillsborough experienced larger
academic gains (in mathematics) than eligible non‑participants. No gains
were found in Anchorage, but the participant group was extremely small
which makes it difficult to detect significant effects.

On average, students receiving supplemental educational services in
Anchorage did not experience larger gains in academic performance than
eligible non‑participants (see Exhibit 17).[^10] The coefficients
associated with the overall gains in mathematics and reading were not
statistically significant.[^11] Note that a negative coefficient does
not *necessarily* imply an achievement decline of the participant group.
It indicates that students receiving supplemental educational services
experienced smaller gains (or potentially, larger declines) than
eligible non‑participants. It is a relative, not an absolute, measure of
performance.

+-----------------------+----------+----------+-----------+-----------+
| []{#__RefH            |          |          |           |           |
| eading___Toc220485737 |          |          |           |           |
| .anchor}Exhibit 17\   |          |          |           |           |
| Achievement Gains on  |          |          |           |           |
| the Benchmark and SBA |          |          |           |           |
| of Students in Grades |          |          |           |           |
| 3--10\                |          |          |           |           |
| by Participation in   |          |          |           |           |
| Supplemental          |          |          |           |           |
| Educational Services, |          |          |           |           |
| in Anchorage,\        |          |          |           |           |
| 2002--03 Through      |          |          |           |           |
| 2006--07              |          |          |           |           |
+-----------------------+----------+----------+-----------+-----------+
|                       | Mat      |          | Reading   |           |
|                       | hematics |          |           |           |
+-----------------------+----------+----------+-----------+-----------+
|                       | Gains    | n        | Gains     | n         |
+-----------------------+----------+----------+-----------+-----------+
|                       |          |          |           |           |
+-----------------------+----------+----------+-----------+-----------+
| Overall effect        | --0.05   | 221      | 0.01      | 258       |
+-----------------------+----------+----------+-----------+-----------+
| First-year effect     | --0.04   | 215      | 0.01      | 255       |
+-----------------------+----------+----------+-----------+-----------+
| **Exhibit reads:** In |          |          |           |           |
| Anchorage, students   |          |          |           |           |
| who participated in   |          |          |           |           |
| supplemental          |          |          |           |           |
| educational services  |          |          |           |           |
| in mathematics showed |          |          |           |           |
| no significant        |          |          |           |           |
| differences in        |          |          |           |           |
| mathematics           |          |          |           |           |
| achievement gains     |          |          |           |           |
| from those of         |          |          |           |           |
| eligible              |          |          |           |           |
| non-participants.     |          |          |           |           |
|                       |          |          |           |           |
| Notes: Scores         |          |          |           |           |
| reported are          |          |          |           |           |
| rank-based z‑scores.  |          |          |           |           |
| Multiple Year Effect  |          |          |           |           |
| results not included  |          |          |           |           |
| due to small n size   |          |          |           |           |
| (6 participants total |          |          |           |           |
| for 2005--06 and      |          |          |           |           |
| 2006--07). N is the   |          |          |           |           |
| number of student     |          |          |           |           |
| observations in       |          |          |           |           |
| treatment             |          |          |           |           |
| contributing to the   |          |          |           |           |
| estimate. All models  |          |          |           |           |
| controlled for        |          |          |           |           |
| student eligibility   |          |          |           |           |
| for services. \*      |          |          |           |           |
| indicates gains that  |          |          |           |           |
| were statistically    |          |          |           |           |
| different form the    |          |          |           |           |
| gains of eligible     |          |          |           |           |
| non-participants at   |          |          |           |           |
| the .05 level.        |          |          |           |           |
|                       |          |          |           |           |
| Source: Anchorage     |          |          |           |           |
| School District       |          |          |           |           |
| Administrative Data,  |          |          |           |           |
| 2002--03 through      |          |          |           |           |
| 2006--07.             |          |          |           |           |
+-----------------------+----------+----------+-----------+-----------+

In Hillsborough, students receiving supplemental educational services
experienced higher achievement gains in mathematics than eligible
students who chose not to participate. This difference was equal to
0.05 standard deviations in this subject; this analysis controls for the
subject in which a student received services (see Exhibit 18). No
statistically significant differences were observed between participants
and eligible non-participants in reading.

In Hillsborough, when the effect of the first year of program
participation is isolated from the subsequent years, participants
experienced larger gains in mathematics than eligible non‑participants
only during the first year of the program. Again, no statistically
significant differences were observed in reading.

+-----------------------+----------+----------+-----------+-----------+
| []{#__RefH            |          |          |           |           |
| eading___Toc220485738 |          |          |           |           |
| .anchor}Exhibit 18\   |          |          |           |           |
| Achievement Gains on  |          |          |           |           |
| the FCAT of Students  |          |          |           |           |
| in Grades 1--12 by    |          |          |           |           |
| Participation\        |          |          |           |           |
| in Supplemental       |          |          |           |           |
| Educational Services, |          |          |           |           |
| in Hillsborough,\     |          |          |           |           |
| 2002--03 Through      |          |          |           |           |
| 2006--07              |          |          |           |           |
+-----------------------+----------+----------+-----------+-----------+
|                       | Mat      |          | Reading   |           |
|                       | hematics |          |           |           |
+-----------------------+----------+----------+-----------+-----------+
|                       | Gains    | n        | Gains     | n         |
+-----------------------+----------+----------+-----------+-----------+
| Overall effect        | 0.05\*   | 1,966    | -0.001    | 7,288     |
+-----------------------+----------+----------+-----------+-----------+
| First-year effect     | 0.04\*   | 1,683    | -0.0001   | 6,092     |
+-----------------------+----------+----------+-----------+-----------+
| Multiple-year effect  | 0.07     | 283      | -0.01     | 1,196     |
+-----------------------+----------+----------+-----------+-----------+
| **Exhibit reads:** In |          |          |           |           |
| Hillsborough,         |          |          |           |           |
| students who          |          |          |           |           |
| participated in       |          |          |           |           |
| supplemental          |          |          |           |           |
| educational services  |          |          |           |           |
| demonstrated greater  |          |          |           |           |
| gains in mathematics  |          |          |           |           |
| ac                    |          |          |           |           |
| hievement---improving |          |          |           |           |
| by .05 standard       |          |          |           |           |
| deviations---relative |          |          |           |           |
| to eligible           |          |          |           |           |
| non‑participants, a   |          |          |           |           |
| statistically         |          |          |           |           |
| significant           |          |          |           |           |
| difference.           |          |          |           |           |
|                       |          |          |           |           |
| Notes: Scores         |          |          |           |           |
| reported are          |          |          |           |           |
| rank-based z‑scores.  |          |          |           |           |
| N is the number of    |          |          |           |           |
| student observations  |          |          |           |           |
| in treatment          |          |          |           |           |
| contributing to the   |          |          |           |           |
| estimate. All models  |          |          |           |           |
| controlled for        |          |          |           |           |
| student eligibility   |          |          |           |           |
| for services.\        |          |          |           |           |
| \* indicates gains    |          |          |           |           |
| that were             |          |          |           |           |
| statistically         |          |          |           |           |
| different form the    |          |          |           |           |
| gains of eligible     |          |          |           |           |
| non-participants at   |          |          |           |           |
| the .05 level.        |          |          |           |           |
|                       |          |          |           |           |
| Source: Hillsborough  |          |          |           |           |
| School District       |          |          |           |           |
| Administrative Data,  |          |          |           |           |
| 2002--03 through      |          |          |           |           |
| 2006--07.             |          |          |           |           |
+-----------------------+----------+----------+-----------+-----------+

##   {#section-6 .unnumbered}

## Gains in Student Achievement by Provider {#gains-in-student-achievement-by-provider .unnumbered}

In the aggregate, only students served by non-district providers in
Anchorage and Hillsborough experienced larger academic gains in
mathematics compared with eligible non-participants.

On average, students who received services from non-district providers
in Anchorage experienced statistically significant achievement gains in
comparison to eligible non‑participants (see Exhibit 19).

+-------------------------+----------+----------+----------+----------+
| []{#__Re                |          |          |          |          |
| fHeading___Toc220485739 |          |          |          |          |
| .anchor}Exhibit 19\     |          |          |          |          |
| Achievement Gains on    |          |          |          |          |
| the Benchmark and SBA   |          |          |          |          |
| of Students in Grades   |          |          |          |          |
| 3--10\                  |          |          |          |          |
| by Provider Type, in    |          |          |          |          |
| Anchorage, 2002--03     |          |          |          |          |
| Through 2006--07        |          |          |          |          |
+-------------------------+----------+----------+----------+----------+
|                         | Mat      |          | Reading  |          |
|                         | hematics |          |          |          |
+-------------------------+----------+----------+----------+----------+
|                         | Gains    | n        | Gains    | n        |
+-------------------------+----------+----------+----------+----------+
| District provider       | --0.17   | 67       | --0.11   | 83       |
+-------------------------+----------+----------+----------+----------+
| Non‑district providers  | 0.18\*   | 62       | 0.12     | 90       |
+-------------------------+----------+----------+----------+----------+
| **Exhibit reads:** In   |          |          |          |          |
| Anchorage, students who |          |          |          |          |
| received supplemental   |          |          |          |          |
| educational services in |          |          |          |          |
| mathematics from the    |          |          |          |          |
| district provider       |          |          |          |          |
| showed no significant   |          |          |          |          |
| difference in           |          |          |          |          |
| mathematics achievement |          |          |          |          |
| gains from those of     |          |          |          |          |
| eligible                |          |          |          |          |
| non-participants.       |          |          |          |          |
| Students who were       |          |          |          |          |
| served by non-district  |          |          |          |          |
| providers demonstrated  |          |          |          |          |
| greater gains in        |          |          |          |          |
| mathematics             |          |          |          |          |
| achievement, on         |          |          |          |          |
| average---improving by  |          |          |          |          |
| 0.18 standard           |          |          |          |          |
| deviations---relative   |          |          |          |          |
| to eligible             |          |          |          |          |
| non-participants, a     |          |          |          |          |
| statistically           |          |          |          |          |
| significant difference. |          |          |          |          |
|                         |          |          |          |          |
| Notes: Scores reported  |          |          |          |          |
| are rank-based          |          |          |          |          |
| z‑scores. All models    |          |          |          |          |
| controlled for student  |          |          |          |          |
| eligibility for         |          |          |          |          |
| services. N is the      |          |          |          |          |
| number of student       |          |          |          |          |
| observations in         |          |          |          |          |
| treatment contributing  |          |          |          |          |
| to the estimate. \*     |          |          |          |          |
| indicates gains that    |          |          |          |          |
| were statistically      |          |          |          |          |
| different from the      |          |          |          |          |
| gains of eligible       |          |          |          |          |
| non-participants at the |          |          |          |          |
| .05 level.              |          |          |          |          |
|                         |          |          |          |          |
| Source: Anchorage       |          |          |          |          |
| School District         |          |          |          |          |
| Administrative Data,    |          |          |          |          |
| 2002--03 through        |          |          |          |          |
| 2006--07.               |          |          |          |          |
+-------------------------+----------+----------+----------+----------+

In Anchorage, no provider showed achievement gains that were
significantly higher than the gains of eligible non-participants. One
provider in Anchorage had gains in mathematics that were higher than the
average gain across all participants. This provider experienced a gain
of 0.17 standard deviations above the performance of eligible
non-participants, and served 46 students for an average of 31 hours. All
the other providers in Anchorage experienced mathematics achievement
gains statistically indistinguishable from the average coefficient
across all participants (--0.05 standard deviations) (see Exhibit 20).
The small sample size of students served by each provider makes it
difficult to detect significant gains.

+------------------------+-------------+---------------+---------------+
| []{#__Ref              |             |               |               |
| Heading___Toc220485740 |             |               |               |
| .anchor}Exhibit 20\    |             |               |               |
| Mathematics            |             |               |               |
| Achievement Gains on   |             |               |               |
| the Benchmark and SBA  |             |               |               |
| of Students in Grades  |             |               |               |
| 3--10 by Eligibility   |             |               |               |
| for and Participation  |             |               |               |
| in Supplemental        |             |               |               |
| Educational Services,  |             |               |               |
| by Provider, in        |             |               |               |
| Anchorage, 2002--03    |             |               |               |
| Through 2006--07       |             |               |               |
+------------------------+-------------+---------------+---------------+
|                        | Number of   | Average Hours | Mathematics   |
|                        | P           | per Student   |               |
|                        | articipants |               |               |
+------------------------+-------------+---------------+---------------+
| **Average Across All   |             |               | --**0.05**    |
| Providers**            |             |               |               |
+------------------------+-------------+---------------+---------------+
| **Providers Showing No |             |               |               |
| Significant Effects**  |             |               |               |
+------------------------+-------------+---------------+---------------+
| Private Provider A     | 46          | 31            | 0.17          |
+------------------------+-------------+---------------+---------------+
| Anchorage School       | 67          | 29            | --0.17        |
| District               |             |               |               |
+------------------------+-------------+---------------+---------------+
| Other providers        | 16          | 28            | 0.21          |
+------------------------+-------------+---------------+---------------+
| Total number of        | 129         |               |               |
| students               |             |               |               |
+------------------------+-------------+---------------+---------------+
| **Exhibit reads:** In  |             |               |               |
| Anchorage, Private     |             |               |               |
| Provider A provided    |             |               |               |
| supplemental           |             |               |               |
| educational services   |             |               |               |
| to 46 students in      |             |               |               |
| 2006--07, with an      |             |               |               |
| average of 31 hours of |             |               |               |
| services per student.  |             |               |               |
| Students served by     |             |               |               |
| Private Provider A     |             |               |               |
| showed a statistically |             |               |               |
| significant difference |             |               |               |
| in mathematics         |             |               |               |
| achievement gains      |             |               |               |
| compared with the      |             |               |               |
| average participant    |             |               |               |
| gain, but not with     |             |               |               |
| respect to eligible    |             |               |               |
| non-participants in    |             |               |               |
| the district.          |             |               |               |
|                        |             |               |               |
| Notes: Scores reported |             |               |               |
| are rank-based         |             |               |               |
| z‑scores. The "Other   |             |               |               |
| Providers" category    |             |               |               |
| includes all providers |             |               |               |
| serving less than 20   |             |               |               |
| students in            |             |               |               |
| mathematics. All       |             |               |               |
| models controlled for  |             |               |               |
| student eligibility    |             |               |               |
| for services.\         |             |               |               |
| \* indicates that the  |             |               |               |
| provider experienced   |             |               |               |
| gains that were        |             |               |               |
| statistically          |             |               |               |
| different from the     |             |               |               |
| gains of eligible      |             |               |               |
| non-participants at    |             |               |               |
| the .05 level. Scores  |             |               |               |
| reported are           |             |               |               |
| rank-based z‑scores.   |             |               |               |
| Shaded bars indicate   |             |               |               |
| that the provider      |             |               |               |
| experienced gains that |             |               |               |
| were statistically     |             |               |               |
| different from the     |             |               |               |
| district average for   |             |               |               |
| all participants.  In  |             |               |               |
| the next exhibit, no   |             |               |               |
| bars are shaded,       |             |               |               |
| indicating that there  |             |               |               |
| are no statistically   |             |               |               |
| significant            |             |               |               |
| differences from the   |             |               |               |
| district average.      |             |               |               |
|                        |             |               |               |
| > Source: Anchorage    |             |               |               |
| > School District      |             |               |               |
| > Administrative Data, |             |               |               |
| > 2002--03 through     |             |               |               |
| > 2006--07.            |             |               |               |
+------------------------+-------------+---------------+---------------+

In Anchorage, no provider---including the Anchorage school
district---had statistically significant gains in reading above or below
the group average gain (see Exhibit 21). In other words, none of these
performance gains was statistically different from the overall average
participant gain of 0.01 standard deviations above that of eligible
non-participants (see Exhibit 17). These providers served between 20 and
70 students in reading for an average of 26 to 33 hours.

+----------------------+---------------+---------------+---------------+
| []{#__RefHe          |               |               |               |
| ading___Toc220485741 |               |               |               |
| .anchor}Exhibit 21\  |               |               |               |
| Reading Achievement  |               |               |               |
| Gains on the         |               |               |               |
| Benchmark and SBA of |               |               |               |
| Students in Grades   |               |               |               |
| 3--10 by Eligibility |               |               |               |
| for and              |               |               |               |
| Participation in     |               |               |               |
| Supplemental         |               |               |               |
| Educational          |               |               |               |
| Services, by         |               |               |               |
| Provider, in         |               |               |               |
| Anchorage, 2002--03  |               |               |               |
| Through 2006--07     |               |               |               |
+----------------------+---------------+---------------+---------------+
|                      | Number of     | Average Hours | Reading       |
|                      | Participants  | per Student   |               |
+----------------------+---------------+---------------+---------------+
| **Average Across All |               |               | **0.01**      |
| Providers**          |               |               |               |
+----------------------+---------------+---------------+---------------+
| **Providers Showing  |               |               |               |
| No Significant       |               |               |               |
| Effects**            |               |               |               |
+----------------------+---------------+---------------+---------------+
| Private Provider A   | 70            | 27            | 0.11          |
+----------------------+---------------+---------------+---------------+
| Anchorage School     | 83            | 33            | --0.11        |
| District             |               |               |               |
+----------------------+---------------+---------------+---------------+
| Other providers      | 20            | 26            | 0.14          |
+----------------------+---------------+---------------+---------------+
| Total number of      | 173           |               |               |
| students             |               |               |               |
+----------------------+---------------+---------------+---------------+
| > **Exhibit reads:** |               |               |               |
| > In Anchorage,      |               |               |               |
| > Private Provider A |               |               |               |
| > showed achievement |               |               |               |
| > gains for          |               |               |               |
| > participating      |               |               |               |
| > students that were |               |               |               |
| > not significantly  |               |               |               |
| > different than     |               |               |               |
| > gains for eligible |               |               |               |
| > non-participants.  |               |               |               |
| > This gain was also |               |               |               |
| > not significantly  |               |               |               |
| > larger than the    |               |               |               |
| > average            |               |               |               |
| > participant gain   |               |               |               |
| > in the district.   |               |               |               |
|                      |               |               |               |
| Notes: Shading       |               |               |               |
| indicates that the   |               |               |               |
| provider experienced |               |               |               |
| gains that were      |               |               |               |
| statistically        |               |               |               |
| different from the   |               |               |               |
| district average. \* |               |               |               |
| indicates that the   |               |               |               |
| provider experienced |               |               |               |
| gains that were      |               |               |               |
| statistically        |               |               |               |
| different from the   |               |               |               |
| gains of eligible    |               |               |               |
| non-participants at  |               |               |               |
| the .05 level.       |               |               |               |
| Scores reported are  |               |               |               |
| rank-based z‑scores. |               |               |               |
| The "Other           |               |               |               |
| Providers" other     |               |               |               |
| category includes    |               |               |               |
| all providers        |               |               |               |
| serving less than 20 |               |               |               |
| students in reading. |               |               |               |
| All models           |               |               |               |
| controlled for       |               |               |               |
| student eligibility  |               |               |               |
| for services.        |               |               |               |
|                      |               |               |               |
| Source: Anchorage    |               |               |               |
| School District      |               |               |               |
| Administrative Data, |               |               |               |
| 2002--03 through     |               |               |               |
| 2006--07.            |               |               |               |
+----------------------+---------------+---------------+---------------+

In Hillsborough, only students served by non-district providers
experienced higher academic gains in mathematics than eligible
non‑participants.

In Hillsborough, only students served by non‑district providers showed
statistically significant achievement gains in mathematics, 0.08
standard deviations, compared to eligible non‑participants (see Exhibit
22). Consistent with the overall program effects observed in
Hillsborough (Exhibit 18), these results indicate no statistically
significant difference between participants and eligible
non-participants in reading.

In Hillsborough a hybrid category existed, composed of students who were
served simultaneously by the district and non-district providers in
2006--07. The data show that this hybrid group of participants did not
experience significantly different academic gains than eligible
non‑participants.

+-------------------------+----------+----------+----------+----------+
| []{#__Re                |          |          |          |          |
| fHeading___Toc220485742 |          |          |          |          |
| .anchor}Exhibit 22\     |          |          |          |          |
| Achievement Gains on    |          |          |          |          |
| the FCAT of Students in |          |          |          |          |
| Grades 1--12\           |          |          |          |          |
| by Provider Type, in    |          |          |          |          |
| Hillsborough, 2002--03  |          |          |          |          |
| Through 2006--07        |          |          |          |          |
+-------------------------+----------+----------+----------+----------+
|                         | Mat      |          | Reading  |          |
|                         | hematics |          |          |          |
+-------------------------+----------+----------+----------+----------+
|                         | Gains    | n        | Gains    | n        |
+-------------------------+----------+----------+----------+----------+
| District provider       | 0.05     | 234      | --0.02   | 447      |
+-------------------------+----------+----------+----------+----------+
| Non‑district providers  | 0.08\*   | 743      | --0.01   | 2,503    |
+-------------------------+----------+----------+----------+----------+
| District and            | \*\*     | 18       | 0.22     | 50       |
| non‑district            |          |          |          |          |
| providers^a^            |          |          |          |          |
+-------------------------+----------+----------+----------+----------+
| **Exhibit reads:** In   |          |          |          |          |
| Hillsborough, students  |          |          |          |          |
| who received            |          |          |          |          |
| supplemental            |          |          |          |          |
| educational services in |          |          |          |          |
| mathematics from the    |          |          |          |          |
| district provider       |          |          |          |          |
| showed no significant   |          |          |          |          |
| difference in           |          |          |          |          |
| mathematics achievement |          |          |          |          |
| gains from those of     |          |          |          |          |
| eligible                |          |          |          |          |
| non-participants.       |          |          |          |          |
| Students who were       |          |          |          |          |
| served by non-district  |          |          |          |          |
| providers demonstrated  |          |          |          |          |
| greater gains in        |          |          |          |          |
| mathematics             |          |          |          |          |
| achievement---improving |          |          |          |          |
| by 0.08 standard        |          |          |          |          |
| deviations---relative   |          |          |          |          |
| to eligible             |          |          |          |          |
| non-participants, a     |          |          |          |          |
| statistically           |          |          |          |          |
| significant difference. |          |          |          |          |
|                         |          |          |          |          |
| Notes: Scores reported  |          |          |          |          |
| are rank-based          |          |          |          |          |
| z‑scores. N is the      |          |          |          |          |
| number of student       |          |          |          |          |
| observations in         |          |          |          |          |
| treatment contributing  |          |          |          |          |
| to the estimate. All    |          |          |          |          |
| models controlled for   |          |          |          |          |
| student eligibility for |          |          |          |          |
| services.\              |          |          |          |          |
| \* indicates that the   |          |          |          |          |
| provider experienced    |          |          |          |          |
| gains that were         |          |          |          |          |
| statistically different |          |          |          |          |
| from the gains of       |          |          |          |          |
| eligible                |          |          |          |          |
| non-participants at the |          |          |          |          |
| .05 level.              |          |          |          |          |
|                         |          |          |          |          |
| \*\* indicates that the |          |          |          |          |
| data are not reported   |          |          |          |          |
| due to small sample     |          |          |          |          |
| size (n\<20 students).\ |          |          |          |          |
| ^a^ Some students in    |          |          |          |          |
| Hillsborough were       |          |          |          |          |
| served in 2006--07      |          |          |          |          |
| simultaneously by the   |          |          |          |          |
| district provider and   |          |          |          |          |
| non-district providers, |          |          |          |          |
| creating a hybrid       |          |          |          |          |
| group.                  |          |          |          |          |
|                         |          |          |          |          |
| Source: Hillsborough    |          |          |          |          |
| School District         |          |          |          |          |
| Administrative Data,    |          |          |          |          |
| 2002--03 through        |          |          |          |          |
| 2006--07.               |          |          |          |          |
+-------------------------+----------+----------+----------+----------+

Out of the 10 providers in Hillsborough that served more than 20
participants in mathematics during the 2006--07 school year, one
provider showed achievement gains that were significantly higher than
the gains of eligible non-participants (provider S, noted with an
asterisk) (see Exhibit 23). However, no provider had achievement gains
in mathematics that were statistically greater than the average gain
across all participants (0.05). These 10 providers served between 20 and
234 students in mathematics in 2006--07 for an average number of hours
that ranged from 9 to 35 hours. One of these providers was the district,
which served 234 participants in mathematics for an average of 20 hours.

+------------------------+----------+---------------+-----------------+
| []{#__Ref              |          |               |                 |
| Heading___Toc220485743 |          |               |                 |
| .anchor}Exhibit 23\    |          |               |                 |
| Mathematics            |          |               |                 |
| Achievement Gains on   |          |               |                 |
| the FCAT of Students   |          |               |                 |
| in Grades 1--12 by     |          |               |                 |
| Eligibility for and    |          |               |                 |
| Participation in       |          |               |                 |
| Supplemental           |          |               |                 |
| Educational Services,  |          |               |                 |
| by Provider, in        |          |               |                 |
| Hillsborough,\         |          |               |                 |
| 2002--03 Through       |          |               |                 |
| 2006--07               |          |               |                 |
+------------------------+----------+---------------+-----------------+
|                        | Number   | Average Hours | Mathematics     |
|                        | of       | per Student   |                 |
|                        | Part     |               |                 |
|                        | icipants |               |                 |
+------------------------+----------+---------------+-----------------+
| **Average Across All   |          |               | **0.05**        |
| Providers**            |          |               |                 |
+------------------------+----------+---------------+-----------------+
| **Providers Showing    |          |               |                 |
| Significant Gains**    |          |               |                 |
+------------------------+----------+---------------+-----------------+
| Private Provider S     | 20       | 27            | 0.32\*          |
+------------------------+----------+---------------+-----------------+
| **Providers Showing No |          |               |                 |
| Significant Gains**    |          |               |                 |
+------------------------+----------+---------------+-----------------+
| Private Provider W     | 22       | 9             | 0.14            |
+------------------------+----------+---------------+-----------------+
| Private Provider Q     | 49       | 24            | 0.08            |
+------------------------+----------+---------------+-----------------+
| Private Provider N     | 40       | 35            | 0.08            |
+------------------------+----------+---------------+-----------------+
| Private Provider I     | 158      | 18            | 0.07            |
+------------------------+----------+---------------+-----------------+
| Private Provider H     | 213      | 24            | 0.05            |
+------------------------+----------+---------------+-----------------+
| Hillsborough School    | 234      | 21            | 0.05            |
| District               |          |               |                 |
+------------------------+----------+---------------+-----------------+
| Private Provider L     | 55       | 16            | 0.05            |
+------------------------+----------+---------------+-----------------+
| Private Provider K     | 52       | 19            | 0.02            |
+------------------------+----------+---------------+-----------------+
| Private Provider B     | 44       | 16            | --0.04          |
+------------------------+----------+---------------+-----------------+
| Other providers        | 128      | 24            | 0.14            |
+------------------------+----------+---------------+-----------------+
| Total number of        | 995      |               |                 |
| students               |          |               |                 |
+------------------------+----------+---------------+-----------------+
| > **Exhibit reads:**   |          |               |                 |
| > In Hillsborough, one |          |               |                 |
| > private provider (S) |          |               |                 |
| > showed achievement   |          |               |                 |
| > gains for            |          |               |                 |
| > participating        |          |               |                 |
| > students that were   |          |               |                 |
| > significantly larger |          |               |                 |
| > than gains for       |          |               |                 |
| > eligible             |          |               |                 |
| > non-participants.    |          |               |                 |
| > However, these gains |          |               |                 |
| > were not             |          |               |                 |
| > statistically larger |          |               |                 |
| > than the average     |          |               |                 |
| > effect size across   |          |               |                 |
| > all supplemental     |          |               |                 |
| > educational service  |          |               |                 |
| > providers.           |          |               |                 |
|                        |          |               |                 |
| Notes: Scores reported |          |               |                 |
| are rank-based         |          |               |                 |
| z‑scores. The "Other   |          |               |                 |
| Providers" category    |          |               |                 |
| includes all providers |          |               |                 |
| serving less than 20   |          |               |                 |
| students in            |          |               |                 |
| mathematics. All       |          |               |                 |
| models controlled for  |          |               |                 |
| student eligibility    |          |               |                 |
| for services.\         |          |               |                 |
| \* indicates that the  |          |               |                 |
| provider experienced   |          |               |                 |
| gains that were        |          |               |                 |
| statistically          |          |               |                 |
| different from the     |          |               |                 |
| gains of eligible      |          |               |                 |
| non-participants at    |          |               |                 |
| the .05 level.\        |          |               |                 |
| Shaded bars indicate   |          |               |                 |
| that the provider      |          |               |                 |
| experienced gains that |          |               |                 |
| were statistically     |          |               |                 |
| different from the     |          |               |                 |
| district average.  In  |          |               |                 |
| this exhibit, no bars  |          |               |                 |
| are shaded, indicating |          |               |                 |
| that there are no      |          |               |                 |
| statistically          |          |               |                 |
| significant            |          |               |                 |
| differences from the   |          |               |                 |
| district average.      |          |               |                 |
|                        |          |               |                 |
| Source: Hillsborough   |          |               |                 |
| School District        |          |               |                 |
| Administrative Data,   |          |               |                 |
| 2002--03 through       |          |               |                 |
| 2006--07.              |          |               |                 |
+------------------------+----------+---------------+-----------------+

In reading, 19 providers in Hillsborough served more than 20
participants during the 2006--07 school year. No provider statistically
performed above the overall average participant gain in the district or
above eligible non-participants (see Exhibit 24). The reading
achievement gains ranged from 0.07 to 0.28 standard deviations above the
performance of eligible non-participants (see Exhibit 18). Nevertheless
none of these differences with the group average were significant at the
.05 level. This group of providers, which included the Hillsborough
school district, served between 24 and 476 students in reading for an
average of 13 to 35 hours in 2006--07.

+-------------------------+-----------+----------------+---------------+
| []{#__Re                |           |                |               |
| fHeading___Toc220485744 |           |                |               |
| .anchor}Exhibit 24\     |           |                |               |
| Reading Achievement     |           |                |               |
| Gains on the FCAT of    |           |                |               |
| Students in Grades      |           |                |               |
| 1--12 by Eligibility    |           |                |               |
| for and Participation   |           |                |               |
| in Supplemental         |           |                |               |
| Educational Services,   |           |                |               |
| by Provider, in         |           |                |               |
| Hillsborough,\          |           |                |               |
| 2002--03 Through        |           |                |               |
| 2006--07                |           |                |               |
+-------------------------+-----------+----------------+---------------+
|                         | Number of | Average Hours  | Reading       |
|                         | Par       | per Student    |               |
|                         | ticipants |                |               |
+-------------------------+-----------+----------------+---------------+
| **Average Across All    |           |                | **-0.001**    |
| Providers**             |           |                |               |
+-------------------------+-----------+----------------+---------------+
| **Providers Showing No  |           |                |               |
| Significant Gains**     |           |                |               |
+-------------------------+-----------+----------------+---------------+
| Private Provider A      | 41        | 31             | 0.28          |
+-------------------------+-----------+----------------+---------------+
| Private Provider S      | 26        | 30             | 0.20          |
+-------------------------+-----------+----------------+---------------+
| Private Provider B      | 144       | 17             | 0.08          |
+-------------------------+-----------+----------------+---------------+
| Private Provider M      | 194       | 17             | 0.07          |
+-------------------------+-----------+----------------+---------------+
| Private Provider K      | 161       | 21             | 0.04          |
+-------------------------+-----------+----------------+---------------+
| Private Provider N      | 102       | 34             | 0.03          |
+-------------------------+-----------+----------------+---------------+
| Private Provider P      | 67        | 28             | 0.02          |
+-------------------------+-----------+----------------+---------------+
| Private Provider L      | 112       | 16             | 0.01          |
+-------------------------+-----------+----------------+---------------+
| Private Provider U      | 49        | 27             | 0.002         |
+-------------------------+-----------+----------------+---------------+
| Private Provider Y      | 24        | 17             | -0.01         |
+-------------------------+-----------+----------------+---------------+
| Hillsborough School     | 447       | 22             | -0.01         |
| District                |           |                |               |
+-------------------------+-----------+----------------+---------------+
| Private Provider J      | 476       | 25             | -0.02         |
+-------------------------+-----------+----------------+---------------+
| Private Provider R      | 48        | 13             | -0.03         |
+-------------------------+-----------+----------------+---------------+
| Private Provider I      | 331       | 19             | -0.05         |
+-------------------------+-----------+----------------+---------------+
| Private Provider H      | 543       | 24             | -0.06         |
+-------------------------+-----------+----------------+---------------+
| Private Provider Q      | 44        | 23             | -0.07         |
+-------------------------+-----------+----------------+---------------+
| Private Provider V      | 31        | 35             | -0.07         |
+-------------------------+-----------+----------------+---------------+
| Private Provider O      | 68        | 29             | -0.07         |
+-------------------------+-----------+----------------+---------------+
| Private Provider T      | 30        | 15             | -0.08         |
+-------------------------+-----------+----------------+---------------+
| Other providers         | 62        | 22             | -0.001        |
+-------------------------+-----------+----------------+---------------+
| Total number of         | 3,000     |                |               |
| students                |           |                |               |
+-------------------------+-----------+----------------+---------------+
| **Exhibit reads:** In   |           |                |               |
| Hillsborough, no        |           |                |               |
| provider showed         |           |                |               |
| achievement gains for   |           |                |               |
| participating students  |           |                |               |
| that were significantly |           |                |               |
| larger than the gains   |           |                |               |
| for eligible            |           |                |               |
| non-participants. All   |           |                |               |
| these gains were also   |           |                |               |
| statistically           |           |                |               |
| indistinguishable from  |           |                |               |
| the average effect size |           |                |               |
| across all supplemental |           |                |               |
| educational services    |           |                |               |
| participants in the     |           |                |               |
| district.               |           |                |               |
|                         |           |                |               |
| Notes: Scores reported  |           |                |               |
| are rank-based          |           |                |               |
| z‑scores. The "Other    |           |                |               |
| Providers" category     |           |                |               |
| includes all providers  |           |                |               |
| serving less than 20    |           |                |               |
| students in             |           |                |               |
| mathematics. All models |           |                |               |
| controlled for student  |           |                |               |
| eligibility for         |           |                |               |
| services.\              |           |                |               |
| \* indicates that the   |           |                |               |
| provider experienced    |           |                |               |
| gains that were         |           |                |               |
| statistically different |           |                |               |
| from the gains of       |           |                |               |
| eligible                |           |                |               |
| non-participants at the |           |                |               |
| .05 level.\             |           |                |               |
| Shaded bars indicate    |           |                |               |
| that the provider       |           |                |               |
| experienced gains that  |           |                |               |
| were statistically      |           |                |               |
| different from the      |           |                |               |
| district average.  In   |           |                |               |
| this exhibit, no bars   |           |                |               |
| are shaded, indicating  |           |                |               |
| that there are no       |           |                |               |
| statistically           |           |                |               |
| significant differences |           |                |               |
| from the district       |           |                |               |
| average.                |           |                |               |
|                         |           |                |               |
| Source: Hillsborough    |           |                |               |
| School District         |           |                |               |
| Administrative Data,    |           |                |               |
| 2002--03 through        |           |                |               |
| 2006--07.               |           |                |               |
+-------------------------+-----------+----------------+---------------+

## Effect of Number of Hours of Services on Student Achievement  {#effect-of-number-of-hours-of-services-on-student-achievement .unnumbered}

In both Anchorage and Hillsborough, a majority of supplemental
educational services participants received fewer than 40 hours of
services.

In Anchorage and Hillsborough, there was a high level of variation in
the number of hours of supplemental educational services students
received from providers. To analyze the relationship between hours of
services and achievement gains, we developed three broad categories: a)
fewer than 20 hours of services, b) between 20 and 39 hours of services,
and c) greater than 39 hours of services (see Exhibit 25).

+------------------+-------+-------+-------+-------+-------+-------+
| []{#__RefHeadin  |       |       |       |       |       |       |
| g___Toc220485745 |       |       |       |       |       |       |
| .an              |       |       |       |       |       |       |
| chor}Exhibit 25\ |       |       |       |       |       |       |
| Distribution of  |       |       |       |       |       |       |
| Supplemental     |       |       |       |       |       |       |
| Educational      |       |       |       |       |       |       |
| Services         |       |       |       |       |       |       |
| Participants by  |       |       |       |       |       |       |
| Number of Hours  |       |       |       |       |       |       |
| of\              |       |       |       |       |       |       |
| Services         |       |       |       |       |       |       |
| Received, by     |       |       |       |       |       |       |
| Provider Type,   |       |       |       |       |       |       |
| in Anchorage and |       |       |       |       |       |       |
| Hillsborough,\   |       |       |       |       |       |       |
| 2004--05 Through |       |       |       |       |       |       |
| 2006--07         |       |       |       |       |       |       |
+------------------+-------+-------+-------+-------+-------+-------+
| Hours of         | All   |       | Dis   |       | No    |       |
|                  | Prov  |       | trict |       | n‑Dis |       |
| Services         | iders |       | Pro   |       | trict |       |
| Received         |       |       | vider |       | Pro   |       |
|                  |       |       |       |       | vider |       |
+------------------+-------+-------+-------+-------+-------+-------+
|                  | 200   |       | 200   |       |       |       |
|                  | 4--05 |       | 6--07 |       |       |       |
|                  | To    |       |       |       |       |       |
|                  | 200   |       |       |       |       |       |
|                  | 6--07 |       |       |       |       |       |
+------------------+-------+-------+-------+-------+-------+-------+
|                  | Pe    | n     | Pe    | n     | Pe    | n     |
|                  | rcent |       | rcent |       | rcent |       |
+------------------+-------+-------+-------+-------+-------+-------+
| Anchorage        |       |       |       |       |       |       |
| (Grades K--12)   |       |       |       |       |       |       |
+------------------+-------+-------+-------+-------+-------+-------+
| Fewer than 20    | 28%   | 121   | 33%   | 45    | 25%   | 52    |
| hours            |       |       |       |       |       |       |
+------------------+-------+-------+-------+-------+-------+-------+
| 20--39 hours     | 43%   | 188   | 35%\* | 48    | 47%\* | 105   |
+------------------+-------+-------+-------+-------+-------+-------+
| More than 39     | 27%   | 129   | 33%\* | 45    | 28%\* | 8     |
| hours            |       |       |       |       |       |       |
+------------------+-------+-------+-------+-------+-------+-------+
| Total            | 100%  | 438   | 100%  | 138   | 100%  | 165   |
+------------------+-------+-------+-------+-------+-------+-------+
|                  |       |       |       |       |       |       |
+------------------+-------+-------+-------+-------+-------+-------+
| Hillsborough     |       |       |       |       |       |       |
| (Grades K--12)   |       |       |       |       |       |       |
+------------------+-------+-------+-------+-------+-------+-------+
| Fewer than 20    | 36%   | 3,654 | 46%\* | 442   | 35%\* | 3,212 |
| hours            |       |       |       |       |       |       |
+------------------+-------+-------+-------+-------+-------+-------+
| 20--39 hours     | 59%   | 5,937 | 45%\* | 441   | 60%\* | 5,496 |
+------------------+-------+-------+-------+-------+-------+-------+
| 40--59 hours     | 5%    | 505   | 9%\*  | 87    | 4%\*  | 405   |
+------------------+-------+-------+-------+-------+-------+-------+
| **Total**        | 100%  | 1     | 100%  | 971   | 100%  | 9,125 |
|                  |       | 0,096 |       |       |       |       |
+------------------+-------+-------+-------+-------+-------+-------+
| **Exhibit        |       |       |       |       |       |       |
| reads:** In      |       |       |       |       |       |       |
| Anchorage, 28    |       |       |       |       |       |       |
| percent of all   |       |       |       |       |       |       |
| participants in  |       |       |       |       |       |       |
| grades           |       |       |       |       |       |       |
| K--12 received   |       |       |       |       |       |       |
| less than        |       |       |       |       |       |       |
| 20 hours of      |       |       |       |       |       |       |
| services. Among  |       |       |       |       |       |       |
| those served by  |       |       |       |       |       |       |
| the district in  |       |       |       |       |       |       |
| 2006--07, 33     |       |       |       |       |       |       |
| percent received |       |       |       |       |       |       |
| less than 20     |       |       |       |       |       |       |
| hours of         |       |       |       |       |       |       |
| services and     |       |       |       |       |       |       |
| among those      |       |       |       |       |       |       |
| served by        |       |       |       |       |       |       |
| non‑district     |       |       |       |       |       |       |
| providers, 25    |       |       |       |       |       |       |
| percent received |       |       |       |       |       |       |
| less than 20     |       |       |       |       |       |       |
| hours of         |       |       |       |       |       |       |
| services during  |       |       |       |       |       |       |
| that year.       |       |       |       |       |       |       |
|                  |       |       |       |       |       |       |
| Notes: \*        |       |       |       |       |       |       |
| indicates a      |       |       |       |       |       |       |
| significant      |       |       |       |       |       |       |
| difference at    |       |       |       |       |       |       |
| the .05 level    |       |       |       |       |       |       |
| between the      |       |       |       |       |       |       |
| percentage of    |       |       |       |       |       |       |
| participants     |       |       |       |       |       |       |
| served by the    |       |       |       |       |       |       |
| district         |       |       |       |       |       |       |
| provider and     |       |       |       |       |       |       |
| non-district     |       |       |       |       |       |       |
| providers.       |       |       |       |       |       |       |
|                  |       |       |       |       |       |       |
| Source:          |       |       |       |       |       |       |
| Anchorage School |       |       |       |       |       |       |
| District         |       |       |       |       |       |       |
| Administrative   |       |       |       |       |       |       |
| Data, 2002--03   |       |       |       |       |       |       |
| through          |       |       |       |       |       |       |
| 2006--07;        |       |       |       |       |       |       |
| Hillsborough     |       |       |       |       |       |       |
| School District  |       |       |       |       |       |       |
| Administrative   |       |       |       |       |       |       |
| Data, 2002--03   |       |       |       |       |       |       |
| through          |       |       |       |       |       |       |
| 2006--07.        |       |       |       |       |       |       |
+------------------+-------+-------+-------+-------+-------+-------+

Between 2004--05 and 2006--07, the majority of participants in both
districts received less than 40 hours of services. In Anchorage, less
than 30 percent of participants received more than 40 hours of services,
while in Hillsborough only 5 percent of participants received more than
40 hours of services.[^12] In Anchorage, there was not a statistically
significant difference between the district provider and non-district
providers in the percentage of their participants who received less than
20 hours of services in 2006--07. In Hillsborough, the district provider
tended to provide students slightly fewer hours of tutoring than other
providers (see Exhibit 25). Note, however, that this exhibit combines
the available data from 2004--05 through 2006--07, in contrast to
Exhibit 10 (overall average number of hours), which only includes data
for the 2006--07 school year. (See Appendix D for an analysis of the
average number of hours of supplemental educational services across
district and non‑district providers using the data of 2004--05 through
2006--07.)

There was not a consistent relationship between the amount of services
received and achievement gains in both districts.

In Anchorage, reading achievement gains of students receiving fewer than
20 hours of tutoring from the district lagged 0.24 standard deviations
below eligible non‑participants. Students served by non‑district
providers had no statistically significant gains or losses compared to
eligible non‑participants, regardless of hours of services received (see
Exhibit 26). Breaking down the analysis by hours of services,
mathematics achievement gains were statistically indistinguishable from
those of eligible non-participants.[^13]

+--------------------+-------+-------+-------+-------+-------+-------+
| []{#__RefHead      |       |       |       |       |       |       |
| ing___Toc220485746 |       |       |       |       |       |       |
| .                  |       |       |       |       |       |       |
| anchor}Exhibit 26\ |       |       |       |       |       |       |
| Achievement Gains  |       |       |       |       |       |       |
| on the Benchmark   |       |       |       |       |       |       |
| and SBA of         |       |       |       |       |       |       |
| Students in Grades |       |       |       |       |       |       |
| 3--10\             |       |       |       |       |       |       |
| by Provider Type   |       |       |       |       |       |       |
| and Hours of       |       |       |       |       |       |       |
| Services Received, |       |       |       |       |       |       |
| in Anchorage,\     |       |       |       |       |       |       |
| 2002--03 Through   |       |       |       |       |       |       |
| 2006--07           |       |       |       |       |       |       |
+====================+=======+=======+=======+=======+=======+=======+
| Hours of           | All   |       | Dis   |       | No    |       |
|                    | Prov  |       | trict |       | n‑Dis |       |
| Services Received  | iders |       | Pro   |       | trict |       |
|                    |       |       | vider |       | Prov  |       |
|                    |       |       |       |       | iders |       |
+--------------------+-------+-------+-------+-------+-------+-------+
|                    | 200   |       | 200   |       |       |       |
|                    | 4--05 |       | 6--07 |       |       |       |
|                    | To    |       |       |       |       |       |
|                    | 200   |       |       |       |       |       |
|                    | 6--07 |       |       |       |       |       |
+--------------------+-------+-------+-------+-------+-------+-------+
|                    | Gains | n     | Gains | n     | Gains | n     |
+--------------------+-------+-------+-------+-------+-------+-------+
| **Mathematics**    |       |       |       |       |       |       |
+--------------------+-------+-------+-------+-------+-------+-------+
| Fewer than 20      | --    | 70    | -     | 31    | 0.18  | 20    |
| hours              | 0.004 |       | -0.16 |       |       |       |
+--------------------+-------+-------+-------+-------+-------+-------+
| 20--39 hours       | -     | 82    | \*\*  | 18    | 0.12  | 37    |
|                    | -0.10 |       |       |       |       |       |
+--------------------+-------+-------+-------+-------+-------+-------+
| 40 or more hours   | -     | 69    | \*\*  | 18    | \*\*  | 5     |
|                    | -0.01 |       |       |       |       |       |
+--------------------+-------+-------+-------+-------+-------+-------+
| Total n            |       | 201   |       | 67    |       | 62    |
+--------------------+-------+-------+-------+-------+-------+-------+
| Reading            |       |       |       |       |       |       |
+--------------------+-------+-------+-------+-------+-------+-------+
| Fewer than 20      | -     | 85    | --0   | 34    | 0.01  | 32    |
| hours              | -0.08 |       | .24\* |       |       |       |
+--------------------+-------+-------+-------+-------+-------+-------+
| 20--39 hours       | 0.07  | 96    | 0.01  | 20    | 0.08  | 53    |
+--------------------+-------+-------+-------+-------+-------+-------+
| 40 or more hours   | 0.05  | 77    | -     | 29    | \*\*  | 5     |
|                    |       |       | -0.05 |       |       |       |
+--------------------+-------+-------+-------+-------+-------+-------+
| Total n            |       | 258   |       | 83    |       | 90    |
+--------------------+-------+-------+-------+-------+-------+-------+
| **Exhibit reads:** |       |       |       |       |       |       |
| In Anchorage,      |       |       |       |       |       |       |
| overall, students  |       |       |       |       |       |       |
| receiving fewer    |       |       |       |       |       |       |
| than 20 hours of   |       |       |       |       |       |       |
| supplemental       |       |       |       |       |       |       |
| educational        |       |       |       |       |       |       |
| services did not   |       |       |       |       |       |       |
| demonstrate        |       |       |       |       |       |       |
| greater gains in   |       |       |       |       |       |       |
| mathematics        |       |       |       |       |       |       |
| achievement than   |       |       |       |       |       |       |
| eligible           |       |       |       |       |       |       |
| non‑participants.  |       |       |       |       |       |       |
| Specifically,      |       |       |       |       |       |       |
| students receiving |       |       |       |       |       |       |
| 20 hours or less   |       |       |       |       |       |       |
| of supplemental    |       |       |       |       |       |       |
| educational        |       |       |       |       |       |       |
| services from the  |       |       |       |       |       |       |
| district in        |       |       |       |       |       |       |
| reading            |       |       |       |       |       |       |
| experienced        |       |       |       |       |       |       |
| smaller gains      |       |       |       |       |       |       |
| (-0.24 standard    |       |       |       |       |       |       |
| deviations) than   |       |       |       |       |       |       |
| eligible           |       |       |       |       |       |       |
| non‑participants.  |       |       |       |       |       |       |
|                    |       |       |       |       |       |       |
| Notes: Scores      |       |       |       |       |       |       |
| reported are       |       |       |       |       |       |       |
| rank-based         |       |       |       |       |       |       |
| z‑scores. N is the |       |       |       |       |       |       |
| number of student  |       |       |       |       |       |       |
| observations in    |       |       |       |       |       |       |
| treatment          |       |       |       |       |       |       |
| contributing to    |       |       |       |       |       |       |
| the estimate. All  |       |       |       |       |       |       |
| models controlled  |       |       |       |       |       |       |
| for student        |       |       |       |       |       |       |
| eligibility for    |       |       |       |       |       |       |
| services.\         |       |       |       |       |       |       |
| \* indicates gains |       |       |       |       |       |       |
| that were          |       |       |       |       |       |       |
| statistically      |       |       |       |       |       |       |
| different form the |       |       |       |       |       |       |
| gains of eligible  |       |       |       |       |       |       |
| non-participants   |       |       |       |       |       |       |
| at the .05 level.  |       |       |       |       |       |       |
|                    |       |       |       |       |       |       |
| \*\* indicates     |       |       |       |       |       |       |
| that the data are  |       |       |       |       |       |       |
| not reported due   |       |       |       |       |       |       |
| to small sample    |       |       |       |       |       |       |
| size (n\<20        |       |       |       |       |       |       |
| students).         |       |       |       |       |       |       |
|                    |       |       |       |       |       |       |
| Source: Anchorage  |       |       |       |       |       |       |
| School District    |       |       |       |       |       |       |
| Administrative     |       |       |       |       |       |       |
| Data, 2002--03     |       |       |       |       |       |       |
| through 2006--07.  |       |       |       |       |       |       |
+--------------------+-------+-------+-------+-------+-------+-------+

In Hillsborough, mathematics achievement gains were found for students
who received less than 20 hours of services and for those receiving 20
to 39 hours of services. Gains for students receiving 40 or more hours
of services in mathematics were not statistically significant, but
participant sizes for this group were very small (see Exhibit 27).
During 2006--07, non-district providers serving students who received
between 20 and 39 hours of services in mathematics experienced gains of
about 0.1 standard deviations larger than those of eligible
non-participants.

Breaking down the analysis by hours of tutoring, no statistically
significant differences were observable in reading in Hillsborough
between participants and eligible non-participants.

+------------------+-------+-------+-------+-------+-------+-------+
| []{#__RefHeadin  |       |       |       |       |       |       |
| g___Toc220485747 |       |       |       |       |       |       |
| .an              |       |       |       |       |       |       |
| chor}Exhibit 27\ |       |       |       |       |       |       |
| Achievement      |       |       |       |       |       |       |
| Gains on the     |       |       |       |       |       |       |
| FCAT of Students |       |       |       |       |       |       |
| in Grades 1--12  |       |       |       |       |       |       |
| by Provider Type |       |       |       |       |       |       |
| and\             |       |       |       |       |       |       |
| Hours of         |       |       |       |       |       |       |
| Services         |       |       |       |       |       |       |
| Received, in     |       |       |       |       |       |       |
| Hillsborough,    |       |       |       |       |       |       |
| 2002--03 Through |       |       |       |       |       |       |
| 2006--07         |       |       |       |       |       |       |
+==================+=======+=======+=======+=======+=======+=======+
| Hours of         | All   |       | Dis   |       | No    |       |
|                  | Prov  |       | trict |       | n‑Dis |       |
| Services         | iders |       | Pro   |       | trict |       |
| Received         |       |       | vider |       | Prov  |       |
|                  |       |       |       |       | iders |       |
+------------------+-------+-------+-------+-------+-------+-------+
|                  | 200   |       | 200   |       |       |       |
|                  | 4--05 |       | 6--07 |       |       |       |
|                  | To    |       |       |       |       |       |
|                  | 200   |       |       |       |       |       |
|                  | 6--07 |       |       |       |       |       |
+------------------+-------+-------+-------+-------+-------+-------+
|                  | Gains | n     | Gains | n     | Gains | n     |
+------------------+-------+-------+-------+-------+-------+-------+
| **Mathematics**  |       |       |       |       |       |       |
+------------------+-------+-------+-------+-------+-------+-------+
| Fewer than 20    | 0     | 761   | 0.03  | 112   | 0.05  | 277   |
| hours            | .05\* |       |       |       |       |       |
+------------------+-------+-------+-------+-------+-------+-------+
| 20--39 hours     | 0     | 1,096 | 0.07  | 104   | 0     | 445   |
|                  | .04\* |       |       |       | .09\* |       |
+------------------+-------+-------+-------+-------+-------+-------+
| 40 or more hours | 0.09  | 106   | 0.08  | 27    | 0.18  | 39    |
+------------------+-------+-------+-------+-------+-------+-------+
| **Total n**      |       | 1,966 |       | 243   |       | 761   |
+------------------+-------+-------+-------+-------+-------+-------+
| Reading          |       |       |       |       |       |       |
+------------------+-------+-------+-------+-------+-------+-------+
| Fewer than 20    | -     | 2,471 | -     | 210   | -     | 795   |
| hours            | -0.01 |       | -0.10 |       | -0.03 |       |
+------------------+-------+-------+-------+-------+-------+-------+
| 20--39 hours     | 0.002 | 4,445 | 0.05  | 211   | 0.01  | 1,673 |
+------------------+-------+-------+-------+-------+-------+-------+
| 40 or more hours | 0.03  | 362   | 0.13  | 84    | -     | 84    |
|                  |       |       |       |       | -0.03 |       |
+------------------+-------+-------+-------+-------+-------+-------+
| **Total n**      |       | 7,288 |       | 505   |       | 2,552 |
+------------------+-------+-------+-------+-------+-------+-------+
| **Exhibit        |       |       |       |       |       |       |
| reads:** In      |       |       |       |       |       |       |
| Hillsborough,    |       |       |       |       |       |       |
| students         |       |       |       |       |       |       |
| receiving fewer  |       |       |       |       |       |       |
| than 20 hours of |       |       |       |       |       |       |
| services in      |       |       |       |       |       |       |
| mathematics      |       |       |       |       |       |       |
| experienced      |       |       |       |       |       |       |
| academic gains   |       |       |       |       |       |       |
| of .05 standard  |       |       |       |       |       |       |
| deviations above |       |       |       |       |       |       |
| eligible         |       |       |       |       |       |       |
| n                |       |       |       |       |       |       |
| on‑participants, |       |       |       |       |       |       |
| a statistically  |       |       |       |       |       |       |
| significant      |       |       |       |       |       |       |
| difference.      |       |       |       |       |       |       |
|                  |       |       |       |       |       |       |
| Notes: Scores    |       |       |       |       |       |       |
| reported are     |       |       |       |       |       |       |
| rank-based       |       |       |       |       |       |       |
| z‑scores. N is   |       |       |       |       |       |       |
| the number of    |       |       |       |       |       |       |
| student          |       |       |       |       |       |       |
| observations in  |       |       |       |       |       |       |
| treatment        |       |       |       |       |       |       |
| contributing to  |       |       |       |       |       |       |
| the estimate.    |       |       |       |       |       |       |
| All models       |       |       |       |       |       |       |
| controlled for   |       |       |       |       |       |       |
| student          |       |       |       |       |       |       |
| eligibility for  |       |       |       |       |       |       |
| services.\       |       |       |       |       |       |       |
| \* indicates     |       |       |       |       |       |       |
| gains that were  |       |       |       |       |       |       |
| statistically    |       |       |       |       |       |       |
| different form   |       |       |       |       |       |       |
| the gains of     |       |       |       |       |       |       |
| eligible         |       |       |       |       |       |       |
| non-participants |       |       |       |       |       |       |
| at the .05       |       |       |       |       |       |       |
| level.           |       |       |       |       |       |       |
|                  |       |       |       |       |       |       |
| Source:          |       |       |       |       |       |       |
| Hillsborough     |       |       |       |       |       |       |
| School District  |       |       |       |       |       |       |
| Administrative   |       |       |       |       |       |       |
| Data, 2002--03   |       |       |       |       |       |       |
| through          |       |       |       |       |       |       |
| 2006--07.        |       |       |       |       |       |       |
+------------------+-------+-------+-------+-------+-------+-------+

##   {#section-7 .unnumbered}

## Effect of Subject of Services on Achievement {#effect-of-subject-of-services-on-achievement .unnumbered}

In this study, only participants in Hillsborough that received tutoring
in mathematics experienced larger academic gains (in mathematics) than
eligible non‑participants.

Not all participants received supplemental educational services in the
same subject. Because of this, it is important to analyze whether the
subject of the service provided was related to the academic gains of
participants. In Anchorage, no statistically significant differences in
academic gains were observable between students who only received
supplemental educational services in mathematics or reading and eligible
non‑participants (see Exhibit 28). However, those who received services
in both subjects experienced smaller academic gains in mathematics than
did eligible non‑participants.

+-----------------------+----------+----------+-----------+-----------+
| []{#__RefH            |          |          |           |           |
| eading___Toc220485748 |          |          |           |           |
| .anchor}Exhibit 28\   |          |          |           |           |
| Achievement Gains on  |          |          |           |           |
| the Benchmark and SBA |          |          |           |           |
| of Students in Grades |          |          |           |           |
| 3--10, by Subject, in |          |          |           |           |
| Anchorage, 2002--03   |          |          |           |           |
| Through 2006--07      |          |          |           |           |
+-----------------------+----------+----------+-----------+-----------+
| **Subject of          | Mat      |          | Reading   |           |
| Services**            | hematics |          |           |           |
+-----------------------+----------+----------+-----------+-----------+
|                       | Gains    | n        | Gains     | n         |
+-----------------------+----------+----------+-----------+-----------+
| Mathematics and       | --0.21\* | 139      | --0.11    | 137       |
| reading               |          |          |           |           |
+-----------------------+----------+----------+-----------+-----------+
| Mathematics only      | 0.04     | 49       | --0.09    | 50        |
+-----------------------+----------+----------+-----------+-----------+
| Reading only          | 0.05     | 92       | 0.11      | 91        |
+-----------------------+----------+----------+-----------+-----------+
| **Exhibit reads: In   |          |          |           |           |
| Anchorage,** students |          |          |           |           |
| who received services |          |          |           |           |
| in both mathematics   |          |          |           |           |
| and reading           |          |          |           |           |
| demonstrated smaller  |          |          |           |           |
| gains in mathematics  |          |          |           |           |
| achievement, with a   |          |          |           |           |
| change in their       |          |          |           |           |
| academic achievement  |          |          |           |           |
| of .21 standard       |          |          |           |           |
| deviations below      |          |          |           |           |
| eligible              |          |          |           |           |
| non‑participants, a   |          |          |           |           |
| statistically         |          |          |           |           |
| significant           |          |          |           |           |
| difference.           |          |          |           |           |
|                       |          |          |           |           |
| Notes: Scores         |          |          |           |           |
| reported are          |          |          |           |           |
| rank-based z‑scores.  |          |          |           |           |
| N is the number of    |          |          |           |           |
| student observations  |          |          |           |           |
| in treatment          |          |          |           |           |
| contributing to the   |          |          |           |           |
| estimate. All models  |          |          |           |           |
| controlled for        |          |          |           |           |
| student eligibility   |          |          |           |           |
| for services.\        |          |          |           |           |
| \* indicates gains    |          |          |           |           |
| that were             |          |          |           |           |
| statistically         |          |          |           |           |
| different form the    |          |          |           |           |
| gains of eligible     |          |          |           |           |
| non-participants at   |          |          |           |           |
| the .05 level.        |          |          |           |           |
|                       |          |          |           |           |
| Source: Anchorage     |          |          |           |           |
| School District       |          |          |           |           |
| Administrative Data,  |          |          |           |           |
| 2002--03 through      |          |          |           |           |
| 2006--07.             |          |          |           |           |
+-----------------------+----------+----------+-----------+-----------+

In Hillsborough, participants who received supplemental educational
services in reading experienced no larger academic gains in reading or
mathematics than eligible non‑participants. However, students who
received supplemental educational services in mathematics only
demonstrated a significant gain in mathematics achievement, 0.07
standard deviations (see Exhibit 29). Students who received supplemental
educational services in both subjects did not experience statistically
significant gains in either subject in comparison to eligible
non‑participants.

+-----------------------+----------+----------+-----------+-----------+
| []{#__RefH            |          |          |           |           |
| eading___Toc220485749 |          |          |           |           |
| .anchor}Exhibit 29\   |          |          |           |           |
| Achievement Gains on  |          |          |           |           |
| the FCAT of Students  |          |          |           |           |
| in Grades 1--12 by    |          |          |           |           |
| Subject, in           |          |          |           |           |
| Hillsborough,         |          |          |           |           |
| 2002--03 Through      |          |          |           |           |
| 2006--07              |          |          |           |           |
+-----------------------+----------+----------+-----------+-----------+
| **Subject of          | Mat      |          | Reading   |           |
| Services**            | hematics |          |           |           |
+-----------------------+----------+----------+-----------+-----------+
|                       | Gains    | n        | Gains     | n         |
+-----------------------+----------+----------+-----------+-----------+
| Mathematics and       | 0.004    | 608      | 0.001     | 608       |
| reading               |          |          |           |           |
+-----------------------+----------+----------+-----------+-----------+
| Mathematics only      | 0.07\*   | 1,552    | --0.02    | 1,552     |
+-----------------------+----------+----------+-----------+-----------+
| Reading only          | --0.01   | 6,810    | 0.03      | 6,810     |
+-----------------------+----------+----------+-----------+-----------+
| **Exhibit reads:** In |          |          |           |           |
| Hillsborough,         |          |          |           |           |
| students who received |          |          |           |           |
| services in both      |          |          |           |           |
| mathematics and       |          |          |           |           |
| reading did not       |          |          |           |           |
| demonstrate greater   |          |          |           |           |
| gains in mathematics  |          |          |           |           |
| achievement than      |          |          |           |           |
| eligible              |          |          |           |           |
| non‑participants.     |          |          |           |           |
|                       |          |          |           |           |
| Notes: Scores         |          |          |           |           |
| reported are          |          |          |           |           |
| rank-based z‑scores.  |          |          |           |           |
| N is the number of    |          |          |           |           |
| student observations  |          |          |           |           |
| in treatment          |          |          |           |           |
| contributing to the   |          |          |           |           |
| estimate. All models  |          |          |           |           |
| controlled for        |          |          |           |           |
| student eligibility   |          |          |           |           |
| for services.\        |          |          |           |           |
| \* indicates gains    |          |          |           |           |
| that were             |          |          |           |           |
| statistically         |          |          |           |           |
| different form the    |          |          |           |           |
| gains of eligible     |          |          |           |           |
| non-participants at   |          |          |           |           |
| the .05 level.        |          |          |           |           |
|                       |          |          |           |           |
| Source: Hillsborough  |          |          |           |           |
| School District       |          |          |           |           |
| Administrative Data,  |          |          |           |           |
| 2002--03 through      |          |          |           |           |
| 2006--07.             |          |          |           |           |
+-----------------------+----------+----------+-----------+-----------+

## Effect of Participation by Student Characteristics {#effect-of-participation-by-student-characteristics .unnumbered}

In both districts, most student subgroups did not show significant
achievement gains.

In Anchorage, where sample sizes were very small, the only student
subgroup showing a significant difference between participants and
non-participants was Hispanic students, where a lower gain than eligible
non-participants was found in mathematics (see Exhibit 30). In
Hillsborough, positive achievement gains were found for Hispanic and
limited English proficient students in mathematics. No significant gains
were observed for African-American, white, or special education students
in mathematics or for any subgroup in reading (see Exhibit 31).

+--------------------+-----------+-----------+-----------+-------------+
| []{#__RefHead      |           |           |           |             |
| ing___Toc220485750 |           |           |           |             |
| .                  |           |           |           |             |
| anchor}Exhibit 30\ |           |           |           |             |
| Achievement Gains  |           |           |           |             |
| on the Benchmark   |           |           |           |             |
| and SBA of         |           |           |           |             |
| Students in Grades |           |           |           |             |
| 3--10\             |           |           |           |             |
| for Supplemental   |           |           |           |             |
| Educational        |           |           |           |             |
| Services           |           |           |           |             |
| Participants,\     |           |           |           |             |
| by Student         |           |           |           |             |
| Characteristics,   |           |           |           |             |
| in Anchorage,      |           |           |           |             |
| 2002--03 Through   |           |           |           |             |
| 2006--07           |           |           |           |             |
+--------------------+-----------+-----------+-----------+-------------+
|                    | Ma        |           | Reading   |             |
|                    | thematics |           |           |             |
+--------------------+-----------+-----------+-----------+-------------+
|                    | Gains     | n         | Gains     | n           |
+--------------------+-----------+-----------+-----------+-------------+
| All Participants   | --0.05    |           | 0.01      |             |
+--------------------+-----------+-----------+-----------+-------------+
| African-American   | 0.13      | 49        | --0.03    | 58          |
| students           |           |           |           |             |
+--------------------+-----------+-----------+-----------+-------------+
| Alaska Native      | --0.01    | 42        | 0.06      | 54          |
| students           |           |           |           |             |
+--------------------+-----------+-----------+-----------+-------------+
| White students     | --0.14    | 35        | --0.17    | 37          |
+--------------------+-----------+-----------+-----------+-------------+
| Asian students     | 0.11      | 37        | 0.24      | 44          |
+--------------------+-----------+-----------+-----------+-------------+
| Hispanic students  | --0.34\*  | 36        | --0.29    | 36          |
+--------------------+-----------+-----------+-----------+-------------+
| **Exhibit reads:** |           |           |           |             |
| In Anchorage,      |           |           |           |             |
| African-American   |           |           |           |             |
| students who       |           |           |           |             |
| participated in    |           |           |           |             |
| supplemental       |           |           |           |             |
| educational        |           |           |           |             |
| services did not   |           |           |           |             |
| demonstrate        |           |           |           |             |
| greater gains in   |           |           |           |             |
| mathematics        |           |           |           |             |
| achievement than   |           |           |           |             |
| eligible           |           |           |           |             |
| non‑participants.  |           |           |           |             |
|                    |           |           |           |             |
| Notes: Scores      |           |           |           |             |
| reported are       |           |           |           |             |
| rank-based         |           |           |           |             |
| z‑scores. N is the |           |           |           |             |
| number of student  |           |           |           |             |
| observations in    |           |           |           |             |
| treatment          |           |           |           |             |
| contributing to    |           |           |           |             |
| the estimate. All  |           |           |           |             |
| models controlled  |           |           |           |             |
| for student        |           |           |           |             |
| eligibility for    |           |           |           |             |
| services.\         |           |           |           |             |
| \* indicates gains |           |           |           |             |
| that were          |           |           |           |             |
| statistically      |           |           |           |             |
| different form the |           |           |           |             |
| gains of eligible  |           |           |           |             |
| non-participants   |           |           |           |             |
| at the .05 level.  |           |           |           |             |
|                    |           |           |           |             |
| Source: Anchorage  |           |           |           |             |
| School District    |           |           |           |             |
| Administrative     |           |           |           |             |
| Data, 2002--03     |           |           |           |             |
| through 2006--07.  |           |           |           |             |
+--------------------+-----------+-----------+-----------+-------------+

+---------------------+----------+----------+-----------+------------+
| []{#__RefHea        |          |          |           |            |
| ding___Toc220485751 |          |          |           |            |
| .anchor}Exhibit 31\ |          |          |           |            |
| Achievement Gains   |          |          |           |            |
| on the FCAT of      |          |          |           |            |
| Students in Grades  |          |          |           |            |
| 1--12\              |          |          |           |            |
| for Supplemental    |          |          |           |            |
| Educational         |          |          |           |            |
| Services            |          |          |           |            |
| Participants,\      |          |          |           |            |
| by Student          |          |          |           |            |
| Characteristics, in |          |          |           |            |
| Hillsborough,       |          |          |           |            |
| 2002--03 Through    |          |          |           |            |
| 2006--07            |          |          |           |            |
+---------------------+----------+----------+-----------+------------+
|                     | Mat      |          | Reading   |            |
|                     | hematics |          |           |            |
+---------------------+----------+----------+-----------+------------+
|                     | Gains    | n        | Gains     | n          |
+---------------------+----------+----------+-----------+------------+
| All Participants    | 0.05\*   |          | --0.001   |            |
+---------------------+----------+----------+-----------+------------+
| African-American    | 0.04     | 879      | --0.01    | 2,448      |
| students            |          |          |           |            |
+---------------------+----------+----------+-----------+------------+
| Hispanic students   | 0.05\*   | 702      | --0.01    | 3,361      |
+---------------------+----------+----------+-----------+------------+
| White students      | 0.08     | 270      | 0.05      | 1,090      |
+---------------------+----------+----------+-----------+------------+
| Special education   | 0.08     | 370      | --0.02    | 1,901      |
| students            |          |          |           |            |
+---------------------+----------+----------+-----------+------------+
| Limited English     | 0.05\*   | 615      | --0.01    | 3,185      |
| proficient          |          |          |           |            |
+---------------------+----------+----------+-----------+------------+
| **Exhibit reads:**  |          |          |           |            |
| In Hillsborough,    |          |          |           |            |
| African-American    |          |          |           |            |
| students who        |          |          |           |            |
| participated in     |          |          |           |            |
| supplemental        |          |          |           |            |
| educational         |          |          |           |            |
| services did not    |          |          |           |            |
| demonstrate greater |          |          |           |            |
| gains in            |          |          |           |            |
| mathematics or      |          |          |           |            |
| reading achievement |          |          |           |            |
| than eligible       |          |          |           |            |
| non‑participants.   |          |          |           |            |
|                     |          |          |           |            |
| Notes: Scores       |          |          |           |            |
| reported are        |          |          |           |            |
| rank-based          |          |          |           |            |
| z‑scores. N is the  |          |          |           |            |
| number of student   |          |          |           |            |
| observations in     |          |          |           |            |
| treatment           |          |          |           |            |
| contributing to the |          |          |           |            |
| estimate. All       |          |          |           |            |
| models controlled   |          |          |           |            |
| for student         |          |          |           |            |
| eligibility for     |          |          |           |            |
| services.\          |          |          |           |            |
| \* indicates gains  |          |          |           |            |
| that were           |          |          |           |            |
| statistically       |          |          |           |            |
| different form the  |          |          |           |            |
| gains of eligible   |          |          |           |            |
| non-participants at |          |          |           |            |
| the .05 level.      |          |          |           |            |
|                     |          |          |           |            |
| Source:             |          |          |           |            |
| Hillsborough School |          |          |           |            |
| District            |          |          |           |            |
| Administrative      |          |          |           |            |
| Data, 2002--03      |          |          |           |            |
| through 2006--07.   |          |          |           |            |
+---------------------+----------+----------+-----------+------------+

# District Communication With Parents

By accepting the supplemental educational service waivers, the four
districts were required to improve communication with parents by using
multiple communications strategies, extend enrollment windows, and
provide balanced information on all providers. These strategies were
intended to achieve one objective of the waivers: to help ensure that
more eligible students received supplemental educational services.

As evidence of meeting this requirement, Anchorage and Hillsborough
submitted examples of their communication with parents about
supplemental educational services eligibility and about their provider
options. Anchorage submitted six documents and notes. Hillsborough
submitted three documents and a parent handbook.[^14] Two study team
members independently reviewed each document looking for clarity
(including general readability, correct use of terms, and use of the
phrase "free tutoring"), bias towards district providers, and use of
jargon. To assess readability, the Flesch‑Kincaid score[^15] was
calculated to determine the grade level at which each document was
written. (The Flesch-Kincaid test measures word length and sentence
length to generate a score, which is translated into a U.S. grade level
indicator, representative of the approximate educational level to which
the text is written.) Reviewers also determined whether the documents
were available in other languages, whether districts used diverse
outreach strategies to distribute information about supplemental
educational services to parents and whether district materials included
key details, such as contact information.

Both Anchorage and Hillsborough used jargon and complex language in
communication with parents.

The parent notification documents districts distributed, such as
letters, handbooks and flyers, demonstrated an effort to communicate
necessary information about supplemental educational services with
parents. However, the style and language used in these letters was
somewhat complex. For example, one letter to parents in Anchorage begins
with "Under the federal education law, *No Child Left Behind*, some
students who attend Title I schools that have reached an *NCLB* School
Improvement Status of Level 2 or higher are eligible for supplemental
educational services." That particular letter rated a 10.2 on the
Flesch‑Kincaid readability test, indicating the letter was written just
above the tenth grade level.

Similarly, the Hillsborough supplemental educational services parent
handbook included questions under the heading "General Information." In
response to the question "What are supplemental educational services?"
the parent handbook explained, "According to the *No Child Left Behind
Act*, supplemental educational services are free tutoring and other
supplemental academic enrichment services that are in addition to
instruction provided during the school day." The Flesch‑Kincaid
readability rating for this document was 13.2, which suggests it would
be challenging for parents who had not completed high school.

Both Anchorage and Hillsborough used multiple communication strategies
to reach eligible families.

Districts receiving the waiver were required to use multiple
communication strategies to inform eligible families about supplemental
educational services in order to increase participation. Anchorage
indicated that it used multiple methods of communication. For example,
it mailed notification letters to all households in which a student was
eligible for supplemental educational services on August 3 and September
7. These letters covered the supplemental educational services options
and specifics and advertised upcoming informational meetings. Anchorage
sent these letters in English, Hmong, Samoan, Spanish, and Tagalog. On
August 4, 2006, the district published AYP results and supplemental
educational services information in the Anchorage Daily News newspaper.
Between September and November Anchorage posted one hundred posters in
ten schools advertising informational meetings, and each school put
notices in their school newsletters during September and October. Also
during September and October, the district held informational meetings
in each of the ten schools in which students were eligible for
supplemental educational services, and on September 19, sent automated
phone calls to households with eligible students. On September 20,
Anchorage held a provider fair at one middle school. In November,
schools required to offer supplemental educational services set up
tables at their entrance on parent‑teacher conference nights. In
addition, information was continually available on the Anchorage school
district website. Finally, in April and May of 2007, the district began
working with the state's Parent Information Resource Center to form
partnerships with faith‑based and community‑based organizations to
provide information to parents about supplemental educational services.

Hillsborough also used several methods to reach families. It sent
families with eligible students a Supplemental Educational Services
Parent Handbook for the 2006--07 school year. The handbook contained a
welcome letter, general information questions, a provider "at‑a‑glance"
chart, a description of delivery models (in‑home, distance learning,
off‑site, on‑site), parent satisfaction survey results from the 2005--06
school year, a list of acronyms and phrases, detailed profiles of each
provider, and a provider fair announcement. The district also offered
ten provider fairs in August. All fairs were held on weeknights from
5:30pm to 7:30pm. The announcement indicated that door prizes and
refreshments would be available. A brochure about the district provider,
Hillsborough County Public Schools Academy for Success (HCPSAFS), was
available in both English and Spanish. Parents who chose HCPSAFS
received a thank you letter dated October 5. The letter informed them
that HCPSAFS was still hiring staff and would contact them within four
to five weeks about scheduling a goal‑setting meeting. In the letter,
HCPSAFS gave parents the option to chose another provider if they
wished. This letter was sent in both English and Spanish.

Both Anchorage and Hillsborough provided extended enrollment windows for
families to sign up for services.

Another stipulation of the waiver was that districts would offer
extended or additional enrollment windows to give more families the
opportunity to sign up and participate in supplemental educational
services. In Anchorage, notification letters were sent in August and
September. Enrollment began September 7 with the application due date of
October 20. The district offered an additional window beginning with the
parent‑teacher conferences November 8 and ending with an application due
date of November 22. The district indicated that applications that
arrived after the due date were approved.

According to Hillsborough's communication with the U.S. Department of
Education, its enrollment began during pre-planning days in July with a
steady recruitment process going through the first soft deadline date of
October 13---a deadline required by state law. Enrollment then continued
until January 26 for any interested parent of an eligible student.

Both Anchorage and Hillsborough provided balanced information on all
providers and showed no bias towards their own programs.

Districts receiving the waiver needed to demonstrate that they provided
eligible families with balanced information on all providers. Anchorage
submitted a one‑page flyer that lists thirteen providers in alphabetical
order. Contact information for each provider was listed. Anchorage
School District was sixth on the list and no bias was evident.

In the Hillsborough Supplemental Educational Services Parent Handbook,
each provider was listed in alphabetical order. The district's Academy
for Success was second in that order, but there was no indication it was
the district provider aside from the contact email address.

# Conclusions

This examination of the implementation of supplemental educational
services shows that in Anchorage and Hillsborough the percentage of
eligible students participating in supplemental educational services
increased over time, reaching 11 and 14 percent, respectively, in
2006--07.

In both Anchorage and Hillsborough, students who were eligible for
supplemental educational services had significantly lower levels of
prior achievement in mathematics and reading than students who were not
eligible. In both districts students who participated in supplemental
educational services in 2006--07 had lower levels of prior academic
achievement in 2005--06 than eligible students who chose not to
participate.

In both districts, students served by non‑district providers showed
larger academic achievement gains than eligible non-participants in
mathematics (but not in reading). Students served by the two district
providers (Anchorage and Hillsborough) did not show gains in either
subject compared with eligible non-participants. Looking at individual
non-district providers, only one such provider in Hillsborough showed an
achievement gain (in mathematics) relative to eligible non-participants.
No individual provider in Anchorage showed any significant gains
relative to non-participants.

Overall, in Anchorage, participants receiving supplemental education
services from any provider did not show a statistically significant
difference in academic achievement gains compared with eligible
non‑participants. In Hillsborough, the overall results across all
providers showed achievement gains in mathematics but not in reading.

While there is room for improvement regarding district communication
with parents about supplemental educational services, both Anchorage and
Hillsborough made efforts to inform eligible families about the program
offerings and to help ensure that more eligible students receive
supplemental educational services. To this end, both districts used
multiple communication methods and provided unbiased program
information.

# Appendix A: Description of District Data {#appendix-a-description-of-district-data .unnumbered}

## Anchorage {#anchorage .unnumbered}

The Anchorage school district provided student‑level data for the school
years 2002--03 through\
2006--07. Demographic and assessment data were provided for all years
and all students in the district, while detailed information on
supplemental educational services was provided for 2004--05 through
2006--07. In these latter years, data were provided on the subjects in
which students were tutored, the total number of hours of tutoring they
received, and the provider they chose. Student eligibility and
participation in supplemental educational services was provided for all
three years.

## Hillsborough {#hillsborough .unnumbered}

The Hillsborough school district provided student‑level data for all
students taking Florida's Comprehensive Assessment Test (FCAT) in
2002--03 through 2006--07, which included students in grades 3--10 in
2005--06 and 2006--07. The district also provided information on student
eligibility, participation, provider used, and number of hours of
tutoring received. These comprehensive databases included all students
in the district during the whole period of analysis, 2002--03 through
2006--07.

# Appendix B: Supplemental Exhibits {#appendix-b-supplemental-exhibits-1 .unnumbered}

+----------------------+----------------+---------------+-------------+
| > []{#__RefHe        |                |               |             |
| ading___Toc220485752 |                |               |             |
| > .anchor}Exhibit    |                |               |             |
| > B.1\               |                |               |             |
| > Participation      |                |               |             |
| > Rates for          |                |               |             |
| > Supplemental       |                |               |             |
| > Educational        |                |               |             |
| > Services in        |                |               |             |
| > Grades K--12, by   |                |               |             |
| > Race‑Ethnicity, in |                |               |             |
| > Anchorage,         |                |               |             |
| > 2006--07           |                |               |             |
+----------------------+----------------+---------------+-------------+
|                      | Eligible       | Participants  | Pa          |
|                      | No             |               | rticipation |
|                      | n‑Participants |               | Rate        |
+----------------------+----------------+---------------+-------------+
| \% African-American  | 334            | 64            | 16.1%       |
+----------------------+----------------+---------------+-------------+
| \% Alaska Native     | 654            | 61            | 8.5%        |
+----------------------+----------------+---------------+-------------+
| \% White             | 417            | 51            | 10.9%       |
+----------------------+----------------+---------------+-------------+
| \% Asian             | 678            | 48            | 6.6%        |
+----------------------+----------------+---------------+-------------+
| \% Multi‑racial      | 271            | 44            | 14.0%       |
+----------------------+----------------+---------------+-------------+
| \% Hispanic          | 197            | 37            | 15.8%       |
+----------------------+----------------+---------------+-------------+
| \% American Indian   | 31             | 3             | 8.8%        |
+----------------------+----------------+---------------+-------------+
| **Total**            | **2,582**      | **308**       | **10.7%**   |
+----------------------+----------------+---------------+-------------+
| **Exhibit reads:**   |                |               |             |
| In 2006--07, 334     |                |               |             |
| African-American     |                |               |             |
| students were        |                |               |             |
| eligible for         |                |               |             |
| supplemental         |                |               |             |
| educational services |                |               |             |
| but did not          |                |               |             |
| participate in the   |                |               |             |
| program in           |                |               |             |
| Anchorage. The       |                |               |             |
| number of            |                |               |             |
| African-American     |                |               |             |
| participants was     |                |               |             |
| equal to 64 during   |                |               |             |
| that year, or        |                |               |             |
| 16.1 percent of all  |                |               |             |
| eligible             |                |               |             |
| African-American     |                |               |             |
| students.            |                |               |             |
|                      |                |               |             |
| Source: Anchorage    |                |               |             |
| School District      |                |               |             |
| Administrative Data, |                |               |             |
| 2002--03 through     |                |               |             |
| 2006--07.            |                |               |             |
+----------------------+----------------+---------------+-------------+

+------------------+-------------+------------+-----------+----------+
| >                |             |            |           |          |
|  []{#__RefHeadin |             |            |           |          |
| g___Toc220485753 |             |            |           |          |
| >                |             |            |           |          |
|  .anchor}Exhibit |             |            |           |          |
| > B.2\           |             |            |           |          |
| > Participation  |             |            |           |          |
| > Rates for      |             |            |           |          |
| > Supplemental   |             |            |           |          |
| > Educational    |             |            |           |          |
| > Services in    |             |            |           |          |
| > Grades K--12,  |             |            |           |          |
| > by             |             |            |           |          |
| >                |             |            |           |          |
|  Race‑Ethnicity, |             |            |           |          |
| > in             |             |            |           |          |
| > Hillsborough,  |             |            |           |          |
| > 2006--07       |             |            |           |          |
+------------------+-------------+------------+-----------+----------+
|                  | Eligible    | Pa         | Part      |          |
|                  | Non‑P       | rticipants | icipation |          |
|                  | articipants |            | Rate      |          |
+------------------+-------------+------------+-----------+----------+
| \%               | 13,113      | 2,034      | 13.4%     |          |
| African-American |             |            |           |          |
+------------------+-------------+------------+-----------+----------+
| \% White         | 6,372       | 838        | 11.6%     |          |
+------------------+-------------+------------+-----------+----------+
| \% Asian         | 510         | 70         | 12.1%     |          |
+------------------+-------------+------------+-----------+----------+
| \% Multi‑racial  | 1,597       | 255        | 13.8%     |          |
+------------------+-------------+------------+-----------+----------+
| \% Hispanic      | 15,441      | 2,966      | 16.1%     |          |
+------------------+-------------+------------+-----------+----------+
| \% American      | 71          | 9          | 11.3%     |          |
| Indian           |             |            |           |          |
+------------------+-------------+------------+-----------+----------+
| **Total**        | **37,104**  | **6,172**  | **14.3%** |          |
+------------------+-------------+------------+-----------+----------+
| **Exhibit        |             |            |           |          |
| reads:** In      |             |            |           |          |
| 2006--07, 13,113 |             |            |           |          |
| African-American |             |            |           |          |
| students were    |             |            |           |          |
| eligible for     |             |            |           |          |
| supplemental     |             |            |           |          |
| educational      |             |            |           |          |
| services but did |             |            |           |          |
| not participate  |             |            |           |          |
| in the program   |             |            |           |          |
| in Hillsborough. |             |            |           |          |
| The number of    |             |            |           |          |
| African-American |             |            |           |          |
| participants was |             |            |           |          |
| equal to 2,034   |             |            |           |          |
| during that      |             |            |           |          |
| year, or         |             |            |           |          |
| 13.4 percent of  |             |            |           |          |
| all eligible     |             |            |           |          |
| African-American |             |            |           |          |
| students.        |             |            |           |          |
|                  |             |            |           |          |
| Source:          |             |            |           |          |
| Hillsborough     |             |            |           |          |
| School District  |             |            |           |          |
| Administrative   |             |            |           |          |
| Data, 2002--03   |             |            |           |          |
| through          |             |            |           |          |
| 2006--07.        |             |            |           |          |
+------------------+-------------+------------+-----------+----------+
| []{#__RefHeadin  |             |            |           |          |
| g___Toc220485754 |             |            |           |          |
| .anc             |             |            |           |          |
| hor}Exhibit B.3\ |             |            |           |          |
| Number and       |             |            |           |          |
| Percentage of    |             |            |           |          |
| Supplemental     |             |            |           |          |
| Educational      |             |            |           |          |
| Services         |             |            |           |          |
| Participants in  |             |            |           |          |
| Grades K--12, by |             |            |           |          |
| Subject, in      |             |            |           |          |
| Anchorage,       |             |            |           |          |
| 2006--07         |             |            |           |          |
+------------------+-------------+------------+-----------+----------+
| ![               |             |            |           |          |
| ](media/image5.p |             |            |           |          |
| ng){width="3.478 |             |            |           |          |
| 4722222222224in" |             |            |           |          |
| height="2.861    |             |            |           |          |
| 111111111111in"} |             |            |           |          |
+------------------+-------------+------------+-----------+----------+
| **Exhibit        |             |            |           |          |
| reads:** In      |             |            |           |          |
| Anchorage in     |             |            |           |          |
| 2006--07, 135    |             |            |           |          |
| students         |             |            |           |          |
| participated in  |             |            |           |          |
| supplemental     |             |            |           |          |
| educational      |             |            |           |          |
| services in      |             |            |           |          |
| reading only,    |             |            |           |          |
| representing 44  |             |            |           |          |
| percent of all   |             |            |           |          |
| participants.    |             |            |           |          |
|                  |             |            |           |          |
| Source:          |             |            |           |          |
| Anchorage School |             |            |           |          |
| District         |             |            |           |          |
| Administrative   |             |            |           |          |
| Data, 2006--07.  |             |            |           |          |
+------------------+-------------+------------+-----------+----------+

+-----------------------------------------------------------------------+
| []{#__RefHeading___Toc220485755 .anchor}Exhibit B.4\                  |
| Number and Percentage of Supplemental Educational Services            |
| Participants in Grades K--12 by Subject, in Hillsborough, 2006--07    |
+-----------------------------------------------------------------------+
| ![](media/image6.png){width="2.7472222222222222in"                    |
| height="2.88125in"}                                                   |
+-----------------------------------------------------------------------+
| **Exhibit reads:** In Hillsborough, 4,857 students received           |
| supplemental educational services in reading only in 2006--07,        |
| representing 79 percent of all participants.                          |
|                                                                       |
| Source: Hillsborough School District Administrative Data, 2006--07.   |
+-----------------------------------------------------------------------+

+-------------+----------+---------+--------+--------+--------+--------+
| []{#__Ref   |          |         |        |        |        |        |
| Heading___T |          |         |        |        |        |        |
| oc220485756 |          |         |        |        |        |        |
| .anc        |          |         |        |        |        |        |
| hor}Exhibit |          |         |        |        |        |        |
| B.5\        |          |         |        |        |        |        |
| Subject of  |          |         |        |        |        |        |
| S           |          |         |        |        |        |        |
| upplemental |          |         |        |        |        |        |
| Services in |          |         |        |        |        |        |
| Grades      |          |         |        |        |        |        |
| K--12, by   |          |         |        |        |        |        |
| Provider    |          |         |        |        |        |        |
| Type, in    |          |         |        |        |        |        |
| Anchorage,  |          |         |        |        |        |        |
| 2006--07    |          |         |        |        |        |        |
+-------------+----------+---------+--------+--------+--------+--------+
|             | All      |         | Anc    |        | Other  |        |
|             | Part     |         | horage |        | Pro    |        |
|             | icipants |         | School |        | viders |        |
|             |          |         | Di     |        |        |        |
|             |          |         | strict |        |        |        |
+-------------+----------+---------+--------+--------+--------+--------+
|             | Number   | Percent | Number | P      | Number | P      |
|             | of       | of      | of     | ercent | of     | ercent |
|             | Students | Total   | St     | of     | St     | of     |
|             |          |         | udents | Total  | udents | Total  |
+-------------+----------+---------+--------+--------+--------+--------+
| Reading     | 135      | 44%     | 52     | 38%    | 83     | 49%    |
| only        |          |         |        |        |        |        |
+-------------+----------+---------+--------+--------+--------+--------+
| Mathematics | 51       | 17%     | 11     | 8%     | 40     | 23%    |
| only        |          |         |        |        |        |        |
+-------------+----------+---------+--------+--------+--------+--------+
| Reading and | 72       | 23%     | 54     | 39%    | 18     | 11%    |
| Math        |          |         |        |        |        |        |
+-------------+----------+---------+--------+--------+--------+--------+
| No          | 50       | 16%     | 21     | 15%    | 29     | 17%    |
| information |          |         |        |        |        |        |
+-------------+----------+---------+--------+--------+--------+--------+
| **Total**   | **308**  | *       | *      | **     | *      | **     |
|             |          | *100%** | *138** | 100%** | *170** | 100%** |
+-------------+----------+---------+--------+--------+--------+--------+
| **Exhibit   |          |         |        |        |        |        |
| reads:** In |          |         |        |        |        |        |
| 2006--07,   |          |         |        |        |        |        |
| 135         |          |         |        |        |        |        |
| students    |          |         |        |        |        |        |
| received    |          |         |        |        |        |        |
| s           |          |         |        |        |        |        |
| upplemental |          |         |        |        |        |        |
| educational |          |         |        |        |        |        |
| services    |          |         |        |        |        |        |
| only in     |          |         |        |        |        |        |
| reading in  |          |         |        |        |        |        |
| Anchorage.  |          |         |        |        |        |        |
| This        |          |         |        |        |        |        |
| represented |          |         |        |        |        |        |
| 44 percent  |          |         |        |        |        |        |
| of all      |          |         |        |        |        |        |
| students    |          |         |        |        |        |        |
| receiving   |          |         |        |        |        |        |
| s           |          |         |        |        |        |        |
| upplemental |          |         |        |        |        |        |
| educational |          |         |        |        |        |        |
| services in |          |         |        |        |        |        |
| any subject |          |         |        |        |        |        |
| during that |          |         |        |        |        |        |
| year.       |          |         |        |        |        |        |
|             |          |         |        |        |        |        |
| Source:     |          |         |        |        |        |        |
| Anchorage   |          |         |        |        |        |        |
| School      |          |         |        |        |        |        |
| District    |          |         |        |        |        |        |
| Adm         |          |         |        |        |        |        |
| inistrative |          |         |        |        |        |        |
| Data,       |          |         |        |        |        |        |
| 2002--03    |          |         |        |        |        |        |
| through     |          |         |        |        |        |        |
| 2006--07.   |          |         |        |        |        |        |
+-------------+----------+---------+--------+--------+--------+--------+

+-------------+----------+---------+--------+--------+--------+--------+
| []{#__Ref   |          |         |        |        |        |        |
| Heading___T |          |         |        |        |        |        |
| oc220485757 |          |         |        |        |        |        |
| .anc        |          |         |        |        |        |        |
| hor}Exhibit |          |         |        |        |        |        |
| B.6\        |          |         |        |        |        |        |
| Subject of  |          |         |        |        |        |        |
| S           |          |         |        |        |        |        |
| upplemental |          |         |        |        |        |        |
| Services in |          |         |        |        |        |        |
| Grades      |          |         |        |        |        |        |
| K--12, by   |          |         |        |        |        |        |
| Provider    |          |         |        |        |        |        |
| Type, in    |          |         |        |        |        |        |
| Hi          |          |         |        |        |        |        |
| llsborough, |          |         |        |        |        |        |
| 2006--07    |          |         |        |        |        |        |
+-------------+----------+---------+--------+--------+--------+--------+
|             | All      |         | Hillsb |        | Other  |        |
|             | Part     |         | orough |        | Pro    |        |
|             | icipants |         | School |        | viders |        |
|             |          |         | Di     |        |        |        |
|             |          |         | strict |        |        |        |
+-------------+----------+---------+--------+--------+--------+--------+
|             | Number   | Percent | Number | P      | Number | P      |
|             | of       | of      | of     | ercent | of     | ercent |
|             | Students | Total   | St     | of     | St     | of     |
|             |          |         | udents | Total  | udents | Total  |
+-------------+----------+---------+--------+--------+--------+--------+
| Reading     | 4,857    | 79%     | 668    | 69%    | 4,204  | 80%    |
| only        |          |         |        |        |        |        |
+-------------+----------+---------+--------+--------+--------+--------+
| Mathematics | 1,283    | 21%     | 296    | 30%    | 999    | 19%    |
| only        |          |         |        |        |        |        |
+-------------+----------+---------+--------+--------+--------+--------+
| Reading and | 32       | 1%      | 7      | 1%     | 27     | 1%     |
| Math        |          |         |        |        |        |        |
+-------------+----------+---------+--------+--------+--------+--------+
| **Total**   | *        | *       | *      | **     | **5    | **     |
|             | *6,172** | *100%** | *971** | 100%** | ,230** | 100%** |
+-------------+----------+---------+--------+--------+--------+--------+
| **Exhibit   |          |         |        |        |        |        |
| reads:** In |          |         |        |        |        |        |
| 2006--07,   |          |         |        |        |        |        |
| 4,857       |          |         |        |        |        |        |
| students    |          |         |        |        |        |        |
| received    |          |         |        |        |        |        |
| s           |          |         |        |        |        |        |
| upplemental |          |         |        |        |        |        |
| educational |          |         |        |        |        |        |
| services    |          |         |        |        |        |        |
| only in     |          |         |        |        |        |        |
| reading in  |          |         |        |        |        |        |
| Hi          |          |         |        |        |        |        |
| llsborough. |          |         |        |        |        |        |
| This        |          |         |        |        |        |        |
| represented |          |         |        |        |        |        |
| 79 percent  |          |         |        |        |        |        |
| of all      |          |         |        |        |        |        |
| students    |          |         |        |        |        |        |
| receiving   |          |         |        |        |        |        |
| s           |          |         |        |        |        |        |
| upplemental |          |         |        |        |        |        |
| educational |          |         |        |        |        |        |
| services in |          |         |        |        |        |        |
| any subject |          |         |        |        |        |        |
| during that |          |         |        |        |        |        |
| year.       |          |         |        |        |        |        |
|             |          |         |        |        |        |        |
| Source:     |          |         |        |        |        |        |
| H           |          |         |        |        |        |        |
| illsborough |          |         |        |        |        |        |
| School      |          |         |        |        |        |        |
| District    |          |         |        |        |        |        |
| Adm         |          |         |        |        |        |        |
| inistrative |          |         |        |        |        |        |
| Data,       |          |         |        |        |        |        |
| 2002--03    |          |         |        |        |        |        |
| through     |          |         |        |        |        |        |
| 2006--07.   |          |         |        |        |        |        |
+-------------+----------+---------+--------+--------+--------+--------+
| []{#__Ref   |          |         |        |        |        |        |
| Heading___T |          |         |        |        |        |        |
| oc220485758 |          |         |        |        |        |        |
| .anchor}E   |          |         |        |        |        |        |
| xhibit B.7\ |          |         |        |        |        |        |
| Average     |          |         |        |        |        |        |
| Number of   |          |         |        |        |        |        |
| Hours of    |          |         |        |        |        |        |
| S           |          |         |        |        |        |        |
| upplemental |          |         |        |        |        |        |
| Educational |          |         |        |        |        |        |
| Services    |          |         |        |        |        |        |
| Received by |          |         |        |        |        |        |
| P           |          |         |        |        |        |        |
| articipants |          |         |        |        |        |        |
| in Grades   |          |         |        |        |        |        |
| K--12, by   |          |         |        |        |        |        |
| Provider,   |          |         |        |        |        |        |
| in          |          |         |        |        |        |        |
| Anchorage   |          |         |        |        |        |        |
| and         |          |         |        |        |        |        |
| H           |          |         |        |        |        |        |
| illsborough |          |         |        |        |        |        |
+-------------+----------+---------+--------+--------+--------+--------+
| ![]         |          |         |        |        |        |        |
| (media/imag |          |         |        |        |        |        |
| e7.png){wid |          |         |        |        |        |        |
| th="2.45in" |          |         |        |        |        |        |
| height="    |          |         |        |        |        |        |
| 3.297916666 |          |         |        |        |        |        |
| 6666666in"} |          |         |        |        |        |        |
+-------------+----------+---------+--------+--------+--------+--------+
| **Exhibit   |          |         |        |        |        |        |
| reads:** In |          |         |        |        |        |        |
| Anchorage,  |          |         |        |        |        |        |
| students    |          |         |        |        |        |        |
| who         |          |         |        |        |        |        |
| received    |          |         |        |        |        |        |
| s           |          |         |        |        |        |        |
| upplemental |          |         |        |        |        |        |
| educational |          |         |        |        |        |        |
| services    |          |         |        |        |        |        |
| from the    |          |         |        |        |        |        |
| district    |          |         |        |        |        |        |
| provider    |          |         |        |        |        |        |
| received,   |          |         |        |        |        |        |
| on average, |          |         |        |        |        |        |
| 31 hours of |          |         |        |        |        |        |
| tutoring.   |          |         |        |        |        |        |
|             |          |         |        |        |        |        |
| \*          |          |         |        |        |        |        |
| indicates   |          |         |        |        |        |        |
| differences |          |         |        |        |        |        |
| in the      |          |         |        |        |        |        |
| hours of    |          |         |        |        |        |        |
| services    |          |         |        |        |        |        |
| provided by |          |         |        |        |        |        |
| district    |          |         |        |        |        |        |
| and         |          |         |        |        |        |        |
| n           |          |         |        |        |        |        |
| on-district |          |         |        |        |        |        |
| providers   |          |         |        |        |        |        |
| that were   |          |         |        |        |        |        |
| st          |          |         |        |        |        |        |
| atistically |          |         |        |        |        |        |
| different   |          |         |        |        |        |        |
| at the .05  |          |         |        |        |        |        |
| level.      |          |         |        |        |        |        |
|             |          |         |        |        |        |        |
| Source:     |          |         |        |        |        |        |
| Anchorage   |          |         |        |        |        |        |
| School      |          |         |        |        |        |        |
| District    |          |         |        |        |        |        |
| Adm         |          |         |        |        |        |        |
| inistrative |          |         |        |        |        |        |
| Data,       |          |         |        |        |        |        |
| 2004--05    |          |         |        |        |        |        |
| through     |          |         |        |        |        |        |
| 2006--07,   |          |         |        |        |        |        |
| and         |          |         |        |        |        |        |
| H           |          |         |        |        |        |        |
| illsborough |          |         |        |        |        |        |
| School      |          |         |        |        |        |        |
| District    |          |         |        |        |        |        |
| Adm         |          |         |        |        |        |        |
| inistrative |          |         |        |        |        |        |
| Data,       |          |         |        |        |        |        |
| 2004--05    |          |         |        |        |        |        |
| through     |          |         |        |        |        |        |
| 2006--07.   |          |         |        |        |        |        |
+-------------+----------+---------+--------+--------+--------+--------+

# Appendix C: Standard Error Exhibits  {#appendix-c-standard-error-exhibits-1 .unnumbered}

  --------------------------------- ------------- ----------- ------------ ------------
  []{#__RefHeading___Toc220485759                                          
  .anchor}Exhibit C.1\                                                     
  Achievement Gains on the                                                 
  Benchmark and SBA of Students in                                         
  Grades 3--10\                                                            
  by Participation in Supplemental                                         
  Educational Services, in                                                 
  Anchorage,\                                                              
  2002--03 Through 2006--07                                                

                                    Mathematics               Reading      

                                    Gains         n           Gains        n

  Overall effect                    (0.075)       221         (0.052)      258

  First-year effect                 (0.076)       215         (0.052)      255

  Source: Anchorage School District                                        
  Administrative Data, 2002--03                                            
  through 2006--07.                                                        
  --------------------------------- ------------- ----------- ------------ ------------

  --------------------------------- ------------- ----------- ------------ ------------
  []{#__RefHeading___Toc220485760                                          
  .anchor}Exhibit C.2\                                                     
  Achievement Gains on the FCAT of                                         
  Students in Grades 1--12 by                                              
  Participation\                                                           
  in Supplemental Educational                                              
  Services, in Hillsborough,\                                              
  2002--03 Through 2006--07                                                

                                    Mathematics               Reading      

                                    Gains         n           Gains        n

  Overall effect                    (0.014)       1,966       (0.009)      7,288

  First-year effect                 (0.014)       1,683       (0.009)      6,092

  Multiple-year effect              (0.043)       283         (0.019)      1,196

  Source: Hillsborough School                                              
  District Administrative Data,                                            
  2002--03 through 2006--07.                                               
  --------------------------------- ------------- ----------- ------------ ------------

  --------------------------------- ------------- ----------- ----------- -----------
  []{#__RefHeading___Toc220485761                                         
  .anchor}Exhibit C.3\                                                    
  Achievement Gains on the                                                
  Benchmark and SBA of Students in                                        
  Grades 3--10\                                                           
  by Provider Type, in Anchorage,                                         
  2002--03 Through 2006--07                                               

  Provider Type                     Mathematics               Reading     

                                    Gains         n           Gains       n

  District provider                 (0.097)       67          (0.109)     83

  Non‑district providers            (0.087)       62          (0.075)     90

  Source: Anchorage School District                                       
  Administrative Data, 2002--03                                           
  through 2006--07.                                                       
  --------------------------------- ------------- ----------- ----------- -----------

  --------------------------------- --------------- --------------- ---------------
  []{#__RefHeading___Toc220485762                                   
  .anchor}Exhibit C.4\                                              
  Mathematics Achievement Gains on                                  
  the Benchmark and SBA of Students                                 
  in Grades 3--10 by Eligibility                                    
  for and Participation in                                          
  Supplemental Educational                                          
  Services, by Provider, in                                         
  Anchorage, 2002--03 Through                                       
  2006--07                                                          

                                    Number of       Average Hours   Mathematics
                                    Participants    per Student     

  **Average Across All Providers**                                  (0.097)

  **Providers Showing No                                            
  Significant Effects**                                             

  Private Provider A                46              31              (0.109)

  Anchorage School District         67              29              (0.097)

  Other providers                   16              28              (0.172)

  Total number of students          129                             

  Source: Anchorage School District                                 
  Administrative Data, 2002--03                                     
  through 2006--07.                                                 
  --------------------------------- --------------- --------------- ---------------

  --------------------------------- ---------------- ---------------- -------------
  []{#__RefHeading___Toc220485763                                     
  .anchor}Exhibit C.5\                                                
  Reading Achievement Gains on the                                    
  Benchmark and SBA of Students in                                    
  Grades 3--10 by Eligibility for                                     
  and Participation in Supplemental                                   
  Educational Services, by                                            
  Provider, in Anchorage, 2002--03                                    
  Through 2006--07                                                    

                                    Number of        Average Hours    Reading
                                    Participants     per Student      

  **Average Across All Providers**                                    (0.109)

  **Providers Showing No                                              
  Significant Effects**                                               

  Private Provider A                70               27               (0.101)

  Anchorage School District         83               33               (0.109)

  Other providers                   20               26               (0.178)

  Total number of students          173                               

  Source: Anchorage School District                                   
  Administrative Data, 2002--03                                       
  through 2006--07.                                                   
  --------------------------------- ---------------- ---------------- -------------

  --------------------------------- ------------- ----------- ----------- -----------
  []{#__RefHeading___Toc220485764                                         
  .anchor}Exhibit C.6\                                                    
  Achievement Gains on the FCAT of                                        
  Students in Grades 1--12\                                               
  by Provider Type, in                                                    
  Hillsborough, 2002--03 Through                                          
  2006--07                                                                

                                    Mathematics               Reading     

                                    Gains         n           Gains       n

  District provider                 (0.037)       234         (0.046)     447

  Non‑district providers            (0.025)       743         (0.019)     2,503

  District and non‑district         (0.119)       18          (0.177)     50
  providers ^a^                                                           

  Source: Hillsborough School                                             
  District Administrative Data,                                           
  2002--03 through 2006--07.                                              
  --------------------------------- ------------- ----------- ----------- -----------

  --------------------------------- -------------- ---------------- ------------------
  []{#__RefHeading___Toc220485765                                   
  .anchor}Exhibit C.7\                                              
  Mathematics Achievement Gains on                                  
  the FCAT of Students in Grades                                    
  1--12 by Eligibility for and                                      
  Participation in Supplemental                                     
  Educational Services, by                                          
  Provider, in Hillsborough,\                                       
  2002--03 Through 2006--07                                         

                                    Number of      Average Hours    Mathematics
                                    Participants   per Student      

  **Average Across All Providers**                                  (0.037)

  **Providers Showing Significant                                   
  Gains**                                                           

  Private Provider S                20             27               (0.143)

  **Providers Showing No                                            
  Significant Gains**                                               

  Private Provider W                22             9                (0.177)

  Private Provider Q                49             24               (0.087)

  Private Provider N                40             35               (0.044)

  Private Provider I                158            18               (0.054)

  Private Provider H                213            24               (0.028)

  Hillsborough School District      234            21               (0.037)

  Private Provider L                55             16               (0.076)

  Private Provider K                52             19               (0.076)

  Private Provider B                44             16               (0.111)

  Other providers                   128            24               (0.080)

  Total number of students          995                             

  Source: Hillsborough School                                       
  District Administrative Data,                                     
  2002--03 through 2006--07.                                        
  --------------------------------- -------------- ---------------- ------------------

  --------------------------------- -------------- ---------------- ---------------
  []{#__RefHeading___Toc220485766                                   
  .anchor}Exhibit C.8\                                              
  Reading Achievement Gains on the                                  
  FCAT of Students in Grades 1--12                                  
  by Eligibility for and                                            
  Participation in Supplemental                                     
  Educational Services, by                                          
  Provider, in Hillsborough,\                                       
  2002--03 Through 2006--07                                         

                                    Number of      Average Hours    Reading
                                    Participants   per Student      

  **Average Across All Providers**                                  (0.046)

  **Providers Showing No                                            
  Significant Gains**                                               

  Private Provider A                41             31               (0.185)

  Private Provider S                26             30               (0.218)

  Private Provider B                144            17               (0.080)

  Private Provider M                194            17               (0.056)

  Private Provider K                161            21               (0.082)

  Private Provider N                102            34               (0.069)

  Private Provider P                67             28               (0.068)

  Private Provider L                112            16               (0.058)

  Private Provider U                49             27               (0.028)

  Private Provider Y                24             17               (0.048)

  Hillsborough School District      447            22               (0.047)

  Private Provider J                476            25               (0.038)

  Private Provider R                48             13               (0.064)

  Private Provider I                331            19               (0.043)

  Private Provider H                543            24               (0.039)

  Private Provider Q                44             23               (0.094)

  Private Provider V                31             35               (0.087)

  Private Provider O                68             29               (0.123)

  Private Provider T                30             15               (0.145)

  Other providers                   62             22               (0.126)

  Total number of students          3,000                           

  Source: Hillsborough School                                       
  District Administrative Data,                                     
  2002--03 through 2006--07.                                        
  --------------------------------- -------------- ---------------- ---------------

+--------------------+-------+-------+-------+-------+-------+-------+
| []{#__RefHead      |       |       |       |       |       |       |
| ing___Toc220485767 |       |       |       |       |       |       |
| .a                 |       |       |       |       |       |       |
| nchor}Exhibit C.9\ |       |       |       |       |       |       |
| Achievement Gains  |       |       |       |       |       |       |
| on the Benchmark   |       |       |       |       |       |       |
| and SBA of         |       |       |       |       |       |       |
| Students in Grades |       |       |       |       |       |       |
| 3--10\             |       |       |       |       |       |       |
| by Provider Type   |       |       |       |       |       |       |
| and Hours of       |       |       |       |       |       |       |
| Services Received, |       |       |       |       |       |       |
| in Anchorage,\     |       |       |       |       |       |       |
| 2002--03 Through   |       |       |       |       |       |       |
| 2006--07           |       |       |       |       |       |       |
+====================+=======+=======+=======+=======+=======+=======+
| Hours of           | All   |       | Dis   |       | No    |       |
|                    | Prov  |       | trict |       | n‑Dis |       |
| Services Received  | iders |       | Pro   |       | trict |       |
|                    |       |       | vider |       | Prov  |       |
|                    |       |       |       |       | iders |       |
+--------------------+-------+-------+-------+-------+-------+-------+
|                    | Gains | n     | Gains | n     | Gains | n     |
+--------------------+-------+-------+-------+-------+-------+-------+
| **Mathematics**    |       |       |       |       |       |       |
+--------------------+-------+-------+-------+-------+-------+-------+
| Less than 20 hours | (0    | 70    | (0    | 31    | (0    | 20    |
|                    | .078) |       | .081) |       | .124) |       |
+--------------------+-------+-------+-------+-------+-------+-------+
| 20--39 hours       | (0    | 82    | (0    | 18    | (0    | 37    |
|                    | .140) |       | .326) |       | .091) |       |
+--------------------+-------+-------+-------+-------+-------+-------+
| 40 or more hours   | (0    | 69    | (0    | 18    | (0    | 5     |
|                    | .167) |       | .071) |       | .216) |       |
+--------------------+-------+-------+-------+-------+-------+-------+
| Total n            |       | 201   |       | 67    |       | 62    |
+--------------------+-------+-------+-------+-------+-------+-------+
| Reading            |       |       |       |       |       |       |
+--------------------+-------+-------+-------+-------+-------+-------+
| Less than 20 hours | (0    | 85    | (0    | 34    | (0    | 32    |
|                    | .075) |       | .080) |       | .068) |       |
+--------------------+-------+-------+-------+-------+-------+-------+
| 20--39 hours       | (0    | 96    | (0    | 20    | (0    | 53    |
|                    | .089) |       | .149) |       | .129) |       |
+--------------------+-------+-------+-------+-------+-------+-------+
| 40 or more hours   | (0    | 77    | (0    | 29    | (0    | 5     |
|                    | .109) |       | .240) |       | .247) |       |
+--------------------+-------+-------+-------+-------+-------+-------+
| Total n            |       | 258   |       | 83    |       | 90    |
+--------------------+-------+-------+-------+-------+-------+-------+
| Source: Anchorage  |       |       |       |       |       |       |
| School District    |       |       |       |       |       |       |
| Administrative     |       |       |       |       |       |       |
| Data, 2002--03     |       |       |       |       |       |       |
| through 2006--07.  |       |       |       |       |       |       |
+--------------------+-------+-------+-------+-------+-------+-------+

+------------------+-------+-------+-------+-------+-------+-------+
| []{#__RefHeadin  |       |       |       |       |       |       |
| g___Toc220485768 |       |       |       |       |       |       |
| .anch            |       |       |       |       |       |       |
| or}Exhibit C.10\ |       |       |       |       |       |       |
| Achievement      |       |       |       |       |       |       |
| Gains on the     |       |       |       |       |       |       |
| FCAT of Students |       |       |       |       |       |       |
| in Grades 1--12  |       |       |       |       |       |       |
| by Provider Type |       |       |       |       |       |       |
| and\             |       |       |       |       |       |       |
| Hours of         |       |       |       |       |       |       |
| Services         |       |       |       |       |       |       |
| Received, in     |       |       |       |       |       |       |
| Hillsborough,    |       |       |       |       |       |       |
| 2002--03 Through |       |       |       |       |       |       |
| 2006--07         |       |       |       |       |       |       |
+==================+=======+=======+=======+=======+=======+=======+
| Hours of         | All   |       | Dis   |       | No    |       |
|                  | Prov  |       | trict |       | n‑Dis |       |
| Services         | iders |       | Pro   |       | trict |       |
| Received         |       |       | vider |       | Prov  |       |
|                  |       |       |       |       | iders |       |
+------------------+-------+-------+-------+-------+-------+-------+
|                  | Gains | n     | Gains | n     | Gains | n     |
+------------------+-------+-------+-------+-------+-------+-------+
| **Mathematics**  |       |       |       |       |       |       |
+------------------+-------+-------+-------+-------+-------+-------+
| Less than 20     | (0    | 761   | (0    | 112   | (0    | 277   |
| hours            | .037) |       | .060) |       | .045) |       |
+------------------+-------+-------+-------+-------+-------+-------+
| 20--39 hours     | (0    | 1,096 | (0    | 104   | (0    | 445   |
|                  | .024) |       | .050) |       | .027) |       |
+------------------+-------+-------+-------+-------+-------+-------+
| 40 or more hours | (0    | 106   | (0    | 27    | (0    | 39    |
|                  | .093) |       | .071) |       | .146) |       |
+------------------+-------+-------+-------+-------+-------+-------+
| Total n          |       | 1,966 |       | 243   |       | 761   |
+------------------+-------+-------+-------+-------+-------+-------+
| Reading          |       |       |       |       |       |       |
+------------------+-------+-------+-------+-------+-------+-------+
| Less than 20     | (0    | 2,471 | (0    | 210   | (0    | 795   |
| hours            | .028) |       | .069) |       | .028) |       |
+------------------+-------+-------+-------+-------+-------+-------+
| 20--39 hours     | (0    | 4,445 | (0    | 211   | (0    | 1,673 |
|                  | .022) |       | .050) |       | .023) |       |
+------------------+-------+-------+-------+-------+-------+-------+
| 40 or more hours | (0    | 362   | (0    | 84    | (0    | 84    |
|                  | .069) |       | .184) |       | .046) |       |
+------------------+-------+-------+-------+-------+-------+-------+
| Total n          |       | 7,288 |       | 505   |       | 2,552 |
+------------------+-------+-------+-------+-------+-------+-------+
| Source:          |       |       |       |       |       |       |
| Hillsborough     |       |       |       |       |       |       |
| School District  |       |       |       |       |       |       |
| Administrative   |       |       |       |       |       |       |
| Data, 2002--03   |       |       |       |       |       |       |
| through          |       |       |       |       |       |       |
| 2006--07.        |       |       |       |       |       |       |
+------------------+-------+-------+-------+-------+-------+-------+

  --------------------------------- ------------- ----------- ------------ ------------
  []{#__RefHeading___Toc220485769                                          
  .anchor}Exhibit C.11\                                                    
  Achievement Gains on the                                                 
  Benchmark and SBA of Students in                                         
  Grades 3--10 by Subject, in                                              
  Anchorage, 2002--03 Through                                              
  2006--07                                                                 

  **Subject of Services**           Mathematics               Reading      

                                    Gains         n           Gains        n

  Mathematics and Reading           (0.069)       139         (0.065)      137

  Mathematics only                  (0.100)       49          (0.079)      50

  Reading only                      (0.095)       92          (0.093)      91

  Source: Anchorage School District                                        
  Administrative Data, 2002--03                                            
  through 2006--07.                                                        
  --------------------------------- ------------- ----------- ------------ ------------

  --------------------------------- ------------- ----------- ------------ ------------
  []{#__RefHeading___Toc220485770                                          
  .anchor}Exhibit C.12\                                                    
  Achievement Gains on the FCAT of                                         
  Students in Grades 1--12 by                                              
  Subject, in Hillsborough,                                                
  2002--03 Through 2006--07                                                

  **Subject of Services**           Mathematics               Reading      

                                    Gains         n           Gains        n

  Mathematics and Reading           (0.018)       608         (0.019)      608

  Mathematics only                  (0.017)       1,552       (0.016)      1,552

  Reading only                      (0.010)       6,810       (0.010)      6,810

  Source: Hillsborough School                                              
  District Administrative Data,                                            
  2002--03 through 2006--07.                                               
  --------------------------------- ------------- ----------- ------------ ------------

  --------------------------------- ------------- ------------ ------------ ------------
  []{#__RefHeading___Toc220485771                                           
  .anchor}Exhibit C.13\                                                     
  Achievement Gains on the                                                  
  Benchmark and SBA of Students in                                          
  Grades 3--10\                                                             
  for Supplemental Educational                                              
  Services Participants,\                                                   
  by Student Characteristics, in                                            
  Anchorage, 2002--03 Through                                               
  2006--07                                                                  

                                    Mathematics                Reading      

                                    Gains         n            Gains        n

  All Participants                  (0.075)       221          (0.052)      258

  African-American students         (0.163)       49           (0.087)      58

  Alaska Native students            (0.136)       42           (0.091)      54

  White students                    (0.190)       35           (0.114)      37

  Asian students                    (0.174)       37           (0.130)      44

  Hispanic students                 (0.131)       36           (0.203)      36

  Source: Anchorage School District                                         
  Administrative Data, 2002--03                                             
  through 2006--07.                                                         
  --------------------------------- ------------- ------------ ------------ ------------

  --------------------------------- ------------- ------------ ------------ ------------
  []{#__RefHeading___Toc220485772                                           
  .anchor}Exhibit C.14\                                                     
  Achievement Gains on the FCAT of                                          
  Students in Grades 1--12\                                                 
  for Supplemental Educational                                              
  Services Participants,\                                                   
  by Student Characteristics, in                                            
  Hillsborough, 2002--03 Through                                            
  2006--07                                                                  

                                    Mathematics                Reading      

                                    Gains         n            Gains        n

  All Participants                  (0.014)       1,966        (0.009)      7,288

  African-American students         (0.019)       879          (0.015)      2,448

  Hispanic students                 (0.021)       702          (0.013)      3,361

  White students                    (0.076)       270          (0.025)      1,090

  Special education students        (0.051)       370          (0.020)      1,901

  Limited English proficient        (0.021)       615          (0.013)      3,185

  Source: Hillsborough School                                               
  District Administrative Data,                                             
  2002--03 through 2006--07.                                                
  --------------------------------- ------------- ------------ ------------ ------------

[^1]: The Benchmark and SBA tests have similar scale scores and
    proficiency markers. As explained later, scale scores were
    standardized within grade.

[^2]: In Title I schools that have been identified for school
    improvement, corrective action, or restructuring, the district must
    offer students the option to transfer to another public school in
    the district that has not been so identified. All students in
    identified schools are eligible for this option, and the district
    must provide transportation for participating students, subject to
    available funding.

[^3]: Zimmer, R., Gill, B., Razquin, P., Booker, K., & Lockwood, J.R.,
    (2007). *State and local implementation of the No Child Left Behind
    Act: Volume I--Title I school choice, supplemental educational
    services, and student achievement*. Washington, D.C.: U.S.
    Department of Education.

[^4]:

[^5]: Private providers are not identified by their real names. However,
    each letter or pair of letters represents a distinct provider and is
    used consistently throughout the report.

[^6]: Adding the number of students served by each provider in 2006--07
    gives a total figure of 6,246, which is higher than the 6,172
    students participating in the program during that year. This
    indicates that some students had more than one provider.

[^7]: This is analogous to a national percentile rank where the mean is
    zero rather than 50 percent, and the population is the district
    rather than the nation.

[^8]: The interpretation of these interaction terms is complicated; they
    are designed to indicate whether a particular group of students is
    making significant gains. For example, to examine gains for Hispanic
    students, we use an interaction term to indicate the specific gain
    for Hispanic participants, while the comparison group is
    non‑Hispanic, non‑participants. Therefore the model estimates the
    gains for all groups, to show which groups made significant gains
    and which did not. The significance test measures whether the effect
    of participation plus the effect of being Hispanic is significantly
    different from zero.

[^9]: Note that this approach does not directly compare the gains of
    students served by district versus non‑district providers; it tests
    whether students served by the district make gains that are
    significantly greater than zero, and concurrently tests whether the
    gains made by students served by non‑district providers were
    significantly greater than zero. The relative size of the gains can
    be seen by comparing the coefficients for each provider type, which
    show the average gains in standard deviation units.

[^10]: Robust standard errors of all fixed-effects regressions are
    presented in Appendix E.

[^11]: In statistical terms, the power to reject the hypothesis of no
    effect (null hypothesis) depends on the number of observations
    available. The larger the treatment and control group, the higher
    the power is to reject the null hypothesis.

[^12]: Hillsborough only provided information on hours of supplemental
    educational services for 2005--06 and 2006--07. 398 students
    received these services in 2004--05 and it was necessary to impute
    the hours of supplemental educational services received for students
    during that school year. We did so by using the average hours of
    service observed by provider during 2005--06 and 2006--07. For those
    providers that did not serve students after 2004--05, we used the
    overall average number of hours across all providers.

[^13]: Note that students served by non-district providers experienced
    overall larger gains in mathematics than eligible non-participants.
    But when these students are separated into different groups by hours
    of tutoring, the sample sizes become too small in Anchorage to
    detect statistical significance.

[^14]: Guidance to districts that received waivers from the U.S.
    Department of Education included a checklist of documents and data
    that districts were encouraged to submit as evidence of meeting the
    waiver requirements. Note that the materials examined were for the
    2006--07 school year.

[^15]: See Flesch, R. (1948); A new readability yardstick, *Journal of
    Applied Psychology*, Vol. 32, pp. 221-233; and Kincaid, J. P.;
    Fishburne, R. P., Jr.; Rogers, R. L.; and Chissom, B. S. (1975);
    *Derivation of new readability formulas (Automated Readability
    Index, Fog Count and Flesch Reading Ease Formula) for Navy enlisted
    personnel,* Research Branch Report 8-75, Millington, TN: Naval
    Technical Training, U. S. Naval Air Station, Memphis, TN