![](media/image1.png){width="1.9902777777777778in"
height="0.625in"}Contract No.: ED-01-CO-0039/0008

MPR Reference No.: 6043-050

> **Impact Evaluation of Charter School Strategies**
>
> **Supporting Statement for Request for OMB Approval of Data
> Collection---Student, Parent, Principal, and Authorizer/State Surveys
> and Case Study Site Visit Forms**
>
> ***December 12, 2005***
>
> ***Philip Gleason***
>
> ***Rita Stapulonis***
>
> ***Martha Bleeker***
>
> ***Emily Dwoyer***
>
> ***Paul Hill***
>
> ***Abigail Ocobock***
>
> ***Rob Olsen***
>
> ***Christina Tuttle***

+---------------------------------------+------------------------------+
| Submitted to:                         | Submitted by:                |
|                                       |                              |
| National Center for Education         | > Mathematica Policy         |
| Evaluation and Regional Assistance    | > Research, Inc.             |
|                                       | >                            |
| > Institute of Education Sciences     | > P.O. Box 2393              |
| >                                     | >                            |
| > U.S. Department of Education        | > Princeton, NJ 08543-2393   |
| >                                     | >                            |
| > 555 New Jersey Ave., NW             | > Telephone: (609) 799-3535  |
| >                                     | >                            |
| > Washington, DC 20208                | > Facsimile: (609) 799-0005  |
|                                       |                              |
| Project Officer:                      | Project Director:            |
|                                       |                              |
| > Marsha Silverberg                   | > Philip Gleason             |
+---------------------------------------+------------------------------+

## CONTENTS

**Chapter Page**

INTRODUCTION 1

OVERVIEW 3

STUDY APPROACH 5

A. JUSTIFICATION 8

1\. Circumstances That Make Data Collection Necessary 8

2\. How, by Whom, and for What Purpose the Information Is To Be Used 9

a\. Intake and Consent Forms 10

b\. Administrative Records 10

c\. Parent Interviews 10

d\. Student Interviews 11

e\. Principal Survey---Study Schools 11

f\. Principal Survey---Non-Study Charter Middle Schools 11

g\. Authorizer and State Surveys 12

h\. Site Visits 12

3\. Use of Information Technology 13

4\. Efforts to Identify and Avoid Duplication 13

5\. Impacts on Small Businesses and Other Small Entities 14

6\. Consequences to Federal Programs or Policies if Data Collection is
Not Conducted 14

7\. Special Circumstances 15

8\. Solicitation of Public Comments and Consultation with People Outside
the Agency 15

a\. Federal Register Announcement 15

b\. Consultations Outside the Agency 15

9\. Respondent Payments 16

10\. Confidentiality Assurances 17

11\. Sensitive Questions 18

12\. Total Annual Cost Burden to Respondents 19

**CONTENTS** *(continued)*

**Chapter Page**

13\. Estimate of the Total Annual Cost Burden to Respondents or Record
Keepers 20

14\. Annualized Cost to the Federal Government, Including a Description
of the Method Used to Estimate Costs 20

15\. Program Changes or Adjustments 21

16\. Tabulation, Analysis and Publication of Results 21

a\. Impacts of Charter Schools 21

b\. Differences Between Charter Schools and Conventional Schools 23

c\. Influence of Policy Environment 26

d\. Publication Plans and Time Schedule 28

17\. Approval Not to Display the Expiration Date for OMB Approval 28

18\. Exception to the Certification Statement 28

B. COLLECTION OF INFORMATION EMPLOYING STATISTICAL

METHODS 31

1\. Respondent Universe and Sampling Procedures 31

2\. Statistical Methods for Sample Selection and Degree of Accuracy
Needed 35

a\. Stratification and Sample Selection 35

b\. Estimation Procedures 35

c\. Degree of Accuracy Needed 35

d\. Unusual Problems Requiring Specialized Sampling Plans 36

e\. Use of Periodic Data Collection Cycles to Reduce Burden 36

3\. Methods to Maximize Response Rates and Deal with Nonresponse 37

a\. Parent and Student Surveys 37

b\. Principal, Authorizer, and State Survey 38

4\. Test of Procedures and Methods to be Undertaken 40

5\. Individuals Involved 42

## exhibits

**Exhibit**

EXHIBIT A PARENT SURVEY INSTRUMENT

EXHIBIT B STUDENT SURVEY INSTRUMENT

EXHIBIT C PRINCIPAL SURVEY INSTRUMENT

EXHIBIT D AUTHORIZER SURVEY INSTRUMENT

EXHIBIT E STATE SURVEY INSTRUMENT

EXHIBIT F CASE STUDY SITE VISIT INTERVIEW GUIDE TOPICS

EXHIBIT G PARENT SURVEY ADVANCE LETTER

EXHIBIT H PRINCIPAL SURVEY ADVANCE LETTERS

# INTRODUCTION

This document presents the Supporting Statement for an Impact Evaluation
of Charter School Strategies, the first large-scale experimental study
of charter schools ever conducted. In particular, we are requesting OMB
approval for a set of data collection activities that includes surveys
of students, their parents, principals, and charter school authorizers.
This submission represents the second in a two-stage forms clearance
process. The first stage resulted in a January 2005 approval (OMB
1850-0799) for the evaluation design, sampling and analysis plans, and
for initial data collection activities necessary for random assignment
(consent form, baseline form, and school records collection). The
current submission requests approval for three modest enhancements to
the design and for the specific instruments that were only briefly
described in the earlier package. These changes will be highlighted on
the subsequent pages, although a broader discussion of the
already-approved components is provided for completeness.

Since their inception in 1992, charter schools have become an
increasingly important part of education reform. Despite growth in the
number of charter schools nationally, few rigorous studies have examined
their impacts on students who attend them. The existing studies have
suffered from methodological problems related to the selection of
suitable comparison groups. In keeping with the *No Child Left Behind
Act* of 2001 (P.L. No. 107-110), which requires that education decision
makers base instructional practices and programs on scientifically based
research, the Impact Evaluation of Charter School Strategies is designed
to use rigorous methods to estimate the impacts of charter schools. In
particular, the evaluation will use an experimental design whereby
random assignment is used to select a treatment group of students
admitted to charter schools and a control group of students not admitted
to charter schools. Random assignment, when properly implemented,
ensures that there are no systematic differences between the treatment
and control groups other than their admission to the charter school.
Thus, systematic differences between the two groups in subsequent
outcomes can be attributed to the effects of the treatment---in this
case, to the effects of being admitted to the charter school. The study
will address three key research questions:

1.  What are the impacts of charter schools on student achievement,
    other indicators of school performance, and parent satisfaction?

2.  In what ways are charter schools and conventional public schools
    different, and what role do these differences play in determining
    student outcomes?

3.  To what extent does the autonomy or policy environment under which
    charter schools operate influence their effectiveness?

This submission provides an overview of all aspects of the planned data
collection. It also provides details on forms used in the student,
parent, and principal surveys, as well as for the site visits. In
addition, the submission includes estimates of respondent burden that
are associated with the data collection efforts that are described here
in detail.

# 

# OVERVIEW

The Impact Evaluation of Charter School Strategies is being conducted by
Mathematica Policy Research, Inc. (MPR) and its subcontractor, Optimal
Solutions Group (OSG). This section provides an overview of the study
and the data to be collected for this study. *Much of this has already
been approved as part of the earlier submission; any changes are
highlighted below.*

The charter school concept is a response to some of the challenges faced
by the existing public education system. In general, charter school
legislation was passed in states as a result of a public demand for more
educational choice and higher-quality schools. Quality in conventional
public schools was seen as being adversely affected by a large
administrative bureaucracy and parents' limited ability to "vote with
their feet" by moving their children across public schools. Charter
schools were seen as a means both to bring greater choice into the
public school system and to escape the constraints imposed by large
public school bureaucracies and regulation.

Charter schools are independent public schools that are established on
the basis of charters granted by local or state educational
organizations (authorizers). Under the charters, authorizers monitor
schools' quality and integrity, but allow them to operate free from some
regulations and policy constraints that apply to traditional public
schools. Charter schools are schools of choice, meaning that any student
within a district or an even broader area can apply for admission.

Charter schools have the potential to create a wide range of effects on
the education system. Most directly, they may influence the educational
outcomes among the students who attend them, as well as the attitudes of
their parents toward their children's schools. Measuring these impacts
is the primary goal of the evaluation. However, charter schools may also
have a broader set of effects. To the extent that they bring innovation
and introduce competition into the conventional public school system,
charter schools may also influence outcomes among students who attend
conventional public schools. While these broader impacts might be
important, the evaluation is not designed to measure them or assess the
success of charter schools on the basis of their role in changing the
public educational system as a whole.[^1]

To better understand how charter schools influence outcomes among
students, we developed the conceptual framework in Figure 1, which shows
a theory of action by which charter schools may influence outcomes.
First, a state's charter school legislation allows charter schools to
form and to operate under greater autonomy than conventional public
schools. This greater autonomy

![](media/image2.wmf)

could take the form of greater freedom to hire/fire staff, set the
budget in accordance with local priorities, determine the curriculum and
instructional approach, and set other school policies.

The charter schools that have formed use greater autonomy to create a
school setting that is different from nearby district schools. The
charter school could use its autonomy to develop a better or more
coherent curriculum and/or instructional approach than is present in
surrounding public schools. Or the entire structure and organization of
the school could be different than the surrounding schools. The approach
taken by the charter school might be designed to be better for all
students, or it might simply be better suited to a group of students not
well served by conventional public schools in the area. Another
mechanism through which charter schools may influence student outcomes
results from the fact that parents have chosen to enroll their children
in these schools, which provides charter schools with an opportunity to
build a more coherent relationship with the families of their students.
If charter schools respond to this opportunity in creative ways, they
may promote parents' involvement in their children's education.

These changes made by charter schools---more coherent instructional
approaches, approaches better matched to students' learning styles, more
coherent school-family relationships---may lead to improved student
outcomes. Ultimately, we are interested in impacts on student
achievement, as measured through test scores. However, charter schools
may influence a range of intermediate outcomes (such as behavior,
attendance, or parent involvement) that, in turn, positively influence
achievement levels.

## STUDY APPROACH

As described in our previously approved OMB submission, the Impact
Evaluation of Charter School Strategies involves the completion of four
basic tasks: (1) selecting and recruiting a set of 50 charter middle
schools for the study; (2) randomly assigning students who apply to
these charter schools into a treatment group admitted to the school and
a control group who will instead attend some other area school; (3)
collecting a variety of forms of data to measure charter school
characteristics and to track student characteristics and outcomes such
as student achievement and parental satisfaction; and (4) analyzing the
data and reporting the results of our analysis.

**Selecting Charter Schools.** The evaluation focuses on the effects of
charter middle schools. Middle schools were chosen primarily because of
the availability of test score data from school records for this group,
reducing evaluation costs and burden on sample members. We are selecting
schools for the study based on their ability to support a random
assignment design (for example, they must be oversubscribed) and their
location in states that provide variation in key aspects of charter
school policy. The schools selected for the evaluation are also required
to have at least two years of experience as charter schools, so as to
minimize the chances that study schools will still be under development
and thus undergoing a substantial amount of change during the evaluation
period.

One difference between the study design described in our original
submission and the design described in this submission involves the time
period over which charter schools are being selected and recruited into
the study. In our original submission, we described selecting 50 charter
middle schools in a single cohort during the 2004-2005 school year.
During our first year of recruiting, however, we learned that some
schools with early admissions lotteries needed to be contacted very
early in the school year to consider participating in the study. We also
discovered that the number of schools that were sufficiently
oversubscribed at the time of their admissions lottery was smaller than
anticipated. Thus, we decided to spread the recruiting effort over two
years. The first cohort of about 25 schools was selected during the
2004-2005 school year (which will be their baseline year), and a second
cohort of another 25 schools will be selected during the 2005-2006
school year. Additional details on the selection of charter schools for
the study, including the decision to focus on middle schools, is
provided in Section B.1.

**Randomly Assigning Students.** At each charter school selected for the
study, a sample of about 60 students who apply to the charter school is
randomly assigned, primarily through school-run lotteries. Although the
number of openings will vary across schools, an average of 30 applicants
have been or will be randomly assigned to the treatment group and
offered admission to the charter school, and 30 randomly assigned to the
control group and not offered admission. The overall sample size will be
about 3,000. The statistical power of this design is discussed in
Section B.2.

**Collecting Data.** For the evaluation, we will collect data from
various sources. We are initially collecting data from a baseline intake
form to be completed by parents applying for admission to the charter
school (and thus entering the random assignment process). We will
collect administrative records data on outcomes such as test scores for
the baseline year (from the school attended when the student applies to
the charter school) and up to two follow-up years (from the charter
school or any other school attended during this period). During the
first follow-up year (spring 2006 and spring 2007 for the first and
second cohorts of schools, respectively), we will conduct surveys of
students, parents, and principals of the charter schools and other
schools in the study. During the spring of 2007, we will conduct surveys
of charter school authorizers, the principals of charter schools not
participating in the study, and state officials. We will also conduct
case study site visits to collect qualitative data at selected charter
schools as well as some of the schools attended by control group
students in those sites. An overview of the data collection plan for the
study, along with key pieces of information to be obtained from each
data source, is shown in Table 1.

**Analyzing Data.** The analysis will focus on estimating the impacts of
charter school attendance on student achievement, parent satisfaction,
and selected other outcomes. We will summarize the findings of the
evaluation in a final report covering impacts through the first
follow-up year for both cohorts and through the second follow-up year
for the first cohort.

TABLE 1

PROPOSED DATA COLLECTION PLAN

+---------+---------------+-----------------------+--------------------+
| Res     | Mode          | Timeline              | Key Data           |
| pondent |               |                       |                    |
+---------+---------------+-----------------------+--------------------+
| Parents | Baseline      | C1: Spring/Summer     | Demographic        |
|         | intake form   | 2005                  | information        |
|         |               |                       |                    |
|         |               | C2: Spring/Summer     | Reasons for        |
|         |               | 2006                  | applying           |
+---------+---------------+-----------------------+--------------------+
| S       | A             | C1 (baseline):        | Test scores        |
| chools/ | dministrative | Summer/Fall 2005      |                    |
| Di      | records data  |                       | Student attendance |
| stricts |               | C2 (baseline):        |                    |
|         |               | Summer/Fall 2006      | Other student      |
|         |               |                       | measures           |
|         |               | C1 (FU-1):            |                    |
|         |               | Summer/Fall 2006      |                    |
|         |               |                       |                    |
|         |               | C2 (FU-1):            |                    |
|         |               | Summer/Fall 2007      |                    |
|         |               |                       |                    |
|         |               | C1 (FU-2):            |                    |
|         |               | Summer/Fall 2007^a^   |                    |
+---------+---------------+-----------------------+--------------------+
| S       | Telephone     | C1: Spring 2006       | Student            |
| tudents | survey        |                       | satisfaction       |
|         |               | C2: Spring 2007       |                    |
|         |               |                       | School climate     |
|         |               |                       |                    |
|         |               |                       | Student attendance |
|         |               |                       | and behavior       |
+---------+---------------+-----------------------+--------------------+
| Parents | Telephone     | C1: Spring 2006       | Parental           |
|         | survey        |                       | involvement        |
|         |               | C2: Spring 2007       |                    |
|         |               |                       | Parental           |
|         |               |                       | satisfaction       |
+---------+---------------+-----------------------+--------------------+
| Pri     | Mail survey   | C1 (study schools):   | Curriculum &       |
| ncipals |               | Spring 2006           | instructional      |
|         |               |                       | approach           |
|         |               | C2 (study schools):   |                    |
|         |               | Spring 2007           | Budgetary &        |
|         |               |                       | operational        |
|         |               | Non-study schools:    | autonomy           |
|         |               | Spring 2007           |                    |
+---------+---------------+-----------------------+--------------------+
| Auth    | Mail survey   | C1 & C2: Spring 2007  | Charter school     |
| orizers |               |                       | autonomy and       |
|         |               |                       | oversight/         |
|         |               |                       | accountability     |
+---------+---------------+-----------------------+--------------------+
| State   | Mail survey   | All study states:     | Charter school     |
| Of      |               | Spring 2007           | autonomy and       |
| ficials |               |                       | oversight/         |
|         |               |                       | accountability     |
+---------+---------------+-----------------------+--------------------+
| School  | Se            | C1 & C2: January --   | Characteristics of |
| Staff   | mi-structured | May 2007              | school funding,    |
|         | interviews    |                       | budgetary and      |
|         | and classroom |                       | operational        |
|         | observations  |                       | autonomy,          |
|         | obtained      |                       | curriculum, and    |
|         |               |                       | classroom          |
|         | during site   |                       | environment        |
|         | visits        |                       |                    |
+---------+---------------+-----------------------+--------------------+

^a^There is no second follow-up year administrative records data
collection for cohort 2.

FU = Follow-up

C1 = Cohort 1

C2 = Cohort 2

## A. JUSTIFICATION

### 1. Circumstances That Make Data Collection Necessary

Although charter schools are barely more than a decade old, they are
becoming increasingly prevalent and the charter school movement is
recognized as an important part of education reform. Authorized through
legislation in 40 states, the District of Columbia, and Puerto Rico,
charter schools are not governed by many state and district regulations
that govern traditional public schools, but are held accountable for the
quality of student outcomes. There are currently nearly 3,400 charter
schools in the United States that serve close to a million students
(Center for Education Reform 2005). Because charter school legislation
is passed at the state level, there is substantial variation across
states (and districts) in the prevalence of charter schools, their
funding, and the extent to which they are exempt from state and district
regulations and policies.

The federal government has played an increasingly important role in the
charter school movement during its brief history. In 1994, the
government created the Public Charter School Program (PCSP) as part of
the reauthorization of the Elementary and Secondary Education Act. The
PCSP, for which funding has grown rapidly since its inception, is
intended to promote charter school growth and development through the
provision of start-up funding and technical assistance. The federal
budget for fiscal year 2002 awarded \$200 million in grants under this
program.

There is substantial interest in understanding the effects of charter
schools. There is also an ongoing debate over whether they are a good
use of public resources and an effective alternative to traditional
public schools. Charter school proponents believe that these schools
contribute to the improvement of the entire school system by promoting
choice and competition. They also argue that large public school
bureaucracies and regulations constrain traditional public schools from
effectively serving students, so the greater autonomy allowed charter
schools will lead to greater effectiveness. Charter school opponents,
however, have expressed concerns that a lack of control and oversight
could lead to the formation of charter schools with ill-conceived
educational plans or little educational management expertise. Some have
argued that charter schools could fail to serve certain groups of
students, lead to increased segregation, or divert resources from
traditional public schools and districts.

Despite the growth in charter schools and great interest in their
effects, relatively little is known about how they affect the students
who attend them. While studies have documented charter school growth and
described aspects of their operations, few studies have rigorously
estimated the impacts of charter schools on student outcomes. The
studies that have estimated charter school impacts have had mixed
results about charter school effects. Further, these existing studies
have used non-experimental methods, whereby the outcomes of charter
school students have been compared with those of students who have
chosen to remain in traditional public schools. The problem with this
method of estimating charter school impacts is that these two groups of
students may differ systematically in ways other than their charter
school attendance. For example, the parents of charter school students
may be more involved in their children's education than parents who have
not chosen to send their children to charter schools. Thus, it is
difficult to determine whether observed differences between the two
groups in outcomes are due to their charter school attendance (or
non-attendance) or to these differences in other student
characteristics.

In addition to this methodological limitation, existing studies of
charter schools have generally not had as an objective determining the
conditions under which charter schools are most likely to be effective.
For example, little is known about the relationship between the policy
environment under which charter schools operate and their impacts on
students. It is important to understand the relationship between charter
school policies and their impacts for several reasons. First, charter
school laws are continuously evolving in the states. States are not only
deciding whether and how to allow charter schools to form, but are also
legislating issues such as the degree of autonomy charter schools should
operate under and systems for funding charter schools. Even states that
have established charter school laws will have opportunities to refine
these laws as new information becomes available. So that the laws may be
structured in a way to promote student achievement, it is important that
we understand the relationship between the policy environment faced by
charter schools and their effectiveness. Second, all schools, including
charter schools must meet Adequate Yearly Progress (AYP) requirements
under *No Child Left Behind*. Thus, district officials and charter
school authorizers, who also set policies that influence charter
schools, would benefit from understanding how these policies may be
related to charter schools' ability to promote student achievement
levels.

To estimate charter school impacts, the study will employ an
experimental design. At each charter school selected for the evaluation,
applicants are randomly assigned into two groups: (1) a treatment group
that is admitted to the charter school, and (2) a control group that is
not admitted to the charter school but is free to attend other area
schools. Systematic differences in subsequent outcomes will then be
attributable to the fact that the treatment group was admitted to the
charter school while the control group was not.

As described in more detail below, the charter schools included in the
study are being selected so that they represent a broad range of charter
school policy environments. In contrast to previous studies, this will
allow us to examine charter school impacts under a broad range of
different policy conditions and examine the relationship between these
conditions and charter school impacts.

### 2. How, by Whom, and for What Purpose the Information Is To Be Used

Information on charter schools and related topics will be collected by
Mathematica Policy Research, Inc. (MPR) and its subcontractor, Optimal
Solutions Group, LLC. (OSG), under contract number ED-01-CO-0039/0008
with the U.S. Department of Education (ED). The data will also be
analyzed by MPR and OSG. The specific data to be collected (as
summarized in Table 1) is being obtained from baseline intake and
consent forms, school or district administrative records data, parent
interviews, student interviews, surveys of principals, authorizers, and
state officials, and case study site visits. In the earlier submission,
we provided a discussion of all of these data collection activities,
with the exception of the two enhancements (sections f. and g. below): a
survey of all non-study charter middle school principals in the 12 study
states and a survey of the authorizers and state accountability
officials who oversee the study schools.

#### a. Intake and Consent Forms

When parents apply for their children to be admitted to charter schools,
we supplement the schools' application materials with two additional
forms for parents to complete during the baseline period---a baseline
intake form and a consent form. The baseline intake form is being used
to obtain demographic and socioeconomic information from parents at the
time of application. The form is also being used to collect information
on parents' reasons for applying to the charter school. At the same
time, we provide parents with an advance letter with information about
the study and a consent form to obtain their permission for their
children to participate in the study. We are using active consent
procedures and obtaining a signed consent form from all parents whose
children enter the sample. The baseline intake form and consent form
have been approved as part of our previous submission (OMB Number
1850-0799).

#### b. Administrative Records

School or district administrative records will be used to measure key
outcome variables in the analysis. Most importantly, we will measure
student achievement using test scores from school records. We will also
measure several intermediate outcomes with these records, including
student attendance and disciplinary incidents. The school records to be
collected will cover the study's baseline year and two follow-up years.
Baseline outcomes will be used in conjunction with follow-up year
outcomes to measure student outcomes in terms of changes over time.

To collect administrative records data, we are using a form that staff
in charter schools and the comparison schools attended by control
students will complete for each sample member enrolled in the school. As
discussed in Section B.1, schools or districts may find it more
convenient to provide information on all district students rather than
separately providing the information only for the students in our
sample. In this case, we will encourage them to provide information on
all students, and we will extract the records of evaluation sample
members ourselves. The administrative records form was approved as part
of our previous submission.

#### c. Parent Interviews

At the end of the first follow-up year---in spring 2006 for the first
cohort of students and spring 2007 for the second cohort---we will
conduct short telephone interviews with the parents of sample members.
These fifteen minute interviews will yield information on parents'
satisfaction with their children's education, their assessment of the
climate of their children's schools, their assessment of their
children's behavior, and a description of their involvement in their
children's education in general and in the schools their children attend
in particular. The parent interview data will be used to develop
intermediate outcome measures. A copy of the parent survey instrument is
shown in Exhibit A.

#### d. Student Interviews

At the same time we conduct the parent interviews, we will also conduct
short telephone interviews with all students in the sample. We will
coordinate the student and parent surveys. At the conclusion of the
parent survey, we will ask the parent whether the student is available
at that time to complete a fifteen minute interview. If they are not
available, we will attempt to schedule a time at which we can complete
the student interview. When we reach the student, we will begin by
seeking their assent to complete the survey. If we get their assent, we
will then collect information from them about their satisfaction with
their educational experiences as well as their assessment of various
aspects of school climate. We will also collect students' reports of
their behavior, both within and outside of school, and their perspective
on the involvement of their parents in their education. This information
will be used to develop intermediate outcome measures. A copy of the
student survey instrument is shown in Exhibit B.

#### e. Principal Survey---Study Schools

Concurrent with the student and parent surveys in spring 2006 and spring
2007, we will conduct a mail survey with telephone follow-up of the
principals of all charter schools in the study---in other words, the
principals of all schools attended by sample members. This survey will
yield information on the characteristics of these schools, with a
particular emphasis on school autonomy, funding, and the
curriculum/instructional approach used in the school. This information
will be used to characterize charter schools relative to comparison
schools for purposes of examining the relationship between charter
school characteristics and charter school impacts. Principals of the
charter schools and the other study schools will receive the very
similar surveys. They will have a set of common questions, and then
there will be additional questions asked of the charter school
principals. Thus, the charter school principal survey will be slightly
longer than that given to the principals of other study schools. On
average, we expect the principal survey to take 30 minutes for
respondents to complete. Copies of the principal survey instruments for
charter schools and for other study schools are shown in Exhibit C.

#### f. Principal Survey---Non-Study Charter Middle Schools

In spring 2007, we will administer the principal survey to an additional
group of principals---those at all charter middle schools not in the
study that are located in the states covered by the study. The
instrument for this survey will be identical to the survey administered
to the principals of charter schools in the study. The data collected
from the non-study charter school principals will allow us to compare
and contrast the charter schools in the study---and for which we are
estimating impacts---with a broader group of charter schools. Since the
charter schools in the study are being selected purposively on the basis
of their suitability for a random assignment study, it is vitally
important to describe how these schools are similar to and different
from charter middle schools in the United States more generally. This
data collection component was added after the previous submission, and
it reflects actual recruiting experience during the first year, when it
became apparent that there are not as many oversubscribed charter
schools and prior studies have suggested.

#### g. Authorizer and State Surveys

Another component recently added to the evaluation is a survey of
officials who oversee charter schools and monitor their performance,
particularly with regard to No Child Left Behind (NCLB) accountability.
The surveys of charter school authorizers and state officials will be
similar in their format and content. Each will be a mail survey with
telephone follow-up conducted in spring 2007. Respondents to the charter
school authorizer survey will be the key staff member responsible for
oversight of the charter schools in the study at each of these schools'
authorizers. Since some of the charter schools in the study share the
same authorizer, we expect there to be about 40 respondents to the
authorizer survey. The state survey respondents will be the staff member
at the department of education in each of the study states who is
responsible for assessment and accountability issues and charter schools
(and possibly other public schools) in the state. In both the authorizer
and state survey, we will collect data that will help us to characterize
the level of autonomy of each charter school in the study, as well as
the degree of performance and accountability oversight from the
authorizer or state experienced by the charter school. The question of
how NCLB accountability, and the stringency with which it is
implemented, is associated with charter school impacts is of increasing
importance to policy makers and was raised recently by the General
Accounting Office as an issue that ED must address. The authorizer
survey is shown in Exhibit D, while Exhibit E contains the state survey.

#### h. Site Visits

Case study site visits will be conducted in spring 2007 to obtain
in-depth information on the policy environment in which charter schools
are operating and on the characteristics of charter schools relative to
the comparison schools attended by control group students. This
information will be used, in part, to help explain the presence or
absence of impacts.

In particular, these visits to selected charter schools and selected
schools attended by control group students will yield information on:

-   The extent to which charter schools have autonomy in hiring/firing
    staff, school spending, determining the curriculum and instructional
    approach used in the school, and setting other school policies
    relative to comparison schools;

-   The level of funding received by the charter school relative to
    comparison schools;

-   The curriculum and instructional approach used by the charter school
    relative to comparison schools;

-   The classroom environment in the charter school relative to
    comparison schools;

-   The relationship between the charter school and their authorizers,
    with a particular focus on the level of authorizer oversight of the
    charter school; and

```{=html}
<!-- -->
```
-   Other characteristics of the charter school relative to comparison
    schools.

While at each charter school, we will observe classroom activities and
conduct semi-structured interviews with selected teachers and school
administrators. During the site visits, we will also go to selected
schools attended by control students denied admission to the charter
schools. At these "comparison schools," we will collect similar
information as at the charter schools. Copies of the interview guides
containing the topics to be covered in the semi-structured interviews
during the site visits are contained in Exhibit F.

### 3. Use of Information Technology

The data collection plan reflects sensitivity to issues of efficiency,
accuracy, and respondent burden. Where feasible, information will be
gathered from existing data sources rather than imposing additional
burden by collecting primary data. Minimizing evaluation costs and
reducing respondent burden were key considerations in the decision to
focus the study on middle schools, since at this level student
achievement data can be collected via existing administrative records as
opposed to administering standardized tests to sample members.

We will use computer assisted telephone interview (CATI) techniques to
conduct the student and parent surveys in the shortest, most efficient
manner possible. The decision was made to use a mail survey to collect
data from principals, authorizers, and state officials so as to give
them greater flexibility in gathering the necessary information and
completing the surveys. In providing principals with information about
the survey, however, we will give them the option of using a toll-free
number and completing the survey by telephone.

### 4. Efforts to Identify and Avoid Duplication

The purpose of the data collection is to estimate the impacts of
attending a charter school on student achievement and parental
satisfaction. While previous studies have estimated the effects of
charter schools, they have done so using non-experimental designs. There
have been no previous studies that systematically and rigorously
evaluate the impact of charter schools using randomized field
trials---the preferred method for answering causal questions about the
effectiveness of programs.

Conducting randomized trials to evaluate the impacts of charter schools
requires randomly assigning students into a treatment group accepted
into the charter school and a control group not accepted. To the extent
possible, we will use existing data for the study rather than
duplicating data collection efforts. For example, if schools'
application forms already collect information that we are requesting on
the baseline intake form and doing so in a similar manner, we will adapt
the intake form to avoid this duplication. Further, a major part of our
data collection effort will be the collection of school records data.
This data source will be used in place of administering standardized
tests ourselves.

The information to be collected as part of student, parent, principal,
authorizer, and state surveys, however, will not be available elsewhere.
The information to be collected will represent the attitudes of these
individuals, as well as information they have that is not currently
available elsewhere. In addition, we will collect information on student
behavior from students and their parents. To avoid duplication within
these surveys, we have designed them (along with the baseline intake
form) such that time invariant data (such as a student's date of birth)
is collected on only one of the surveys.

On the other hand, to collect information on a key charter school
characteristic such as autonomy, we will collect data related to
autonomy from principals, charter school authorizers, and state
officials. There are two reasons for this. First, these three informants
will be able to report on different aspects of charter school autonomy.
For example, while charter school authorizers will provide information
related to requirements they may or may not impose on charter schools,
state officials will provide information on the extent to which charter
schools must comply with various state laws and regulations. Second, it
is important that we collect information on autonomy from respondents
who may have very different perspectives. The extent to which charter
school principals believe that their freedom to set their own policies
along specific dimensions is restricted by their authorizer may be very
different from the extent to which the authorizers believe that the
freedom of charter schools is restricted along these same dimensions. We
would like to be able to measure any conflict in the level of charter
school autonomy reported by different informants.

### 5. Impacts on Small Businesses and Other Small Entities

The primary entities for this study are schools and the districts to
which they belong, along with the children who attend them and their
parents. Burden is reduced for all respondents by requesting only the
minimum information required to meet the study objectives. The burden on
schools and districts has been minimized through the careful
specification of information needs, restricting questions to generally
available information, and designing the data collection
strategy---particularly the survey methods---to minimize burden on
respondents. All data collection will be coordinated by the evaluation
contractors, Mathematica Policy Research (MPR), and its subcontractor,
Optimal Solutions Group (OSG), so as to minimize burden on school and
district staff, children, and their parents.

### 6. Consequences to Federal Programs or Policies if Data Collection is Not Conducted

Not conducting this data collection would significantly impede ED's
ability to assess the impacts of charter schools on student achievement,
other student outcomes, and parental satisfaction. As noted previously,
the *No Child Left Behind* Act requires that education decision makers
base policies and programs on scientifically based research. ED operates
the Public Charter School Program (PCSP), a \$200 million discretionary
grant program intended to promote charter school growth and development
through provision of start-up funding and technical assistance. Without
information on the impacts of charter schools and the conditions under
which charter schools are most likely to be effective, it will be
difficult for the PCSP to efficiently award program dollars and provide
technical assistance.

More generally, the growth in the number charter schools nationally is
viewed as an important part of the broader school choice movement and a
key component of NCLB. Understanding the effects of allowing parents to
choose to enroll their children in charter schools will help ED
understand the implications of at least one part of the movement to
expand parents' educational options for their children. This should
allow ED to develop sensible approaches for understanding other aspects
of school choice.

### 7. Special Circumstances

*Inconsistent with Usual Policy:* Explanations of special circumstances
which may cause the information collection to be conducted in a manner
requiring reporting more often than quarterly, responding in less than
30 days, requiring more than the original and two copies of documents,
maintaining records for more than 3 years, using unsupported
confidentiality, or requiring submission of trade secrets or
confidential data, all covered under 5CFR.

None of the issues listed as inconsistent with usual policy for this
section are relevant in the current study.

### 8. Solicitation of Public Comments and Consultation with People Outside the Agency

#### a. Federal Register Announcement

A 60-day notice to solicit public comments was published in the Federal
Register on September 14, 2005 (p. 54372-54373). We have addressed the
comments received during this comment period.

#### b. Consultations Outside the Agency

Consultations on the research design, sample design, data sources and
needs, and study reports have occurred during the study's design phase
and will continue to take place throughout the study. The purpose of
such consultations is to ensure the technical soundness of the study and
the relevance of its findings, and to verify the importance, relevance,
and accessibility of the information sought in the study.

MPR, its subcontractor, OSG, and a study consultant, Paul Hill of the
Center on Reinventing Public Education, have provided substantial input
to ED for the study. Senior technical staff from these organizations who
are conducting the study are listed below:

Mathematica Policy Research, Inc. Dr. Phil Gleason 315-781-8495

Dr. Mark Dynarski 609-275-2397

Ms. Rita Stapulonis 609-275-2363

Optimal Solutions Group, LLC. Dr. Mark Turner 443-451-7060

Center on Reinventing Public Education Professor Paul Hill 206-685-2214

In addition to the above, an evaluation advisory panel has provided
substantial input on the study design and data collection plan. The
advisory panel members represent a number of the nation's leading
researchers on charter schools, educational policy, and evaluation
design. The panel includes:

-   Professor Thomas Cook, Northwestern University

-   Dr. Jay Greene, Manhattan Institute for Policy Research

-   Dr. Laura Hamilton, RAND Corporation

-   Dr. Eric Hanushek, Hoover Institute, Stanford University

-   Professor Helen Ladd, Duke University

-   Professor Rebecca Maynard, University of Pennsylvania

-   Dr. Robert Meyer, Wisconsin Center for Education Research,
    University of Wisconsin

```{=html}
<!-- -->
```
-   Dr. Larry Orr, Abt Associates

### 9. Respondent Payments

The parent survey will provide a key source of information on a variety
of key outcomes. Many of the parents of charter school students are very
busy, and it may be challenging to contact these individuals and get
enough of their time to complete the parent survey and help arrange a
time for their child to complete the child survey. We propose a \$10
payment for the parents of sample members who complete the parent survey
and agree to assist us in arranging a time for their child to complete
the student survey. The purpose of this payment is to reduce the
potential for nonresponse bias. Students in our sample who complete the
student survey will not receive any payment.

Data from the principal, authorizer, and state surveys will be used to
develop measures of charter school characteristics that can be
contrasted with the characteristics of the other schools attended by
control group students. The purpose of these measures will be to measure
the relationship between the characteristics of charter schools and the
impacts of these schools on student outcomes, a key objective of the
study. The principal, authorizer, and state surveys will be particularly
important in measuring the policy environment in which charter schools
operate---their degree of autonomy and the extent to which they are held
accountable for their performance.

Furthermore, obtaining high response rates on the principal survey will
be particularly challenging given the populations of principals being
targeted for the survey. Charter school principals frequently play
multiple roles within charter schools, handling both administrative
duties at the school and teaching classes. In addition, these principals
often play a role in managing the longer term viability of charter
schools, by managing issues related to attracting a sufficient number of
student applicants, securing adequate facilities for the school, and
interacting with their authorizers and---sometimes---the local public
school district. As a result, these principals tend to work long hours.
Given their busy schedules, many charter school principals complete
surveys such as the one we will ask them to complete on their own time.

Getting the principals of regular public schools that sample members
attend to respond to the survey will be challenging for other reasons.
The principals of traditional public schools may feel that charter
schools drain funding from traditional public school districts and do
not serve some of the most difficult-to-serve (and expensive) students
in the system, such as special needs students. Thus, these principals
may be hostile to charter schools and not receptive to an effort to
study their effects.

As part of our strategy to obtain a good response rate to these surveys,
we propose a \$10 pre-payment to the principals of study schools.
Research suggests that pre-paid incentives yield higher response rates
than promised incentive payments in mail surveys. Building on social
exchange theory, pre-payments motivate respondents to complete a survey
accompanied by a payment since it fulfills their role in the implied
deal. The up front payment has the additional potential benefit of
calling the principals' attention to the survey. Since principals
typically receive a large volume of mail, including the payment will
help reduce the possibility that the survey will end up getting treated
as junk mail. Finally, to reduce the possibility of non-response bias
and differential response rates among the principals of charter schools
in the study and other study schools, we propose to use an additional
\$10 payment as necessary for refusal conversion.

### 10. Confidentiality Assurances

The Impact Evaluation of Charter School Strategies will be conducted in
accordance with all relevant regulations and requirements, including the
Privacy Act of 1974 (5 usc 552 a), the Family Educational Rights And
Privacy Act Of 1974 (20 usc 1232 g), the Freedom Of Information Act (5
usc 522), The Protection Of Pupil Rights Act (20 usc 1232 h), the
Confidentiality Provisions Of The Education Sciences Reform Act (20 usc
9573), related regulations (41 cfr part 1-1 and 45 cfr part 5b), and, as
appropriate, other federal or ED regulations on the protection of human
subjects. Research participants are being informed about the nature of
the information that will be requested and confidentiality protection,
and they are being assured that information will be reported only in
aggregate, statistical form in reports and public use data files.
Respondents are also being informed that their names will not be
associated with their answers and that no one will have access to this
information except as may be required by law, regulation, or subpoena or
unless permission is given by both the parent and participating child.

It is very important that parents or legal guardians of sample members
understand that information is being collected regarding their children,
and that this information is being held confidential. In obtaining the
consent of parents/guardians for the participation of their children in
this study, we are using the following process to inform them of our
procedures for maintaining confidentiality in the data collection,
analysis, and reporting of results for the study. First, these
parents/guardians were informed of the nature of the study through an
advance letter either sent or given by the school to the parent
(depending on whether the parent picked up an application in person or
got the application in some other way) at the time that they expressed
interest in applying to the school. As part of participating in the
study, participating charter schools asked parents to provide their
consent for study activities at the time parents applied to the school.
Parents were informed at this time that the study is voluntary. We then
used active consent procedures and obtained signed consent forms from
parents of children entering the sample. When we conduct the parent
interviews, respondents will be reminded of the data confidentiality
procedures being used in the study.

MPR has a long history of protecting confidentiality and privacy of
survey and records data, and considers such practice a critical aspect
of the scientific and legal integrity of any survey. The integrity MPR
brings to protecting data confidentiality and privacy will extend to
every aspect of data collection and data handling for the Charter
Schools Evaluation. We plan to use ongoing, long-standing techniques
that have proven effective in the past. Every MPR employee is required
to sign a pledge to protect the confidentiality of respondent data. The
pledge indicates that any violation or unauthorized disclosure may
result in legal action or other sanctions by MPR. In addition to the
confidentiality pledge, MPR requires all interviewers to view a
videotape about the Belmont Report for the protection of human subjects,
and includes a discussion of human subject protection as part of their
training. After participating in this training, interviewers sign a form
certifying that they have received this training. A copy of both pledges
will be kept on file and will be available upon request.

In addition, the following safeguards are routinely employed by MPR to
carry out confidentiality assurances:

-   Access to confidential sample selection data is limited to those who
    have direct responsibility for selecting the sample. At the
    conclusion of the research, these data are destroyed.

-   Identifying information on schools, students, and parents is
    maintained on separate forms which are linked to the interviews only
    by a sample identification number. These forms are separated from
    the interviews as soon as possible.

-   Access to the file linking sample identification numbers with
    respondents' identification and contact information is limited to a
    small number of individuals who have a need to know this
    information.

-   Access to the hard copy documents collected from respondents is
    strictly limited. Documents are stored in locked files and cabinets.
    Discarded material is shredded.

-   Computer data files are protected with passwords and access is
    limited to specific users. With especially sensitive data, the data
    are maintained on removable storage devices that are kept physically
    secure when not in use.

### 11. Sensitive Questions 

In this OMB request for data collection, we are requesting OMB
permission to conduct student, parent, principal, authorizer, and state
surveys. Both the student and parent surveys will include questions
about students' behavior, including questions about delinquent behavior.
These questions will be used to define outcome measures so that we will
be able to estimate the impact of charter schools on students' behavior,
both within and outside of school.

These questions are vital to the study because of the ongoing debate
about the effects of charter schools. Measures of student
behavior/discipline are a particularly important student outcome at the
middle school level, as some experts have suggested that the small size
and focused mission of charter schools, along with their emphasis on
discipline, might have a particularly important effect on the behavior
of middle school students. We will be testing this hypothesis using data
from these questions.

### 12. Total Annual Cost Burden to Respondents 

As reported in our initial clearance request, the estimated burden for
the baseline collection effort for 2005 was 1,900 hours. Because the
number of schools brought into the study was one-half the number
originally targeted, and thus we have continued our school recruiting
effort (and associated baseline collection) into the current school
year, we estimate that one half of the originally planned baseline
collection burden was completed in 2005 and the remain burden for the
baseline collection effort is to be expected in 2006. In other words,
the expected burden from the portion of data collection for which we
have already obtained clearance is 950 hours.

Table 2 presents our estimate of respondent burden for the data
collection for which we are requesting approval in this submission. Our
previous OMB submission included burden estimates for the collection of
baseline intake forms (from parents) and administrative records data
(from school district staff). This submission includes burden estimates
for the student, parent, principal, authorizer, and state surveys and
for school staff involved in our case study site visits. The surveys
will take place in the spring of 2006 and spring of 2007. The 25 case
study site visits will involve interviews with 2 teachers and 1
administrator at the charter school and 2 teachers and 1 administrator
at a regular public school. Overall, there will be a total of 7,002
respondents to the data collection instruments covered by this
submission, with a total number of respondent hours of 2,076.

The surveys of parents, students, and study school principals will be
split evenly across the two survey years of 2006 and 2007. The surveys
of non-study school principals, charter school authorizers, and state
officials, along with the case study site visits will take place in the
spring of 2007. Thus, there will be 3,150 respondents and 825 respondent
hours in 2006. When combined with the 950 respondent hours for the
baseline data collection in 2006, this implies a total of 1,775
respondent hours for that year. In 2007, there will be 3,852 respondents
and 1,251 respondent hours. We have averaged the estimated burden across
the two remaining years of data collection, so that the estimated yearly
hours burden is 1,513 hours.

TABLE 2

RESPONDENT BURDEN ESTIMATES

+--------------------+--------+--------+--------+--------+------------+
| Informant          | Number | A      | Total  | Est    | Estimated  |
|                    |        | verage | Resp   | imated | Lost       |
|                    | of     | Time   | ondent | Hourly | Burden to  |
|                    |        | Per    | Time   | Wage   | R          |
|                    | Res    | Re     |        | (Do    | espondents |
|                    | ponses | sponse | (      | llars) | (Dollars)  |
|                    |        | (      | Hours) |        |            |
|                    |        | Hours) |        |        |            |
+--------------------+--------+--------+--------+--------+------------+
| Parents            | 3,000  | ¼ (15  | 750    | \      | \          |
|                    |        | min)   |        | $14.95 | $11,212.50 |
|                    |        |        |        | ^a^    |            |
+--------------------+--------+--------+--------+--------+------------+
| Students           | 3,000  | ¼ (15  | 750    | 0      | \$0.00     |
|                    |        | min)   |        |        |            |
+--------------------+--------+--------+--------+--------+------------+
| Principals---Study | 300    | ½ (30  | 150    | \      | \$5,712.00 |
| Schools            |        | min)   |        | $38.08 |            |
|                    |        |        |        | ^b^    |            |
+--------------------+--------+--------+--------+--------+------------+
| Prin               | 500    | ½ (30  | 250    | \      | \$9,520.00 |
| cipals---Non-Study |        | min)   |        | $38.08 |            |
| School             |        |        |        | ^b^    |            |
+--------------------+--------+--------+--------+--------+------------+
| Authorizers        | 40     | ½ (30  | 20     | \      | \$761.60   |
|                    |        | min)   |        | $38.08 |            |
|                    |        |        |        | ^b^    |            |
+--------------------+--------+--------+--------+--------+------------+
| State Officials    | 12     | ½ (30  | 6      | \      | \$184.44   |
|                    |        | min)   |        | $30.74 |            |
|                    |        |        |        | ^c^    |            |
+--------------------+--------+--------+--------+--------+------------+
| Teachers           | 100    | 1      | 100    | \      | \$2,190.00 |
|                    |        |        |        | $21.90 |            |
|                    |        |        |        | ^b^    |            |
+--------------------+--------+--------+--------+--------+------------+
| School             | 50     | 1      | 50     | \      | \$1,904.00 |
| Administrator      |        |        |        | $38.08 |            |
|                    |        |        |        | ^b^    |            |
+--------------------+--------+--------+--------+--------+------------+

^a^ 2003 Statistical Abstract of the U.S. Table No. 636: Average Hourly
Earnings by Private Industry Group: 1980-2002 (estimate in table is for
2002).

^b^ 2003 Statistical Abstract of the U.S. Table No. 251: Average Salary
and Wages Paid in Public School Systems: 1980-2002 (estimate in table is
for 2002).

^c^ 2003 Statistical Abstract of the U.S. Table No. 469: State and Local
Government: Employer Cost Per Hour Worked: 2002.

### 13. Estimate of the Total Annual Cost Burden to Respondents or Record Keepers 

There are no direct costs to participants. The only costs are the
opportunity costs of respondents' time required to provide information
as explained in item 12 above. The Impact Evaluation of Charter School
Strategies does not place any capital equipment, start-up, or record
maintenance requirements on respondents.

### 14. Annualized Cost to the Federal Government, Including a Description of the Method Used to Estimate Costs

The estimated cost to the federal government of conducting the Impact
Evaluation of Charter School Strategies is based on the government's
contracted cost of the data collection and related study activities
along with personnel cost of government employees involved in oversight
and/or analysis. For the data collection activities for which OMB
approval is currently being requested, the overall cost to the
government is \$1,088,981. This includes:

-   \$234,990 for the collection of student survey data (including
    \$117,495 for the 1st cohort of students in spring 2006 and
    \$117,495 from the 2nd cohort in spring 2007)

-   \$250,320 for the collection of parent survey data (including
    \$125,160 for the 1st cohort in spring 2006 and \$125,160 for the
    2nd cohort in spring 2007)

-   \$96,404 for the collection of study school principal survey data
    (including \$48,202 for the 1st cohort in spring 2006 and \$48,202
    for the 2nd cohort in spring 2007)

-   \$74,669 for the collection of non-study school principal survey
    data in spring 2007

-   \$14,248 for the collection of authorizer and state official survey
    data in spring 2007

```{=html}
<!-- -->
```
-   \$418,350 for the collection of site visit case study data in the
    spring of 2007

Thus, the costs to the government for the first year of data collection
covered under this submission (spring 2006) will be \$290,857. The costs
to the government for the second year of data collection covered under
this submission (spring 2007) will be \$798,124.

The overall costs to the government of the full range of evaluation
activities over the entire study period will be \$6,052,302 over a
five-year period. When annualized, this cost amounts to \$1,210,460.10
per year. This estimate is based on the evaluation contractor's previous
experience managing other research and data collection activities of
this type.

### 15. Program Changes or Adjustments

A program change of -387 hours is shown. The reason for this program
change is that the original estimate of 1,900 hours covered baseline
collection for the full sample. Since half of the baseline collection
was completed in 2005, only half remains for the coming program year. In
addition to the remaining baseline collection burden, we have added the
respondent hours estimated for the completion of the data collection
forms for which we are requesting clearance in this package. These forms
are being added as part of our request for approval of three modest
enhancements to the design.

### 16. Tabulation, Analysis and Publication of Results

Our discussion of analysis, tabulation, and publication of results
focuses on the three main study questions outlined in the introduction.

#### a. Impacts of Charter Schools

The first question to be addressed involves estimating the impacts of
charter schools on student achievement, other student outcomes, and
parental satisfaction. A simple estimator of charter school impacts is
the difference between mean treatment and control group outcomes. For
example, the simple difference between the average test scores of
students randomly assigned to the treatment group and the average test
scores of those randomly assigned to the control group provides an
unbiased estimate of the impact of admission to charter schools relative
to what similar children would have experienced without the opportunity
to attend charter schools. A t-test of the difference in average test
scores enables the evaluation to assess the likelihood that the
difference was due to chance or to the charter schools.

Building on the simple differences-in-means estimator, we plan to
estimate analytic models. The analytic models will use information we
will collect about student and school baseline characteristics, along
with an indicator of treatment status, to predict student outcomes. The
covariates we plan to use will include gender, race, ethnicity, poverty,
English-language proficiency, and baseline standardized test scores, as
well as additional information on the attributes of the school the
student attends at baseline (such a operational age and size). The use
of information beyond treatment status allows for the calculation of
estimates of charter school impacts that are more precise, because this
information may account for some of the variation in outcome measures
unrelated to the charter school impacts. The basic form of the model is:

\(1\) ![](media/image3.wmf)

where *y~ij~* is the outcome of interest for student *i* in site *j*
(where a site is defined according to the charter school to which sample
members originally applied and were randomly assigned), *X~ij~* is a
vector of characteristics of student *i* in site *j* (including a
site-specific intercept), *T~ij~* is a binary variable that indicates
whether student *i* was admitted to the treatment group in site *j*,
*e~ij~* is a random error term that reflects the influence of unobserved
factors on the outcome, and ![](media/image4.wmf)are parameters or
vectors of parameters to be estimated. In particular, the parameter
![](media/image5.wmf) (*j=1,...,50)* represents the estimated impact of
the treatment (admission to the charter school) in site j.

In the simplest version of this model, the estimated impact is assumed
to be the same across sites, so that ![](media/image6.wmf) is the
overall impact estimate. If we allow the impact to vary across sites,
the overall impact estimate can be calculated as the average value of
these site-specific estimates. The model estimation technique will
depend on the form of the dependent variable. If the dependent variable
is binary, we will estimate a logistic regression model. If the
dependent variable is continuous, we will estimate a linear regression
model. Finally, the standard errors estimated and significance tests
conducted will account for the fact that student observations are
clustered within the charter school sites and thus are not statistically
independent of one another. Estimating a hierarchical linear model, as
described below, is one approach to taking account of such clustering.

The impact analysis described above measures the impact of being
admitted to a charter school, commonly referred to as an
*intent-to-treat*, or ITT, impact estimate. It is useful to policy
makers because it indicates the expected effect of a charter school on
the group of students given the opportunity to attend that school. And
if compliance with random assignment is high---with treatment students
actually attending the charter school and control students not attending
the school (or another charter school)---this ITT impact estimate will
also indicate the effect of attending the charter school. Noncompliance
with random assignment is an issue in many program evaluations, however.
In this case, noncompliance would occur if a substantial number of
treatment students never attended the charter school or if some of the
control students did attend a charter school. With noncompliance, an
insignificant ITT impact estimate could either result from the charter
school not influencing outcomes among its enrolled students or, for
example, from students admitted to the charter school never attending
classes at the school. Thus, policy makers will benefit from knowing
both the ITT impact estimate and the estimated impact of the *treatment
on the treated,* or TOT. By estimating equation (1) on the full sample,
we will generate an ITT impact estimate. We plan to use information on
the proportion of treatment group members and control group members who
attend the charter school to generate a TOT impact estimate as well.[^2]

In addition to estimating charter school impacts among all students, we
will examine whether impacts differ among particular subgroups of
students, such as those defined by gender, race, ethnicity, grade,
poverty, use of English as a second language, or prior achievement.
Estimating subgroup impacts entails adding interaction terms between the
treatment group indicator and an indicator of whether a student is a
member of the relevant subgroup under consideration.

After estimating charter school impacts, overall and for particular
subgroups, we will present these impact estimates in tables that show
four values for each outcome measure: (1) the regression-adjusted mean
value of the outcome among the treatment group; (2) the
regression-adjusted mean value of the outcome among the control group;
(3) the estimated impact of admission to a charter school on the outcome
(the ITT impact estimate); and (4) the estimated impact of attending a
charter school on the outcome (the TOT impact estimate). Tables 3 and 4
show how these impact estimates may be presented in the evaluation
reports. To show impact estimates for subgroups, additional tables will
be prepared that show impacts on a similar set of outcomes for sets of
related subgroups (such as males versus females, low versus medium
versus high income students, and so on).

#### b. Differences Between Charter Schools and Conventional Schools

The second research question in the study involves examining differences
in the characteristics of charter schools and nearby conventional public
schools and how these differences influence charter school
effectiveness. Our analysis of this question will include two parts.
First, we will measure the characteristics of the charter schools in the
study and the schools that control group students are attending (which
may well include both conventional public schools and private schools).
The primary source of this information will be the survey of school
principals shown in Exhibit C. Second, we will characterize the
differences between charter schools and comparison schools, and examine
the relationship of the charter school characteristics (relative to
those of comparison schools) with charter school impacts.

TABLE 3

IMPACT OF CHARTER SCHOOLS ON STUDENT OUTCOMES

+----------------------------------+-------+--------+------+---------+
| Outcome                          | Regr  |        | E    | Es      |
|                                  | essio |        | stim | timated |
|                                  | n-Adj |        | ated | Impact  |
|                                  | usted |        | Cha  | on      |
|                                  |       |        | rter | Charter |
|                                  | Value |        | Sc   | School  |
|                                  |       |        | hool | At      |
|                                  |       |        | Im   | tenders |
|                                  |       |        | pact |         |
+----------------------------------+-------+--------+------+---------+
|                                  | Trea  | C      |      |         |
|                                  | tment | ontrol |      |         |
|                                  | Group | Group  |      |         |
+----------------------------------+-------+--------+------+---------+
| Test Scores                      |       |        |      |         |
+----------------------------------+-------+--------+------+---------+
| > Mean reading score             |       |        |      |         |
+----------------------------------+-------+--------+------+---------+
| > Mean math score                |       |        |      |         |
+----------------------------------+-------+--------+------+---------+
| Other School Outcomes            |       |        |      |         |
+----------------------------------+-------+--------+------+---------+
| > Percentage of students who     |       |        |      |         |
| > report that they "often" or    |       |        |      |         |
| > "always" do the homework       |       |        |      |         |
| > teachers assign                |       |        |      |         |
+----------------------------------+-------+--------+------+---------+
| > Percentage of students whose   |       |        |      |         |
| > parents "agree" or "strongly   |       |        |      |         |
| > agree" that their child works  |       |        |      |         |
| > hard at school                 |       |        |      |         |
+----------------------------------+-------+--------+------+---------+
| Mean number of days absent       |       |        |      |         |
+----------------------------------+-------+--------+------+---------+
| > Mean number of days late       |       |        |      |         |
+----------------------------------+-------+--------+------+---------+
| Student Behavior in School       |       |        |      |         |
+----------------------------------+-------+--------+------+---------+
| > Percentage of students who     |       |        |      |         |
| > report being sent to the       |       |        |      |         |
| > office for misbehaving         |       |        |      |         |
+----------------------------------+-------+--------+------+---------+
| > Percentage of students whose   |       |        |      |         |
| > parents report being called by |       |        |      |         |
| > school about behavior          |       |        |      |         |
+----------------------------------+-------+--------+------+---------+
| > Percentage of students         |       |        |      |         |
| > suspended                      |       |        |      |         |
+----------------------------------+-------+--------+------+---------+
| Student Attitudes Toward School  |       |        |      |         |
+----------------------------------+-------+--------+------+---------+
| > Percentage of students who     |       |        |      |         |
| > "strongly agree" that they     |       |        |      |         |
| > like school                    |       |        |      |         |
+----------------------------------+-------+--------+------+---------+
| > Percentage of students who     |       |        |      |         |
| > "strongly agree" that their    |       |        |      |         |
| > teachers are very good         |       |        |      |         |
+----------------------------------+-------+--------+------+---------+
| > Mean overall grade they give   |       |        |      |         |
| > school                         |       |        |      |         |
+----------------------------------+-------+--------+------+---------+
| Student Behavior Outside of      |       |        |      |         |
| School---Percentage of Students  |       |        |      |         |
| Who:                             |       |        |      |         |
+----------------------------------+-------+--------+------+---------+
| > Consumed alcohol in previous   |       |        |      |         |
| > month                          |       |        |      |         |
+----------------------------------+-------+--------+------+---------+
| > Smoked cigarettes in previous  |       |        |      |         |
| > month                          |       |        |      |         |
+----------------------------------+-------+--------+------+---------+
| Used drugs in previous month     |       |        |      |         |
+----------------------------------+-------+--------+------+---------+
| > Were arrested/detained by      |       |        |      |         |
| > police in previous year        |       |        |      |         |
+----------------------------------+-------+--------+------+---------+
| > Got in fight in previous week  |       |        |      |         |
+----------------------------------+-------+--------+------+---------+
| **Sample Size**                  |       |        |      |         |
+----------------------------------+-------+--------+------+---------+

[Source:]{.smallcaps} Student Survey, Parent Survey, School Records.

\*Significantly different from zero at the .10 level, two-tailed test

\*\*Significantly different from zero at the .05 level, two-tailed test

\*\*\*Significantly different from zero at the .01 level, two-tailed
test

TABLE 4

IMPACT OF CHARTER SCHOOLS ON PARENT OUTCOMES

+----------------------------------+-------+--------+------+---------+
| Outcome                          | Regr  |        | E    | Es      |
|                                  | essio |        | stim | timated |
|                                  | n-Adj |        | ated | Impact  |
|                                  | usted |        | Cha  | on      |
|                                  |       |        | rter | Charter |
|                                  | Value |        | Sc   | School  |
|                                  |       |        | hool | At      |
|                                  |       |        | Im   | tenders |
|                                  |       |        | pact |         |
+----------------------------------+-------+--------+------+---------+
|                                  | Trea  | C      |      |         |
|                                  | tment | ontrol |      |         |
|                                  | Group | Group  |      |         |
+----------------------------------+-------+--------+------+---------+
| Parent Attitudes Toward School   |       |        |      |         |
+----------------------------------+-------+--------+------+---------+
| > Percentage of parents who      |       |        |      |         |
| > "strongly agree" that their    |       |        |      |         |
| > child likes school             |       |        |      |         |
+----------------------------------+-------+--------+------+---------+
| > Percentage of parents who      |       |        |      |         |
| > "strongly agree" that teachers |       |        |      |         |
| > in their child's school are    |       |        |      |         |
| > better than teachers in other  |       |        |      |         |
| > local schools                  |       |        |      |         |
+----------------------------------+-------+--------+------+---------+
| > Mean overall grade they give   |       |        |      |         |
| > school                         |       |        |      |         |
+----------------------------------+-------+--------+------+---------+
| Parent Involvement in School:    |       |        |      |         |
| Percentage of Parents Who Did    |       |        |      |         |
| the Following At Least Three     |       |        |      |         |
| Times During Year                |       |        |      |         |
+----------------------------------+-------+--------+------+---------+
| > Attended an open house at the  |       |        |      |         |
| > school                         |       |        |      |         |
+----------------------------------+-------+--------+------+---------+
| > Attended parent-teacher        |       |        |      |         |
| > organization meetings          |       |        |      |         |
+----------------------------------+-------+--------+------+---------+
| > Attended an after-school event |       |        |      |         |
+----------------------------------+-------+--------+------+---------+
| > Volunteered to help out at     |       |        |      |         |
| > school                         |       |        |      |         |
+----------------------------------+-------+--------+------+---------+
| > Involved in decision-making at |       |        |      |         |
| > school                         |       |        |      |         |
+----------------------------------+-------+--------+------+---------+
| Parent Involvement in Child's    |       |        |      |         |
| Education: Percentage of Parents |       |        |      |         |
| Who Report Doing the Following   |       |        |      |         |
+----------------------------------+-------+--------+------+---------+
| > Helped their child with        |       |        |      |         |
| > homework at least three times  |       |        |      |         |
| > last week                      |       |        |      |         |
+----------------------------------+-------+--------+------+---------+
| > Checked on their child's       |       |        |      |         |
| > homework completion at least   |       |        |      |         |
| > three times last week          |       |        |      |         |
+----------------------------------+-------+--------+------+---------+
| > Asked their child about things |       |        |      |         |
| > they were doing in class at    |       |        |      |         |
| > least seven times last month   |       |        |      |         |
+----------------------------------+-------+--------+------+---------+
| **Sample Size**                  |       |        |      |         |
+----------------------------------+-------+--------+------+---------+

[Source:]{.smallcaps} Student Survey, Parent Survey, School Records.

\*Significantly different from zero at the .10 level, two-tailed test

\*\*Significantly different from zero at the .05 level, two-tailed test

\*\*\*Significantly different from zero at the .01 level, two-tailed
test

We will examine how these charter school characteristics are related to
charter school impacts using the following hierarchical linear model:

\(2\) ![](media/image7.wmf)

\(3\) ![](media/image8.wmf)

Equation (2) is identical to equation (1) above. Equation (3), the
"second-level" equation, allows these site-level impacts to vary
according to characteristics of the site (included in the vector W~j~).
By measuring these site characteristics in terms of differences between
the charter school and comparison schools, we can examine the
relationship between these factors and charter school impacts. Suppose,
for example, we hypothesized that charter school impacts were driven by
the fact that charter schools are typically smaller than traditional
public schools. In this case, a variable in the vector W~j~ might
measure the size of the charter school relative to the size of the
average comparison school attended by students in that site. Thus, a
positive and significant coefficient on this variable would suggest that
a larger average size of charter schools leads to more positive impacts
of charter schools on student outcomes.[^3]

For example, we may wish to examine whether charter schools that use an
innovative curriculum or instructional approach relative to comparison
schools have different impacts than charter schools that do not use an
innovative approach.[^4] By including in W~j~ a binary variable
indicating whether the charter school uses an innovative approach, the
coefficient on this variable can be used to measure its relationship
with the impacts of charter schools. An example of how we would present
these results (for impacts on student outcomes) is shown in Table 5.

#### c. Influence of Policy Environment

An important objective of the evaluation will involve examining how the
policy environment in which charter schools operate influences their
effectiveness. Two key dimensions of the policy environment will be the
autonomy with which charter schools operate and their funding from state
and local government sources. Each of these dimensions will be measured
relative to policy conditions faced by the conventional public schools
with which the charter schools are competing for students.

TABLE 5

IMPACT OF CHARTER SCHOOLS ON STUDENT OUTCOMES,

BY CHARACTERISTICS OF CHARTER SCHOOLS

RELATIVE TO COMPARISON SCHOOLS

+----------------------------+--------------+--------------+---------+
| Outcome                    | Estimated    | Estimated    | Dif     |
|                            | Impact of    | Impact of    | ference |
|                            | Charter      | Charter      |         |
|                            | Schools      | Schools Not  |         |
|                            | Using        | Using        |         |
|                            | Innovative   | Innovative   |         |
|                            | Approach to  | Approach to  |         |
|                            | Curriculum/  | Curriculum/  |         |
|                            | Instruction  | Instruction  |         |
+----------------------------+--------------+--------------+---------+
| Test Scores                |              |              |         |
+----------------------------+--------------+--------------+---------+
| > Mean reading score       |              |              |         |
+----------------------------+--------------+--------------+---------+
| > Mean math score          |              |              |         |
+----------------------------+--------------+--------------+---------+
| Other School Outcomes      |              |              |         |
+----------------------------+--------------+--------------+---------+
| > Percentage of students   |              |              |         |
| > who report that they     |              |              |         |
| > "often" or "always" do   |              |              |         |
| > the homework teachers    |              |              |         |
| > assign                   |              |              |         |
+----------------------------+--------------+--------------+---------+
| > Percentage of students   |              |              |         |
| > whose parents "agree" or |              |              |         |
| > "strongly agree" that    |              |              |         |
| > their child works hard   |              |              |         |
| > at school                |              |              |         |
+----------------------------+--------------+--------------+---------+
| > Mean number of days      |              |              |         |
| > absent                   |              |              |         |
+----------------------------+--------------+--------------+---------+
| > Mean number of days late |              |              |         |
+----------------------------+--------------+--------------+---------+
| Student Behavior in School |              |              |         |
+----------------------------+--------------+--------------+---------+
| > Percentage of students   |              |              |         |
| > who report being sent to |              |              |         |
| > the office for           |              |              |         |
| > misbehaving              |              |              |         |
+----------------------------+--------------+--------------+---------+
| > Percentage of students   |              |              |         |
| > whose parents report     |              |              |         |
| > being called by school   |              |              |         |
| > about behavior           |              |              |         |
+----------------------------+--------------+--------------+---------+
| > Percentage of students   |              |              |         |
| > suspended                |              |              |         |
+----------------------------+--------------+--------------+---------+
| Student Attitudes Toward   |              |              |         |
| School                     |              |              |         |
+----------------------------+--------------+--------------+---------+
| > Percentage of students   |              |              |         |
| > who "strongly agree"     |              |              |         |
| > that they like school    |              |              |         |
+----------------------------+--------------+--------------+---------+
| > Percentage of students   |              |              |         |
| > who "strongly agree"     |              |              |         |
| > that their teachers are  |              |              |         |
| > very good                |              |              |         |
+----------------------------+--------------+--------------+---------+
| > Mean overall grade they  |              |              |         |
| > give school              |              |              |         |
+----------------------------+--------------+--------------+---------+
| Student Behavior Outside   |              |              |         |
| of School---Percentage of  |              |              |         |
| Students Who:              |              |              |         |
+----------------------------+--------------+--------------+---------+
| > Consumed alcohol in      |              |              |         |
| > previous month           |              |              |         |
+----------------------------+--------------+--------------+---------+
| > Smoked cigarettes in     |              |              |         |
| > previous month           |              |              |         |
+----------------------------+--------------+--------------+---------+
| > Used drugs in previous   |              |              |         |
| > month                    |              |              |         |
+----------------------------+--------------+--------------+---------+
| > Were arrested/detained   |              |              |         |
| > by police in previous    |              |              |         |
| > year                     |              |              |         |
+----------------------------+--------------+--------------+---------+
| > Got in fight in previous |              |              |         |
| > week                     |              |              |         |
+----------------------------+--------------+--------------+---------+
| **Sample Size**            |              |              |         |
+----------------------------+--------------+--------------+---------+

[Source:]{.smallcaps} Student Survey, Parent Survey, School Records.

\*Significantly different from zero at the .10 level, two-tailed test

\*\*Significantly different from zero at the .05 level, two-tailed test

\*\*\*Significantly different from zero at the .01 level, two-tailed
test

Our selection of charter schools for the evaluation has been conducted
thus far in a way designed to promote variability along these policy
dimensions. We are selecting some charter schools from states that
provide charter schools with greater freedom to hire/fire staff, set
budgets, and determine other school policies, while selecting other
charter schools from states that provide charter school with less
freedom or autonomy in these areas. However, a given charter school's
level of autonomy may be affected by factors other than state
legislation and policy, such as the actions and policies taken by the
entity that provided a particular school with its charter (that is, the
school's authorizer). Even within states, as a result, there may be
variation in the level of autonomy experienced by charter schools.
During the data collection phase of the study (and after sample
selection has been completed), we will collect more detailed information
regarding charter school autonomy as well as other characteristics
potentially influenced by policy (such as its level of funding), to
confirm or revise our initial categorization of charter schools' policy
environments.[^5] Finally, we will create variables that indicate the
policy environment facing each charter school. Like school
characteristics, these policy variables will enter the model in the
vector W~j~. We will then measure how these variables are related to
charter school impacts using the hierarchical linear model shown in
equations (2) and (3). Table 6 shows how we plan to present the results
of this analysis.

#### d. Publication Plans and Time Schedule

The primary product of the evaluation will a final report. This report
will cover our findings after one follow-up year for the full sample and
after two follow-up years for the first cohort of students. A draft of
this report will be submitted in Spring 2008. The final report will be
completed in September 2008.

The full timeline for the evaluation in shown in Table 7. The timeline
calls for major design and school selection activities between October
2003 and September 2005. Data collection for the baseline year begins in
Spring 2005, and school records data will be collected following the
baseline year and each of the two follow-up years. Site visits will be
conducted during Spring 2007, and the student, parent, and principal
surveys will also be conducted during spring 2006 and spring 2007.

### 17. Approval Not to Display the Expiration Date for OMB Approval

Approval not to display the expiration date for OMB approval is not
requested.

### 18. Exception to the Certification Statement

No exceptions to the certification statement are requested or required.

TABLE 6

IMPACT OF CHARTER SCHOOLS ON STUDENT OUTCOMES,

BY CHARTER SCHOOL POLICY ENVIRONMENT

<table>
<colgroup>
<col style="width: 48%" />
<col style="width: 6%" />
<col style="width: 0%" />
<col style="width: 6%" />
<col style="width: 11%" />
<col style="width: 2%" />
<col style="width: 7%" />
<col style="width: 0%" />
<col style="width: 5%" />
<col style="width: 2%" />
<col style="width: 8%" />
</colgroup>
<tbody>
<tr class="odd">
<td rowspan="2">Outcome</td>
<td colspan="4"><p>Charter School</p>
<p>Autonomy</p></td>
<td rowspan="2"></td>
<td colspan="5"><p>Charter School</p>
<p>Funding</p></td>
</tr>
<tr class="even">
<td>High</td>
<td colspan="2">Low</td>
<td>Difference</td>
<td>High</td>
<td colspan="2">Low</td>
<td colspan="2">Difference</td>
</tr>
<tr class="odd">
<td>Test Scores</td>
<td colspan="2"></td>
<td></td>
<td></td>
<td></td>
<td colspan="2"></td>
<td colspan="2"></td>
<td></td>
</tr>
<tr class="even">
<td><blockquote>
<p>Mean reading score</p>
</blockquote></td>
<td colspan="2"></td>
<td></td>
<td></td>
<td></td>
<td colspan="2"></td>
<td colspan="2"></td>
<td></td>
</tr>
<tr class="odd">
<td><blockquote>
<p>Mean math score</p>
</blockquote></td>
<td colspan="2"></td>
<td></td>
<td></td>
<td></td>
<td colspan="2"></td>
<td colspan="2"></td>
<td></td>
</tr>
<tr class="even">
<td>Other School Outcomes</td>
<td colspan="2"></td>
<td></td>
<td></td>
<td></td>
<td colspan="2"></td>
<td colspan="2"></td>
<td></td>
</tr>
<tr class="odd">
<td><blockquote>
<p>Percentage of students who report that they “often” or “always” do
the homework teachers assign</p>
</blockquote></td>
<td colspan="2"></td>
<td></td>
<td></td>
<td></td>
<td colspan="2"></td>
<td colspan="2"></td>
<td></td>
</tr>
<tr class="even">
<td><blockquote>
<p>Percentage of students whose parents “agree” or “strongly agree” that
their child works hard at school</p>
</blockquote></td>
<td colspan="2"></td>
<td></td>
<td></td>
<td></td>
<td colspan="2"></td>
<td colspan="2"></td>
<td></td>
</tr>
<tr class="odd">
<td><blockquote>
<p>Mean number of days absent</p>
</blockquote></td>
<td colspan="2"></td>
<td></td>
<td></td>
<td></td>
<td colspan="2"></td>
<td colspan="2"></td>
<td></td>
</tr>
<tr class="even">
<td><blockquote>
<p>Mean number of days late</p>
</blockquote></td>
<td colspan="2"></td>
<td></td>
<td></td>
<td></td>
<td colspan="2"></td>
<td colspan="2"></td>
<td></td>
</tr>
<tr class="odd">
<td>Student Behavior in School</td>
<td colspan="2"></td>
<td></td>
<td></td>
<td></td>
<td colspan="2"></td>
<td colspan="2"></td>
<td></td>
</tr>
<tr class="even">
<td><blockquote>
<p>Percentage of students who report being sent to the office for
misbehaving</p>
</blockquote></td>
<td colspan="2"></td>
<td></td>
<td></td>
<td></td>
<td colspan="2"></td>
<td colspan="2"></td>
<td></td>
</tr>
<tr class="odd">
<td><blockquote>
<p>Percentage of students whose parents report being called by school
about behavior</p>
</blockquote></td>
<td colspan="2"></td>
<td></td>
<td></td>
<td></td>
<td colspan="2"></td>
<td colspan="2"></td>
<td></td>
</tr>
<tr class="even">
<td><blockquote>
<p>Percentage of students suspended</p>
</blockquote></td>
<td colspan="2"></td>
<td></td>
<td></td>
<td></td>
<td colspan="2"></td>
<td colspan="2"></td>
<td></td>
</tr>
<tr class="odd">
<td>Student Attitudes Toward School</td>
<td colspan="2"></td>
<td></td>
<td></td>
<td></td>
<td colspan="2"></td>
<td colspan="2"></td>
<td></td>
</tr>
<tr class="even">
<td><blockquote>
<p>Percentage of students who “strongly agree” that they like school</p>
</blockquote></td>
<td colspan="2"></td>
<td></td>
<td></td>
<td></td>
<td colspan="2"></td>
<td colspan="2"></td>
<td></td>
</tr>
<tr class="odd">
<td><blockquote>
<p>Percentage of students who “strongly agree” that their teachers are
very good</p>
</blockquote></td>
<td colspan="2"></td>
<td></td>
<td></td>
<td></td>
<td colspan="2"></td>
<td colspan="2"></td>
<td></td>
</tr>
<tr class="even">
<td><blockquote>
<p>Mean overall grade they give school</p>
</blockquote></td>
<td colspan="2"></td>
<td></td>
<td></td>
<td></td>
<td colspan="2"></td>
<td colspan="2"></td>
<td></td>
</tr>
<tr class="odd">
<td>Student Behavior Outside of School—Percentage of Students Who:</td>
<td colspan="2"></td>
<td></td>
<td></td>
<td></td>
<td colspan="2"></td>
<td colspan="2"></td>
<td></td>
</tr>
<tr class="even">
<td><blockquote>
<p>Consumed alcohol in previous month</p>
</blockquote></td>
<td colspan="2"></td>
<td></td>
<td></td>
<td></td>
<td colspan="2"></td>
<td colspan="2"></td>
<td></td>
</tr>
<tr class="odd">
<td><blockquote>
<p>Smoked cigarettes in previous month</p>
</blockquote></td>
<td colspan="2"></td>
<td></td>
<td></td>
<td></td>
<td colspan="2"></td>
<td colspan="2"></td>
<td></td>
</tr>
<tr class="even">
<td><blockquote>
<p>Used drugs in previous month</p>
</blockquote></td>
<td colspan="2"></td>
<td></td>
<td></td>
<td></td>
<td colspan="2"></td>
<td colspan="2"></td>
<td></td>
</tr>
<tr class="odd">
<td><blockquote>
<p>Were arrested/detained by police in previous year</p>
</blockquote></td>
<td colspan="2"></td>
<td></td>
<td></td>
<td></td>
<td colspan="2"></td>
<td colspan="2"></td>
<td></td>
</tr>
<tr class="even">
<td><blockquote>
<p>Got in fight in previous week</p>
</blockquote></td>
<td colspan="2"></td>
<td></td>
<td></td>
<td></td>
<td colspan="2"></td>
<td colspan="2"></td>
<td></td>
</tr>
<tr class="odd">
<td><strong>Sample Size</strong></td>
<td colspan="2"></td>
<td></td>
<td></td>
<td></td>
<td colspan="2"></td>
<td colspan="2"></td>
<td></td>
</tr>
</tbody>
</table>

[Source:]{.smallcaps} Student Survey, Parent Survey, School Records.

\*Significantly different from zero at the .10 level, two-tailed test

\*\*Significantly different from zero at the .05 level, two-tailed test

\*\*\*Significantly different from zero at the .01 level, two-tailed
test

TABLE 7

SCHEDULE OF ACTIVITIES

  ---------------------------------- ----------------- ----------- -------------
                                     Schedule                      

  Activity                           Cohort 1          Cohort 2    

  Study Design                       October 2003-June             
                                     2004                          

  Selection of Schools               April 2004-Jan    Aug         
                                     2005              2005-Sept   
                                                       2005        

  Recruitment of Schools             Oct 2004-June     Aug         
                                     2005              2005-June   
                                                       2006        

  Random Assignment and Baseline     Jan 2005-Aug 2005 Jan         
  Data Collection                                      2006-Aug    
                                                       2006        

  Baseline Records Data Collection   June 2005-Nov     June        
                                     2005              2006-Nov    
                                                       2006        

  Student/Parent/Study School        Spring 2006       Spring 2007 
  Principal Surveys                                                

  First Follow-up Records Data       June 2006-Sep     June        
  Collection                         2006              2007-Sep    
                                                       2007        

  Site Visits                        Spring 2007                   

  Authorizer/State/Non-Study School  Spring 2007                   
  Principal Surveys                                                

  Second Follow-up Records Data      June 2007-Sep                 None
  Collection                         2007                          

  Final Report                       Sep 2008                      
  ---------------------------------- ----------------- ----------- -------------

## B. COLLECTION OF INFORMATION EMPLOYING STATISTICAL METHODS

This section is as previously described in the earlier submission. No
changes have been made to the approved sampling and other statistical
procedures.

### 1. Respondent Universe and Sampling Procedures

The Impact Evaluation of Charter School Strategies will include 50
charter schools selected from among the approximately 3,000 charter
schools nationally. A purposive strategy is being used for selecting
these charter schools. There are three main reasons we are using a
purposive strategy for selecting schools. First, since a key policy
question involves examining the relationship between the policy
environment in which charter schools operate and their impacts, we wish
to ensure that our sample of charter schools has a sufficient amount of
variation in policy conditions such as their level of autonomy. Second,
to implement the random assignment experimental design the study uses,
we need to ensure that the schools selected for the evaluation are
oversubscribed; that is, that they have a larger number of applicants
than they have openings for students. This oversubscription ensures that
we can randomly assign students to the control group that is not
admitted to the school without denying services to students who would
otherwise have received them. Third, we did not want to conduct an
experimental study of charter school impacts using schools that were
recently opened as they may have still been in a state of flux, and so
decided to focus on charter schools that had been operating for more
than two years. This purposive sampling strategy prevents us from
generalizing to a broader population of charter schools, though the
random assignment design provides good internal validity on the impacts
of the schools in the sample.

The selected schools will all be charter middle schools in that the
grade in which a cohort of students enters the school will be either
grade 4, 5, 6, or 7. Charter middle schools were chosen as the focus
because of the likely availability of administrative records with
student test scores for sample members in the middle school grades. The
*No Child Left Behind* legislation requires public schools to test
students in grades 3 through 8 on a yearly basis. Thus, by selecting a
sample of students entering a charter middle school in grades 4 to 7, we
will have test scores from a baseline year and two follow-up years. For
students entering charter elementary schools or charter high schools for
the first time, on the other hand, we would not have been able to rely
on administrative records for test score data. Thus, to avoid the cost
and burden on respondents of administering a test to sample members as
part of the study, we decided to examine charter middle schools in this
study.

In addition, the schools are being selected from among the eleven states
with the largest number of charter middle schools, plus one additional
state chosen to increase the number of charter schools in the evaluation
from states that provide relatively low levels of autonomy to charter
schools. In particular, we developed a measure of the degree to which
charter schools enjoy autonomy in a given state. This autonomy measure
was created as the sum of the following five individual components,
primarily reflecting state charter school policies:

1\. Whether charter schools are exempt from most state laws governing
public schools

2\. Whether charter schools are exempt from public school collective
bargaining agreements

3\. Whether charter schools are exempt from teacher certification
requirements

4\. Whether charter schools are legally independent of the public school
district in which they are located

5\. Whether charter schools authorized by entity other than their local
public school district

If the sum of these components (that is, the autonomy index) for charter
schools in a given state was 0 or 1, the state was labeled as a "Low
Autonomy" state. If the autonomy index was 2 or 3, it was a "Medium
Autonomy" state. And if the index was 4 or 5, it was a "High Autonomy"
state. Among the eleven states with the largest number of charter middle
schools, two were labeled as low, six were medium, and three were high
autonomy states. Thus, we included an additional state (the next largest
low autonomy state) in the evaluation.

As a check on our autonomy index, we also examined the "state grades"
for charter school policies from the Center for Education Reform (CER),
obtained at http:\\\\www.edreform.com. We focused on the five components
most closely related to charter school autonomy. After summing these
components, we adapted the CER's "grading scale" and assigned a grade
for the state's policies related to charter school autonomy. States
receiving an 'A' were treated as high autonomy states, while those
receiving a 'B' were treated as medium autonomy states and those
receiving a 'C' were low autonomy states.

A list of the states chosen for the evaluation, the number of charter
middle schools in the state as of the 2001-2002 school year (based on
data from the National Center for Education Statistics' Common Core of
Data, or CCD), their autonomy classification according to both our
autonomy index and the CER grade, and the number of charter middle
schools targeted for inclusion in the evaluation are shown in Table 8.

The number of charter middle schools to be selected for the evaluation
was chosen according to two main criteria. First, we wanted to select an
approximately equal number of charter schools from low, medium, and high
autonomy states. This criterion allows us to examine the relationship
between charter school autonomy and charter school impacts more
precisely than if charter schools were selected from each category in
proportion to their numbers across the country. Second, we wanted to
minimize the number of charter schools selected from states in which the
two autonomy categories do not agree, since these are the states in
which our uncertainty about actual charter school autonomy is
greatest.[^6]

TABLE 8

STATES FROM WHICH CHARTER SCHOOLS WILL BE SELECTED FOR EVALUATION

+----------------+--------------+-----------+----------+--------------+
| State          | Total Number | Autonomy  | CER      | Number of    |
|                | of Charter   | Index     | Autonomy | Charter      |
|                | Middle       |           | Grade    | Middle       |
|                | Schools      | Category  | Category | Schools to   |
|                |              |           |          | be Selected  |
|                |              |           |          | for          |
|                |              |           |          | E            |
|                |              |           |          | valuation^a^ |
+----------------+--------------+-----------+----------+--------------+
| AZ             | 72           | High      | High     | 9            |
+----------------+--------------+-----------+----------+--------------+
| CA             | 76           | Medium    | Medium   | 4            |
+----------------+--------------+-----------+----------+--------------+
| TX             | 69           | High      | Low      | 4            |
+----------------+--------------+-----------+----------+--------------+
| FL             | 50           | Medium    | Medium   | 3            |
+----------------+--------------+-----------+----------+--------------+
| MI             | 30           | Medium    | High     | 3            |
+----------------+--------------+-----------+----------+--------------+
| WI             | 35           | Low       | Low      | 10           |
+----------------+--------------+-----------+----------+--------------+
| NC             | 17           | High      | Medium   | 4            |
+----------------+--------------+-----------+----------+--------------+
| MN             | 19           | Medium    | High     | 3            |
+----------------+--------------+-----------+----------+--------------+
| PA             | 17           | Medium    | Medium   | 3            |
+----------------+--------------+-----------+----------+--------------+
| MA             | 18           | Medium    | Medium   | 3            |
+----------------+--------------+-----------+----------+--------------+
| CO             | 14           | Low       | Medium   | 4            |
+----------------+--------------+-----------+----------+--------------+
| NM             | 9            | Low       | Low      | 3            |
+----------------+--------------+-----------+----------+--------------+
| **Total**      | **409**      |           |          | **53**       |
+----------------+--------------+-----------+----------+--------------+

^a^ We targeted more than 50 charter schools in the expectation that
some of the schools selected would end up dropping out of the study.

Within each state, we are selecting charter middle schools that have the
following characteristics:

-   The school expects to have a cohort of at least 15 to 20 students
    entering the school for the first time in the fall of 2005 or 2006

-   It has at least two years of experience operating as a charter
    school as of the beginning of the 2005-2006 or 2006-2007 school
    year.

-   It expects to have a waiting list with a substantial number of
    students

Based on data examined from the National Center for Education
Statistics' Common Core of Data (CCD), we counted the number of charter
middle schools in each of the twelve states in the evaluation as of the
2001-2002 school year. We then verified that these schools are still in
existence by gathering information from internet sources. The resulting
counts of the number of charter middle schools in each state that will
have at least two years of experience as of the 2004-2005 school year
are provided in Table 8. Based on CCD enrollment data, we expect that a
large majority of these schools will have more than 20 entering students
each year. The Table 8 counts suggest that there will be enough charter
middle schools in these states to meet our needs so long as a reasonably
large percentage of these schools have waiting lists. To maximize our
chances of successfully recruiting 50 charter schools into the study, we
are recruiting schools during both the 2004-2005 and 2005-2006 school
years.

The student universe consists of applicants to the eligible charter
schools who participate in the random assignment (or lotteries) for
admission.[^7] Random assignment is being conducted at the student
level, with student applicants randomly assigned into either a treatment
group (and accepted for admission into the charter school) or a control
group (not admitted to the charter school but able to attend other area
schools).

The average sample size in a given charter school site is roughly 30
treatment students and 30 control students, though there is variation
around these averages.[^8] Should a school (or its waiting list) not be
sufficiently large to generate a sample of this size, we will select
more than 30 treatment and 30 control students from some other site.
Thus, we anticipate a combined sample size across the 50 sites of 3,000
students, including 1,500 treatment students and 1,500 control students.
For outcomes based on school records data, we anticipate an 80 percent
response rate.

### 2. Statistical Methods for Sample Selection and Degree of Accuracy Needed

#### a. Stratification and Sample Selection

The student sample will be representative of students applying to and
participating in charter school lotteries at participating charter
schools for the 2005-2006 and/or 2006-2007 school years. We are not
using any stratification within a given charter school site, unless the
charter school already uses a stratified system of admitting new
students into the school. For example, if the "entry cohort" of new
students into the school includes students in more than one grade and
there are a fixed number of slots for these students in each grade, we
will stratify the random assignment process by grade level.

#### b. Estimation Procedures

The plans for the statistical analysis of the data are presented in
Section A.16.

#### c. Degree of Accuracy Needed

In order to assess appropriate sample sizes for the evaluation, we adopt
a precision standard using impact results found in other evaluations. In
examining past research on charter school impacts, it is important to
realize that these impacts may be positive or negative. Among three
well-designed non-experimental studies (Bifulco and Ladd 2004; Hanushek
et al. 2002; Solmon et al. 2001), two found negative impacts and one
found a positive impact. Solmon et al. (2001) found a positive impact of
charter schools of 0.12 of a standard deviation on reading tests in
Arizona. By contrast, Hanushek et al. (2002) found a negative effect
size of 0.17 for math tests in Texas and Bifulco and Ladd (2004) found a
negative effect size of 0.16 for math tests in North Carolina. Since we
would expect an effect size of 0.12 to 0.17 to be sufficiently large to
be of substantive interest to educators and policy-makers, we have set
our target minimum detectable effect size (MDE) for the overall impact
of charter schools to be 0.15, or roughly in the middle of this range.

Table 9 displays the minimum detectable impacts on test scores measured
in effect size units at 80 percent power at a 5 percent level of
statistical significance. These calculations incorporate design effects
due to clustering at the school level. On the basis of findings from
previous education-related evaluations, we assume a value for the
coefficient of intraclass school-level correlation
(![](media/image9.wmf)) of 0.085. We also take into account that not all
students assigned to the treatment group will attend the charter
school---we assume a "show rate" of 80 percent---and that we will not be
able to collect data from all students---we assume an 80 percent
response rate. Other assumptions are displayed at the bottom of the
table.

TABLE 9

MINIMUM DETECTABLE EFFECT SIZES

+------------------------+-------------+------------+-----------------+
|                        | Total       | Sample     | Minimum         |
|                        |             | with       | Detectable      |
|                        | Sample      |            |                 |
|                        |             | Data       | Effect Size     |
+========================+=============+============+=================+
| Overall Sample         | 3,000       | 2,400      | 0.15            |
+------------------------+-------------+------------+-----------------+
| 50% Subgroup           | 1,500       | 1,200      | 0.21            |
|                        |             |            |                 |
| (25 sites; 60 students |             |            |                 |
| per site)              |             |            |                 |
+------------------------+-------------+------------+-----------------+
| 50% Subgroup           | 1,500       | 1,200      | 0.18            |
|                        |             |            |                 |
| (50 sites; 30 students |             |            |                 |
| per site)              |             |            |                 |
+------------------------+-------------+------------+-----------------+
| 33% Subgroup           | 1,020       | 816        | 0.26            |
|                        |             |            |                 |
| (17 sites; 60 students |             |            |                 |
| per site)              |             |            |                 |
+------------------------+-------------+------------+-----------------+
| 33% Subgroup           | 1,000       | 800        | 0.20            |
|                        |             |            |                 |
| (50 sites; 20 students |             |            |                 |
| per site)              |             |            |                 |
+------------------------+-------------+------------+-----------------+

Note: We assumed the following for the power calculations: a two-tailed
test at 80 percent power and a 5 percent level of statistical
significance, an R^2^ value of 0.50, the proportion of total variance
accounted for by between-school variance (![](media/image9.wmf)) of
0.085, and a correlation between treatment and control means within a
site (*corr*) of 0.35. The calculations are based on a design with J=50
charter school sites and n=30 treatment group and 30 control group
members per site. We assumed that we would obtain test score data for 80
percent of the sample and that 80 percent of those randomly assigned to
the treatment group would actually attend the charter school. The
equation we used to calculate MDEs was:

$\text{MDE} = 2\text{.}\text{802}\sqrt{(1 - R^{2})\left\lbrack \frac{2\rho(1 - \text{corr})}{J} + \frac{2(1 - \rho)}{\text{nJ}} \right\rbrack} \ast \left( \frac{1}{\text{showrate}} \right)$

This design meets our target MDE of 0.15 for the overall sample. The MDE
for a subgroup varies in the range of 0.18 to 0.26 depending on the size
of the subgroup and whether it is selected across all sites or consists
of all sample members at a limited number of sites. Although these
subgroup MDEs are higher than the target MDE for the overall sample, we
suspect that impacts for some subgroups may well be larger than the
overall impact. For example, certain groups of charter schools may have
stronger impacts than others, and thus, the impact of these charter
schools would be larger than the overall impact.

#### d. Unusual Problems Requiring Specialized Sampling Plans

We do not anticipate any unusual problems requiring specialized sampling
procedures.

#### e. Use of Periodic Data Collection Cycles to Reduce Burden

With the exception of the school records data (discussed in our previous
OMB submission), all forms of data collection at any given site will be
one time only.

#### 3. Methods to Maximize Response Rates and Deal with Nonresponse

Obtaining high response rates in the Impact Evaluation of Charter School
Strategies will be critical to the success of the study. It will be
particularly important to obtain response rates that are not only high
overall, but that are approximately equal in the treatment and control
groups. This will be challenging due to the fact that while most of the
treatment group will presumably be in the charter school, control group
students will potentially attend a number of different schools, and the
identities of these schools will not be known in advance. In this
section, we discuss steps taken to obtain high response rates in both
the treatment and control groups with respect to the collection of the
survey data.

#### a. Parent and Student Surveys

A key to the successful collection of parent survey data will be that we
will have already obtained parental consent for the student's
participation in the study. Students whose parents did not give consent
will not be in our sample, and thus their parents will not be approached
to complete the parent survey. Those who have provided consent will have
indicated their willingness to participate in the study and for their
child to participate in the study. Nevertheless, we will take a number
of steps to ensure a high response rate among these groups.

To successfully interview sample members and their parents, we need to
be able to locate them. Our best source of information on their location
will be the contact information they provided on the baseline
information form. Should this information be incomplete or out of date,
however, we will use information on student mobility across schools that
we collect in our efforts to obtain administrative records data. As
described in our previous OMB submission, we will track student mobility
across schools using regular communications with each student's last
known school of attendance to confirm that student's enrollment or
obtain their transfer information. In particular, two times a year we
will submit lists of students in the sample to their last known school
of attendance, requesting confirmation that the students remain enrolled
there. For students no longer enrolled at that school, we will request
any transfer record information schools have to determine the student's
current school. We then will use the information on the student's
current school to obtain contact information for the student and his or
her parents.

Once we have located a sample member, our approach to conducting the
parent and student surveys will involve coordinating our effort to
interview the parent and that parent's child (who is the student in our
sample). We will attempt to conduct the parent survey first. At the
conclusion of the parent survey, we will ask whether the student is
available to complete their interview. If they are not available, we
will attempt to schedule a time when the student in likely to be
available and have the time to complete the interview. We will then
follow up with the student at that time. At the beginning of the student
survey, we will introduce the purpose of the survey and obtain their
assent to proceed with the interview.

A key aspect of our approach to these surveys will involve providing
parents with information about the study and about the parent and
student surveys in advance of our attempts to contact them and conduct
the survey by telephone. We will send an advance mailing to them that
includes a letter reminding them of the study, describing the purposes
of the surveys, and noting the steps that will be taken to ensure the
confidentiality of their responses. This mailing will also include the
glossy brochure we have developed for the study that may answer some of
their basic questions and a toll-free number to call if they have
additional questions.

Another key to minimizing nonresponse among study participants will be
the use of experienced and highly skilled interviewers. Interviewers
hired for this study will be selected based on their experience
conducting telephone interviews with similar populations. Preference
will be given to telephone interviewers who have worked on other studies
that have educational topics as a focus. For the student surveys, we
will seek interviewers with experience in conducting surveys of
children. Bilingual interviewers will be hired where there is likely to
be a concentration of non-English speaking respondents.

Interviewers will also receive extensive training. They will receive
training about the study in general and about gathering data on the
topics covered during the parent and student surveys in particular. As
part of the training, interviewers will be asked to complete practice
exercises using CATI prior to the start of interviewing.

Obtaining cooperation from parents on the questions about their
involvement in their child's education, their satisfaction with the
child's school, and the behavior of their child are very important to
the success of the study. Many of these parents' children will be
attending schools other than the charter schools being evaluated, and
thus may feel little connection to the study. Respondents will be
offered \$10 to complete the survey, which we feel will help improve the
response rate.

Several other techniques will be used to minimize nonresponse. We will
contact respondents at different times of day and different days during
the week to maximize our chances of getting in touch with them. As noted
previously, all respondents will be assured of the confidentiality of
the data they provide. We will also use standard locating procedures if
we are unable to locate parents and students at their last known phone
number. The survey will be conducted using computer-assisted telephone
interviewing software, which will ensure that all questions are asked
with the appropriate prompts and that skip patterns are followed. The
computer programs also make the interviews go faster and thus reduce
burden.

#### b. Principal, Authorizer, and State Survey

Obtaining high response rates on the principal, authorizer, and state
surveys will also be challenging, since the respondents on these
surveys, particularly the charter school principals, have highly
demanding jobs and may already feel overworked. Working with the
principals of traditional public schools will be challenging due to the
fact that these schools will not have a direct connection to the
study---they are not the schools being evaluated. In addition, these
principals may feel a strong aversion to charter schools, since these
schools may draw students---and the associated funding for these
students---away from traditional public schools. Our approach to
maximizing the response rates on these surveys will include:

-   Providing respondents with sufficient information in advance about
    the study in general and their surveys in particular,

-   Giving respondents flexibility to gather the information being
    requested and completing the survey according to their own schedule,

-   Giving respondents the flexibility to either complete a hard copy of
    the survey and mail in the results, to complete the survey by phone
    by calling a toll-free number, or to complete the survey by phone by
    responding to a call from an MPR interviewer, and

```{=html}
<!-- -->
```
-   Providing a small payment for their time completing the survey.

As with the parent survey, we will send each of these respondents an
advance mailing with information about both the overall study and the
survey they will be completing. The mailing will include a letter
detailing the data collection effort, including information on data
confidentiality; a letter of support for the study from the U.S.
Department of Education; and a glossy brochure with basic information
about the study and a toll-free number to call with questions. The
letter in this mailing will indicate that we will send them another
mailing with the survey to be completed.

To accommodate respondents' busy schedules, we will build flexibility
into the survey in two main ways. First, because the primary means by
which we expect respondents to complete the survey is by mail, they will
have time to gather the necessary information and complete the survey
according to their own schedule (as opposed to having to complete it
whenever we happen to reach them by phone). Second, we will give them
the flexibility to either complete the survey by hand and return it to
us in the mail or to complete it over the phone by calling a toll free
number at their convenience. For respondents who have not completed a
survey within a reasonable time frame, we will attempt to contact them
by phone and either complete the survey by phone when we reach them or
ask them to complete the hard copy version of the survey and return it
to us in a pre-paid envelope that we have provided for them.

As with the student and parent surveys, obtaining data from respondents
on the principal, authorizer, and state surveys will be critical to the
success off the study. These surveys will also present challenges that
we will not face in the student and parent surveys. For example, we
anticipate that the principal, authorizer, and state surveys will take
30 minutes to complete, compared with 10 to 15 minutes for the student
and parent surveys. And unlike the parents, most of the respondents on
the principal, authorizer, and state surveys will not have previously
given us consent for their participation in the study, and some will not
have previously been aware of the study. For these reasons, we plan to
provide an up-front payment of \$10 to principals to encourage their
completion of the survey, and to have \$10 payments available for
refusal conversion.

### 4. Test of Procedures and Methods to be Undertaken

In designing the surveys, we drew heavily on questions used in prior
studies; many with previous OMB clearance. The parent survey instrument
is based upon instruments administered during baseline data collection
on the National Evaluation of Charter Schools (OMB no. 1850-0799,
expiration date 1/31/2008), and on the Evaluation of the New York City
School Choice Scholarships (NYC School Choice) which was not federally
funded. The student survey instrument is based upon instruments
administered in the 21st Century Student Follow-Up Survey: After School
Activities (2001-2002) as part of the Evaluation of the 21st century
Community Learning Centers Program (OMB No: 1875-0183, expiration date
08/31/03), the Educational Longitudinal Study of 2002 - Student
Questionnaire, Base Year, 10th Grade (OMB No: 1850-0652, expiration date
06/30/2003), as well as the School Choice Scholarship Student
Questionnaire (1999-2000) as part of the NYC School Choice study. The
principal survey instrument is based largely upon instruments
administered in the Schools and Staffing Survey (OMB No. 1850-0598)
including the School Questionnaire, 2003-04 School Year (approval
expires 08/31/2004); the Public Charter School Questionnaire, 1999-2000
School Year (approval expires 09/30/2001); the Principal Questionnaire
2003-04 School Year (approval expires 08/31/2004); and the National
Longitudinal Study Of No Child Left Behind: Principal Survey, 2004-2005
School Year (OMB No: 1875-0227).

The State Accountability Office questionnaire is largely based upon the
Survey of Charter School Accountability and Flexibility sponsored by the
United States General Accounting Office (GAO). The charter school
authorizer survey instrument also includes items from the GAO survey.

**Pre-Test Procedures and Sample.** The parent and student surveys were
pretested by telephone on hard-copy questionnaires using respondents
from schools that had been recruited for the study, but later deemed
ineligible. This approach had the added benefit of contacting
respondents who had previously signed consent forms as part of the
baseline process. The principal pretest sample was partially comprised
of principals from ineligible charter schools, and referrals to public
schools principals.[^9] The principal survey was conducted using two
different approaches to mirror the planned mail with telephone interview
follow up methodology. The first approach entailed mailing the
questionnaire to principals and asking them to complete it and then
engage in a debriefing interview. The second approach utilized telephone
interviewers administering the questionnaire to principals.

Each pretest interview was monitored to determine if the response to
questions was consistent with the intent of each question. The pretest
also enabled MPR to check the accuracy of the internal skip patterns of
the surveys, instructions to interviewers, the ease with which
interviewers could read question wording, and the apparent ability of
respondents to follow question wording and sequence. Project staff
monitored all pretest interviews and took extensive notes on question
wording, skip logic, timing, and overall flow of the instruments.
Additionally, the principal debriefing interview was conducted by a
member of the project staff.

**Pre-Test Results.** As a result of the pretest interviews, several
question wording modifications and sequence changes were made to the
instruments. The pretest for the parent questionnaire resulted in minor
clarifications in the questionnaire, with no major changes necessary.
The parents appeared to understand the questions and were able to
provide answers for all questions they were asked. One point of
confusion was the response to questions on frequency such as "how many
times during the school **year**" and "how often/how many times during a
typical **month** in the school year." To resolve this issue both the
response options and question wording were made clearer and response
options were changed so they are now read to the respondent.

The student questionnaire also resulted in only minor modifications. The
scales throughout the questionnaire were at first excessively
repetitive, which was unnecessary for the respondents and added length
to the interview time. In response, we modified most of the transitional
statements. On the scale questions we were also concerned about
acquiescence or students providing socially desirable responses. We
tested the 5 point scales as well as 10 point scales with only the end
points defined and concluded that the 5 point scales worked well.
Finally, for the satisfaction questions, we changed from a 4 point "very
satisfied to very dissatisfied" scale to asking the students to give a
grade to each aspect of the school; a concept we felt they would be able
to relate to more closely. This also worked well.

Several areas were modified in the principal questionnaire so that
research objectives could more accurately be met. For example, we added
\"the state\" as a response category to the questions dealing with
school policies. We also changed the time periods on several questions
from this school year (a time period that would vary depending on when
the principal was interviewed) to "in a typical school year". We also
clarified several questions and response categories in order to make the
questionnaire easier to understand and respond to. For example, more
examples were provided under other specify categories in light of some
responses and we made more use of bolding and underlining of key words
and phrases that would focus the intent of the question.

Interview administration time in the pretest, adjusted for the expected
efficiencies to be achieved in CATI operations for the parent and
student surveys, was found to be consistent with the survey plans.
Administration time for the pretest interviews averaged 19.7 minutes for
the parent questionnaire and 18.3 minutes for the student questionnaire.
However, the parent and student pretests were conducted using a hardcopy
version of the questionnaire, which requires interviewers to follow skip
pattern instructions and other instructions for pre-filling certain
items from earlier responses. Based on experience in other surveys, we
are therefore estimating that the average administration time when the
surveys are converted to a CATI program will be around 15 minutes.

The principal survey that was mailed and self-administered was reported
to take 30 minutes and those completed by telephone averaged 28.6
minutes (20.5 for public school principals and 34 minutes for charter
school principals). Although the principal surveys conducted by
telephone will not be conducted using CATI, efficiencies will still be
achieved once the hardcopy questionnaire is designed to be "interviewer
friendly" and once interviewers achieve a learning curve and become
familiar with the skip instructions.

### 5. Individuals Involved

The statistical aspects of the design have been reviewed thoroughly by
staff at the Institute of Education Sciences and Office of Innovation
and Improvement, as well as by members of the study's expert panel
listed in Section A.8. The following individuals have worked closely in
developing the statistical procedures and will be responsible for data
collection and data analysis.

  ------------------------ ------------------------ ---------------------
  Name                     Title                    Telephone

  Mark Dynarski            MPR Senior Fellow        609-275-2397

  Phil Gleason             MPR Senior Researcher    315-781-8495

  Rita Stapulonis          MPR Senior Survey        609-275-2363
                           Researcher               

  David Myers              MPR Chief Strategy       202-484-4523
                           Officer                  

  Robert Olsen             MPR Senior Researcher    202-484-4223

  Allen Schirm             MPR Senior Fellow        202-484-4686

  Marsha Silverberg        IES Economist            202-708-7178

  Mark Turner              OSG President            443-451-7060
  ------------------------ ------------------------ ---------------------

[^1]: In addition to the study not measuring the broader impacts of
    charter schools---impacts on outcomes among students in traditional
    public schools---the possibility of these broader impacts lead to
    the possibility that the study will understate the impacts of
    charter schools on the students who attend charter schools. If
    charter schools positively influence outcomes among students in
    other nearby schools, then the usual assumption that control
    students (who are attending these nearby schools) are not influenced
    by the treatment is violated. In this case, the estimated impact
    will reflect the true impact of attending a charter school minus the
    (positive) spillover effect of the charter school on control group
    members in other nearby schools.

[^2]: In its simplest form, after a series of additional assumptions
    about the nature of the impact being estimated have been made, the
    TOT estimator adjusts the ITT impact estimate by an amount inversely
    proportional to the percentage of students who comply with the study
    protocol---by attending the charter school if in the treatment group
    and not attending the charter school if in the control group. In
    practice, we will use an instrumental variables approach to generate
    this TOT impact estimate, as described in Angrist, Imbens, and Rubin
    (1996).

[^3]: Since different numbers of control students may be attending
    different comparison schools, we would use a weighted average of the
    number of students in the comparison school. For example, if 20 of
    the 30 control students attended comparison school A and the
    remainder attended comparison school B, the size of the average
    comparison school would be calculated as 0.67\*(size of school A) +
    0.33\*(size of school B).

[^4]: We plan to define what constitutes an innovative approach based on
    information collected from the site visits and principal interviews
    and with substantial input from our advisory panel.

[^5]: This data collection is being conducted primarily through site
    visits and the principal survey

[^6]: Once we get data from charter school principals and authorizers,
    we should be able to resolve some of the uncertainty about the
    autonomy level of charter schools in particular states.

[^7]: Charter schools typically admit some applicants---such as siblings
    of current students---automatically, without requiring them to
    participate in an admissions lottery.

[^8]: In addition, these students will be spread across two school years
    in some sites.

[^9]: The principals' schools were at least ten miles away from any of
    the current study schools to avoid the risk of being included in the
    study as a control school.