Implementation of Title I/II Program 

Initiatives

Supporting Statement for Paperwork Reduction 

Act Submission

PART B: Collection of Information Employing 

Statistical Methods

Contract ED-IES-11-C-0063

June 2013

Prepared for:

Institute of Education Sciences

U.S. Department of Education

Prepared by:

Westat

Mathematica Policy Research
i

Table of Contents

Introduction ......................................................................................................................................1

Overview of the Study........................................................................................................................1

B.1. Respondent Universe and Sampling Methods............................................................................3

B.1.1. State Sample ....................................................................................................................... 3

B.1.2. School District Sample ........................................................................................................3

B.1.3. School Sample.....................................................................................................................4

B.1.4. Teacher Sample...................................................................................................................5

B.2. Information Collection Procedures............................................................................................6

B.2.1. Notification of the sample, recruitment and data collection.............................................6

B.2.2. Statistical Methodology for Stratification and Sample Selection.......................................8

B.2.2.1. Nationally representative sample of school districts..................................................8

B.2.2.2. Nationally representative sample of schools............................................................12

B.2.2.3. Nationally representative sample of teachers..........................................................17

B.2.3. Estimation Procedures......................................................................................................23

B.2.4. Degree of Accuracy Needed .............................................................................................24

B.2.5. Unusual Problems Requiring Specialized Sampling Procedures.......................................25

B.2.6. Use of Periodic (less than annual) Data Collection to Reduce Burden.............................25

B.3. Methods to Maximize Response Rates ....................................................................................25

B.3.1. Weighting the district, school, and teacher samples........................................................26

B.4. Test of Procedures..................................................................................................................26

B.5. Individuals Consulted on Statistical Aspects of Design..............................................................27

References.......................................................................................................................................28

Appendix F Notification Letters ......................................................................................................F-1

Appendix G Study Brochure........................................................................................................... G-1
ii

Tables

B - 1. Definitions of district size strata ......................................................................................9

B - 2. Proposed stratification design for district sampling........................................................9

B - 3. Properties of proposed stratification design for district sampling................................10

B - 4. Proposed schools span-size subgroups for differential sampling rates.........................13

B - 5. Subgroups for the school sample design.......................................................................14

B - 6. Expected school sample sizes for the subgroups in the school sample design.............15

B - 7. Power properties of school design ................................................................................16

B - 8. Effective sample sizes for national school-level estimates under the 3/4th root 

school design .................................................................................................................16

B - 9. Expected teacher sample sizes......................................................................................18

B - 10. Effective sample sizes for national teacher estimates ..................................................19

B - 11. Sample sizes and effective sample sizes for teachers for elementary, middle, and 

high schools ...................................................................................................................20

B - 12. Sample sizes and effective sample sizes for teachers for elementary, middle, and 

high school grade spans.................................................................................................20

B - 13. Selected MDES for teachers by elementary, middle, and high school grade spans......23
1

Part B. Collection of Information Employing Statistical Methods

This package is the first of two for the Implementation of Title I/II Program Initiatives study. This 

package requests approval for an initial round of recruitment and data collection that will include 

surveys of all states and nationally representative samples of school districts, schools, and Kindergarten 

through 12th grade teachers. We anticipate that the state education agency (SEA) and school district 

surveys will begin in November 2013, and the principal and teacher surveys will begin in January 2014. 

The second package will request approval for the follow-up survey, which will survey the same states, 

districts, and schools as well as a new nationally representative sample of teachers within the sampled 

schools during the 2015-16 school year. 

Title I is one of the U.S. Department of Education’s (ED) largest elementary and secondary education 

programs. Historically, Title I provides financial assistance to schools and districts with a high percentage 

of students from low-income families to help these students increase achievement. Title I also includes 

requirements that states hold schools and districts accountable for improvements in student 

achievement. During the 2009-2010 school year, more than 56,000 public schools used Title I funds, and 

the program served 21 million children (U.S. Department of Education, 2012a). Title II provides funds to 

increase academic achievement by improving teacher and principal quality including educator 

preparation and professional development, as well as providing funds for class size reduction. An 

estimated 95 percent of districts nationally received Title II, Part A funding for the 2011-12 school year 

(U.S. Department of Education, 2012b). 

The last National Assessment of the Title I program concluded in 2006. Since that time, there have been 

changes in Title I provisions, such as allowing states to incorporate proficiency improvement (alongside 

proficiency levels) into school accountability measures and providing more resources to the lowest 

performing schools through the expansion of eligibility and funding for School Improvement Grants 

(SIG). Title II guidance allows more flexibility for certain teachers (e.g., special education teachers) to 

meet the standards to become highly qualified teachers (HQT). 

The most recent change related to Title I and Title II was the introduction of the Elementary and 

Secondary Education Act (ESEA) Flexibility requests in 2011, allowing states to waive a number of 

provisions in exchange for a commitment to key reform principles. The Title I/II study will provide policy 

makers with detailed information on how these initiatives are playing out in states, districts, schools, 

and classrooms. 

The Implementation of Title I/II Program Initiatives study is commissioned by the Institute of Education 

Sciences (IES), ED’s independent research and evaluation arm. The Title I/II study will examine the 

implementation of policies promoted through ESEA at the state, district, school, and classroom levels. 

Through surveys at each level and selected extant data and documents, the study will provide 

Introduction

Overview of the Study
2

information on activities in four core areas: state content standards, aligned assessments, accountability 

and school turnaround, and development of effective teachers and leaders.

The study will reflect changes in Title I and Title II provisions since the last National Assessment of the 

Title I program concluded in 2006 (including ESEA Flexibility provided to states with approved requests). 

In addition, this study will supplement findings from ED’s annual survey of a nationally representative 

sample of districts on their uses of Title II, Part A funds. This study will provide more detail on districtimplemented workforce development policies and practices for educators as well as teachers’ and 

principals’ access to and receipt of professional development.

State content standards

State content standards have been a foundational element of ESEA since the 1994 reauthorization of the 

Act, which required states to establish statewide standards in reading and mathematics in the 3 through 

5, 6 through 8, and high school grade ranges and implement statewide accountability systems for 

evaluating school-level performance. The No Child Left Behind Act of 2001 (NCLB), the subsequent 

reauthorization of ESEA, extended and strengthened these concepts, incorporating science alongside 

reading and math, and establishing more prescriptive rules for state accountability systems. More 

recently, states approved for ESEA Flexibility have agreed to adopt especially rigorous standards; over 40 

states have adopted the Common Core State Standards (CCSS). 

Aligned assessments

The 1994 reauthorization of ESEA required states to administer annual academic assessments aligned to 

content standards in reading and mathematics in at least one elementary grade, one middle grade, and 

one high-school grade. In 2001, NCLB extended this requirement from three grades to seven (grades 3-8 

and at least once in high school) and added a requirement for science assessments in each of the 3-5, 6-

8, and high school grade ranges. 

These assessments are meant to provide information on whether students have acquired the knowledge 

and skills identified by the standards and, if not, the areas where students fall short and may require 

additional instruction and assistance. Annual assessment data are also used to hold schools and districts, 

and increasingly teachers, accountable for student performance. The content and rigor of these 

assessments vary across states.

Accountability and school turnaround

NCLB standardized state accountability policies by requiring states to establish “adequate yearly 

progress” (AYP) goals and objectives for student proficiency on the mandated assessments, with targets 

rising over time so that by 2014, all students would be proficient. Schools and school districts are held 

accountable for achieving rates of student proficiency in reading and mathematics that meet the annual 

objectives, as well as for the proficiency rates of specific subgroups of students. Schools that failed to 

meet AYP for either all students or a subgroup of students over successive years were subject to an 

increasingly aggressive set of interventions spelled out in the law, beginning with parental choice and 

culminating with school closing or state takeover.
3

The federal government has adjusted accountability requirements since the last National Assessment. 

The Growth Model Pilot Project, initiated in 2005, permitted nine states to use accountability measures 

that incorporate proficiency improvement (rather than only proficiency levels) under a pilot in 2007-08. 

When the approach was written into regulations in 2008, 15 states participated. The ESEA Flexibility 

regulations introduced in 2011 allow states to develop more flexible systems of school accountability 

that incorporate measures of the achievement growth of individual students, broaden achievement 

measures beyond mathematics and reading, and include other student outcomes such as graduation 

rates. In addition, recent policy changes have led to more intensive focus on the lowest-performing 

schools and differentiated support for schools depending on their performance levels

Development of effective teachers and leaders 

Improving and supporting the effectiveness of teachers and school leaders is a key part of any strategy 

for improving student achievement and college and career readiness. Research has shown that teachers 

(e.g., Rowan, Correnti, & Miller, 2002; Rockoff, 2004; Rivkin, Hanushek, & Kain, 2005) and school 

administrators (e.g., Witziers, Bosker, & Kruger, 2003) have measurable effects on student achievement.

Title I and Title II incorporate a number of requirements and incentives for states and districts to 

implement policies that improve certification, support educators during all phases of their careers, hold 

them accountable for contributing to student achievement, and ensure their equitable distribution—all 

based on the premise that improvement in student achievement requires improving educator quality. 

However, there has been substantial development in the national discourse about teacher quality since 

the passage of the NCLB and the initiation of the last National Assessment of Title I.

Most importantly, thinking about educator quality has shifted away from educator inputs (e.g., degrees 

and certifications) toward educator performance, defined in terms of practice and especially in terms of 

contribution to growth in student achievement. This shift has been facilitated by the improvements in 

educator practice assessments (Porter, Youngs, & Odden, 2001; Milanowski, Heneman & Kimball, 2011) 

and in “value-added” methods to measure school and teacher contributions to student achievement 

(Meyer, 1997; Harris, 2009; Glazerman et al., 2010). 

The study sample will include the universe of states and the District of Columbia, and nationally 

representative samples of districts, schools, and teachers. The school sample will be nested in the 

district sample and the teacher sample will be nested in the school sample.

B.1.1. State Sample

We will survey all 50 states and the District of Columbia. 

B.1.2. School District Sample

A nationally representative sample of districts will be selected. This will provide unbiased estimators of 

district characteristics, and will provide the first stage of selection for nationally representative samples 

B.1. Respondent Universe and Sampling Methods
4

of schools and teachers. A nationally representative sample is necessary as Title I/II covers most of the 

U.S. public school educational system. 

We also are interested in statistically comparing the implementation of initiatives promoted by Title I 

and Title II by district level of poverty and size of districts based on student enrollment. Poverty is 

included because Title I is specifically intended to ameliorate the effects of poverty on local funding 

constraints and education opportunity. District capacity to implement initiatives also will be of interest 

for this study. Success in implementation of initiatives, particularly those that build upon each other 

such as having longitudinal data systems and identifying effective teachers, might be tied to district 

organizational capacity. Therefore, we also will examine implementation by district size. 

For the district sample, we will implement a “minimax” sample design that strikes a balance between 

producing efficient student-weighted estimates and efficient district-weighted estimates. This seems 

appropriate because the policy levers of Title I and Title II are directed at districts (and schools and 

states). For some purposes, it will be useful to understand the average experience of students across the 

country, but for other purposes, we will want to understand the experience and behavior of the average 

districts (and schools)—the units that the U.S. Department of Education (ED) expects its policies to 

immediately influence. The minimax design is a compromise between a design that is relatively efficient 

(i.e., allows estimates with narrow confidence intervals) for answering questions about the number or 

proportion of U.S. public school students in districts implementing initiatives of interest, and one that is 

relatively efficient for answering questions about the number or proportion of U.S. school districts 

implementing such initiatives.1 

To construct the sampling frame, we will use data primarily from the National Center for Education 

Statistics’ (NCES) Common Core of Data (CCD), with supplementary data from sources such as the U.S.

Bureau of the Census’s district-level SAIPE (Small Area Income and Poverty Estimates) program for 

school-district percentages of families with children in poverty. 

We will draw a district sample of 570 districts out of 15,629 school districts. See Table B - 2 in Section 

B.2 for the universe counts by stratification classifications.

B.1.3. School Sample

A nationally representative sample of schools will be selected. The school sample will be a two-stage 

sample, nested within the sampled districts. This will provide unbiased estimators of school 

characteristics, and will provide the second stage of selection for a nationally representative sample of 

teachers. 

In addition to examining initiatives in schools nationwide, we also will statistically compare policy 

implementation by school Title I status, school poverty level, and the cross-classification of Title I and 

poverty status (high-poverty Title I schools, low-/medium-poverty Title I schools, and non-Title I 

 

1

The minimax design differs from the one used for the last Title I study. The previous study selected districts probability proportional to size 

(PPS), with size measured by student enrollment. The PPS design is quite efficient for estimating the proportion of students enrolled in 

districts implementing policies of interest. However, when estimating the percent of districts implementing a policy, the PPS design is 

relatively inefficient, compared to a simple random sample. This is because relatively few small and medium-sized districts are included in a 

PPS design. This, in turn, requires the small and medium-sized districts in the sample to be given greater weight to better represent the 

population of districts nationwide and can lead to relatively wide confidence intervals around estimates of proportions of districts. 
5

schools).2The school sample is not limited to Title I schools. While a key part of the study focuses on 

initiatives promoted by Title I, the study is not looking exclusively at initiatives funded by Title I. 

Furthermore, non-Title I schools may benefit from professional development funded by district Title II 

funds. In addition, we anticipate that implementation status and types of initiatives may vary by school 

grade span (elementary, middle schools, and high schools). For example, implementation of state 

content standards and aligned assessments, as well as responses to accountability systems, likely differs 

by grade level. As a result, we will look for differences between Title I and non-Title I schools within 

grade span. There is variation in the poverty levels of Title I schools, with 37.5 percent of Title I schools 

considered high-poverty schools.3 Our analysis will examine whether implementation of initiatives is 

different in the neediest Title I schools

We will also implement a “minimax” sample design for the school sample. This will balance between the 

needs to produce efficient student-weighted estimates and efficient school-weighted estimates. To 

construct the sampling frame, we will use data from the CCD.

We will draw a sample of 1,300 schools nested within the nationally representative sample of 570

school districts. (The universe of schools is 92,589 schools.) See Table B - 5 in section B.2 for the universe 

counts by stratification classification.

B.1.4. Teacher Sample

A nationally representative sample of teachers will be selected. The teacher sample will be a three-stage 

sample, nested within sampled districts and schools. This will provide unbiased estimators of teacher 

characteristics. The sample will be drawn from a comprehensive list of teachers provided to us from 

each sampled school. 

We believe there will be considerable interest in the responses of teachers who teach classes in which 

students are tested for accountability requirements for ESEA4as they have been the focus of the federal 

accountability system. In addition, these teachers are most likely to be affected by improved measures 

of educator effectiveness based on student growth. Teachers in these subjects and grades are also the 

most likely to be affected by state-set achievable annual measurable objectives (AMOs) promoted by 

the ESEA Flexibility requests, and will most likely receive student growth data for use in improving 

practice and differentiating instruction (U.S. Department of Education, 2011). The sample will thus allow 

statistical comparisons of ESEA-tested and non-ESEA-tested teachers by school grade span on such 

topics as teachers’ use of student achievement, and especially growth in achievement, for evaluation, 

accountability, and practice improvement.

Our sample will also allow sufficient numbers of teachers per district and school to allow examination of 

the relationship between implementation of various initiatives promoted by Title I and Title II as 

described by principals and district officials and teachers’ reports of their experiences and responses. 

For example, we may examine whether teachers in states that provide more extensive support for state 

 

2

Since the majority of non-Title I schools are low/medium-poverty schools, these schools will not be broken out by poverty status.

3

The cutoff for high-poverty schools is the enrollment-weighted 75th percentile: this was 66.12 percent of students eligible for free or reduced 

lunch on an earlier frame.

4

Under ESEA, as reauthorized by the No Child Left behind Act (NCLB) of 2001, public school students are tested annually in reading/English

language arts and mathematics in grades 3 through 8, and once in high school. Students are tested in science once in each of the following 

grade groupings: grades 3-5, grades 6-9, and grades 10-12. 
6

content standards are more likely to report using the state content standards in their classrooms than 

teachers in other states.

The study will use an approach similar to the teacher sampling in the NCES Schools and Staffing Survey 

(SASS) to determine the overall number of teachers to select from each sampled school. In general, the 

SASS approach sets a teacher sample size for each sampled school using the teacher roster count from 

the school and the school’s probability of selection, in such a way that yields a “self-weighting” sample 

(each teacher in each school has an equal probability of selection within the three major school 

subgroups (non-Title I, Title I high poverty, Title I low poverty)). Effectively, the teacher sample size is 

selected in a way that cancels out any oversampling or undersampling of the school given its teacher 

count, bringing the teacher probabilities back to equality across schools within each major school 

subgroup, to the extent possible. This avoids large differences in the weight given to individual teachers, 

differences which complicate estimating standard errors and can reduce stability of estimates.

We will draw a sample of 9,100 teachers from the 1,300 sampled schools—an average of seven teachers 

per sampled school. See Table B - 9 in Section B.2 for the expected teacher sample sizes by school 

classification.

We expect all states and the District of Columbia to participate in the study. We expect to obtain at least 

an 85 percent response rate from districts, schools, and teachers.

B.2.1. Notification of the sample, recruitment and data collection 

Introduce the Study to State Education Agencies. We will begin by sending the chief state school 

officer and the state Title I administrator a notification letter (see Appendix F) and study brochure (see 

Appendix G) explaining the study, the importance of the state’s involvement, and the mandatory nature 

of the state’s response. We will then follow up with a phone call to the state Title I administrator to 

answer questions about the study and identify additional state-level respondents based on areas of 

expertise. Once we have secured the correct contacts, we will mail the Title I administrator a hard copy 

instrument with instructions to have other staff complete relevant sections.5

The mailing will include a version of the extant data form that will be pre-filled with the data that we 

have collected from publicly available sources. We will ask the respondents to review the form to 

confirm the accuracy of the data, correct any data that were not correct on the public website, and 

provide any data that were not available publicly.

The mailing will include a password and secure web address to access an electronic version of the 

questionnaire and instructions on how to submit the questionnaire via a secure SharePoint site. After 

the initial mailing, we will also send a follow-up email that also includes the web address and password. 

Project staff will monitor completion rates, review the instruments for completeness throughout the 

field period, and follow up by email and telephone as needed to answer questions and encourage 

completion. During these calls, state representatives will be given the option of completing the module 

 

5

If requested, the study team will contact each relevant state-level staff member directly to complete the survey.

B.2. Information Collection Procedures
7

by telephone with the researcher. Each of the four topic areas (state content standards, aligned

assessments, accountability and school turnaround, and development of effective teachers and leaders)

will on average require between 30 and 45 minutes to complete. 

Researchers knowledgeable about the four content areas will review the completed questionnaire and 

documentation downloaded from state and other publicly available web sites for completeness. We will 

then conduct follow up calls with states to clarify any ambiguous answers in the survey. 

Introduce the Study to District and School Leaders. We will send notification letters (see Appendix F)

and the study brochure (see Appendix G) by email and mail to sampled districts and schools. Notification 

letters will be customized for each type of respondent, informing them of the study’s importance and 

benefits. Sending both email and mail will increase the likelihood that addressees will receive our 

communications in a timely manner. States and districts receiving Title I and Title II funds have an 

obligation to participate in Department evaluations (Education Department General Administrative 

Regulations (EDGAR) (34 C.F.R. § 76.591)), and virtually all states and districts receive Title I and/or Title 

II funds. Since we are not able to discern which schools receive Title I and/or Title II funds, survey 

participation is not mandatory for school principals.

Mailings to district superintendents will include the names of the sampled schools. Because we 

anticipate that district surveys will require input from several key individuals, we will ask the district 

superintendent to provide contact information (including email) for a person designated by the 

superintendent as the study liaison, who will coordinate the completion of the survey by the 

appropriate staff. Once the district liaison is identified, we will conduct all follow-up directly with the 

district liaison. 

For those districts that do not respond and identify a study liaison online within five days, we will make 

follow-up telephone calls to district superintendents to confirm receipt of the letter, answer any 

questions, confirm the identity of the study liaison, and obtain the designated liaison’s contact 

information. We will follow all required procedures, and as necessary, we will obtain the approval of the 

district for principal and teacher participation through submission of the required research application.

Notification letters will not be sent to principals or teachers prior to district approval.

Principal mailings will begin once we have received any required district approval. The principal mailings 

will inform principals of their schools’ selection and also will include directions for completing the 

teacher roster via the web from which we will draw the teacher sample. Principals will be asked to verify

their schools’ current grade spans and list all teachers along with teachers’ email addresses, primary

grades and subjects taught, (e.g., reading/English/language arts, mathematics, science and other), and 

whether the teachers teach classes in which students are tested for accountability requirements under 

ESEA. Mockups of web pages that could be used to collect this information are shown in Appendix E

(Part A). Online edits in the web roster will ensure that all data items required for teacher sampling are 

entered. The principal mailing will include a hardcopy roster listing all of these items and a fax number 

and business reply envelope, should school staff prefer to submit hardcopy. We will also make available

at the school’s request an Excel template should the school staff prefer to enter teacher information in 

Excel. We will also accept roster output from the school’s database, either electronically or on hardcopy. 

We will work with the principal or his/her designee to be sure all fields required for teacher sampling are 

captured. As teachers are selected from a school, their notification letters will be mailed.
8

Administer Surveys. By email and mail, we will send a notification letter to all respondents (i.e., school 

principals, and teachers). The letters and emails will underscore the purpose of the study and the 

importance of participation. Letters and emails will be tailored to the respondents with the district 

communication informing district-level respondents that completing the web-based survey is mandatory 

and required by law. Principals will be informed that their participation is voluntary. We will emphasize 

the survey is an opportunity for principals to provide valuable information about how the policies of 

ESEA influence teaching and learning. Teachers also will be informed that their participation is voluntary 

and that they will receive $20 as a thank-you for their participation. In districts that require research 

applications for principal and teacher participation, we will send the principal and teacher emails once 

we obtain district approval to begin data collection. Mailed letters will include the survey URL and login 

information for responding to the survey as a web-based instrument. For security purposes, the first 

email will provide the URL and the User ID with the respondent’s password provided in a second email. 

All communications will include a toll-free study number and a study email address for respondents’ 

questions and technical support. Based on Westat’s experience on large-scale data collections, we will 

assign several trained research staff to answer the study hotline and reply to emails in the study 

mailbox. We will train them on the purpose of the study, the obligations of district respondents to 

participate in the evaluation, and the details for completing the web-based survey. Content questions 

will be referred to the study leadership. An internal FAQ document will be developed and updated as 

needed throughout the course of data collection to ensure that the research staff has the most current

information on the study.

The web will be our primary method of data collection for the district, school and teacher surveys. We 

will offer respondents the option of emailing them an electronic version of the survey (e.g., PDF or Word 

document) to complete and return by email or completing a paper-and-pencil instrument. However, we 

have found that the vast majority of users prefer the web-based approach. A phone survey option will 

be offered to respondents as part of the nonresponse follow-up effort. Since the web-based surveys will 

include data checks, we will use the web-based surveys to enter any surveys received on hard copy or by 

phone.

Westat will develop a web-based data monitoring system (DMS) to track the sample for each 

instrument, record the status of district approvals, generate materials for mailings, and monitor survey 

response rates.

B.2.2. Statistical Methodology for Stratification and Sample Selection

B.2.2.1. Nationally representative sample of school districts

The district sample will be stratified by poverty status and district size. The poverty strata are defined 

based on the percent of families with children in poverty. The high-poverty stratum consists of the 

roughly 25 percent of districts with percentages greater than the national 75th percentile.

6

The 

low/medium-poverty stratum consists of the complement set (roughly 75 percent of the districts). The 

district size strata are given in Table B - 1. It should be noted that for comparing adjacent classes, each 

 

6

This percentile is weighted by enrollment, and is found using the U.S. Bureau of the Census school district SAIPE (Small Area Income and 

Poverty Estimates). In a recent national district sample, we found that this 75th percentile was 16.7 percent of families in poverty. The exact 

percentile will be computed using the new frame when it is developed. We will utilize SAIPE in assigning districts to these poverty strata. 
9

class has an enrollment range roughly three times greater than the preceding class (in terms of 

minimums, mean value, or maximums). 

Table B - 1. Definitions of district size strata

District 

oversampling class

Lower bound district 

enrollment

Upper bound district 

enrollment

G 1 500

F 501 1,500

E 1,501 5,000

D 5,001 15,000

C 15,001 50,000

B 50,001 150,000

A 150,001 no limit

A total of 570 districts will be sampled, with oversampling of high-poverty districts by a factor of 3. This 

oversampling will bring the expected sample size for high-poverty districts in line with the expected 

sample size for low/medium-poverty districts (285 per each stratum). 

Table B - 2 presents the proposed strata for sampling districts, with relative sampling rates (as compared 

to the stratum with the lowest sampling rate). Note that the counts are based on the 2008-2009 schoolyear CCD frame, and may be slightly different when the final frame is developed. Note that under a 

probability proportionate to size by enrollment design, the relative sampling rates between neighboring 

district size classes would be 3, as that is roughly the enrollment ratio. By using powers of 1.87 rather 

than powers of 3 as relative sampling factors, we are oversampling the strata with the higher 

enrollments, but not to the full extent justified by the ratios of enrollment means. The largest nine 

low/medium-poverty stratum and the largest 25 high-poverty stratum districts are sampled with 

certainty (indicated with a relative sampling rate of infinity).

7

The exceptionally large size of these 

districts will make them larger than the sampling interval under the minimax design (described below), 

and they will be taken as certainties to maintain efficiency. 

Table B - 2. Proposed stratification design for district sampling

Poverty 

stratum

District 

size 

class

District 

count

Percent 

of 

districts

Relative 

sampling 

rate

Expected 

district 

sample 

size

Percent 

of 

district 

sample

Low/med G 3,669 23.5% 1.00 31.6 5.5%

Low/med F 3,163 20.2% 1.87 50.9 8.9%

Low/med E 3,048 19.5% 3.50 91.7 16.1%

Low/med D 1,131 7.2% 6.54 63.6 11.2%

Low/med C 348 2.2% 12.23 36.6 6.4%

Low/med B 55 0.4% 22.87 10.8 1.9%

Low/med A 9 0.1% Inf 9 1.6%

Low/med Total 11,423 73.1% 0.00 294 51.6%

 

7

These counts may change somewhat after reviewing the distribution of districts on the final frame, and their relationship to the sampling 

intervals.
10

High G 1,920 12.3% 3.00 49.6 8.7%

High F 1,046 6.7% 5.61 50.5 8.9%

High E 858 5.5% 10.49 77.4 13.6%

High D 268 1.7% 19.62 45.2 7.9%

High C 89 0.6% 36.68 28.1 4.9%

High B 19 0.1% Inf 19 3.3%

High A 6 0.0% Inf 6 1.1%

High Total 4,206 26.9% 276 48.4%

Total Total 15,629 100.0% 570 100.0%

This sample design we call a ‘minimax’ design, as it is designed to equalize the efficiency for two types of 

estimates. The first type of estimate counts each district as one in the population, so that the base 

weight is the inverse of the district probability of selection. This type of ‘count-based’ estimate answers 

questions such as “What percentage of districts have characteristic X?” The second type of estimates 

includes enrollment of the district, so that the sampling base weight is the enrollment divided by the 

probability of selection. This type of ‘enrollment-based’ estimate answers questions such as “What 

percentage of students are in enrolled districts which have characteristic X?” A probability proportionate 

to enrollment design will lead to optimal efficiency for the second type of estimate, but will have poor 

efficiency for the first type of estimate (as the district weights will be close to equality for the 

enrollment-based estimate, but will vary considerably for the count-based estimate). On the other hand, 

a simple stratified design with no oversampling of larger district-size strata will have high efficiency for 

count-based estimates, but poor efficiency for enrollment-based estimates. This ‘middle-ground’ design 

oversamples the higher enrollment district-size strata, but proportional to the 0.56 root8of the 

enrollment mean in the stratum, rather than to enrollment directly,

9

and will have reasonable efficiency 

for both count-based estimates and enrollment-based estimates (the design is set up to equalize the 

efficiency for both types of estimates, at the cost of not being as good for each type of estimate as the 

optimal design for that type of estimate). Table B - 3 summarizes the properties of this design as they 

would be if implemented on the 2008/2009 CCD frame.

10

Table B - 3. Properties of proposed stratification design for district sampling

Power Property

Enrollmentbased 

weight 

estimates

Countbased 

weight 

estimates

Effective sample size: All districts 286.9 318.3

Effective sample size: High-poverty districts 179.3 187.7

Effective sample size: Low/medium-poverty 

districts 184.7 183.1

Minimum detectable effect size (MDES) comparing 

Poverty District Strata 29.4% 29.1%

 

8

1.87 is the 0.56 root of 3. 

9

This design is close to a ‘square root’ design, except that it is stratified design rather than a fully PPS design (sampling rates are equal within 

strata), and the root used is slightly larger than 1⁄2. 

10 As processed to drop ineligible schools and entities, schools with no enrollment, etc.
11

The effective sample sizes are the sample sizes for a simple random sample which would provide the 

same precision as the design.

11

 Note that the effective sample size for all-district estimates is about half 

of the district sample size of 570. This large ratio is caused partially by the oversampling of high-poverty 

districts. Note also an equalization of effective sample sizes for the two types of estimates. This is the 

‘minimax’ aspect. The MDES (minimum detectable effect size) is computed for evaluating the null 

hypothesis of no difference between the high-poverty and the low/medium-poverty districts for a range 

of district-level characteristics.

12 This is a very important comparison at the district level for this study, 

and the sample design is carefully tailored with this comparison in mind (we effectively equate the 

sample designs for the two strata, though the high-poverty districts are only one quarter of enrollment). 

The sample design does achieve an MDES lower than 30 percent for both types of estimates.

13

The district frame will be based on the most recent NCES CCD frame, as processed through National 

Assessment of Educational Progress (NAEP) macros to purge entities that are not in scope (e.g., 

administrative districts, district consortiums, entities devoted to auxiliary educational services, etc.), as 

well as a canvassing of new districts from the preliminary CCD frame from the following year (only 

districts with positive enrollments on the CCD frame, with other out-of-scope entities purged out). All 

school districts and independent charter districts with at least one eligible school and at least one 

enrolled student will be included in the frame.14

Further Details of District Sampling. The primary strata are by district-size class and poverty status as 

described in the previous section. We also plan to define a stratum for small states (states with an 

expected district sample size less than 1), and carry out a systematic sample within this stratum by state 

in order to guarantee that every state has at least one selected district. The remaining complement 

stratum will be stratified by poverty status (high and medium/low) and district-size class as described 

above. Within these primary strata, there will be further implicit stratification by ordering the sample. 

This ordering will be defined by urbanicity, Census region, and finally district enrollment. 

The high-poverty stratum will be formally defined by SAIPE estimates of percentages of 5 to 17 year old 

children in poverty for the school district. We will compute the weighted 75th percentile percentage over 

all districts in the U.S., and the mean value will become the cutoff. Districts with percentages lower than 

the cutoff will be designated ‘low/medium-poverty’ and districts with percentages higher than the 

cutoff will be designated ‘high poverty.’ Independent districts such as charter school districts will be 

associated with the public school district that they are associated with geographically, as only the 

primary geographically-based public school districts have poverty estimates from SAIPE. 

Any districts with only one school will have a sampling rate set to be 1/4 of the sampling rate they would 

otherwise receive. Even with this undersampling, they will still be represented correctly in the 

population, as their weights will reflect their reduced probabilities of selection (the weights will be four

times larger than they would otherwise be), but we will have fewer of these districts. This method of 

undersampling is similar to that done in the NAEP for schools with very small numbers of students. 

 

11 The effective sample size is equal to the population variance divided by the sampling variance under the design. 

12 We assume a null hypothesis of no difference with a two-sided critical region with a 5 percent alpha level. We find the smallest population 

difference that would be detectable with this test with 80 percent power. The MDES is this population difference divided by the (assumed) 

common population standard deviation for each subgroup.

13 This assumes 100 percent district response. District nonresponse will degrade these power results, but we expect minimal district 

nonresponse as this study is by law mandatory for the districts. 

14 In defining district eligibility, we follow the criteria from the NAEP. The NAEP macros for excluding districts are applied also in the generation 

of the district frame here. 
12

B.2.2.2. Nationally representative sample of schools

The school sample is a two-stage sample of 1,300 schools, nesting within the sampled districts. There 

will be sampling rates assigned for sampling groups defined by school Title I status,

15 school poverty 

status, and by school span and school size. Table B - 4 presents the proposed 10 span-size subgroups 

with differential sampling rates.

School grade span is defined as follows, following the CCD definition:

 elementary is defined to have a low grade of Pre-K through 3rd grade, and a high grade of 

Pre-K through 8th grade; 

 middle is defined to have a low grade of 4th through 7th grade, and a high grade of 4th 

through 9th grade; 

 high school is defined to have a low grade of 7th through 12th, and a high grade of 12th only; 

and

 other schools is defined to include all other schools.

 

15 For sampling purposes, we will use the Title I status variable that is on the CCD that indicates whether the school is a Title I eligible school. 

We will confirm whether a sampled school received Title I for the 2013-2014 school year or remained Title I eligible. 
13

Table B - 4. Proposed school span-size subgroups for differential sampling rates

School span School size 

class

School 

count

Percent 

of 

schools

Enrollment

Percent 

of 

enrollment

Minimum 

enrollment

Maximum 

enrollment

Mean 

enrollment

Elementary C-Small 22,120 23.9% 5,581,817 11.4% 1 400 252

Elementary B-Medium 17,162 18.5% 8,453,902 17.3% 401 600 493

Elementary A-Large 11,887 12.8% 9,123,017 18.7% 601 4,595 767

Middle C-Small 8,016 8.7% 2,350,833 4.8% 1 525 293

Middle B-Medium 4,491 4.9% 2,947,351 6.0% 526 800 656

Middle A-Large 3,912 4.2% 4,128,550 8.4% 801 3,152 1055

High C-Small 11,114 12.0% 2,957,123 6.0% 1 700 266

High B-Medium 3,961 4.3% 4,155,302 8.5% 701 1,450 1049

High A-Large 3,379 3.6% 7,075,509 14.5% 1,451 5,017 2094

Other 6,547 7.1% 2,140,757 4.4% 1 7,874 327

Total 92,589 100.0% 48,914,161 100.0%

The school design will define three major subgroups: non-Title I schools, Title I high-poverty schools, and 

Title I low/medium-poverty schools. Given the importance of Title I schools and the need to compare 

non-Title I schools to Title I schools, these subgroups are very important. Poverty status is defined for 

schools in terms of percentage of students eligible for free or reduced price lunch. The cutoff for highpoverty schools is the enrollment-weighted 75th percentile: 66.12 percent of students eligible for free or 

reduced lunch. Table B - 5 presents the breakdown of school counts and enrollment according to a 

frame based on the 2008/2009 CCD frame. 
14

Table B - 5. Subgroups for the school sample design

Major school 

group Span

School 

size group

Frame 

count

Percent 

of 

frame

Total 

enrollment

Percent 

of 

enrollment

Mean 

enrollment

Enrollment 

ratio

3/4 

root 

enrollment 

ratio

Non-Title I Elem C-Small 4,227 4.57% 997,967 2.04% 236 0.94 0.953

Non-Title I Elem B-Medium 4,207 4.54% 2,092,331 4.28% 497 1.98 1.666

Non-Title I Elem A-Large 3,611 3.90% 2,809,827 5.74% 778 3.09 2.331

Non-Title I Middle C-Small 2,146 2.32% 636,414 1.30% 297 1.18 1.131

Non-Title I Middle B-Medium 1,788 1.93% 1,182,990 2.42% 662 2.63 2.064

Non-Title I Middle A-Large 1,810 1.95% 1,905,772 3.90% 1,053 4.18 2.924

Non-Title I High C-Small 5,466 5.90% 1,405,599 2.87% 257 1.02 1.016

Non-Title I High B-Medium 2,272 2.45% 2,408,530 4.92% 1,060 4.21 2.939

Non-Title I High A-Large 2,032 2.19% 4,200,144 8.59% 2,067 8.21 4.850

Non-Title I Comb 3,026 3.27% 866,858 1.77% 286 1.14 1.102

Non-Title I Total Total 30,585 33.03% 18,506,432 37.83%

Title I high pov Elem C-Small 6,551 7.08% 1,727,962 3.53% 264 1.05 1.035

Title I high pov Elem B-Medium 5,251 5.67% 2,587,753 5.29% 493 1.96 1.655

Title I high pov Elem A-Large 4,096 4.42% 3,186,122 6.51% 778 3.09 2.330

Title I high pov Middle C-Small 1,834 1.98% 550,176 1.12% 300 1.19 1.140

Title I high pov Middle B-Medium 831 0.90% 541,395 1.11% 651 2.59 2.040

Title I high pov Middle A-Large 759 0.82% 829,877 1.70% 1,093 4.34 3.008

Title I high pov High C-Small 1,647 1.78% 428,940 0.88% 260 1.03 1.026

Title I high pov High B-Medium 429 0.46% 438,544 0.90% 1,022 4.06 2.860

Title I high pov High A-Large 317 0.34% 710,281 1.45% 2,241 8.90 5.152

Title I high pov Comb 1,072 1.16% 314,460 0.64% 293 1.16 1.121

Title I high pov Total Total 22,787 24.61% 11,315,510 23.13%

Title I low pov Elem C-Small 11,342 12.25% 2,855,888 5.84% 252 1.00 1.000

Title I low pov Elem B-Medium 7,704 8.32% 3,773,818 7.72% 490 1.95 1.647

Title I low pov Elem A-Large 4,180 4.51% 3,127,068 6.39% 748 2.97 2.263

Title I low pov Middle C-Small 4,036 4.36% 1,164,243 2.38% 288 1.15 1.107

Title I low pov Middle B-Medium 1,872 2.02% 1,222,966 2.50% 653 2.59 2.044

Title I low pov Middle A-Large 1,343 1.45% 1,392,901 2.85% 1,037 4.12 2.891

Title I low pov High C-Small 4,001 4.32% 1,122,584 2.30% 281 1.11 1.085

Title I low pov High B-Medium 1,260 1.36% 1,308,228 2.67% 1,038 4.12 2.894

Title I low pov High A-Large 1,030 1.11% 2,165,084 4.43% 2,102 8.35 4.911

Title I low pov Comb 2,449 2.65% 959,439 1.96% 392 1.56 1.393

Title I low pov Total Total 39,217 42.36% 19,092,219 39.03%

Total Total Total 92,589 100.00% 48,914,161 100.00%

The enrollment ratio is the ratio of the mean enrollment for the subgroup as compared to the subgroup 

Title I low-poverty, small elementary schools. The 3⁄4 root of this ratio is the basis for the sampling rate 

for the subgroups. This power is selected as it equalizes the precision for enrollment-based and countbased estimates (see the discussion below). The second part of this sampling rate is a multiplier for the 

three major subgroups. This multiplier is given in Table B - 6 below. 
15

Table B - 6. Expected school sample sizes for the subgroups in the school sample design

Major school 

group Span

School 

size 

group

Percent

of

enrollmentPercent

of frame

Relative

sample

rate 

Major 

school 

group 

multiplier

Final 

relative

sample

rate

Expected 

sample 

size

Percent

of 

sample

Non-Title I Elem C 2.04% 4.57% 0.953 1.1 1.05 27.8 2.14%

Non-Title I Elem B 4.28% 4.54% 1.666 1.1 1.83 48.4 3.73%

Non-Title I Elem A 5.74% 3.90% 2.331 1.1 2.56 58.2 4.47%

Non-Title I Middle C 1.30% 2.32% 1.131 1.1 1.24 16.8 1.29%

Non-Title I Middle B 2.42% 1.93% 2.064 1.1 2.27 25.5 1.96%

Non-Title I Middle A 3.90% 1.95% 2.924 1.1 3.22 36.6 2.81%

Non-Title I High C 2.87% 5.90% 1.016 1.1 1.12 38.4 2.95%

Non-Title I High B 4.92% 2.45% 2.939 1.1 3.23 46.1 3.55%

Non-Title I High A 8.59% 2.19% 4.850 1.1 5.33 68.1 5.24%

Non-Title I Comb 1.77% 3.27% 1.102 1.1 1.21 23.0 1.77%

Non-Title I 37.83% 33.03% 389.0 29.92%

Title I high pov Elem C 3.53% 7.08% 1.035 2.25 2.33 95.9 7.38%

Title I high pov Elem B 5.29% 5.67% 1.655 2.25 3.72 122.8 9.45%

Title I high pov Elem A 6.51% 4.42% 2.330 2.25 5.24 134.9 10.38%

Title I high pov Middle C 1.12% 1.98% 1.140 2.25 2.57 29.6 2.27%

Title I high pov Middle B 1.11% 0.90% 2.040 2.25 4.59 24.0 1.84%

Title I high pov Middle A 1.70% 0.82% 3.008 2.25 6.77 32.3 2.48%

Title I high pov High C 0.88% 1.78% 1.026 2.25 2.31 23.9 1.84%

Title I high pov High B 0.90% 0.46% 2.860 2.25 6.44 17.3 1.33%

Title I high pov High A 1.45% 0.34% 5.152 2.25 11.59 23.1 1.78%

Title I high pov Comb 0.64% 1.16% 1.121 2.25 2.52 17.0 1.31%

Title I high pov 23.13% 24.61% 520.7 40.06%

Title I low pov Elem C 5.84% 12.25% 1.000 1 1.00 71.3 5.48%

Title I low pov Elem B 7.72% 8.32% 1.647 1 1.65 79.7 6.13%

Title I low pov Elem A 6.39% 4.51% 2.263 1 2.26 59.4 4.57%

Title I low pov Middle C 2.38% 4.36% 1.107 1 1.11 28.1 2.16%

Title I low pov Middle B 2.50% 2.02% 2.044 1 2.04 24.0 1.85%

Title I low pov Middle A 2.85% 1.45% 2.891 1 2.89 24.4 1.88%

Title I low pov High C 2.30% 4.32% 1.085 1 1.08 27.3 2.10%

Title I low pov High B 2.67% 1.36% 2.894 1 2.89 22.9 1.76%

Title I low pov High A 4.43% 1.11% 4.911 1 4.91 31.8 2.44%

Title I low pov Comb 1.96% 2.65% 1.393 1 1.39 21.4 1.65%

Title I low pov 39.03% 42.36% 390.3 30.02%

Total 100.00% 100.00% 1,300 100.00%

The major subgroup multipliers were generated with the goal of making the non-Title I, Title I 

low/medium -poverty, and Title I high-poverty schools 30 percent, 30 percent, and 40 percent of the 

overall school sample respectively and thus, close to equal in size. To do this, the Title I high-poverty 

schools need to be sampled at a 2.25 higher rate than the Title I low-poverty schools, and the non-Title I 

schools at a 1.1 times higher rate than the Title I low-poverty schools. Table B - 7 presents the effective 

sample sizes for all schools and for the three major subgroups, and for enrollment-based and countbased school estimates. As can be seen the precision for the two types of estimates is close: that is the 
16

‘minimax’ property. The effective sample sizes across the three subgroups are roughly the 30-30-40 

breakdown which was desired. 

Table B - 7. Power properties of school design

Power Property Sample size

Enrollmentbased 

weight 

estimates

Countbased 

weight 

estimates

Effective sample size: All schools 1,300 982 970

Effective sample size: Non-Title I schools 389 325 305

Effective sample size: Title I low/medium-poverty schools 390 330 333

Effective sample size: Title I high-poverty schools 521 461 450

The school sample is nested within the district sample, so this constitutes a clustered design. This will 

facilitate school recruiting and will allow for a comparison of school and district responses within the 

sampled districts. The final school probabilities are equal to the unconditional probabilities developed 

based on the school design described in Table B - 6, divided by the district probabilities of selection. This 

clustered design means that intra-district correlation among the schools in each district will affect the 

precision of the estimates. Table B - 8 presents the effective sample sizes under a variety of different 

intra-district correlation coefficients. The mean number of sampled schools per sampled district is 2.28 

(1,300 schools divided by 570 districts). We will approximate the design effect due to district clustering 

as . . = 1 + ( − 1), with  the intradistrict correlation coefficient, and  the mean cluster size 

2.28. 

Table B - 8. Effective sample sizes for national school-level estimates under the 3/4th root school 

design

Intra-district 

correlation 

coefficient

Withindistrict 

mean 

school 

sample 

size

Design 

effect

Effective 

enrollmentbased 

weight 

school 

sample size

Effective 

countbased 

weight 

school 

sample 

size

CV of 

enrollmentbased 

weight 

estimate

CV of 

countbased 

weight 

estimate

0% 2.28 1.000 981.9 970.2 3.19% 3.21%

5% 2.28 1.064 828.8 828.1 3.47% 3.48%

10% 2.28 1.128 781.8 781.1 3.58% 3.58%

15% 2.28 1.192 739.8 739.2 3.68% 3.68%

20% 2.28 1.256 702.1 701.5 3.77% 3.78%

25% 2.28 1.320 668.1 667.5 3.87% 3.87%

30% 2.28 1.384 637.2 636.6 3.96% 3.96%

35% 2.28 1.448 609.0 608.5 4.05% 4.05%

40% 2.28 1.512 583.2 582.7 4.14% 4.14%

45% 2.28 1.576 559.6 559.1 4.23% 4.23%

50% 2.28 1.640 537.7 537.3 4.31% 4.31%

The effective sample sizes comparing non-Title I, Title I high-poverty, and Title I low-poverty schools are 

given in Table B - 7, depending on the type of estimate (enrollment-based or count-based). The lowest 
17

precision is for the pair non-Title I schools vs. Title I low/medium-poverty schools with count-based 

estimates, where the effective sample sizes are 305 and 333 respectively. Assuming no effect from intradistrict correlation, this translates to an MDES of 22.2 percent for comparing non-Title I and Title I highpoverty schools, and slightly lower MDES for the other possible pairs.

16

Further Details of School Sampling. The school sample is nested within the district sample, and districts 

will then be the major stratum for school sampling. The mean school sample size per district is very 

small (2.28), so there is not much room for stratification at the school level beyond districts, but we will 

implicitly stratify by Title I status (i.e., within districts, systematically sample the schools sorted by Title I 

status), and if there is sufficient sample size, by school span (elementary, middle/combined, high).

17

B.2.2.3. Nationally representative sample of teachers

The teacher sample will be 9,100 teachers, sampled in 1,300 sampled schools: a mean of 7.0 teachers 

per sampled school. Expecting at least 85 percent teacher response, this will result in 7,735 expected 

completed surveys, a mean value of 5.95 teachers per sampled school. 

The teacher sample sizes per school will vary across schools. The plan is to set the teacher sample size in 

a way to equalize the final teacher weights within the four basic school-span strata (elementary, middle, 

high, and combined schools), to the extent possible, given the need to have integer teacher sample 

sizes. Our plan will be similar to that used for teacher sampling in SASS (Schools and Staffing Survey), 

carried out by the NCES. Documentation regarding SASS teacher sampling can be found in 

http://www.nces.ed.gov/surveys/sass. SASS is done once every four years (2003/2004, 2007/2008, 

2011/2012). 

We define a school weight  for stratum  (=1,...,) and school  (=1,...,). This is the inverse of the 

school probability of selection, disregarding oversampling for the three major school subgroups (we 

adjust for differential probabilities due to enrollment, but not due to school Title I status and school 

poverty status).

18 Define  as the teacher count for school . Define  as the mean per-school teacher 

sample size for stratum  (e.g., 6.2 teachers for elementary schools) and  as the school sample size in 

stratum . The teacher sample size for school  will be

   ∗  ∗

 ∗ 

∑  ∗ 





 , 10

 

16 Note that the effect of a positive intra-district correlation could be to either increase or decrease the precision, as we expect many 

low/medium-poverty districts to have at least one non-Title I school and one Title I school. For these districts, the presence of intra-district 

correlation actually increases the precision of the difference between the two subgroups. For other districts which have more than one of 

one subgroup, but none of the other, positive intra-district correlation will reduce the precision of the difference. Thus the overall effect will 

be ambiguous (depends on the exact distribution of the sampled schools across the sampled districts). 

17 We would implicitly stratify by school span for example if all schools in a district were Title I, or non-Title I, or if the district sample size 

happened to be larger. 

18 This means that teacher sample sizes will be larger for schools which were undersampled due to enrollment, and smaller for schools which 

were oversampled due to enrollment, bringing teacher sample sizes back to population percentages. However, for the three major 

subgroups, this reversion to population percentages will not take place. All other things being equal, teacher sample sizes within each school 

across the three major subgroups will not be different, resulting in an oversample of teachers in high-poverty Title I schools, roughly 

proportional to the 2.2 oversampling rate for these schools, and a slight oversample in non-Title I schools (a 1.1 oversampling rate), as 

compared to low/medium-poverty Title I schools. 
18

where INT is the function that rounds to the nearest positive integer. Note that if the school probability 

of selection was directly proportional to teacher count within each stratum, then  and  would be 

equal to a constant within each stratum , and the teacher sample size would be simply for each 

school in stratum . The schools are sampled very roughly proportionally to the 3⁄4-root of enrollment 

within the two major strata (high-poverty districts and low/medium-poverty districts), so assuming that 

teacher count is proportional to enrollment (roughly) in the school strata,  will tend to be inversely 

proportional to the 3⁄4-root of , making teacher sample size roughly proportional to the 1⁄4 root of 

teacher count. Thus, the teacher count will be slightly larger for larger schools, and slightly smaller for 

smaller schools, but not far from  unless the teacher-student ratio is unusual for the school, or the 

school’s enrollment is on the extreme end of the distribution for the school stratum. An upper bound of 

10 is placed on the teacher sample size per school to minimize burden.

Within the two major district strata, the design for teachers should be close to self-weighting. We will 

assume design effects of 1.15 to allow for distortions from rounding to integers, respecting the 

maximum burden of 10, and to also allow for the effect of necessary teacher nonresponse adjustments. 

Table B-9 below summarizes the design and calculates effective teacher sample sizes assuming the 

design effect of 1.15 within the three major school subgroups, and the design effect from oversampling 

the high-poverty Title I and non-Title I schools by factors of 2.25 and 1.1 respectively (relative to lowpoverty Title I schools). In Table B - 9 we also assume an ICC of 0 percent for clustering of teachers 

within sampled schools. 

Table B - 9. Expected teacher sample sizes

School 

subgroup

Expected 

school 

sample 

size

Mean 

sampled 

teachers 

per school

Total 

sampled 

teachers

Percent of 

sample 

teachers

Percent of 

pop'n 

teachers

Expected 

teacher 

response 

rate

Expected 

final 

teachers 

per 

school

Total 

final 

teacher 

count

Effective 

teacher 

sample 

size

Non-Title I 389 7 2,723 29.90% 37.80% 85% 5.95 2,315 2,013

Title I low pov 390 7 2,730 30.00% 39.00% 85% 5.95 2,321 2,018

Title I high 

pov

521 7 3,647 40.10% 23.10% 85% 5.95 3,100 2,696

Total 1,300 7 9,100 100.00% 100.00% 85% 5.95 7,735 6,018

Table B - 10 presents the effective sample sizes for national teacher estimates under a variety of 

different intra-district correlation coefficients. The mean number of final teachers per sampled school is 

5.95. We will approximate the design effect due to district clustering as . . = 1 + ( − 1), with 

the intradistrict correlation coefficient, and  the mean cluster size 5.95. 
19

Table B - 10. Effective sample sizes for national teacher estimates

Intraschool 

correlation 

coefficient

Withinschool 

mean 

teacher 

sample 

size

Design 

effect

Effective 

teacher 

sample 

size

0% 5.95 1 6,018

5% 5.95 1.2475 4,824

10% 5.95 1.495 4,025

15% 5.95 1.7425 3,454

20% 5.95 1.99 3,024

25% 5.95 2.2375 2,690

30% 5.95 2.485 2,422

35% 5.95 2.7325 2,202

40% 5.95 2.98 2,019

45% 5.95 3.2275 1,865

50% 5.95 3.475 1,732

We are also interested in generating estimates for teachers separately for elementary, middle, and high 

schools. Table B - 11 provides expected sample sizes for each of these three span subgroups. The 

expected final teachers per school are our expectations for the average number of teacher surveys given 

the expected school sample sizes and an assumed allocation of teachers by school span based on 

enrollment percentages19 (6.2, 7.3, and 8.8 sampled teachers for elementary, middle, and high schools 

respectively, with an 85 percent response rate to obtain the expected surveyed teacher means of 5.27, 

6.21, and 7.48 respectively). The effective sample size for the major subgroups assumes a design effect 

of 1.15 (accounting for teacher nonresponse adjustments and varying teacher sample sizes). The 

effective sample size for the three subgroups combined within grade span also includes the design effect 

induced from oversampling non-Title I schools and high-poverty Title I schools, by benchmarking by 

subgroup enrollment figures (assuming that student enrollment is exactly proportional to teacher 

counts). 

 

19 Teacher allocations by grade span based on enrollment percentages are only provisional (it cannot be expected that teacher/student ratios 

are exactly equal across grade spans). Final allocations of teacher sample sizes by grade span will be produced when the frame for this study 

is generated.
20

Table B - 11. Sample sizes and effective sample sizes for teachers for elementary, middle, and high 

schools

Span Major school group Expected 

sample size

Expected 

final 

teachers 

per school

Total final 

teachers

Effective 

sample 

size

Enrollment

Elementary Non-Title I 134 5.27 706 614 5,900,125

Elementary Title I low pov 210 5.27 1,107 962 9,756,774

Elementary Title I high pov 354 5.27 1,866 1,622 7,501,837

Elementary Total 698 3,678 2,818 23,158,736

Middle Non-Title I 79 6.205 490 426 3,725,176

Middle Title I low pov 77 6.205 478 415 3,780,110

Middle Title I high pov 86 6.205 534 464 1,921,448

Middle Total 241 1,502 1,186 9,426,734

High Non-Title I 153 7.48 1,144 995 8,014,273

High Title I low pov 82 7.48 613 533 4,595,896

High Title I high pov 64 7.48 479 416 1,577,765

High Total 299 2,237 1,828 14,187,934

Total Total 1,239 5.99 7,417 46,773,404

Table B - 12 presents power calculations for each grade span grouping, with the effective teacher 

sample sizes as given in Table B - 11. The first effective sample size accounts for school sampling and 

teacher weighting effects. The ‘final effective sample size’ also accounts for intra-school correlation, 

using the design effect . . = 1 + ( − 1) with  equal to the intra-school correlation and  equal to 

the cluster size (the average number of interviewed teachers per school. The assumed intra-school 

correlations for this table are 50 percent for elementary schools, 40 percent for middle schools, and 30 

percent for high schools (we assume higher clustering due to the small size and assumed ‘homogeneity’ 

within elementary schools). 

Table B - 12. Sample sizes and effective sample sizes for teachers for elementary, middle, and high 

school grade spans

Span

Expected 

final 

teachers

Effective 

sample 

size 

(0% ICC)

Assumed 

intraschool 

correlation

Cluster 

size

Design 

effect

Final 

effective 

sample 

size

Elementary 3,678 2,818 50% 5.27 3.135 899

Middle 1,502 1,186 40% 6.205 3.082 385

High 2,237 1,828 30% 7.48 2.944 621

Within each sampled school, the teachers will be selected in a stratified sample. The overall teacher 

sample size will be determined by the SASS algorithm. The selection process is designed to allow for 
21

sufficient sample sizes for comparisons of teachers of ESEA-tested and non-ESEA tested grades and 

subjects, and support a nationally representative sample. 

For elementary schools, teachers in Kindergarten through 2nd grade are non-ESEA tested teachers. 

Math, reading/English/language arts (ELA), and science teachers in 3rd grade and up are teachers of 

tested subjects and grades. Other teachers in 3rd grade and up are teachers of non-tested subjects. The 

specific sampling allocations will be carried out:

 Teachers of Kindergarten through 2nd grade selected proportional to grade count;

 Teachers of upper grades reading, math, and science selected proportional to grade count 

divided by 2;

 Teachers of upper grades other subjects selected proportional to grade count divided by 2.

For example, if the school is K through 6, 3/7 of the sample will be allocated to Kindergarten through 2nd

grade, 2/7 to upper grades reading, math, and science, and 2/7 to upper grades other subjects.

For middle schools, because most students are tested annually through grade 8, there are no non-tested 

grades, though there are teachers who teach non-tested subjects. Because most states test in reading, 

math, and science, teachers of these subjects will be taken as representing teachers of tested subjects, 

and the others as representing teachers of non-tested subjects. To ensure enough teachers to compare 

responses across tested versus non-tested subjects, the sampling allocation will be as follows:

 Teachers of reading/English/language arts, math, and science selected at a rate of 3 in 7;

 Teachers of other subjects selected at a rate of 4 in 7.

For middle schools, the allocated mean teacher sample size will be 7 (in some cases, the sample size will 

be lower, in some cases higher). Thus, in most cases, the 3 in 7 allocation should yield close to three 

teachers. Allocation of three teachers in this design should provide for a good representation of each of 

the three subjects. Selection of 4 in 7 other teachers should yield close to four other teachers in most 

cases, and this allow sufficient representation of the larger variety of non-tested subjects taught at the 

middle school level, compared with the elementary school level. 

For high schools, testing patterns vary by state, and even by district. ESEA requires students be tested 

once in math, reading/ELA, and science, but some states and districts test more often. Again, we will 

allocate the teacher sample to provide for sufficient numbers of teachers of both tested and non-tested 

subjects. The allocation will be as follows:

 Teachers of reading/English/language arts, math, and science in tested class/grade selected 

at a rate of 1 in 3;

 Teachers of reading/English/language arts, math, and science in non-tested class/grade 

selected at a rate of 1 in 3;

 Teachers of other subjects selected at a rate of 1 in 3.

This allocation allows the comparison of responses from teachers who experience state standardized 

testing, with responses of those who do not. The comparisons are within core subject areas and 

separate possible influences due to testing from those due to subject area. The mean allocated teacher 

sample size for high schools will be close to nine. The one in three allocation should in most cases yield 
22

close to three teachers. Taking three teachers from each group is a reasonable way to improve the 

likelihood of representing each of the core subjects in tested and non-tested conditions, and to reflect 

the diversity of non-core, non-tested subjects taught at the high school level. The larger number of 

teachers sampled in high schools also allows for better representation of the expected greater variability 

of teacher experiences in high schools, which are larger, cover more subjects, and are more likely to be 

departmentalized than the other levels.

For combined schools, allocations will be made to ‘elementary’, ‘middle’, and ‘high’ grades based on the 

grade span of the combined school, and then these allocations will be further separated out as above. 

After the allocations are made, a randomized process will be carried out to assign integer sample sizes 

to the strata defined above in such a way so that the integer sample sizes will have expectation equal to 

the allocated (fractional) sample sizes, and the overall teacher sample sizes add up to the teacher 

sample size assigned to the school by the SASS algorithm. 

This design is intended to maximize the efficiency of comparisons between subgroups such as teachers 

in ESEA-tested subjects vs. non-tested subjects, and teachers in ESEA-tested grades vs. non-tested 

grades, by equalizing the sample sizes for each subgroup. This comes somewhat at the expense of 

overall estimates for teachers, which will have an extra design effect. The calculations in Table B - 10 and 

Table B - 12 do not account for this loss of efficiency from oversampling certain teacher subgroups 

within each school to equalize sample sizes across these teacher subgroups within schools, but the loss 

of efficiency that does occur should be limited in extent, as we believe these allocations are not far from 

the actual population percentages. 

Actually, comparing subgroups which are both present in every school (or in most schools, at least) will 

benefit from the presence of positive intra-school correlation. The estimate in this case is in effect a 

comparison of the two subgroups within each school, aggregated over all schools. The variance in this 

case will be proportional to 









+





 (1 − ), where  is the effective school sample size 

(accounting for design effects from oversampling of the three major school subgroups, but not 

accounting for design effects from enrollment-based oversampling, as the latter is removed in the 

setting of teacher sample sizes),  and  are the sample sizes for the subgroups, and  is the intraschool correlation for the characteristic. Note that the variance decreases linearly with . 

Table B - 13 provides MDES calculations for the following comparisons:

 Elementary schools: comparing two upper grade tested subject to two upper grade nontested subject, with 85 percent response and 50 percent ICC;

 Middle schools: comparing three tested subject to three non-tested subject, with 85 percent

response and 40 percent ICC;

 High schools: comparing three tested class/grade and three non-tested class/grade, with 85

percent response and 30 percent ICC. 

In all cases, the null hypothesis is no difference, with a two-sided test with alpha-level 5 percent, and 80 

percent power for the alternative. The MDES are very good in all cases. 
23

Table B - 13. Selected MDES for teachers by elementary, middle, and high school grade spans

Span

Effective 

school 

sample 

size

Assumed 

intraschool 

correlation

Final 

interview 

sample 

size 

subgroup 

1

Final 

interview 

sample 

size 

subgroup 

2

MDES

Elementary 616 50% 1.7 1.7 8.65%

Middle 219 40% 2.55 2.55 12.98%

High 281 30% 2.55 2.55 12.38%

B.2.3. Estimation Procedures

The focus of the report will be on the progress being made on the core policies promoted by Titles I and 

II. While simple descriptive statistics such as means and percentages will provide answers to many of 

our questions, cross-tabulations will be important to answering questions about variation across state, 

district, schools, and teacher characteristics.

 Whether a state received an approved ESEA Flexibility request from the U.S. Department of 

Education. This is of interest because ESEA Flexibility is expected to influence the design of state 

accountability systems and approaches to educator quality measurement and policy makers will 

be interested in what states do with the ESEA Flexibility. 

 District poverty level, size, urbanicity, and concentration of English learners. Poverty is included 

because Title I is specifically intended to ameliorate the effects of poverty on local funding 

constraints and educational opportunity. Urbanicity is included because of the relationships 

between educational opportunity and rural isolation and the concentration of poverty in urban 

schools. Size is included because it may be related to district capacity to develop and implement 

programs. Concentration of English learners is included because of the increased emphasis on 

ensuring that this group of students also meets state content standards and recognition that 

modifications in testing as well as instruction will be needed to facilitate progress of these 

students. 

 School Title I status, grade span (high school, middle, and elementary grades), poverty level, and 

cross classifications of Title I status and poverty level. Grade span is included because the 

implementation of state content standards and aligned assessments as well as responses to 

accountability systems likely differs by grade level. Title I status is included because the focus of 

Title I funds and requirements is to influence state and district policy and school effectiveness. 

We plan to cross-classify Title I status with school poverty because there may be substantial 

differences between what happens in high-poverty Title I schools in comparison with Title I 

schools with low/medium-poverty levels, which may be more similar to schools which do not 

receive Title I funding. 

 Teacher grade span taught and whether the teacher teaches a grade or subject in which state 

testing is required by ESEA. These characteristics are included because teachers’ responses to 

state content standards, aligned assessments, accountability systems, and teacher evaluations 
24

based on student growth may vary by grade span, and by whether they teach tested subjects or 

grades.

The use of stratification (and oversampling when necessary) in our sample design was introduced to 

ensure reasonable power for specific subgroup comparisons, and our samples are expected to include 

units that vary on other characteristics (e.g., urbanicity) to allow the comparisons described above. 

Because of the use of a statistical sample, survey data presented for districts, schools, and teachers will 

be weighted to national totals (tabulations will, provide standard errors for the reported estimated 

statistics). In addition, the descriptive tables will indicate where differences between subgroups are 

statistically significant. We will use Chi-Square tests to test for significant differences among 

distributions and t-tests for differences in means. Tabulations will be included in the reports where 

appropriate.

Please see Section B.3.1 for a discussion of our weighting procedures.

B.2.4. Degree of Accuracy Needed

We require statistical precision at all three levels in this study: the district level, the school level, and the 

teacher level. This makes this study different from a study which is focused for example on teachers 

alone. The hardest level to achieve precision for will be the district level, as the sample sizes in the 

three-stage sample design have to be smallest at this level (being the first stage of selection). The 

sample size of 570 districts evenly split between high-poverty districts and the complement stratum will 

provide an MDES of 30% for comparing these two important district subgroups. 

At the school level, we are sampling 1,300 schools. An important analysis at the school level compares 

Title I high-poverty schools (roughly 25% of the population), Title I low/medium-poverty schools 

(roughly 40% of the population), and non-Title I schools (roughly 35% of the population). We 

oversample Title I high-poverty schools considerably (a factor of 2.25), and non-Title I schools slightly (a 

factor of 1.1), to roughly equalize sample sizes. The MDES for the three possible pair-wise comparisons 

ranges between 22% and 25%, depending on the degree of intra-district correlation.

At the teacher level, we are sampling 9,100 teachers, and expecting 7,735 respondents. This sample size 

will be sufficient to provide MDESs in the 10%-15% range for comparisons among elementary schools, 

middle schools, and high schools separately, for two subgroups comprising 1⁄4 of the population teachers 

in each subgroup. The MDESs within this 10%-15% range depend on the intra-school correlation and 

how the subgroups break across the schools (each subgroup represented in each school, or each 

subgroup is more concentrated in particular schools). The highest precision is achieved for comparing 

subgroups which are equally within every school (tested subjects vs. non-tested subgroups), and the 

lowest precision is for comparing subgroups which are entirely segregated within school subgroups 

(teachers in Title I schools vs. teachers in non-Title I schools).
25

B.2.5. Unusual Problems Requiring Specialized Sampling Procedures

There are no unusual problems requiring specialized sampling procedures.

B.2.6. Use of Periodic (less than annual) Data Collection to Reduce Burden

This data will be collected during the 2013-14 school year with a follow up survey and a possible 

optional year. This OMB package addresses only the 2013-14 school year data collection.

We plan to work with states, school districts, schools, and teachers to explain the importance of this 

data collection effort and to make it as easy as possible to comply. For all respondents, a clear 

description of the study design, the nature and importance of the study, and the OMB clearance 

information will be provided.

For the states, the data collection’s reliance whenever possible on administrative and extant data,

thereby limiting the data collection from state representatives, will encourage cooperation with 

evaluation efforts. We will be courteous but persistent in follow-up with participants who do not 

respond in a timely manner to our attempts. As noted earlier, we will also be very flexible gathering our 

data, allowing different people to respond to the different content areas and in whichever mode is 

easiest -- electronic, hard copy or telephone format. 

For the school districts, schools, and teachers, we will initiate several forms of follow-up contacts with 

respondents who have not responded to our communication. We will use a combination of reminder 

postcards, emails and follow-up letters to encourage respondents to complete the surveys. The project 

management system developed for this study will be the primary tool for monitoring whether surveys 

have been initiated. After 10 days, we will send an email message (or postcard for those without email) 

to all non-respondents indicating that we have not received a completed survey and encouraging them 

to submit one soon. Within seven business days of this first follow-up, we will mail non-respondents a 

hard copy package including all materials in the initial mailing. Ten days after the second follow-up, we 

will telephone the remaining non-respondents to ask that they complete the survey and offer them the 

option to answer the survey by phone, either at that time or at a time to be scheduled during the call. 

To maximize response rates, we also will (1) provide clear instructions and user-friendly materials, (2) 

offer technical assistance for survey respondents using a toll-free telephone number or email, and (3)

monitor progress regularly. In recognition of the fact that district and school administrators have many 

demands on their time, and typically, these administrators receive numerous requests to participate in 

studies and complete surveys for federal and state governments, district offices, and independent 

researchers, we plan to identify a district liaison. For most districts, completion of the district survey will 

require input from multiple respondents, and the district liaison’s role will be pivotal in positively 

impacting participation, collecting high quality data, and achieving an 85 percent response rate. 

B.3. Methods to Maximize Response Rates
26

B.3.1. Weighting the district, school, and teacher samples

After completion of the field data collection in each year, we plan to weight the data to provide a 

nationally representative estimator at the district, school, and teacher level. Replicate weights will be 

generated to provide consistent jackknife replicate variance estimators (statistical packages such as 

STATA and SAS Version 9.2 allow for easy computation of replicate variance estimates). The 

development of replicate weights will facilitate the computation of standard errors for the complex 

analyses necessary for this survey. The replicates will be based fundamentally on the district sample (the 

first stage of selection), with an exception for certainty districts, for which the first stage of selection is 

at the school level. We anticipate limited nonresponse at the district and school level, which we will 

adjust for by utilizing information about the non-responding districts and schools from the frame. This 

information will be used to generate nonresponse cells with differential response propensities. The 

nonresponse adjustments will be equal to the ratio of the frame weighted count to the sum of weights 

for respondents. This will adjust for bias from nonresponse, and also adjust for differences from the 

frame (accomplishing poststratification). If the cell structure is too rich, we may utilize raking (multidimensional adjustment). The cell structure for districts will include district poverty status, Census 

region, urbanicity, and district size. The cell structure for schools will include district poverty status, 

school poverty status, Title I status, Census region, urbanicity, and school size. 

There are two types of district and school weights that will be generated: enrollment-based weights and 

count-based weights. The first set of weights will add to total enrollment and the second to the simple 

count of districts and schools on the frame. A separate set of nonresponse adjustments will be 

calculated for each set of weights. Trimming may be utilized if necessary to scale back extreme weights. 

We expect a larger degree of teacher nonresponse (though we plan to provide an incentive). Teacher 

nonresponse adjustments will be computed based on an analysis of district and school characteristics, 

and teacher characteristics known from the teacher rosters, which correlate to teacher propensity to 

respond. A ‘frame of teachers’ will be created by weighting the teacher rosters from the sampled 

schools using the final teacher weights. The starting point for teacher weights will be the enrollmentbased final school weights. Teacher adjustments will then be made equaling the teacher roster totals 

divided by the weighted totals over teacher respondents. Raking may be utilized if the cell structure is 

too rich. The cell structure will include district poverty status, school poverty status, Title I status, Census 

region, and school size. Trimming may be utilized if necessary to scale back extreme weights, if this 

occurs. 

The state survey will be pretested with up to three states, and prior to the pretest, we will gather 

documentation and extant data from their state websites to better understand how current is the 

information obtained from secondary sources. The school district, school, and teacher surveys will be 

pretested with nine or fewer respondents. We will also pretest the teacher roster with principals to 

determine that the requested data is sufficient for selecting the teachers.

B.4. Test of Procedures
27

The individuals consulted on the statistical aspects of the design include:

Camilla Heid, Westat, Project Director, 301-294-4413

Patty Troppe, Westat, Deputy Project Director, 301-294-3924

Anthony Milanowski, Westat, Co-Principal Investigator, 240-453-2718

Brian Gill, Mathematica, Co-Principal Investigator, 617-301-8962

Lou Rizzo, Westat, Senior Statistician, 301-294-4486

David Morganstein, Westat, Vice-President Statistical Group, 301-251-8215

Tom Cook, Northwestern University, 847-491-3776

Sean Reardon, Stanford University, Member of Technical Working Group, 650-736-8517

B.5. Individuals Consulted on Statistical Aspects of Design
28

U.S. Department of Education. (2011, September 23). ESEA Flexibility. Retrieved from: 

http://www.ed.gov/esea/flexibility.

U.S. Department of Education. (2012a). Improving Basic Programs Operated by Local Educational 

Agencies (TITLE I, PART A). Retrieved from:

http://www2.ed.gov/programs/titleiparta/index.html

U.S. Department of Education. (2012b). Findings from the 2011-2012 Survey on the Use of Funds Under 

Title II, Part A. Retrieved from: http://www2.ed.gov/programs/teacherqual/resources.html

References