# Implementation of Title I/II Program **Initiatives Supporting**  **Statement for Paperwork Reduction Act Submission**  **PART B: Collection of Information Employing**  **Statistical Methods Contract**  **ED -**  **IES -**  **11 -** **C -** **0063**

**June 2013**  Prepared for: Institute of Education Sciences U.S. Department of Education Prepared by: Westat Mathematica Policy Research

### i



## Table of Contents **Introduction ................................**  **................................ ................................**  **................................ ......**  **1 Overview of the Study**  **................................ ................................**  **................................ ........................**  **1 B.1.**  **Respondent Universe and Sampling Methods ................................**  **................................ ............**  **3** B.1.1. State Sample ................................ ................................ ................................ ....................... 3 B.1.2. School District Sample ................................ ................................ ................................ ........ 3 B.1.3. School Sample ................................ ................................ ................................ ..................... 4 B.1.4. Teacher Sample ................................ ................................ ................................ ................... 5 **B.2. Information Collection Procedures**  **................................ ................................**  **............................ 6**  B.2.1. Notification of the sample, recruitment and data collection ................................ ............. 6 B.2.2. Statistical Methodology for Stratification and Sample Selection ................................ ....... 8 B.2.2.1. Nationally representative sample of school districts ................................ .................. 8 B.2.2.2. Nationally representative sample of schools ................................ ............................ 12 B.2.2.3. Nationally representative sample of teachers ................................ .......................... 17 B.2.3. Estimation Procedures ................................ ................................ ................................ ...... 23 B.2.4. Degree of Accuracy Needed ................................ ................................ ............................. 24 B.2.5. Unusual Problems Requiring Specialized Sampling Procedures ................................ ....... 25 B.2.6. Use of Periodic (less than annual) Data Collection to Reduce Burden ............................. 25 **B.3. Methods to Maximize Response Rates**  **................................ ................................**  **.................... 25**  B.3.1. Weighting the district, school, and teacher samples ................................ ........................ 26 **B.4. Test of Procedures**  **................................ ................................**  **................................ ..................**  **26 B.5.**  **Individuals Consulted on Statistical Aspects of Design ................................**  **.............................. 27**  **References ................................**  **................................ ................................**  **................................ .......**  **28 Appendix F Notification Letters**  **................................ ................................**  **................................ ......**  **F -**  **1 Appendix G Study Brochure**  **................................ ................................**  **................................ ...........**  **G -**  **1**

### ii



## Tables B - 1. Definitions of district size strata ................................ ................................ ...................... 9 B - 2. Proposed stratification design for district sampling ................................ ........................ 9 B - 3. Properties of proposed stratification design for district sampling ................................ 10 B - 4. Proposed schools span - size subgroups for differential sampling rates ......................... 13 B - 5. Subgroups for the school sample design ................................ ................................ ....... 14 B - 6. Expected school sample sizes for the subgroups in the school sample design ............. 15 B - 7. Powe r properties of school design ................................ ................................ ................ 16 B - 8. Effective sample sizes for national school - level estimates under the 3/4th root

school design ................................ ................................ ................................ ................. 16 B - 9. Expected teacher sample sizes ................................ ................................ ...................... 18 B - 10. Effective sample sizes for national teacher estimates ................................ .................. 19 B - 11. Sample sizes and effective sample sizes for teachers for elementary, middle, and

high schools ................................ ................................ ................................ ................... 20 B - 12. Sample sizes and effective sample sizes for teachers for elementary, middle, and

high school grade spans ................................ ................................ ................................ . 20 B - 13. Selected MDES for teachers by elementary, middle, and high school grade spans ...... 231



## Part B. Collection of Information Employing Statistical Methods This package is the first of two for the Implementation of Title I/II Program Initiatives study. This package requests approval for an initial round of recruitment and data collection that will include surveys of all states and nationally representative s amples of school districts, schools, and Kindergarten through 12 th grade teachers . We anticipate that the state education agency (SEA) and school district surveys will begin in November 2013, and the principal and teacher surveys will begin in January 2014 . The second pa ckage will request approval for the follow- up survey, which will survey the same states, districts, and schools as well as a new nationally representative sample of teachers within the sampled schools during the 2015 - 16 school year . Title I is one of the U.S. Department of Education’s (ED) largest elementary and secondary education programs. Historically, Title I provides financial assistance to schools and districts with a high percentage of students from low- income families to help these students increase achievement. Title I also includes requirements that states hold schools and districts accountable for improvements i n student achievement. During the 2009 - 2010 school year, more than 56,000 public schools used Title I funds, and the program served 21 million children (U.S. Department of Education, 2012a). Title II provides funds to increase academic achievement by improving teacher and principal quality including educator preparation and professional development, as well as providing funds for class size reduction. An estimated 95 percent of districts nationally received Title II, Part A funding for the 2011 - 12 school ye ar (U.S. Department of Education, 2012b). The last National Assessment of the Title I program concluded in 2006. Since that time, there have been changes in Title I provisions, such as allowing states to incorporate proficiency improvement (alongside proficiency levels) into school accountability measures and providing more resources to the lowest performing schools through the expansion of eligibility and funding for School Improvement Grants (SIG). Title II guidance allows more flexibility for certain t eachers (e.g., special education teachers) to meet the standards to become highly qualified teachers (HQT). The most recent change related to Title I and Title II was the introduction of the Elementary and Secondary Educatio n Act (ESEA) Flexibility reque sts in 2011 , allow ing states to waive a number of provisions in exchange for a commitment to key reform principles. The Title I/II study will provide policy makers with detailed information on how these initiatives are playing out in states, districts, sch ools, and classrooms. The Implementation of Title I/II Program Init iatives study is commissioned by the Institute of Education Sciences ( IES ) , ED’s independent research and evaluation ar m. The Title I/II study will examine the imple mentation of policies promoted through ESEA at the st ate, district, school , and classroom levels. Through surveys at each level and selected extant data and documents, the study will provide 

## Intr**oduction**

**Overview of the Study**2

information on activities in four core areas: state content standards, aligned assessments, accountability and school turnaround , and development of effective teachers and leaders. The study will reflect changes in Title I and Title II provisions since the last National Assessme nt of the Title I program concluded in 2006 (including ESEA F lexibility provide d to states with approved requests ). In addition, this study will supplement findings from ED’s annual survey of a nationally representative sample of districts on their uses of Title II, Part A funds. This study will provide more detail on districtimplemented workforce development policies and practices for educators as well as teachers’ and principals’ access to and receipt of professional development. 

### State content standards State content standards have been a foundational element of ESEA since the 1994 reauthorization of the Act, which required states to establish statewide standards in reading and mathematics in the 3 through 5, 6 through 8, and high school grade ranges and implement statewide accountability system s for evaluating school - level performance. The No Child Left Behind Act of 2001 ( NCLB ) , the subsequent reauthorization of ESEA, extended and strengthened these concepts, incorporating science alongside reading and math, and establishing more prescriptive rules for state accountability systems . More recently, states approved for ESEA Flexibility have agreed to adopt especially rigorous standards; over 40 states have adopted the Common Core State Standards (CCSS). 

### A **ligned a ssessments**  T he 1994 reauthorization of ESEA required states to administer annual academic assessments aligned to content standards in reading and mathematics in at least one elementary grade, one middle grade, and one high - school grade . In 2 001, NCLB extended this requirement from three grades to seven (grades 3 - 8 and at least once in high school) and added a requirement for science assessments in each of the 3 - 5, 6 - 8, and high school grade ranges. These assessments are meant to provide information on whether students have acquired the knowledge and skills identified by the standards and, if not, the areas where students fall short and may require additional instruction and assistance. Annual assessment data are also used to hold schools and districts, and increasingly teachers, accountable for student performance. T he content and rigor of these assessments vary across states. 

### Accountability and school turnaround NCLB standardized state accountability policies by requiring states to establi sh “adequate yearly progress” (AYP) goals and objectives for student proficiency on the mandated assessments, with targets rising over time so that by 2014, all students would be proficient. Schools and school districts are held accountable for achieving rates of student proficiency in reading and mathematics that meet the annual objectives, as well as for the proficiency rates of specific subgroups of students. Schools that failed to meet AYP for either all students or a subgroup of students over successive years were subject to an increasingly aggressive set of interventions spelled out in the law, beginning with parental choice and culminating with school closing or state takeover.3

The federal government has adjusted accountability requirements since the last National Assessment . The Growth Model Pilot Project, initiated in 2005, permitted n ine states to use accountability measures that incorporate proficiency improvement (rather than only proficiency levels) under a pilot in 2007 -08 . W hen the approach wa s written into regulations in 2008, 15 states participated. The ESEA Flexibility regulations introduced in 2011 allow states to develop more flexible systems of school accountability that incorporate measures of the achievement growth of individual student s, broaden achievement measures beyond mathematics and reading, and include other student outcomes such as graduation rates. In addition, recent policy changes have led to more intensive focus on the lowest- performing schools and differentiated support for schools depending on their performance levels 

### Development of effective t**eachers and leaders** Improving and supporting the effectiveness of teachers and school leaders is a key part of any strategy for improving student achievement and college and career readiness. Research has shown that teachers (e.g., Rowan, Correnti, & Miller, 2002; Rockoff, 2004; Rivkin, Hanushek, & Kain, 2005) and school administrators (e.g., W itziers, Bosker, & Kruger, 2003) have measurable effects on student achievement. Title I and Title II incorporate a number of requirements and incentives for states and districts to implement policies that improve certification , support educators during all phases of their careers, hold them accountable for contributing to student achi evement, and ensure their equitable distribution — all based on the premise that improvement in student achievement requires improving educator quality. However, there has been substantial development in the national discourse about teacher quality since the passage of the NCLB and the init iation of the last National Assessment of Title I . Most importantly, thinking about educator quality has shifted away from educator inputs (e.g., degrees and certifications) toward educator performance, defined in terms of practice and especially in terms of contribution to growth in student achievement. This shift has been facilitated by the improvements in educator practice assessments (Porter, Youngs, & Odden, 2001; Milanowski, Heneman & Kimball, 2011) and in “value - adde d” methods to measure school and teacher contributions to student achievement (Meyer, 1997; Harris, 2009; Glazerman et al., 2010).

The study sample will include the universe of states and the District of Columbia , and nationally representative samples of districts, schools, and teachers. The school sample will be nested in the district sample and the teacher sample will be nested in the school sample. 

### B.1.1. **State Sample** We will survey a ll 50 states and the District of Columbia . 

### B.1.2. **School District Sample** A nationally representative sample of districts will be selected . This will provide unbiased estimators of district characteristics, and will provide the first stage of selection for nationally representative samples 

## B.1 **. Respondent Universe and Sampling Methods** 4

of schools and teachers. A nationally representative sample is necessary as Title I/II covers most of t he U . S . public school educational system. We also are interested in statistically comparing the implementation of initiatives promoted by Title I and Title II by **district level of poverty** and **size of districts** based on student enrollment. Poverty is incl uded because Title I is specifically intended to ameliorate the effects of poverty on local funding constrai nts and education opportunity. D istrict capacity to implement initiatives also will be of interest for this study. Success in implementation of initiatives, particularly those that build upon each other such as having longitudinal data systems and identifying effective teachers, might be tied to district organizational capacity. The refore, we also will examine implementation by district size. For the district sample, we will implement a “minimax” sample design that strikes a balance between producing efficient student-weighted estimates and efficient district-weighted estimates. Th is seems appropriate because the policy levers of Title I and Title II are directed at districts (and schools and states). For some purposes, it will be useful to understand the average experience of students across the country, but for other purposes, we will want to understand the experience and behavior of the average districts (and schools) — the units that the U.S. Department of Education (ED) expects its policies to immediately influence. The minimax design is a compromise between a design that is relat ively efficient (i.e., allows estimates with narrow confidence intervals) for answering questions about the number or proportion of U.S. public school students in districts implementing initiatives of interest, and one that is relatively efficient for answering questions about the number or proportion of U.S. school districts implementing such initiatives. 1 To construct the sampling frame, we will use data primarily from the National Center for Education Statistics’ (NCES) Common Core of Data (CCD), with supplementary data from sources such as the U . S . Bureau of the Census’s district- level SAIPE (Small Area Income and Poverty Estimates) program for school - district percentages of families with children in poverty. We will draw a district sample of 570 di stricts out of 15,629 school districts. See Table B - 2 in Section B.2 for the universe counts by stratification classifications. 

### B.1.3. **School Sample** A nationally representative sample of schools will be selected . The school sample will be a two - stage sample, nested with in the sampled districts. This will provide unbiased estimators of school characteristics, and will provide the second stage of selection for a nationally representative sample of teachers. In addition to examining initiatives in schools nationwide, we also will statistically compare policy implementation by **school Title I status, school poverty level, and the cross -** **classification of Title I and poverty status**  (high - poverty Title I schools, low-/medium - poverty Title I schools, and non -Title I

1 The minimax design differs from the one used for the last Title I study. The previous study selected districts probability proportional to size (PPS), with size measured by student enrollment. The PPS design is quite efficient for estimating the proportion of students enrolled in districts implementing policies of interest. However, when estimating the percent of districts implementing a policy, the PPS design is relatively inefficient, compared to a simple random sample. This is because relatively few smal l and medium - sized districts are included in a PPS design. This, in turn, requires the small and medium - sized districts in the sample to be given greater weight to better represent the population of districts nationwide and can lead to relatively wide confidence intervals around estimates of proportions of districts.5

schools). 2 The school sample is not limited to Title I schools. While a key part of the study focuses on initiatives promoted by Title I, the study is not looking exclusively at initiatives funded by Title I. Furthermore, non -Title I schools may benefit from professional development funded by district Title II funds. In addition, we anticipate that implementation status and types of initiatives may vary by school grade span (elementary, middle schools, and high schools). For example, implementation of state content standards and aligned assessments, as well as responses to accountability systems, likely differs by grade level. As a result, we will look for differences betwee n Title I and non -Titl e I schools within grade span. There is variation in the poverty levels of Title I schools, with 37.5 percent of Title I schools considered high - poverty schools. 3 Our analysis will examine whether implementation of initiatives is different in the neediest Title I schools We will also implement a “minimax” sample design for the school sample. This will balance between the needs to produce efficient student-weighted estimates and effici ent school -weighted estimates. To construct the sam pling frame, we will use data from the C CD. We will draw a sample of 1,300 schools nested with in the nationally representative sample of 5 70 school districts. (The universe of schools is 92,589 schools.) See Table B - 5 in section B.2 for the universe counts by stratification classification. 

### B.1.4. **Teacher Sample** A nationally representative sample of teachers will be selected . The teacher sample will be a three - stage sample, nested within sampled districts and schools. This will provide unbiased estim ators of teacher characteristics. The sample will be drawn from a comprehensive list of teachers provided to us from each sampled school. We believe there will be considerable interest in the responses of teachers who teach classes in which students are tested for accountability requirements for ESEA4 as they have been the focus of the federal accountability system. In addition, these teachers are most likely to be affected by improved measures of educator effectiveness based on student growth. Teachers in these subjects and grades are also the most li kely to be affected by state - set achievable annual measurable objectives (AMOs) promoted by the ESEA Flexibility requests, and will most likely receive student growth data for use in improving practice and differentiating instruction (U.S. Department of E ducation, 2011) . The sample will thus allow statistical comparisons of **ESEA -** **tested and non -**  **ESEA -** **tested teachers** by school grade span on such topics as teachers’ use of student achievement, and especially growth in achievement, for evaluation, accountability, and practice improvement. Our sample will also allow sufficient numbers of teachers per district and school to allow examination of the relationship between implementation of various initiatives promoted by Title I and Title II as described by principa ls and district officials and teachers’ reports of their experiences and responses. For example, we may examine whether teachers in states that provide more extensive support for state

2 Since the majority of non -Title I schools are low/medium - poverty schools, these schools will not be broken out by poverty status. 3 The cutoff for high - poverty schools is the enrollment-w eighted 75 th percentile: this was 66.12 percent of students eligible for free or reduced lunch on an earlier frame. 4 Under ESEA, as reauthorized by the No Child Left behind Act (NCLB) of 2001, public school students are tested annually in rea ding/English language arts and mathematics in grades 3 through 8, and once in high school. Students are tested in science once in each of the following grade groupings: grades 3 - 5, grades 6 - 9, and grades 10- 12.6

content standards are more likely to report using the state content sta ndards in their classrooms than teachers in other states. The study will use an approach similar to the teacher sampling in the NCES Schools and Staffing Survey (SASS) to determine the overall number of teachers to select from each sampled school. In gen eral, the SASS approach sets a teacher sample size for each sampled school using the teacher roster count from the school and the school’s probability of selection, in such a way that yields a “self-weighting” sample (each teacher in each school has an equ al probability of selection within the three major school subgroups (non -Title I, Title I high poverty, Title I low poverty)). Effectively, the teacher sample size is selected in a way that cancels out any oversampling or undersampling of the school given its teacher count, bringing the teacher probabilities back to equality across schools within each major school subgroup, to the extent possible. This avoids large differences in the weight given to individual teachers, differences which complicate estimati ng standard errors and can reduce stability of estimates. We will draw a sample of 9,100 teachers from the 1,300 sampled schools — an average of seven teachers per sampled school. See Table B - 9 in Section B.2 for the expected teacher sample size s by school classification. We expect all states and the District of Columbia to participate in the study . We expect to obtain at least an 85 p ercent response rate from distr icts, schools, and teachers .



### B.2.1. **Notification of the sample, recruitment and data collection Introduce the Study to State Education Agencies.**  We will begin by sending the chief state school officer and the state Title I a dministrator a notification letter (see Appendix F ) and study bro chure (see Appendix G) explaining the study, the importance of the state’s involvement, and the mandatory nature of the state’s response . We will then follow up wit h a phone call to the state Title I a dministrator to answer questions about the study and id entify additional state - level respondents based on areas of expertise. Once we have secured the correct contacts, we will mail the Title I administrator a hard copy instrument with instructions to have other staff complete relevant sections. 5 The mailing will include a version of the extant data form that will be pre -filled with the data that we have collected from publicly available sources. We will ask the respondents to review the form to confirm the accuracy of the data, correct any data that were not correct on the public website, and provide any data that were not available publicly. The mailing will include a password and secure web address to access an electronic version of the questionnaire and instructions on how to submit the questionnaire via a secure SharePoint site. After the initial mailing, we will also send a follow- up email that also includes the web address and password. Project staff will monitor completion rates, review the instruments for completeness throughout the field period, and follow up by email and telephone as needed to answer questions and encourage completion. During these calls, state representatives will be given the option of completing the module

5 If requested, the study team will contact each releva nt state - level staff member directly to complete the survey. 

## B.2 **. I**  **nformation Collection Procedures**7

by telephone with the researcher. Each of the four topic areas ( state conte nt standards, aligned assessment s, accountability and school turnaround , and development of effective teachers and leaders ) will on average require between 30 and 45 minutes to complete. Researchers knowledgeable about the four content areas will review the completed questionnaire and documentation downloaded from state and other publicly available web sites for completeness. We will then conduct follow up calls with states to clarify any ambiguous answers in the survey. **Introduce the Study to District and School Leaders.**  We will send notification letters (see Appendix F ) and the study brochure (see Appendix G) by email and mail to sampled districts and school s. N otification letters will be customized for each type of respondent, informing them of the study’s importance and benefits. Sending both email and mail will increase the likelihood that addressees will receive our communications in a timely manner. States and districts receiving Title I and Title II funds have an obligation to participate in Department evaluations (Education Department General Administrative Regulations (EDGAR) ( 34 C.F.R. § 76.591)) , and virtually all states and districts receive Title I and/o r Title II funds . Since we are not able to discern which schools receive Title I and/or Title II funds, survey participation is not mandatory for school principals. Mailings to district s uperintendents will include the names of the sampled schools. Becau se we anticipate that district surveys will require input from several key individuals, we will ask the d istrict s uperintendent to provide contact information (including email) for a person d esignated by the s uperintendent as the study liaison , wh o will co ordinate the completion of the survey by the appropriate staff. Once the district liaison is identified, we will conduct all follow - up directly with the district liaison. For those districts that do not respond and identify a study liaison online within five days, we will make follow- up telephone calls to district s uperintendents to confirm receipt of the letter, answer any questions, confirm the identity of the study liais on, and obtain the designated liaison’s contact information . We will follow all required procedures, and as necessary, we will obtain the approval of the district for principal and teacher participation through submission of the required research applicati on. Notification letters will not be sent to principals or teachers prior to district approval. Principal mailings will begin once we have received any required district approval. The principal mailings will inform principals of their schools’ selection a nd also will include directions for completing the teacher roster via the web from which we will draw the teacher sample. Principals will be asked to verify their schools’ current grade span s and list all teachers along with teachers’ email address es, prim ary grade s and subjects taught, ( e.g. , reading/English/language arts, mathematics, s cience and other ), and whether the teachers teach class es in which students are tested for accountability requirements under ESEA . Mockups of web pages that could be used to collect this information are shown in Appendix E (Part A). Online edits in the web roster will ensure that all data items required for teacher sampling are entered. The principal mailing will include a hardcopy roster listing all of these items and a fax number and business reply envelope, should school staff prefer to submit hardcopy. We will also make available at the school’s request an Excel template should the school staff prefer to enter teacher informati on in Excel . We will also accept roster output from the school’s database, either electronically or on hardcopy. We will work with the principal or his/her designee to be sure all fields required for teacher sampling are captured. As teachers are selected from a school, their notification letters will be mailed.8

**Administer Surveys.** By email and mail , we will send a notification letter to all respondents ( i.e., school principa ls, and teachers). The letters and emails will underscore the purpose of the study and the importance of participation . Letters and emails will be tailored to the respondents with the district communication informing district- level respondents that completing the web - based survey is mandatory and required by law. Principal s will be informed that their participation is voluntary . We will emphasize the survey is an opportunity for principals to provide valuable information ab out how the policies of ESEA influence teaching and learning. Teachers also will be informed that their particip ation is voluntary and that they will receive $20 as a than k-you for their participation. In districts that require research applications for principal and teacher participation , we will send the principal and teacher emails once we obtain district approva l to begin data collection. Mailed letters will include the survey URL and login information for responding to the survey as a web - based instrument . For security purposes, the first email will provide the URL and the User ID with the respondent’s password provided in a second email . All communications will include a toll -free study number and a study email address for respondents’ questions and technical support. Based on Westat’s experience on large - scale data collections, we will assign several trained research staff to answer the study hotline and reply to emails in the study mailb ox. We will train them on the purpose of the study, the obligations of district respondents to participate in the evaluation, and the details for completing the web - based survey. Content questions will be referred to the study leadership. An internal FAQ document will be developed and updated as needed throughout the course of data collection to ens ure that the research staff has the most current information on the study. The web will be our primary method of data collection for the district, school and teacher surveys. We will offer resp ondents the option of emailing them an electronic version of the survey (e.g., PDF or Word document) to complete and return by email or completing a paper- and - pencil instrument. However, we have found that the vast majority of users prefer the web - based approach. A phone survey option will be offered to respondents as part of the nonresponse follow- up effort. Since the web - based surveys will include data checks, we will use the web - based surveys to enter an y surveys received on hard copy or by phone. Westat will develop a web - based data monitoring system (DMS) to track the sample for each instrument, record the status of district approvals, genera te materials for mailings, and monitor survey response rates. 

### B **. 2.2**  **. Statistical Methodology for Stratification and Sample Selection**  **B .**  **2.2 .1.**  **Nationally representative sample of school districts** The district sample will be stratified by poverty status and district size. The poverty strata are defined based on the percent of families with children in poverty. The high - poverty stratum consists of the roughly 25 percent of districts with percentages greater than the national 75 th percentile . 6 The low/medium - poverty stratum consists of the complement set (roughly 75 percent of the districts). The district size strata are given in Table B - 1. It should be noted that for comparing adjacent classes, each

6 This percentile is weighted by enrollment, and is found using t he U . S . Bureau of the Census school district SAIPE (Small Area Income and Poverty Estimates). In a recent national district sample, we found that this 75 th percentile was 16.7 percent of families in poverty. The exact percentile will be computed using the new fra me when it is developed. We will utilize SAIPE in assigning districts to these poverty strata.9

class has an enrollment range roughly three times greater than the preceding class (in terms of minimums, mean value, or maximums). **Table B -**  **1 . Definitions of district size strata**  **District oversampling class**  **Lower bound district enrollment**  **Upper bound district**

**enrollment** G 1 500 F 501 1,500 E 1,501 5,000 D 5,001 15,000 C 15,001 50,000 B 50,001 150,000 A 150,001 no limit A total of 570 districts will be sampled, with oversampling of high - poverty districts by a factor of 3. This oversampling will bring the expected sample size for high - poverty districts in line with the expected sample size for low/medium - poverty districts (285 per each stratum). Table B - 2 presents the proposed strata for sampling districts, with relative sampling rates (as compared to the stratum with the lowest sampling rate). Note that the counts are based on the 2008 - 2009 school year CCD frame, and m ay be slightly different when the final frame is developed. Note that under a probability proportionate to size by enrollment design, the relative sampling rates between neighboring district size classes would be 3, as that is roughly the enrollment ratio. By using powers of 1.87 rather than powers of 3 as relative sampling factors, we are oversampling the strata with the higher enrollments, but not to the full extent justified by the ratios of enrollment means. The largest nine low/medium - poverty stratum a nd the largest 25 high - poverty stratum districts are sampled with certainty (indicated with a relative sampling rate of infinity) . 7 The exceptionally large size of these districts will make them larger than the sampling interval under the minimax design (d escribed below) , and they will be taken as certainties to maintain efficiency. **Table B -**  **2 . Proposed stratification design for district sampling**  **Poverty stratum**  **District**

**size class**  **District count**  **Percent**

**of districts**  **Relative**

**sampling rate**  **Expected**

**district sample**  **size Percent** 

**of district**  **sample** Low/med G 3,669 23.5% 1.00 31.6 5.5% Low/med F 3,163 20.2% 1.87 50.9 8.9% Low/med E 3,048 19.5% 3.50 91.7 16.1% Low/med D 1,131 7.2% 6.54 63.6 11.2% Low/med C 348 2.2% 12.23 36.6 6.4% Low/med B 55 0.4% 22.87 10.8 1.9% Low/med A 9 0.1% Inf 9 1.6% Low/med Total 11,423 73.1% 0.00 294 51.6%

7 These counts may change somewhat after reviewing the distribution of districts on the final frame, and their relationship to the sampling intervals.10

High G 1,920 12.3% 3.00 49.6 8.7% High F 1,046 6.7% 5.61 50.5 8.9% High E 858 5.5% 10.49 77.4 13.6% High D 268 1.7% 19.62 45.2 7.9% High C 89 0.6% 36.68 28.1 4.9% High B 19 0.1% Inf 19 3.3% High A 6 0.0% Inf 6 1.1% High Total 4,206 26.9% 276 48.4% Total Total 15,629 100.0% 570 100.0% This sample design we call a ‘minimax’ design, as it is designed to equalize the efficiency for two types of estimates. The first type of estimate counts each district as one in the population, so that the base weight is the inverse of the district probabi lity of selection. This type of ‘count - based’ estimate answers questions such as “What percentage of districts have characteristic X?” The second type of estimates includes enrollment of the district, so that the sampling base weight is the enrollment divi ded by the probability of selection. This type of ‘enrollment- based’ estimate answers questions such as “What percentage of students are in enrolled districts which have characteristic X?” A probability proportionate to enrollment design will lead to optimal efficiency for the second type of estimate, but will have poor efficiency for the first type of estimate (as the district weights will be close to equality for the enrollment- based esti mate, but will vary considerably for the count- based estimate). On the other hand, a simple stratified design with no oversampling of larger district-size strata will have high efficiency for count- based estimates, but poor efficiency for enrollment- based estimates. This ‘middle -ground’ design oversamples the higher enrollment district- size strata, but proportional to the 0.56 root 8 of the enrollment mean in the stratum, rather than to enrollment directly, 9 and will have reasonable efficiency for both count- based estimates and enrollment- based estimates (the design is set up to equalize the efficiency for both types of estimates, at the cost of not being as good for each type of estimate as the optimal design for that type of estimate). Table B - 3 summariz es the properties of this design as they would be if implemented on the 2008/2009 CCD frame . 10 **Table B -**  **3 . Properties of proposed stratification design for district sampling** 

**Power Property Enrollment** **** **based weight**  **estimates Count** ****

**based weight**  **estimates** Effective sample size : All districts 286.9 318.3 Effective sample size : High - poverty districts 179.3 187.7 Effective sample size : Low/medium - poverty districts 184.7 183.1 Minimum detectable effect size (MDES) comparing Poverty District Strata 29.4% 29.1%

8 1.87 is the 0.56 root of 3. 9 This design is close to a ‘square root’ design, except that it is stratified design rather than a fully PPS design (sampling rates are equal within strata), and the root used is slightly larger than 1⁄2. 10 As processed to drop ineligible schools and entities, schools with no enrollment, etc.11

The effective sample sizes are the sample sizes for a simple random sample which would provide the same precision as the design . 11 Note that the effective sample size for all - district estimates is about half of the district sample size of 570. This large ratio is caused partially by the oversampling of high - poverty districts. Note also an equalization of effective sample sizes for t he two types of estimates. This is the ‘minimax’ aspect. The MDES (minimum detectable effect size) is computed for evaluating the null hypothesis of no difference between the high - poverty and the low/medium - poverty districts for a range of district- level c haracteristics . 12 This is a very important comparison at the district level for this study, and the sample design is carefully tailored with this comparison in mind ( we effectively equate the sample designs for the two strata, though the high - poverty distri cts are only one quarter of enrollment). The sample design does achieve an MDES lower than 30 percent for both types of estimates . 13 The district frame will be based on the most recent N CES CCD frame, as processed through National Assessment of Educationa l Progress (NAEP) macros to purge entities that are not in scope (e.g., administrative districts, district consortiums, entities devoted to auxiliary educational services, etc.), as well as a canvassing of new districts from the preliminary CCD frame from the following year (only districts with positive enrollments on the CCD frame, with other out -of- scope entities purged out). All school districts and independent charter districts with at least one eligible school and at least one enrolled student will be included in the frame. 14 **Further Details**  **of District**  **Sampling .**  The primary strata are by district- size class and poverty status as described in the previous section. We also plan to define a stratum for small states (states with an expected district sampl e size less than 1), and carry out a systematic sample within this stratum by state in order to guarantee that every state has at least one selected district. The remaining complement stratum will be stratified by poverty status (high and medium/low) and d istrict- size class as described above. Within these primary strata, there will be further implicit stratification by ordering the sample. This ordering will be defined by urbanicity, Census region, and finally district enrollment. The high - poverty stratum will be formally defined by SAIPE estimates of percentages of 5 to 17 year old children in poverty for the school district. We will compute the weighted 75 th percentile percentage over all districts in the U.S., and the mean value will become the c utoff. Districts with percentages lower than the cutoff will be designated ‘low/medium - poverty’ and districts with percentages higher than the cutoff will be designated ‘high poverty.’ Independent districts such as charter school districts will be associa ted with the public school district that they are associated with geographically, as only the primary geographically- based public school districts have poverty estimates from SAIPE. Any districts with only one school will have a sampling rate set to be 1/4 of the sampling rate they would otherwise receive. Even with this undersampling, they will still be represented correctly in the population, as their weights will reflect their reduced probabilities of selection (the weights will be four times larger th an they would otherwise be), but we will have fewer of these districts. This method of undersampling is similar to that done in the N AEP for schools with very small numbers of students.

11 The effective sample size is equal to the population variance divided by the sampling variance under the design. 12 We assume a null hypothesis of no difference with a two - sided critical region with a 5 percent alpha level. We find the smallest population difference that would be detectable with this test with 80 percent power. The MDES is this population difference divided by the (assumed) common population standard deviation for each su bgroup. 13 This assumes 100 percent district response. District nonresponse will degrade these power results, but we expect minimal district nonresponse as this study is by law mandatory for the districts. 14 In defining district eligibility, we follow the criteria from the N AEP . The NAEP macros for excluding districts are applied also in the generation of the district frame here.12

**B .**  **2. 2**  **.2. Nationally representative sample of schools**  The school sample is a two - stage sample of 1,300 schools, nesting within the sampled districts. There will be sampling rates assigned for sampling groups defined by school Title I status, 15 school poverty status, and by school span and school size. Table B - 4 present s the proposed 10 span - size subgroups with differential sampling rates. School grade span is defined as follows, following the C CD definition:

 elementary is defined to have a low grade of P re - K through 3 rd grade, and a high grade of P re - K through 8 th gra de;  middle is defined to have a low grade of 4 th through 7 th grade, and a high grade of 4th through 9 th grade;  high school is defined to have a low grade of 7 th through 12 th, and a high grade of 12 th only; and  other schools is defined to include all oth er schools.

15 For sampling purposes, we will use the Title I status variable that is on the CCD that indicates whether the school is a Title I eligible school. We will confirm whether a sampled school received Title I for the 201 3 - 201 4 school year or remained Title I eligible.13

**Table B -**  **4 . Proposed school span**  **- size subgroups for differential sampling rates**  **School span School size** 

**class School** 

**count Percent** 

**of schools**  **Enrollment Percent**  **of enroll**  **** **ment Minimum** 

**e nroll**  **** **ment Maximum** 

**e nroll**  **** **ment Mean** 

**e nroll**  **** **ment** Elementary C-Small 22,120 23.9% 5,581,817 11.4% 1 400 252 Elementary B - Medium 17,162 18.5% 8,453,902 17.3% 401 600 493 Elementary A- Large 11,887 12.8% 9,123,017 18.7% 601 4,595 767 Middle C-Small 8,016 8.7% 2,350,833 4.8% 1 525 293 Middle B - Medium 4,491 4.9% 2,947,351 6.0% 526 800 656 Middle A- Large 3,912 4.2% 4,128,550 8.4% 801 3,152 1055 High C-Small 11,114 12.0% 2,957,123 6.0% 1 700 266 High B - Medium 3,961 4.3% 4,155,302 8.5% 701 1,450 1049 High A- Large 3,379 3.6% 7,075,509 14.5% 1,451 5,017 2094 Other 6,547 7.1% 2,140,757 4.4% 1 7,874 327 Total 92,589 100.0% 48,914,161 100.0% The school design will define three major subgroups: non -Title I schools, Title I high - poverty schools, and Title I low/medium - poverty schools. Given the importance of Title I schools and the need to compare non -Title I schools to Title I schools, these subgroups are very important. Poverty status is defined for schools in terms of percentage of students eligible for free or reduc ed price lunch. The cutoff for high poverty schools is the enrollment-weighted 75 th percentile: 66.12 percent of students eligible for free or reduced lunch. Table B - 5 presents the breakdown of school counts and enrollment according to a frame based on t he 2008/2009 C CD frame.14

**Table B -**  **5 . Subgroups for the school sample design** 

**Major school group**  **Span School**  **size group Frame** 

**count Percent** 

**of frame**  **Total**

**enrollment Percent** 

**of enroll**  **** **ment Mean** 

**enroll**   ** ment** **Enroll**  

** ment** **ratio 3/4** 

**root enroll**  **** **ment ratio**  Non -Title I Elem C-Small 4,227 4.57% 997,967 2.04% 236 0.94 0.953 Non -Title I Elem B - Medium 4,207 4.54% 2,092,331 4.28% 497 1.98 1.666 Non -Title I Elem A- Large 3,611 3.90% 2,809,827 5.74% 778 3.09 2.331 Non -Title I Middle C-Small 2,146 2.32% 636,414 1.30% 297 1.18 1.131 Non -Title I Middle B - Medium 1,788 1.93% 1,182,990 2.42% 662 2.63 2.064 Non -Title I Middle A- Large 1,810 1.95% 1,905,772 3.90% 1,053 4.18 2.924 Non -Title I High C-Small 5,466 5.90% 1,405,599 2.87% 257 1.02 1.016 Non -Title I High B - Medium 2,272 2.45% 2,408,530 4.92% 1,060 4.21 2.939 Non -Title I High A- Large 2,032 2.19% 4,200,144 8.59% 2,067 8.21 4.850 Non -Title I Comb 3,026 3.27% 866,858 1.77% 286 1.14 1.102 Non -Title I Total Total 30,585 33.03% 18,506,432 37.83% Title I high pov Elem C-Small 6,551 7.08% 1,727,962 3.53% 264 1.05 1.035 Title I high pov Elem B - Medium 5,251 5.67% 2,587,753 5.29% 493 1.96 1.655 Title I high pov Elem A- Large 4,096 4.42% 3,186,122 6.51% 778 3.09 2.330 Title I high pov Middle C-Small 1,834 1.98% 550,176 1.12% 300 1.19 1.140 Title I high pov Middle B - Medium 831 0.90% 541,395 1.11% 651 2.59 2.040 Title I high pov Middle A- Large 759 0.82% 829,877 1.70% 1,093 4.34 3.008 Title I high pov High C-Small 1,647 1.78% 428,940 0.88% 260 1.03 1.026 Title I high pov High B - Medium 429 0.46% 438,544 0.90% 1,022 4.06 2.860 Title I high pov High A- Large 317 0.34% 710,281 1.45% 2,241 8.90 5.152 Title I high pov Comb 1,072 1.16% 314,460 0.64% 293 1.16 1.121 Title I high pov Total Total 22,787 24.61% 11,315,510 23.13% Title I low pov Elem C-Small 11,342 12.25% 2,855,888 5.84% 252 1.00 1.000 Title I low pov Elem B - Medium 7,704 8.32% 3,773,818 7.72% 490 1.95 1.647 Title I low pov Elem A- Large 4,180 4.51% 3,127,068 6.39% 748 2.97 2.263 Title I low pov Middle C-Small 4,036 4.36% 1,164,243 2.38% 288 1.15 1.107 Title I low pov Middle B - Medium 1,872 2.02% 1,222,966 2.50% 653 2.59 2.044 Title I low pov Middle A- Large 1,343 1.45% 1,392,901 2.85% 1,037 4.12 2.891 Title I low pov High C-Small 4,001 4.32% 1,122,584 2.30% 281 1.11 1.085 Title I low pov High B - Medium 1,260 1.36% 1,308,228 2.67% 1,038 4.12 2.894 Title I low pov High A- Large 1,030 1.11% 2,165,084 4.43% 2,102 8.35 4.911 Title I low pov Comb 2,449 2.65% 959,439 1.96% 392 1.56 1.393 Title I low pov Total Total 39,217 42.36% 19,092,219 39.03% Total Total Total 92,589 100.00% 48,914,161 100.00% The enrollment ratio is the ratio of the mean enrollment for the subgroup as compared to the subgroup Title I low- poverty, small elementary schools. The 3⁄4 root of this ratio is the basis for the sampling rate for the subgroups. This power is selected as it equalizes the precision for enrollment- based and countbased estimates (see the discussion b elow). The second part of this sampling rate is a multiplier for the three major subgroups. This multiplier is given in Table B - 6 below.15

**Table B -**  **6 . Expected school sample sizes for the subgroups in the school sample**  **design**

**Major school group**  **Span School**  **size group**  **Percent**

**of enroll**  **** **ment Percent** 

**of frame Rel**  **ative**

**samp le**  **rate Major** 

**school group**  **multi**   ** plier** **Final**

**rel ative**  **samp le**  **rate Expected** 

**sample size**  **Percent**

**of sample**  Non -Title I Elem C 2.04% 4.57% 0.953 1.1 1.05 27.8 2.14% Non -Title I Elem B 4.28% 4.54% 1.666 1.1 1.83 48.4 3.73% Non -Title I Elem A 5.74% 3.90% 2.331 1.1 2.56 58.2 4.47% Non -Title I Middle C 1.30% 2.32% 1.131 1.1 1.24 16.8 1.29% Non -Title I Middle B 2.42% 1.93% 2.064 1.1 2.27 25.5 1.96% Non -Title I Middle A 3.90% 1.95% 2.924 1.1 3.22 36.6 2.81% Non -Title I High C 2.87% 5.90% 1.016 1.1 1.12 38.4 2.95% Non -Title I High B 4.92% 2.45% 2.939 1.1 3.23 46.1 3.55% Non -Title I High A 8.59% 2.19% 4.850 1.1 5.33 68.1 5.24% Non -Title I Comb 1.77% 3.27% 1.102 1.1 1.21 23.0 1.77% No n -Title I 37.83% 33.03% 389.0 29.92% Title I high pov Elem C 3.53% 7.08% 1.035 2.25 2.33 95.9 7.38% Title I high pov Elem B 5.29% 5.67% 1.655 2.25 3.72 122.8 9.45% Title I high pov Elem A 6.51% 4.42% 2.330 2.25 5.24 134.9 10.38% Title I high pov Middle C 1.12% 1.98% 1.140 2.25 2.57 29.6 2.27% Title I high pov Middle B 1.11% 0.90% 2.040 2.25 4.59 24.0 1.84% Title I high pov Middle A 1.70% 0.82% 3.008 2.25 6.77 32.3 2.48% Title I high pov High C 0.88% 1.78% 1.026 2.25 2.31 23.9 1.84% Title I high pov High B 0.90% 0.46% 2.860 2.25 6.44 17.3 1.33% Title I high pov High A 1.45% 0.34% 5.152 2.25 11.59 23.1 1.78% Title I high pov Comb 0.64% 1.16% 1.121 2.25 2.52 17.0 1.31% Title I high pov 23.13% 24.61% 520.7 40.06% Title I low pov Elem C 5.84% 12.25% 1.000 1 1.00 71.3 5.48% Title I low pov Elem B 7.72% 8.32% 1.647 1 1.65 79.7 6.13% Title I low pov Elem A 6.39% 4.51% 2.263 1 2.26 59.4 4.57% Title I low pov Middle C 2.38% 4.36% 1.107 1 1.11 28.1 2.16% Title I low pov Middle B 2.50% 2.02% 2.044 1 2.04 24.0 1.85% Title I low pov Middle A 2.85% 1.45% 2.891 1 2.89 24.4 1.88% Title I low pov High C 2.30% 4.32% 1.085 1 1.08 27.3 2.10% Title I low pov High B 2.67% 1.36% 2.894 1 2.89 22.9 1.76% Title I low pov High A 4.43% 1.11% 4.911 1 4.91 31.8 2.44% Title I low pov Comb 1.96% 2.65% 1.393 1 1.39 21.4 1.65% Title I low pov 39.03% 42.36% 390.3 30.02% Total 100.00% 100.00% 1,300 100.00% The major subgroup multipliers were generated with the goal of making the non -Title I, Title I low/medium - poverty, and Title I high - poverty schools 30 percent, 30 percent, and 40 percent of the overall school sample respectively and thus, close to equal in size . To do this, the Title I high - poverty schools need to be sampled at a 2.25 higher rate than the Title I low- poverty schools, and the non -Title I schools at a 1.1 times higher rate than the Title I low- poverty schools. Table B - 7 presents the effective sample sizes for all schools and for the three major subgroups, and for enrollment- based and countbased school estimates. As can be seen the precision for the two types of estimates is close: that is the16

‘minimax’ property. The effective sample sizes across the three subgroups are roughly the 30 - 30 -40 breakdown which was desired. **Table B -**  **7 . Power properties of school design** 

**Power Property Sample size**  **Enrollment****** **based weight**  **estimates Count** ****

**based weight**  **estimates** Effective sample size : A ll schools 1,300 982 970 Effective sample size : N on -Title I schools 389 325 305 Effective sample size : Title I low/medium - poverty schools 390 330 333 Effective sample size: Title I high - poverty schools 521 461 450 The school sample is nested within the district sample, so this constitutes a clustered design. This will facilitate school recruiting and will allow for a comparison of school and district responses within the sampled districts. The final school probabilities are equal to the unconditional probabilities developed bas ed on the school design described in Table B - 6, divided by the district probabilities of selection. This clustered design means that intra - district correlation among the schools in each district will affect the precision of the estimates. Table B - 8 pre sents the effective sample sizes under a variety of different intra - district correlatio n coefficients. The mean number of sampled schools per sampled district is 2.28 (1,300 schools divided by 570 districts). We will approximate the design effect due to di strict clustering as . . = 1 + ( − 1 ) , with the intradistrict correlation coefficient, and the mean cluster size 2.28. **Table B -**  **8 . Effective sample sizes for national school**  **- level estimates under the 3/4** **th root school** 

**design**

**Intra -** **district correlation**  **coefficient Within**  ****

**district mean**  **school sample**  **size Design** 

**effect Effective** 

**enrollment****** **based weight**  **school sample size**  **Effective**

**count****** **based weight**  **school sample**  **size CV of** 

**enrollment****** **based weight**  **estimate CV of** 

**count****** **based weight**  **estimate** 0% 2.28 1.000 981.9 970.2 3.19% 3.21% 5% 2.28 1.064 828.8 828.1 3.47% 3.48% 10% 2.28 1.128 781.8 781.1 3.58% 3.58% 15% 2.28 1.192 739.8 739.2 3.68% 3.68% 20% 2.28 1.256 702.1 701.5 3.77% 3.78% 25% 2.28 1.320 668.1 667.5 3.87% 3.87% 30% 2.28 1.384 637.2 636.6 3.96% 3.96% 35% 2.28 1.448 609.0 608.5 4.05% 4.05% 40% 2.28 1.512 583.2 582.7 4.14% 4.14% 45% 2.28 1.576 559.6 559.1 4.23% 4.23% 50% 2.28 1.640 537.7 537.3 4.31% 4.31% The effective sample sizes comparing non -Title I, Title I high - poverty, and Title I low- poverty schools are given in Table B - 7, depending on the type of estimate (enrollment- based or count- based). The lowest17

precision is for the pair non -Title I schools vs. Title I low/medium - pove rty schools with count- based estimates, where the effective sample sizes are 305 and 333 respectively. Assuming no effect from intra distr i ct correlation, this translates to an MDES of 22.2 percent for comparing non -Title I and Title I high poverty school s, and slightly lower MDES for the other possible pairs . 16 **Further Details of School Sampling .**  The school sample is nested within the district sample, and districts will then be the major stratum for school sampling. The mean school sample size per district is very small (2.28), so there is not much room for stratification at the school level beyond districts, but we will implicitly stratify by Title I status (i.e., within districts, systematically sample the schools sorted by Title I status) , and if there is sufficient sample size, by school span (elementary, middle/combined, high) . 17 **B .**  **2. 2**  **.3. Nationally representative sample of teachers**  The teacher sample will be 9,100 teachers, sampled in 1,300 sampled schools: a mean of 7.0 teachers per sampled school. Expecting at least 8 5 percent teacher response, this will result in 7,735 expected completed surveys, a mean value of 5.95 teachers per sampled school. The teacher sample sizes per school will vary across schools. The plan is to set the teacher sample size in a way to equalize the final teacher weights within the four basic school - span strata (elementary, middle, high, and combined schoo ls), to the extent possible, given the need to have integer teacher sample sizes. Our plan will be similar to that used for teacher sampling in SASS (Schools and Staffing Survey), carried out by the N CES . Documentation regarding SASS teacher sampling can b e found in http://www.nces.ed.gov/surveys/sass . SASS is done once every four years (2003/2004, 2007/2008, 2011/2012). We define a school weight for stratum ( =1,..., ) and school ( =1,..., ). This is the inverse of the school probability of selection, disregarding oversampling for the three major school subgroups (we adjust for differential probabilities due to enrollment, but not due to school Title I status and school poverty status) . 18 Define as the teacher count for school . Define as the mean per-school teacher sample size for stratum (e.g . , 6.2 teachers for elementary schools) and as the school sample size in stratum . The teacher sample size for school will be

∗ ∗ ∗ ∑ ∗ , 10

16 Note that the effect of a positive intra - district correlation could be to either increase or decrease the precision, as we expect many low/medium - poverty districts to have at least one non -Title I school and one Title I school. For these districts, the presence of intra - district correlation actually increases the precision of the difference between the two subgroups. For other districts which have more than one of one subgroup, but none of the other, positive intra - district correlation will reduce the precision of the difference. Thus the overall effect will be ambig uous (depends on the exact distribution of the sampled schools across the sampled districts). 17 We would implicitly stratify by school span for example if all schools in a district were Title I, or non -Title I, or if the district sample size happened to be larger. 18 This means that teacher sample sizes will be larger for schools which were undersampled due to enrollment, and smaller for sc hools which were oversampled due to enrollment, bringing teacher sample sizes back to population percentages. Howeve r, for the three major subgroups, this reversion to population percentages will not take place. All other things being equal, teacher sample sizes w ithin each school across the three major subgroups will not be different, resulting in an oversample of teac hers in high - poverty Title I schools, roughly proportional to the 2.2 oversampling rate for these schools, and a slight oversample in non -Title I schools (a 1.1 oversampling rate), as compared to low/medium - poverty Title I schools.18

where INT is the function that rounds to the nearest positive integer. Note that if the school probability of selection was directly proportional to teacher count within each stratum, t hen and would be equal to a constant within each stratum , and the teacher sample size would be simply for each school in stratum . The schools are sampled very roughly proportionally to the 3⁄4 - root of enrollment within the two major strata (high - poverty districts and low/ medium - poverty districts), so assuming that teacher count is proportional to enrollment (roughly) in the school strata, will tend to be inversely proportional to the 3⁄4- root of , making teacher sample size roughly proportional to the 1⁄4 root of teacher count. Thus, the teacher count will be slightly larger for larger schools, and slightly smaller for smaller schools, but not far from unless the teacher- student ratio is unusual for the school, or the sch ool’s enrollment is on the extreme end of the distri bution for the school stratum. An upper bound of 10 is placed on the teacher sample size per school to minimize burden. Within the two major district strata, the design for teachers should be close to self-weighting. We will assume design effects of 1.15 to allow for distortions from rounding to integers, respecting the maximum burden of 10, and to also allow for the effect of necessary teacher nonresponse adjustments. Table B - 9 below summarizes the des ign and calculates effective teacher sample sizes assuming the design effect of 1.15 within the three major school subgroups, and the design effect from oversampling t he high - poverty Title I and non -Title I schools by factors of 2.25 and 1.1 respectively ( relative to lowpoverty Title I schools). In Table B - 9 we also assume an ICC of 0 percent for clustering of teachers within sampled schools. **Table B -**  **9 . Expected teacher sample sizes**  **School subgroup**  **Expected**

**school sample**  **size Mean** 

**sampled teachers**  **per school Total** 

**sampled teachers**  **Percent of**

**sample teachers**  **Percent of**

**pop'n teachers**  **Expected**

**teacher response**  **rate Expected** 

**final teachers**  **per school**  **Total**

**final teacher**  **count Effective** 

**teacher sample**  **size** Non -Title I 389 7 2,723 29.90% 37.80% 85% 5.95 2,315 2,013 Title I low pov 390 7 2,730 30.00% 39.00% 85% 5.95 2,321 2,018 Title I high pov 521 7 3,647 40.10% 23.10% 85% 5.95 3,100 2,696 Total 1,300 7 9,100 100.00% 100.00% 85% 5.95 7,735 6,018 Table B - 10 presents the effective sample sizes for national teacher estimates under a variety of different intra - district correlation coefficients. The mean number of final teachers per sampled school is 5. 95 . We will approximate the design effect due to district clustering as . . = 1 + ( − 1 ) , with the intradistrict correlation coefficient, and the mean cluster size 5.95 .19

**Table B -**  **10 . Effective sample sizes for national t** **eacher estimates Intra**  **** **school correlation**  **coefficient Within**  ****

**school mean**  **teacher sample**  **size Design** 

**effect Effective** 

**teacher sample**  **size** 0% 5.95 1 6,018 5% 5.95 1.2475 4,824 10% 5.95 1.495 4,025 15% 5.95 1.7425 3,454 20% 5.95 1.99 3,024 25% 5.95 2.2375 2,690 30% 5.95 2.485 2,422 35% 5.95 2.7325 2,202 40% 5.95 2.98 2,019 45% 5.95 3.2275 1,865 50% 5.95 3.475 1,732 We are also interested in generating estimates for teachers separately for elementary, middle, and high schools. Table B - 11 provides expected sample sizes for each of these three span subgroups. The expected final teachers per school are our expectations for the average number of teacher surveys given the expected school sample sizes and an assumed allocation of teachers by school span based on enrollment percentages 19 (6.2, 7.3, and 8.8 sampled teachers for elementary, middle, and high schools respectively, with an 8 5 percent response rate to obtain the expected surveyed teacher means of 5.27 , 6.21, and 7.48 respectively). Th e effective sample size for the major subgroups assume s a design effect of 1.15 (accounting for teacher nonresponse adjustments and varying teacher sample sizes). The effective sample size for the three subgroups combined within grade span also includes the design effect induced from oversampling non -Title I s chools and high - poverty Title I schools, by benchmarking by subgroup enrollment figures (assuming that student enrollment is exactly proportional to teacher counts).

19 Teacher allocations by grade span based on enrollment percentage s are only provisional (it cannot be expected that teacher/student ratios are exactly equal across grade spans). Final allocations of teacher sample sizes by grade span will be produced when the fram e for this st udy is generated.20

**Table B -**  **11 . Sample sizes and effective sample sizes for t** **eachers for elementary, middle, and high**

**schools Span**  **Major school group Expected**  **sample size Expected** 

**final teachers**  **per school Total final** 

**teachers Effective** 

**sample size**  **Enrollment** Elementary Non -Title I 134 5.27 706 614 5,900,125 Elementary Title I low pov 210 5.27 1,107 962 9,756,774 Elementary Title I high pov 354 5.27 1,866 1,622 7,501,837 Elementary Total 698 3,678 2,818 23,158,736 Middle Non -Title I 79 6.205 490 426 3,725,176 Middle Title I low pov 77 6.205 478 415 3,780,110 Middle Title I high pov 86 6.205 534 464 1,921,448 Middle Total 241 1,502 1,186 9,426,734 High Non -Title I 153 7.48 1,144 995 8,014,273 High Title I low pov 82 7.48 613 533 4,595,896 High Title I high pov 64 7.48 479 416 1,577,765 High Total 299 2,237 1,828 14,187,934 Total Total 1,239 5.99 7,417 46,773,404 Table B - 12 presents power calculations for each grade span grouping, with the effective teacher sample sizes as given in Table B - 11. The first effective sample size accounts for school sampling and teacher weighting effects. The ‘final effective sample size’ also accounts for intra - school correlation, using the design effect . . = 1 + ( − 1 ) with equal to the intra - school correlation and equal to the cluster size (the average number of interviewed teachers per school. The assumed intra -school cor relations for this table are 50 percent for elementary schools, 40 percent for middle schools, and 3 0 percent for high schools (we assume higher clustering due to the small size and assumed ‘homogeneity’ within elementary schools). **Table B -**  **12 . Sample sizes and effective sample sizes for teachers for elementary, middle, and**  **high**

**school grade spans Span**  **Expected**

**final teachers**  **Effective**

**sample size**  **(0% ICC) Assumed** 

**intra**   ** school** **correlation Cluster** 

**size Design**  **effect Final**  **effective sample**  **size** Elementary 3,678 2,818 50% 5.27 3.135 899 Middle 1,502 1,186 40% 6.205 3.082 385 High 2,237 1,828 30% 7.48 2.944 621 Within each sampled school, the teachers will be selected in a stratified sample. The overall teacher sample size will be determined by the SASS algorithm. The selection process is designed to allow for21

sufficient sample sizes for comparisons of teachers of ESEA-tested and non - ESEA tested grades and subjects, and support a nationally representative sample. For elementary schools, teachers in Kindergarten through 2 nd grade are non - ESEA tested teachers. Math, reading/English/language arts (ELA), and science teachers in 3 rd grade and up are teachers of tested subjects and grades. Other teachers in 3 rd grade and up are teachers of non -tested subjects. The specific sampling allocations will be carried out:

 Teachers of Kindergarten through 2 nd grade selected proportional to grade count;  Teachers of upper grades reading, math, and science selected proportional to grade count divided by 2;  Teachers of upper grades other subjects selected proportional to grade count divided by 2. For example, if the school is K through 6, 3/7 of the sample will be allocated to Kindergarten through 2 nd grade, 2/7 to upper grades reading, math, and science, and 2/7 to upper grades other subjects. For middle schools, because most students are tested annually through grade 8, there are no non -tested grades, though there are teachers who teach non -tested subjects. Because most states test in reading, math, and science, teachers of these subjects will be taken as representing teache rs of tested subjects, and the others as representing teachers of non -tested subjects. To ensure enough teachers to compare responses across tested versus non -tested subjects, the sampling allocation will be as follows:

 Teachers of reading/English/languag e arts, math, and science selected at a rate of 3 in 7;  Teachers of other subjects selected at a rate of 4 in 7. For middle schools, the allocated mean teacher sample size will be 7 (in some cases, the sample size will be lower, in some cases higher). Thu s, in most cases, the 3 in 7 allocation should yield close to three teachers. Allocation of three teachers in this design should provide for a good representation of each of the three subjects. Selection of 4 in 7 other teachers should yield close to four other teachers in most cases, and this allow sufficient representation of the larger variety of non -tested subjects taught at the middle school level, compared with the elementary school level. For high schools, testing patterns vary by state, and even by district. ESEA requires students be tested once in math, reading/ELA, and science, but some states and districts test more often. Again, we will allocate the teacher sample to provide for sufficient numbers of teachers of both tested and non -tested subj ects. The allocation will be as follows:

 Teachers of reading/English/language arts, math, and science in tested class/grade selected at a rate of 1 in 3;  Teachers of reading/English/language arts, math, and science in non -tested class/grade selected at a rate of 1 in 3;  Teachers of other subjects selected at a rate of 1 in 3. This allocation allows the comparison of responses from teachers who experience state standardized testing, with responses of those who do not . The comparisons are within core subject areas and separat e possible influences due to testing from those due to subject area. The mean allocated teacher sample size for high schools will be close to nine. The one in three allocation should in most cases yield22

close to three teachers. Taking th ree teachers from each group is a reasonable way to improve the likelihood of representing each of the core subjects in tested and non -tested conditions, and to reflect the diversity of non - core, non -tested subjects taught at the high school level. The larger number of teachers sampled in high schools also allows for better representation of the expected greater variability of teacher experiences in high schools, which are larger, cover more subjects, and are more likely to be departmentalized than the othe r levels. For combined schools, allocations will be made to ‘elementary’, ‘middle’, and ‘high’ grades based on the grade span of the combined school, and then these allocations will be further separated out as above. After the allocations are made, a rand omized process will be carried out to assign integer sample sizes to the strata defined above in such a way so that the integer sample sizes will have expectation equal to the allocated (fractional) sample sizes, and the overall teacher sample sizes add up to the teacher sample size assigned to the school by the SASS algorithm. This design is intended to maximize the efficiency of comparisons between subgroups such as teachers in ESEA-tested subjects vs. non -tested subjects, and teachers in ESEA-tested grades vs. non -tested grades, by equalizing the sample sizes for each subgroup. This comes somewhat at the expense of overall estimates for teachers, which will have an extra design effect. The calculations in Table B - 10 and Table B - 12 do not account for this loss of efficiency from oversampling certain teacher subgroups within each school to equalize sample sizes across these teacher subgroups within schools, but the loss of efficiency that does occur should be limited in extent, as we believe these allocations are not far from the actual population percentages. Actually, comparing subgroups which are both present in every school (or in most schools, at least) will benefit from the presence of positive intra - s chool correlation. The estimate in this case is in effect a comparison of the two subgroups within each school, aggregated over all schools. The variance in this case will be proportional to + ( 1 − ) , where is the effective school sample size (accounting for design effects from oversampling of the three major school subgroups, but not accounting for design effects from enrollment- based oversampling, as the latter is removed in the setting of teacher sample sizes), and are the sample sizes for the su bgroups, and is the intra school correlation for the characteristic. Note that the variance decreases linearly with . Table B - 13 provides MDES calculations for the following comparisons:

 Elementary schools: comparing two upper grade tested subject to two upper grade non tested subject, with 85 percent response and 50 percent ICC;  Middle schools: comparing three tested subject to three non -tested su bject, with 85 percent response and 40 percent ICC;  High schools: comparing three tested class/grade and three non -te sted class/grade, with 85 percent response and 30 percent ICC. In all cases, the null hypothesis is no difference, with a two - sided test with alpha - level 5 percent, and 80 percent power for the alternative. The MDES are very good in all c ases.23

**Table B -**  **13 . Selected MDES for teachers by elementary, middle, and high school grade spans** 

**Span Effective** 

**school sample**  **size Assumed** 

**intra**   ** school** **correlation Final** 

**interview sample**  **size subgroup**  **1 Final**  **interview sample**  **size subgroup**  **2 MDES**  Elementary 616 50% 1.7 1.7 8.65% Middle 219 40% 2.55 2.55 12.98% High 281 30% 2.55 2.55 12.38% 

### B.2.3 **. Estimation Procedures**  The focus of the report will be on the progress being made on the core policies promoted by Titles I and II. While simple descriptive statistics such as means and percentages will provide answers to many of our questions, cross -tabulations will be important to answering questions about variation across state, district, schools, and teacher characteristics.

 Whether a state received an approved ESEA Flexibility request from the U.S. Department of Education. This is of interest because ESEA F lexibility is expected to influence the design of state accountability systems and approaches to educa tor quality measurement and policy makers will be interested in what states do with the ESEA F lexibility .  District poverty level, size, urbanicity, and concentration of English learners. P overty is included because Title I is specifically intended to ame liorate the effects of poverty on local funding constraints and educational opportunity. U rbanicity is included because of the relationships between educational opportunity and rural isolation and the concentration of poverty in urban schools. Size is incl uded because it may be related to district capacity to develop and implement programs. Concentration of English learners is included because of the increased emphasis on ensuring that this group of students also meets state content standards and recognitio n that modifications in testing as well as instruction will be needed to facilitate progress of these students.

 School Title I status, grade span (high school, middle, and elementary grades), poverty level, and cross classifications of Title I status and poverty level. G rade span is included because the implementation of state content standards and aligned assessments as well as responses to accountability systems likely differs by grade level. Title I status is included because the focus of Title I funds and requirements is to influence state and district policy and school effectiveness. We plan to cross - classify Title I status with school poverty because there may be substantial differences between what happens in high - poverty Title I schools in comparison with Title I schoo ls with low/medium - poverty levels, which may be more similar to schools which do not receive Title I funding.

 Teacher grade span taught and whether the teacher teaches a grade or subject in which state testing is required by ESEA. These characteristics a re included because teachers’ responses to state content standards, aligned assessment s, accountability systems, and teacher evaluation s24

based on student growth may vary by grade span, and by whether they teach tested subjects or grades. The use of strat ification (and oversampling when necessary) in our sample design was introduced to ensure reasonable power for specific subgroup comparisons, and our samples are expected to include units that vary on other characteristics (e.g., urbanicity) to allow the c omparisons described above. Because of the use of a statistical sample, survey data presented for districts, schools, and teachers will be weighted to national totals (tabulations will, provide standard errors for the reported estimated statistics). In a ddition, the descriptive tables will indicate where differences between subgroups are statistically significant. We will use Chi -Square tests to test for significant differences among distributions and t-tests for differences in means. Tabulations will be included in the reports where appropriate. Please see Section B.3.1 for a discussion of our weighting procedures. 

### B.2.4 **. Degree of Accuracy**  **Needed** We require statistical precision at all three levels in this study: the district level, the school level, and the teacher level. This makes this study different from a study which is focused for example on teachers alone. The hardest level to achieve preci sion for will be the district level, as the sample sizes in the three - stage sample design have to be smallest at this level (being the first stage of selection). The sample size of 570 districts evenly split between high - poverty districts and the complemen t stratum will provide an MDES of 30% for comparing these two important district subgroups. At the school level, we are sampling 1,300 schools. An important analysis at the school level compares Title I high - poverty schools (roughly 25% of the populatio n), Title I low/medium - poverty schools (roughly 40% of the population), and non -Title I schools (roughly 35% of the population). We oversample Title I high - poverty schools considerably (a factor of 2.25), and non -Title I schools slightly (a factor of 1.1), to roughly equalize sample sizes. The MDES for the three possible pair-wise comparisons ranges between 22% and 25%, depending on the degree of intra - district correlation. At the teacher level, we are sampling 9,100 teachers, and expecting 7,735 responden ts . This sample size will be sufficient to provide MDESs in the 10% - 15% range for comparisons among elementary schools, middle schools, and high schools separately, for two subgroups comprising 1⁄4 of the population teachers in each subgroup. The MDESs withi n this 10% - 15% range depend on the intra - school correlation and how the subgroups break across the schools (each subgroup represented in each school, or each subgroup is more concentrated in particular schools). The highest precision is achieved for compar ing subgroups which are equally within every school (tested subjects vs. non -tested subgroups), and the lowest precision is for comparing subgroups which are entirely segregated within school subgroups (teachers in Title I schools vs . teachers in non -Title I schools).25



### B.2.5 **. Unusual Problems Requiring Specialized Sampling Procedures**  There are no unusual problems requiring specialized sampling procedures. 

### B.2.6 **. Use of Periodic (less than annual) Data Collection to Reduce Burden**  This data will be collected during the 201 3 - 14 school year with a follow up survey and a possible optional year . This OMB package addresses only the 201 3 - 14 school year data collection.

We plan to work with states, school districts, schools, and teachers to explain the importance of this data collection effort and to make it as easy as possible to comply. For all respondents, a clear description of the study design, the nature and importance of the study, and the OMB clearance information will be provided. For the states, the data collection’s reliance whenever possible on administrative and extant data , thereby limiting the data collection from state representatives, will encourage cooperation with evaluation efforts. We will be courteous but persistent in follow- up with participants who do not respond in a timely manner to our attempts. As noted earlier, we will also be very flexible gathering our data, allowing different people to respond to the different content areas and in whichever mode is easiest -- electronic, hard copy or telephone format. For the school districts, schools, and teachers, we will initiate several forms of follow- up contacts with respondents w ho have not responded to our communication. We will use a combination of reminder postcards, emails and follow- up letters to encourage respon dents to complete the surveys. The project management system developed for this study will be the primary tool for monitoring whether surveys have been initiated. After 10 days, we will send an email message (or postcard for those without email) to all non - respondents indicating that we have not received a completed survey and encouraging them to submit one soon. Withi n seven business days of this first follow- up, we will mail non - respondents a hard copy package including all materials in the initial mailing. Ten days after the second follow- up, we will telephone the remaining non - respondents to ask that they complete t he survey and offer them the option to answer the survey by phone, either at that time or at a time to be scheduled during the call. To maximize response rates, we also will (1) provide clear instructions and user-friendly materials, (2) offer technical assistance for survey respondents using a toll -free telephone number or email, and (3) monitor progress regularly. In recognition of the fact that district and school administrators have many demands on their time, and typically, these administrators rece ive numerous requests to participate in studies and complete surveys for federal and state governments, district offices, and independent researchers, we plan to identify a district liaison. For most districts, completion of the district survey will require input from multiple respondents, and the district liaison’s role will be pivotal in positively impacting participation, collecting high quality data, and achieving an 85 percent response rate. 

## B.3 **. Methods to Maximiz** **e Response Rates**26



### B.3.1. **Weighting the district, school, and teacher samples** After completion of the field data collection in each year, we plan to weight the data to provide a nationally representative estimator at the distri ct, school, and teacher level. Replicate weights will be generated to provide consistent jackknife replica te variance estimators (statistical packages such as STATA and SAS Version 9.2 allow for easy computation of replicate variance estimates) . The development of replicate weights will facilitate the computation of standard errors for the complex analyses nec essary for this survey. The replicates will be based fundamentally on the district sample (the first stage of selection), with an exception for certainty districts, for which the first stage of sel ection is at the school level. We anticipate limited nonres ponse at the district and school level, which we will adjust for by utilizing information about the non - responding districts and schools from the frame. This information will be used to generate nonresponse cells with diffe rential response propensities. Th e nonresponse adjustments will be equal to the ratio of the frame weighted count to the s um of weights for respondents. This will adjust for bias from nonresponse, and also adjust for differences from the frame (acco mplishing poststratification). If the ce ll structure is too rich, we may utilize raking (multi dimensional adjustment). The cell structure for districts will include district poverty status, Census region, urbanicity, and district size. The cell structure for schools will include district poverty status, school poverty status, Title I status, Census region, urbanicity, and school size. There are two types of district and school weights that will be generated: enrollment- based weights and count- based weights. The first set of weights will add to total enrollment and the second to the simple count of districts and schools on the frame. A separate set of nonresponse adjustments will be calculated for each set of weights. Trimming may be utilized if necessary to scale back extreme weights. We expect a larger degree of teacher nonresponse (though we plan to provide an incentive). Teacher nonresponse adjustments will be computed based on an analysis of district and school characteristics, and teacher characteristics known from the teacher rost ers, which correlate to teacher propensity to respond. A ‘frame of teachers’ will be created by weighting the teacher rosters from the sampled schools using th e final teacher weights . The starting point for teacher weights will be the enrollmentbased fina l school weights. Teacher adjustments will then be made equaling the teacher roster totals divided by the weighted totals over teacher respondents. Raking may be utilized if the cell structure is too rich. The cell structure will include district poverty status, school poverty status, Title I status, Census region, and school size. Trimming may be utilized if necessary to scale back extreme weights, if this occurs.

The state survey will be pretested with up to three states, and prior to the pretest, we will gather documentation and extant data from their state websites to better understand h ow current is the information obtain ed from secondary sources. The school district, school , and teacher surveys will be pretested with n ine or fewer respondents. We will also pretest the teacher roster with principals to determine that the requested data is sufficient for selecting the teachers. 

## B.4 **. Te**  **st of Procedures** 27

The individuals consulted on the statistical aspects of the design include: Camilla Heid, Westat, Project Director, 301 - 294-4413 Patty Troppe, Westat, Deputy Project Director, 301 - 294- 3924 Anthony Milanowski, Westat, Co - Principal Investigator, 240 -453 - 2718 Brian Gill, Mathematica, Co - Pr incipal Investigator, 617 - 301 - 8962 Lou Rizzo, Westat, Senior Statistician, 301 - 294-4486 David Morganstein, Westat, Vice - President Statistical Group, 301 - 251 -8215 Tom Cook, Northwestern University, 847 -491 - 3776 Sean Reardon, Stanford University, Member of Technical Working Group, 650 -736 - 8517 

## B.5 **. Individuals Consulted on Statistical Aspects of Design** 28

U.S. Department of Education . (2011, September 23). ESEA Flexibility. Retrieved from : http://www.ed.gov/esea/flexibility . U.S. Department of Education. (2012a). Improving Basic Programs Operated by Local Educational Agencies (TITLE I, PART A). Retrieved from: http://www2.ed.gov/programs/titleiparta/index.html U.S. Department of Education. (2012b). Findings from the 2011 - 2012 Survey on the Use of Funds Under Title II, Part A. Retrieved from: http://www2.ed.gov/programs/teacherqual/resources.html 

## References