# Roads to Success in North Dakota:

**A Randomized Study of a College and Career Preparation Curriculum** 
**Supporting Statement**

**Part B** 

**Request for OMB Review**

**OMB # 1830**  **- NEW** 

Prepared by RTI International and FHI 360 for: **Office of Career, Technical, and Adult Education U.S. Department of Education** i 

### TABLE OF CONTENTS **Section Page**  B. Collection of Information Employing Statistical Methods ................................ ................. 1 B.1 Respondent Universe and Sampling Methods ................................ .......................... 1 B.2 Procedures for Collecting Information ................................ ................................ ..... 1 B.3 Methods for Maximizing Response Rate s and Deal with Nonresponse .................. 3 B.4 Tests of Procedures or Methods to be Undertaken ................................ ................... 5 B.5 Individuals Consulted and Individuals Responsible for Collecting and Analyzing the Information ................................ ................................ ................................ ......... 51

**B. COLLECTION OF INFORM** **ATION EMPLOYING STAT ISTICAL METHODS**  This section describes the respondent universe for this study and the sampling and statistical methodologies proposed. This section also address es suggested methods for maximizing response rates and for tests of procedures and methods, and introduce s the s tatisticians and other technical staff responsible for design and administration of the study . **B.1 Respondent**  **Universe and Sampling Methods**  The target universe for the study includes all North Dakota public high schools and Bureau of Indian Education (BIE) schools with both an 11 th and 12 th grade. Schools with very small grade enrollments (fewer than 10) will be excluded from the universe and sampling frame due to the fact that many of these schools provide instruction in ungraded classes, and therefor e differentiated 11 th

- and 12 th

- grade RTS instruction would not be possible. According to the ND Educational Directory for 2013
- 2014, there are 167 public schools with a 12 th and 11 th grade in North Dakota (33 schools with just grade 9
- 12, and 134 with a combination of high school and lower grades). There are 7,348 12th
- grade students and 7,414 11th
- grade students. Assuming about 75 percent of schools have not implemented RTS in any grade (based on communication with ND officials), about 125 schools (conta ining about 5,511 12 th graders and 5,560 11 th graders) are potentially eligible for the study , with an average 12 th

- grade and 11 th

- grade class of 44 students (each grade) per school. With about 30 of these 125 schools excluded due to size, approximately 95 schools and 10,600 students will be eligible for the study . The school sampling frame will come from the North Dakota Department of Public Instruction, and North Dakota CTE staff and regional CTE coordinators will assist in identifying, in advance, which schools are currentl y implementing RTS or have implemented RTS in recent years (so that, for example, schools who have provided some RTS programming to grade 10 or 11 students are not included in the study). To prepare for the contingency of encountering s chools that have implemented RTS but are unknown to state staff and thus to the research team, our initial contact with schools will screen for prior RTS provision. **B.2 Procedures for Collecting Information**  A randomized design will be employed in which tar get schools are assigned to the treatment or control conditions and all 11 th

- and 12 th

- grade students in a treatment school provided with RTS instruction. This design is necessitated b y the relatively small size of schools in North Dakota. It also provides the advantage that cross
- condition contamination within a school will not be possible.B. Collection of Information Employing Statistical Methods Supporting Statement Request for OMB Review (SF83I)

2 To achieve adequate statistical power, the optimal number of schools for the study would be between 64 and 80 participating schools, depending on various assumptions. However, for maximum statistical power and to better account for the possibility of some schools refusing to participate or dropping out, all 95 estimated eligible schools (125 schools minus the 30 schools with very small 11 th

- and 12 th

- grade enrollments) will be included in the study . Half of the schools will be randomly selected into the treatment group, with the remaining placed in the control group (it will randomly be decided if the odd case is in the treatment or control group). This allows the study to have 80% power with an alpha of 0.05. The sampling design assumptions are shown in table 1. The average class size of 44 masks the fact that there are onl y a few larger schools and man y smaller schools; the median 12 th

- or 11 th

- grade school class is 19 students, and there are onl y about 15 schools with over 100 12 th

- grade or 11 th

- grade students. Stratification of schools will be by school size (below/above median enrollment) to ensure that large schools have an equal chance of being in either the treatm ent or control condition. **Table 1**  **. Sample D**  **esign A**  **ssumptions Assumption**  α = 0.05 44 12 th grade rs and 44 11 th graders per school , on average Power of 80% Intraclass correlation of 0.15; Proportion of variance explained by the REAs is 0.3 Proportion of variance explained by school covariates is 0.2 Estimated proportions of students being tested are 0.3 for the control group Most schools in the study sample would be in their own school districts; i.e., assigning two or more schools within a district to different treatment and control conditions will likely not be possible. For example, ND contains a high number of school districts (148) relative to its number of schools with a 12 th grade (167), meaning that most school districts only conta in one school with a 12 th grade. Though this theoreticall y limits the ability to reduce variation prior to assignment, in practice the student population of ND is homogeneous and the district policies and supports are fairl y consistent. The primary diversi ty is between White and Native American students; White students make up about 83 percent of the ND student population and Native American students make up about 9 percent (Black and Hispanic studentsSupporting Statement Request for OMB Review (SF83I) B. Collection of Information Employing Statistical Methods

3 make up 2
- 3 percent of the population each; the remaini ng 2 percent are divided between Asian and Two or More Races). About 31 percent of the student population is free or reduced
- price lunch eligible. Nevertheless, schools will be assigned to different treatment and control conditions within a regional educational association (REA, a kind of super
- district). There are eight REAs which provide support services to member school districts; two are small and will be combined into one, yielding seven REAs to use in stratifying the schools (with the size stratification being within REA). The REAs cooperate on programs and services to increase district operating efficiencies. Stratifying by REA enables us to split the schools in a REA between the treatment and control groups and can help ensure balance across experim ental conditions. In addition to the fact that there are few districts with multiple high schools, there are also few schools with multiple 12 th

- grade or 11 th

- grade classrooms, due to the small size of most ND high schools. Even where there are multiple classrooms or teachers in schools, the schools are still relativel y small, which makes spillover effects likel y. Thus classrooms within schools will not be assigned to different treatment and control conditions. Therefore, within each sampled school, all 1 1 th and 12 th graders will be included in the study . In the western part of the state, where school growth (mostly at the elementary

and middle school level) has been very

rapid in recent years, there may also be a relativel y greater number of transfer stud ents; however, the numbers of 11 th

- and 12 th

- grade transfer students is expected to be low because workers in the gas industry are often not bringing their families with them initially

— and would be less likely to do so if they

have children nearing high sc hool completion. Nevertheless, district
- level differences in population are another aspect of the students’ environments about which information will be gathered. **B.3 Methods for Maximizing Response Rates**  **and Deal with Nonresponse** Procedures for maximizin g response rates at the institution and respondent levels are based on successful experiences in similar school
- based studies. In this section, methods for maximizing response rates for schools, students, teachers, and school administrators are discussed. The recruitment of schools into the study will begin with the dissemination of information to school counselors and district leaders to notify districts and schools about the study and to express the support of the state of North Dakota. School administr ator conferences in late January /earl y February 2015, and a midwinter counselor’s conference in late February 2015 will be the vehicles for informing potential school participants of the study and its general outlines. The ND Department of CTE will announc e the project in general terms and help build awareness of the importance of the study , as well asB. Collection of Information Employing Statistical Methods Supporting Statement Request for OMB Review (SF83I)

4 interest in participating, at these events. However, no information about selected schools or which schools are in the experimental or control group will be shared prior to the receipt of OMB approval. After OMB clearance and IRB approval are received, we will contact districts and initiate the process of obtaining an y required research applications for conducting research in their schools. Our contact with districts and schools will include cover letters that present the endorsement of the ND Department of CTE and the Department of Education, emphasizing the importance of the study and the value of its findings. All contacting mate rials are included in App endix A . Once an y required district approval is obtained, we will begin contacting schools selected into the study through packets sent through Federal Express informing them of the nature of the study , its purpose, and its value to them and to the state, and asking for their participation. Packets will include endorsement letters, a sheet of frequently asked questions, and contact information to answer further questions. Several day s after receipt of the package, a recruitment specialist from the NCI CTE re search team will contact the school, ask for their participation, address an y questions or concerns they may have, and, if they

agree to participate, begin the process of identifying a school staff member or members to serve as the RTS instructor and a sep arate (where possible) school coordinator to assist in the administration of the survey s and site visits. Recruitment specialists will document any refusals to participate and follow
- up an additional time via email and phone to attempt to convert refusals into acceptance. The recruitment specialist will remain the primary point of contact between NCICTE and the school, coordinating site visits and ensuring continuity in communications and a single source of information and help for any issues that arise. W e will provide school coordinators with a website for the uploading and tracking of student survey participation and parental consent. North Dakota staff inform us that passive (implicit) parental consent is used statewide; therefore, consent forms will be sent to parents via students. Parents or guardians will receive a letter informing them of the nature of the study , the authorization to conduct it, the protection of their child’s data, and other information as required. They will also receive a consent form that they

must return if they

do not want their student to participate; such implicit consent usually yi elds higher participation rates than explicit consent. A sample cover letter and implicit consent form is also inc luded in Appendix A . School coord inators will track parental consent forms and note exclusions for data collection (such students can receive the treatment but not be surveyed or otherwise have data gathered about them). School coordinators will also provide lists of names and other identifying information for students in the 11 th and 12 th grades, from which individual usernames and passwords will be generated, to enable the linking of student responses across the two survey s, and enabling eventualSupporting Statement Request for OMB Review (SF83I) B. Collection of Information Employing Statistical Methods

5 linking with administrative data. Since a ll students in the 11 th and 12 th grades will be included in the sample and survey

participation will be a class activity , we anticipate a high level of student participation. Plans for incentives are provided in Part A. Multiple incentives will be given to RTS instructors, school coordinators, and principals to motivate participation in the study . For teachers, counselors, or career advisors who will be delivering RTS, we will offer between $ 50 - $ 100 for participation, in addition to paying for costs assoc iated with training totaling no more than $200 per instructor (e.g., registration fees). Staff in ND will also work to enable instructors to receive college credit for participating in RTS training; this is believed to be a strong incentive for participati on in the training (and therefore the study as a whole). This size of incentive is commensurate with the burden placed on instructors for completing two survey s throughout the school y ear for each class ($25 per survey). Principals at both treatment and co ntrol schools will receive a small incentive ($25) for their participation in principal interviews. School coordinators will receive up to a $150 incentive for managing consent forms, scheduling the student survey s, and coordinating with NCICTE researchers to plan site visit logistics where applicable. In addition, consistent with our experience on other interventions, we propose a school
- level incentive for control schools which otherwise have little motivation to participate in the study . **B. 4**  **Test s**  **of Procedures or Methods to be Undertaken**  As noted in the content justifications in Part A of this submission, items developed for the survey s drew upon existing surveys and RTS documentation to the greatest extent possible , and site visit protocols derive fr om prior experience conducting numerous similar case studies . Li kewise, recruitment strategies and other communications materials were developed based on procedures utilized for NCES’s secondary longitudinal studies (High School L ongitudinal Study of 2009; Education Longitudinal Study of 2002). Therefore, no tests of procedures or methods are planned for the study . **B.5 Individuals Consulted**  **and Individuals Responsible fo r**  **Collecting and Analyzing the Information**  The following individuals in the U.S. Department of Education were consulted on the design, data collection, and analy sis approach: Braden Goetz in the Office of Career, Technical, and Adult Education and Marsha Silverberg in the Institute of Education Sciences. T he following RTI International staff work on the statistical and methodological aspects of the study design : Ben Dalton, Peter Siegel, Deborah Herget, and Michael Nelson. They can be reached at (919) 541
- 7228 . The following FHI 360 staff work on methodological and analy ti cal components pertaining to the site visits: Ivan Charner and Caitlin Rose Dailey .B. Collection of Information Employing Statistical Methods Supporting Statement Request for OMB Review (SF83I)

6