Roads to Success in North Dakota: 

A Randomized Study of a College 

and Career Preparation Curriculum

Supporting Statement

Part B

Request for OMB Review

OMB #1830-NEW

Prepared by RTI International and FHI 360 for:

Office of Career, Technical, and Adult Education

U.S. Department of Education

i

TABLE OF CONTENTS

Section Page

B. Collection of Information Employing Statistical Methods................................................. 1

B.1 Respondent Universe and Sampling Methods.......................................................... 1

B.2 Procedures for Collecting Information..................................................................... 1

B.3 Methods for Maximizing Response Rates and Deal with Nonresponse .................. 3

B.4 Tests of Procedures or Methods to be Undertaken................................................... 5

B.5 Individuals Consulted and Individuals Responsible for Collecting and Analyzing 

the Information ......................................................................................................... 5

1

B. COLLECTION OF INFORMATION EMPLOYING STATISTICAL METHODS

This section describes the respondent universe for this study and the sampling and statistical 

methodologies proposed. This section also addresses suggested methods for maximizing response rates 

and for tests of procedures and methods, and introduces the statisticians and other technical staff 

responsible for design and administration of the study.

B.1 Respondent Universe and Sampling Methods

The target universe for the study includes all North Dakota public high schools and Bureau of 

Indian Education (BIE) schools with both an 11th and 12th grade. Schools with very small grade 

enrollments (fewer than 10) will be excluded from the universe and sampling frame due to the fact that 

many of these schools provide instruction in ungraded classes, and therefore differentiated 11th- and 12th-

grade RTS instruction would not be possible. 

According to the ND Educational Directory for 2013-2014, there are 167 public schools with a 

12th and 11th grade in North Dakota (33 schools with just grade 9-12, and 134 with a combination of high 

school and lower grades). There are 7,348 12th-grade students and 7,414 11th-grade students. Assuming 

about 75 percent of schools have not implemented RTS in any grade (based on communication with ND 

officials), about 125 schools (containing about 5,511 12th graders and 5,560 11th graders) are potentially 

eligible for the study, with an average 12th-grade and 11th-grade class of 44 students (each grade) per 

school. With about 30 of these 125 schools excluded due to size, approximately 95 schools and 10,600 

students will be eligible for the study.

The school sampling frame will come from the North Dakota Department of Public Instruction, 

and North Dakota CTE staff and regional CTE coordinators will assist in identifying, in advance, which 

schools are currently implementing RTS or have implemented RTS in recent years (so that, for example, 

schools who have provided some RTS programming to grade 10 or 11 students are not included in the 

study). To prepare for the contingency of encountering schools that have implemented RTS but are 

unknown to state staff and thus to the research team, our initial contact with schools will screen for prior 

RTS provision.

B.2 Procedures for Collecting Information

A randomized design will be employed in which target schools are assigned to the treatment or 

control conditions and all 11th- and 12th-grade students in a treatment school provided with RTS 

instruction. This design is necessitated by the relatively small size of schools in North Dakota. It also 

provides the advantage that cross-condition contamination within a school will not be possible.
B. Collection of Information Employing Statistical Methods Supporting Statement Request for OMB Review (SF83I)

2

To achieve adequate statistical power, the optimal number of schools for the study would be 

between 64 and 80 participating schools, depending on various assumptions. However, for maximum 

statistical power and to better account for the possibility of some schools refusing to participate or 

dropping out, all 95 estimated eligible schools (125 schools minus the 30 schools with very small 11th-

and 12th-grade enrollments) will be included in the study. Half of the schools will be randomly selected 

into the treatment group, with the remaining placed in the control group (it will randomly be decided if 

the odd case is in the treatment or control group). This allows the study to have 80% power with an alpha 

of 0.05. The sampling design assumptions are shown in table 1. 

The average class size of 44 masks the fact that there are only a few larger schools and many 

smaller schools; the median 12th- or 11th-grade school class is 19 students, and there are only about 15 

schools with over 100 12th-grade or 11th-grade students. Stratification of schools will be by school size 

(below/above median enrollment) to ensure that large schools have an equal chance of being in either the 

treatment or control condition. 

Table 1. Sample Design Assumptions

Assumption

α = 0.05

44 12th graders and 44 11th graders per 

school, on average

Power of 80%

Intraclass correlation of 0.15; 

Proportion of variance explained by the 

REAs is 0.3

Proportion of variance explained by

school covariates is 0.2

Estimated proportions of students being 

tested are 0.3 for the control group

Most schools in the study sample would be in their own school districts; i.e., assigning two or 

more schools within a district to different treatment and control conditions will likely not be possible. For 

example, ND contains a high number of school districts (148) relative to its number of schools with a 12th

grade (167), meaning that most school districts only contain one school with a 12th grade. Though this 

theoretically limits the ability to reduce variation prior to assignment, in practice the student population of 

ND is homogeneous and the district policies and supports are fairly consistent. The primary diversity is 

between White and Native American students; White students make up about 83 percent of the ND 

student population and Native American students make up about 9 percent (Black and Hispanic students 
Supporting Statement Request for OMB Review (SF83I) B. Collection of Information Employing Statistical Methods

3

make up 2-3 percent of the population each; the remaining 2 percent are divided between Asian and Two 

or More Races). About 31 percent of the student population is free or reduced-price lunch eligible. 

Nevertheless, schools will be assigned to different treatment and control conditions within a 

regional educational association (REA, a kind of super-district). There are eight REAs which provide 

support services to member school districts; two are small and will be combined into one, yielding seven 

REAs to use in stratifying the schools (with the size stratification being within REA). The REAs 

cooperate on programs and services to increase district operating efficiencies. Stratifying by REA enables 

us to split the schools in a REA between the treatment and control groups and can help ensure balance 

across experimental conditions. 

In addition to the fact that there are few districts with multiple high schools, there are also few 

schools with multiple 12th-grade or 11th-grade classrooms, due to the small size of most ND high schools. 

Even where there are multiple classrooms or teachers in schools, the schools are still relatively small, 

which makes spillover effects likely. Thus classrooms within schools will not be assigned to different 

treatment and control conditions. Therefore, within each sampled school, all 11

th and 12th graders will be 

included in the study.

In the western part of the state, where school growth (mostly at the elementary and middle school 

level) has been very rapid in recent years, there may also be a relatively greater number of transfer 

students; however, the numbers of 11th- and 12th-grade transfer students is expected to be low because 

workers in the gas industry are often not bringing their families with them initially—and would be less 

likely to do so if they have children nearing high school completion. Nevertheless, district-level 

differences in population are another aspect of the students’ environments about which information will 

be gathered. 

B.3 Methods for Maximizing Response Rates and Deal with Nonresponse

Procedures for maximizing response rates at the institution and respondent levels are based on 

successful experiences in similar school-based studies. In this section, methods for maximizing response 

rates for schools, students, teachers, and school administrators are discussed. 

The recruitment of schools into the study will begin with the dissemination of information to 

school counselors and district leaders to notify districts and schools about the study and to express the 

support of the state of North Dakota. School administrator conferences in late January/early February 

2015, and a midwinter counselor’s conference in late February 2015 will be the vehicles for informing 

potential school participants of the study and its general outlines. The ND Department of CTE will 

announce the project in general terms and help build awareness of the importance of the study, as well as 
B. Collection of Information Employing Statistical Methods Supporting Statement Request for OMB Review (SF83I)

4

interest in participating, at these events. However, no information about selected schools or which schools 

are in the experimental or control group will be shared prior to the receipt of OMB approval. 

After OMB clearance and IRB approval are received, we will contact districts and initiate the 

process of obtaining any required research applications for conducting research in their schools. Our 

contact with districts and schools will include cover letters that present the endorsement of the ND 

Department of CTE and the Department of Education, emphasizing the importance of the study and the 

value of its findings. All contacting materials are included in Appendix A.

Once any required district approval is obtained, we will begin contacting schools selected into the 

study through packets sent through Federal Express informing them of the nature of the study, its purpose, 

and its value to them and to the state, and asking for their participation. Packets will include endorsement 

letters, a sheet of frequently asked questions, and contact information to answer further questions. Several 

days after receipt of the package, a recruitment specialist from the NCICTE research team will contact the 

school, ask for their participation, address any questions or concerns they may have, and, if they agree to 

participate, begin the process of identifying a school staff member or members to serve as the RTS 

instructor and a separate (where possible) school coordinator to assist in the administration of the surveys 

and site visits. Recruitment specialists will document any refusals to participate and follow-up an 

additional time via email and phone to attempt to convert refusals into acceptance. The recruitment 

specialist will remain the primary point of contact between NCICTE and the school, coordinating site 

visits and ensuring continuity in communications and a single source of information and help for any

issues that arise. 

We will provide school coordinators with a website for the uploading and tracking of student 

survey participation and parental consent. North Dakota staff inform us that passive (implicit) parental 

consent is used statewide; therefore, consent forms will be sent to parents via students. Parents or 

guardians will receive a letter informing them of the nature of the study, the authorization to conduct it, 

the protection of their child’s data, and other information as required. They will also receive a consent 

form that they must return if they do not want their student to participate; such implicit consent usually

yields higher participation rates than explicit consent. A sample cover letter and implicit consent form is 

also included in Appendix A. School coordinators will track parental consent forms and note exclusions 

for data collection (such students can receive the treatment but not be surveyed or otherwise have data 

gathered about them). School coordinators will also provide lists of names and other identifying 

information for students in the 11th and 12th grades, from which individual usernames and passwords will 

be generated, to enable the linking of student responses across the two surveys, and enabling eventual 
Supporting Statement Request for OMB Review (SF83I) B. Collection of Information Employing Statistical Methods

5

linking with administrative data. Since all students in the 11th and 12th grades will be included in the 

sample and survey participation will be a class activity, we anticipate a high level of student participation. 

Plans for incentives are provided in Part A. Multiple incentives will be given to RTS instructors, 

school coordinators, and principals to motivate participation in the study. For teachers, counselors, or 

career advisors who will be delivering RTS, we will offer between $50-$100 for participation, in addition 

to paying for costs associated with training totaling no more than $200 per instructor (e.g., registration 

fees). Staff in ND will also work to enable instructors to receive college credit for participating in RTS 

training; this is believed to be a strong incentive for participation in the training (and therefore the study

as a whole). This size of incentive is commensurate with the burden placed on instructors for completing 

two surveys throughout the school year for each class ($25 per survey). Principals at both treatment and 

control schools will receive a small incentive ($25) for their participation in principal interviews. School 

coordinators will receive up to a $150 incentive for managing consent forms, scheduling the student 

surveys, and coordinating with NCICTE researchers to plan site visit logistics where applicable. In 

addition, consistent with our experience on other interventions, we propose a school-level incentive for 

control schools which otherwise have little motivation to participate in the study.

B.4 Tests of Procedures or Methods to be Undertaken

As noted in the content justifications in Part A of this submission, items developed for the surveys 

drew upon existing surveys and RTS documentation to the greatest extent possible, and site visit protocols 

derive from prior experience conducting numerous similar case studies. Likewise, recruitment strategies 

and other communications materials were developed based on procedures utilized for NCES’s secondary

longitudinal studies (High School Longitudinal Study of 2009; Education Longitudinal Study of 2002). 

Therefore, no tests of procedures or methods are planned for the study.

B.5 Individuals Consulted and Individuals Responsible for Collecting and Analyzing the 

Information

The following individuals in the U.S. Department of Education were consulted on the design, data 

collection, and analysis approach: Braden Goetz in the Office of Career, Technical, and Adult Education 

and Marsha Silverberg in the Institute of Education Sciences. The following RTI International staff work 

on the statistical and methodological aspects of the study design: Ben Dalton, Peter Siegel, Deborah 

Herget, and Michael Nelson. They can be reached at (919) 541-7228. The following FHI 360 staff work 

on methodological and analytical components pertaining to the site visits: Ivan Charner and Caitlin Rose 

Dailey.
B. Collection of Information Employing Statistical Methods Supporting Statement Request for OMB Review (SF83I)

6