# Roads to Success in North Dakota:

**A Randomized Study of a College and Career Preparation Curriculum** 
**Supporting Statement**

**Part A**

**Request for OMB Review**

**OMB # 1830**  **- NEW** 

Prepared by RTI International and FHI 360 for: **Office of Career, Technical, and Adult Education U.S. Department of Education** i 

### TABLE OF CONTENTS **Section Page**  A. Justification
................................ ................................ ................................ ......................... 1 A.1 Circumstances Necessitating Collection of Information
................................ .......... 1 A.1.a Purpose of this Submission
................................ ................................ ....... 1 A.1.b Statement of Need for a Rigorous Evaluation of Roads to Suc cess in North Dakota
................................ ................................ ........... 1 A.1.c Research Questions
................................ ................................ ................... 3 A.1.d Description of Intervention
................................ ................................ ....... 4 A.1.e. Study

Design
................................ ................................ ............................. 8 A.1.f D ata Collection Needs
................................ ................................ .............. 9 A.1.h Legislative Authorization
................................ ................................ ........ 12 A.1.i Prior and Related Studies
................................ ................................ ........ 12 A.2 Purpose and Use of Information Col lection
................................ ........................... 13 A.2.a Content Justifications
................................ ................................ .............. 14 A.3 Use of Information Technology
................................ ................................ ............. 19 A.4 Efforts to Identify

Duplication and Use of Similar Information
............................ 19 A.5 Impact on Small Businesses or Other Small Entities
................................ ............. 20 A.6 Consequences of Not Collecting Data
................................ ................................ .... 20 A.7 Special Circumstances
................................ ................................ ............................ 21 A.8 Federal Register Announcements and Consultations
................................ ............. 21 A.8.a Federal Register Announcement
................................ ............................. 21 A.8.b Consultations Outside the Agency
................................ .......................... 21 A.9 Provision of Pay

ment or Gift to Respondents
................................ ........................ 21 A.10 Assurance of Confidentiality Provided to Respondents
................................ ......... 22 A.11 Justification for Sensitive Questions
................................ ................................ ...... 23 A.12 Estimates of Hours Burden
................................ ................................ ..................... 23 A.13 Estimates of Cost Burden to Respondents
................................ ............................. 24 A.14 Annualized Cost to the Federal Government
................................ ......................... 24 A.15 Reasons for Program Changes
................................ ................................ ............... 24 A.16 Publication Plans and Project Schedule
................................ ................................ . 24 A.17 Reason(s) Display

of OMB Expiration Date Is Inappropriate
............................... 27 A.18 Exceptions to Certification for Paperwork Reduction Act Statement
.................... 27 References
................................ ................................ ................................ .............. 28ii 

### ATTACHMENTS Appendix A Respondent Communication Materials Appendix B Facsimile Student, Instructor, and Principal Surveys Appendix C Site Visit Procedures and ProtocolsSupporting Statement Request for OMB Review (SF83I) A. Justification

1 **A. JUSTIFICATION**  **A.1 Circumstances Necessitating Collection of Information**  **A.1.a Purpose of this Submission**  This package requests approval from the Office of Management and Budget (OMB) for data collection activities for a rigorous evaluation of a college and career preparation curriculum instituted in North Dakota (RTS
-ND) . This study is conducted by

the Office of Career, T echnical, and Adult Education (OCTAE), part of the U.S. Department of Education, under the umbrella of OCTAE’s National Center for Innovation in Career and Technical Education (NCICTE). The contractor for this study is RTI International (also the prime contractor for NCICTE) , and the sole subcontractor is FHI 360
. This request is to conduct an experimental evaluation of a college and career preparation curriculum

— provided as part of the Roads to Success program

—in enhancing 11 th

- and 12 th

- graders’ college and career aspirations, planning for postsecondary transitions and life, and attitudes toward education and careers in North Dakota. Roads to Success (RTS) offers after
- school programs, workforce and leadership development programs, and a college and career planning curriculum designed to enhance the academic and career readiness of students from 7 th grade through the end of high school. This study would specificall y examine the implementation in North Dakota of the college access and career development curriculum provided to 11 th and 12 th graders as part of RTS, which includes elements of college and career advising, career research, job searching basics, interviewing preparation, college essay practice, and financial aid navigation. As part of this submiss ion, OCTAE is publishing a notice in the Federal Register allowing first a 60
- and then a 30
- day

public comment period. **A.1.b Statement of Need for a Rigorous Evaluation of Roads to Success in North Dakota**  College and career planning programs vary signif icantly in their scope and reach. Nationall y, the most prominent programs are programs such as GEAR UP, Upward Bound, and Talent Search. These programs primaril y emphasize college planning and preparation and are delivered b y local grant recipients

—either schools, districts, colleges or a combination of them. Because of the diversity of grant recipients and the broad mandates of the programs, the programs are not consistent across grantees, offering different mixes of services to different groups of student s at different ages and grades. In contrast, RTS was developed in part to provide a long
- term program that serves all students in a school, not just disadvantaged students, and is s ys tematic and consistent in its approach (Bowers and Hatch 2005). RTS addr esses two critical needs. First, as a long
- term program beginning before high school, it provides the necessary resources and guidance to orient students toward postsecondaryA. Justification Supporting Statement Request for OMB Review (SF83I)

2 planning. Without such assistance, students, particularly those whose parents hav e limited education or are from disadvantaged backgrounds, may fail to establish a long
- term horizon for their efforts in school. Orienting students toward postsecondary life can help maintain school engagement and reduce behaviors that threaten good acade mic progress (Rosenbaum 2001). Second, and more importantly for the current study , RTS specificall y addresses informational deficits that may hinder students in taking concrete steps toward college or careers. RTS helps students to understand the courses that will be required to prepare for college and the skills necessary to secure stable employ

ment; it walks students through the steps of applying to college, securing a job, and managing finances independentl y. Informational deficits such as these have been shown to be a large part of why students fail to apply

or attend to college (Grodsky and Riegle
- Crumb 2010), especiall y academicall y excellent students from poor backgrounds (Hoxby and Avery 2012; Radford 2013). From 2007 to 2009, Mathematica Policy Res earch conducted a two
- year evaluation of the portion of RTS delivered to 7 th and 8 th graders (Duncan, Bleeker, and Booker 2010). Conducted in rural, predominatel y white schools in New York and West Virginia, the study randomly assigned schools within sampl ing blocks (defined by free and reduced
- price lunch eligibility rates and average test scores) to either a treatment (13 schools) or control (12 schools) condition. Using baseline and end
- of
- intervention student surveys combined with attendance and mobility data, the study found that “RTS does not have clearl y positive short
- term impacts” on school and class engagement measures (motivation in school; learning and study habits; and attendance and negative behaviors) (p. 43). RTS was found to positively impac t some individual measures of attendance and behavior (specifically school absences and detention), but to also negativel y impact classroom attendance. The study did find impacts for several career outcomes related to the type of informational deficits cit ed above: career exploration with school staff, knowing how to find jobs that fit their needs, and knowing what is required for different careers. Effect sizes for these measures ranged from 0.15 to 0.50. Duncan, Bleeker, and Booker’s study is the only kn own evaluation of RTS. Though it did not find statistically

significant differences for its main outcomes of interest (engagement measures), it is promising that specific career exploration activities were found to be positively affected by

RTS. The design of the current study , focusing on the opposite end of RTS’s grade range and on concrete activities undertaken in those grades, provides an opportunity to confirm and extend the earlier findings and rigorousl y assess the effects of a well
- designed, intensi ve, and integrated curriculum can have on college and career planning and preparedness.Supporting Statement Request for OMB Review (SF83I) A. Justification

3 North Dakota is an excellent site in which to implement such a rigorous evaluation. North Dakota has passed legislation requiring students to develop college and career plans and, with funding from the Hess Foundation, has implemented the RTS program in a small number of schools. Therefore, state and district staff are poised to implement RTS more broadly as part of a randomized evaluation. As part of this study , high sc hool juniors and seniors will be provided the 11 th

- and 12 th

- grade curriculum in the academic year 2015
- 2016 and be administered a baseline and post
- intervention survey to measure the growth of key

indicators of college and career planning and development. The survey draws on existing student surveys available in the RTS Program Manual (and used by Mathematica Policy

Research in a previous study

of RTS). These survey s measure immediate postsecondary plans, educational and occupation aspirations, career goal s and attitudes, student demographics, learning and study habits, and school and community

involvement. Two survey s of instructors in treatment schools will track topics covered to assess the fidelity of implementation, and principal survey s will provide c ontext for the school’s college and career preparation agenda. Inaddition, the study will conduct site visits of control and treatment schools to develop a thorough understanding of how the curriculum is being implemented and the successes and challenges faced b y instructors and students in learning from the material. **A.1.c Research Questions**  As described in section **A.1.d**, RTS is a specific set of instructional modules tailored to different grades. The intervention planned as part of this study will be f or one academic year and examine the end
of
- year outcomes of 11 th

- and 12 th

- graders. Given this time horizon, the research questions for this study focus on concrete activities or specific student interests that are the target of each grade’s intervention. These outcomes are what the RTS L ogic Model refers to as “second
- stage intermediate” outcomes, in the context of providing RTS instruction over many grades. Here, they

are immediate short
- term outcomes directly addressed in the RTS modules that will be implemented. At the 11 th grade, does one year of RTS instruction increase the percentage of students:

 With specific senior
- year coursetaking plans?  Having visited a college or university? Does RTS increase the number of visits?  Having taken or registered for workforce development assessments such as the Test of Adult Basic Education (TABE), the Armed Services Vocational Aptitude Battery (ASVAB), and the ACT WorkKey s Foundational and Personal Assessments?  Reporting preparation of a resume?  Reporting career exploration activities such as job shadowing, attendance at career fairs, internships, or apprenticeships?  With any

interest in one or more specific careers?  With high interest in a specific career?A. Justification Supporting Statement Request for OMB Review (SF83I)

4 At the 12 th grade, does one ye ar of RTS instruction increase the percentage of students:

 Applying to college, overall and by level (technical or trade school, two
- ye ar community college, or four
- ye ar college)? Does RTS increase the number of college applications, overall and b y level?  Completing the Free Application for Federal Student Aid (FAFSA)?  Accepted to or registered at college for the fall semester, overall and b y level? Does RTS increase the number of acceptances?  Reporting preparation of a resume?  Applying for a job?  With a job offer?  With any

interest in one or more specific careers?  With high interest in a specific career?  Planning to major in an area related to career interests? **A.1.d Description of Intervention**  The RTS curriculum is designed to influence college admissio ns behaviors and postsecondary work plans beginning in 7 th grade and continuing through 12 th grade. RTS is organized as a set of instructional modules for each grade, with topics repeating across grades and work in prior grades occasionall y providing the f oundation for later work

— for example, grade 11 modules for preparing a resume are followed b y grade 12 modules for revising the same resume. The program is designed to be delivered throughout the school year, for 45 minutes each week. The program can be de livered by teachers, counselors, or designated RTS facilitators. In North Dakota, instructors are trained by state or regional career and technical education ( CTE ) staff associated with the state CTE office.Supporting Statement Request for OMB Review (SF83I) A. Justification

5 Source: Roads to S uccess Affiliate Program Manual. As a long
- term, in
- depth intervention, RTS is designed to affect a wide array of short
- , medium
- , and longer
- term outcomes that range from pre
- secondary student beliefs about the importance of college and career preparatio n to concrete goals of taking college admissions tests such as the ACT or SAT, applying to and enrolling in college, and preparing resumes and cover letters. The logic model for the full intervention (drawn directly from the RTS
-ND Program Manual) is prese nted in figure 1. The intervention is seen as having both content elements (i.e., the focus of the instruction) as well as structural elements that support the delivery of the content (e.g., regularly scheduled sessions, project
- based learning, RTS facilit ators). Within the logic model, outcomes are divided into short
- term outcomes about the importance of school for postsecondary life; intermediate outcomes involving both diffuse concepts of motivation and college and career awareness, and specific goals of college preparatory activities and high school completion; and long
- term outcomes including college enrollment, persistence, and careerA. Justification Supporting Statement Request for OMB Review (SF83I)

6 attainment. The model incorporates the understanding that contexts of pre
- program student skills and beliefs, school cl imate and peers, and parent and community support can affect the impact that the program has on the desired outcomes. Although the intervention and its expected outcomes are designed to be multi
- year, instructional modules and attendant goals are specifie d for each grade. At the 11 th grade, the intervention consists of seven topic areas and 30 different modules. At the 12 th grade, there are 6 topic areas with 30 distinct instructional modules. Table 1 outlines the topic areas and instructional modules under each at the 11 th and 12 th grades. In general, college preparation in grade 11 consists of college admissions test preparation (the modules are either SAT or ACT, not a combination) and college research (“Education after High School”); college preparation in grade 12 consists of applying to college, financial aid, and first
- ye ar preparation (under “Money Matters” and “Next Steps”). Career preparation in grade 11 consists of career research, resumes, and interviews; while career preparation in grade 12 cons ists of company research, networking, resumes, cover letters, and interviews. The remaining instructional modules cover management of finances more generall y (e.g., car shopping and health insurance), and are not the focus of the current study
. Working wi th ND CTE staff and RTS coordinators, this study will implement each of these modules in 11 th and 12 th grade. However, because RTS is designed as a multi
- year intervention that builds off of each ye ar’s prior work with students, some tailoring of the instr uctional modules is necessary in 11 th and 12 th grades to ensure instructional coherence. Specifically, the following modules will be incorporated from earlier grades: from grade 8, an introductory module explaining different types of postsecondary

programs (e.g., two
- ye ar, four
- ye ar); from grade 9, a work and values module that helps student inventory

what they

value about work, as well as a module covering how to complete job applications; and at grade 10, a module that helps students plan for visiting and understanding college visits, and a module preparing students for a job interview.Supporting Statement Request for OMB Review (SF83I) A. Justification

7 **Table 1. Roads to Success Topic A reas and Instructional Modules for Grades 11 and 12**  **Topic A rea**  **Instructional Modules Grade 11**  **Introduction** Introduction **Test Prep** Introduction to the ACT/SAT ACT/SAT Registration ACT/SAT Practice Questions Wo rkforce and Adult Ed Tests **Careers** Interest Inventor y Career Choice My Career Research **Job Shadow** Introduction to Job Shadow Informational Interview I and II Creating Resumes I and II Reflection and Thank - Yo u Note Wo rk place Behavior **Education after High School** Choosing Courses for Senior Year Choosing a College Choosing a College Major College Research I through III Other Education Paths Letters of Recommendation Evaluating Postsecondar y Options **Money Matters** Understanding Credit Pa ying for a Car Credit Cards Renting Your First Place Let’s Go Car Shopping Signing a Lease **Portfolio Review** Year in Review **Grade 12 Introduction**  Introduction I and II **Ap plying to College**  Completing a College Application Essay W riting I through V **Finding a Job** Wh o Gets Jobs? Cover Letters I and II Your Network The Interview Company Research I and II Follow- Up Your Resume **Financial A id**  Financial Aid Options FAFSA I and II Scholarship I and II Comparing Aid Packages **Money Matters** “Freshman Year” Budget Health Insurance Big Ticket Expenses Budgeting Details **Next Steps** Freshman Year Survival Guide Getting Ahead at Wo rk Advice from the Experts Source: Roads to Success Affiliate Program Manual. The modules and lessons described above are a framework for instruction as well as an organizational tool for RTS materials and lesson plans. In other words, RTS instructors are encouraged to adapt lesson plans to fit class schedules, resource availability , teaching styl es, and needs of their students

— for example, introducing new materials, altering delivery methods, or splitting or doubling up lessons as necessary . Since this flexibility is an integral part of the program, this study will not attempt to enforce unreasonable uniformity by insisting on lockstep application of the curriculum. RTS also encourages instructors to collect and assess student work, but not in a ty pical graded manner. Students will not receive a letter grade for their participation in RTS classes, but they will receive a Pass/No
- Pass designation , which will be collected as part of this study . Students will also receive feedback on collected work using a simple check, check plus, and check minus system. Information on individual students’A. Justification Supporting Statement Request for OMB Review (SF83I)

8 work and grades will not be tracked or gathered for this study , both because such detail is not needed to evaluate the program and because of the logistical difficulties and high costs of doing so. **A.1.e. Study Design**  To answer the research questions, this study will employ

an experimental design in which target schools (defined as public high schools and Bureau of Indian Education [BIE] schools with both an 11 th and 12 th grade) are randoml y assigned to the treatment or control conditions and all 11 th and 12 th grade students in a treatment school provided with RTS instruction. This design is necessitated by th e relativel y small size of schools in North Dakota, but it also provides the advantage that cross
- condition contamination within a school will not be possible. In a random assignment design, the simple difference between the outcomes of treatment and control students serves as an unbiased estimate of the impact of RTS on college and career preparation activities. To achieve adequate statistical power, the minimal number of schools for the study would be between 64 and 80 participating schools, depending on various assumptions. However, for maximum statistical power and to better account for the possibility of some schools refusing to participate or dropping out, we will select 95 schools, which includes all estimated eligible schools (those that have not implemented RTS at any

grade level) minus 30 schools with very small 11 th

- and 12 th

- grade enrollments. After recruitment into the study , half of schools will be randomly placed into the treatment group and half into the control group (if the number of eligib le schools is odd, we will randomly decide if the odd case is in the treatment or control group). This allows us to have 80% power with an alpha of 0.05. Further information on sample design can be found in Part B, section 1 of this OMB request. Most scho ols in the study sample would be in their own school districts; i.e., assigning two or more schools within a district to different treatment and control conditions will not be possible. For example, ND contains a high number of school districts (148) relative to its number of schools with a 12 th grade (167), meaning that most school districts only contain one school with an 11 th or 12 th grade. In addition, b ecause of the ty pically small size of ND high schools (the median 11 th or 12 th

- grade class size is 19 ), all 11 th

- and 12 th

- graders in a selected treatment school will receive the RTS curriculum. In most schools, this will mean that one teacher will be able to deliver the curriculum to one of their classes. However, instruction may be delivered b y t eachers , guidance counselors, or hired career advisors. If teachers are providing the instruction, English or social studies teachers are the most likely to do so; at the 11 th and 12 th grades, they are the onl y teachers likely to have all of the students in the grade regularly
. Instructors will receive training from REA coordinators or one of the four career resource coordinators (CRC) in the state. Training will be delivered in a two
- day session at different regional sitesSupporting Statement Request for OMB Review (SF83I) A. Justification

9 about prior to the start of the 2015
- 20 16 school ye ar; those being trained may be able to apply

the training to professional development requirements or college credit, increasing the motivation to attend training. Training will be organized and managed b y ND Department of CTE staff. In additio n, principals will receive the program manual and guidelines on how to support teachers in their RTS delivery . ND Department of CTE staff, along with REA coordinators, will also provide support for the instruction throughout the ye ar, such as answering que stions about the RTS curriculum and providing or pointing to additional RTS resources. This support will be on an as
- requested basis. Control schools will not receive RTS training or support. Because assignment is by school, and because most treatment sch ools will be the only one in their district receiving RTS, spillover effects should be largel y nonexistent. Substitution may occur if some schools decide to implement RTS on their own; however, given the resources required, control schools are very unlikel y to independentl y adopt RTS at these grade levels. **A.1.f Data Collection Needs**  As part of the evaluation, the study includes multiple data collection efforts, summarized in Table 1 and described below. **Table 1. Data Collection Needs** Instrument Data Need Respondent(s) Mode Schedule Baseline student survey Baseline college and

career preparation Background characteristics All treatment and control 11 th graders and 12 th graders We b or paper/ pencil First two weeks of 2015 - 16

school year End - of - year student surveys College and career

preparation o utcomes All treatment and control 11 th graders and 12 th graders We b or paper/ pencil Last two weeks of 2015 -

2016 school year First instructor survey Fidelity of implementation

Instructor background Training efficacy RTS instructor (treatment schools) We b January 2016 Second instructor survey Fidelity of implementation

Instructor background Support for RTS RTS instructor (treatment schools) We b June 2016 Principal survey RTS support

Other college and career support Principal or designee in treatment or control schools We b Ma y 2016 Site visits Perceptions and attitudes

toward RTS Obstacles and supports for RTS Students, instructors, counselors, and principals in 20 selected treatment or control schools In - person April - May 2016
Administrative data Academic background

Attendance/engagement School - level FAFSA applications All treatment and control 11 th graders and 12 th graders Electronic records July 2016A. Justification Supporting Statement Request for OMB Review (SF83I)

10 **Student surveys.** The primary data source will be baseline and outcome student survey s tailored for each grade (11 and 12). The survey s will be based on a 12 th

- grade survey previousl y developed for RTS that asks about college and career activities and plans (the original survey can be seen on pp. 142
- 148 of the RTS Program Manual here: http://www.roadstosuccess.org/ourcurriculum/category /18
- program - manual ). The original survey asks about behaviors related to college application (including two
ye ar and technical/trade schools), college enrollment if accepted (including name of school, living arrangements, and costs), financial aid, and immediate plans for work (including jobs applied for, job offers), in addition to lo ng - term career and educational goals. This survey has been adapted to produce separate 11 th

- and 12 th

- grade survey s that ask about the specific RTS objectives of each grade’s curriculum, in addition to modifying question wording, response options, and ques tion sequence to conform to best survey

practices (e.g., simplified language, placing more sensitive questions toward the end). The baseline survey s, to be administered within the first month of schooling, will ask the same questions as the outcome survey , with modifications reflecting the student’s place in the college and career planning process . The baseline survey will also ask consent to link survey and participation data to administrative data. **Instructor surveys.**  We will ask RTS instructors in int ervention schools to complete a survey asking about RTS topics covered and activities completed at two time points: at the end of the first half of the school year (earl y J anuary) and at the end of the second half of the school year (earl y June). These sur vey s will monitor the fidelity of implementation of RTS and provide measures of exposure to RTS concepts and activities for the impact anal ys is. Two survey s allows for better recall of RTS topics covered and time spent on RTS instruction. The instructor survey s will include a list of major RTS modules and sub
- components, with corresponding check boxes for items covered. In addition, the survey will gather the average amount of time spent in RTS each week of the relevant semester. The first instructor survey will include additional items about instructor background and prior experiences teaching college and career preparation lessons. The second instructor survey will add items providing general feedback about their experiences teaching the RTS modules, and ask them to record the final pass/no
- pass (or satisfactory

/unsatisfactory) designation for each student (RTS only provides pass/no
- pass grades for partic ipation in RTS classes). **Principal survey.** Principals or staff they may designate, such as assistant principals or guidance counselors, will be surveyed at all treatment and control schools to measure the college and career planning activities offered to students in these schools at a high level. The treatment school questionnaireSupporting Statement Request for OMB Review (SF83I) A. Justification

11 will be designed to elicit additional information about college and career planning beyond or in addition to what is being offered as part of the adapted RTS curriculum; in cont rol schools, the questionnaire will be broader and investigate all college and career planning activities in and outside of the classroom. All survey s will be delivered as online modules using existing school computer equipment to record and secure student, teacher, and principal responses. Facsimile versions of the baseline and end
- of
grade student survey s, the first and second semester teacher survey s, and the principal s urvey can be found in appendix B of this OMB request. **Site visits.** Fourteen treatment schools and six control schools will be visited one time, at the end of the 2015
- 16 school ye ar, to assess perceptions of RTS curriculum, instruction; attitudes toward college and career preparation generall y and RTS specifically; and obstacles to and su pports for learning RTS material. Treatment school site visits will provide rich background and context for survey findings and opportunities to gauge the fidelity of implementation of the intervention. Site visit data will be collected from schools in both the treatment and control groups to allow the study team to examine how RTS in context might differ from other college and career planning services provided in schools in the state. The site visits are designed to help researchers understand how and why RTS may or may not have impacted 11 th

- and 12 th

- graders’ perceptions of their college and career preparedness and their plans after high school. Because of the greater level of explicit college and career planning and instruction to be conducted in treatment schools as part of RTS, the study will visit more treatment schools than control schools; the number of treatment site visits (14) is designed to enable examination of variation within districts or REAs with multiple treatment schools. The number of control schools (6) is designed to provide insight into other modes of college and career preparation but not systematic knowledge of variations in preparedness activities. The site visits will draw on data including informal, open
- ended interviews, focus groups, direct observations, and classroom and school documentation. The bulk of information from the site visits will be collected via interviews and focus groups with individuals participating in RTS. These interviews and focus groups will allow researcher s to develop a holistic understanding of the curriculum and its implementation from multiple points of view. Researchers will interview stakeholders including: academic teachers; teachers and career counselors involved in the implementation of RTS; school administrators; and other school staff as appropriate. Researchers will also conduct focus groups with small groups of 11 th

- and 12 th

- grade students who participate in RTS. The number of informants in each case will be determined by the number of cases bei ng investigated and available resources. The majority of interviews and all fo cus groups will be conducted in
- person during site visits , supplemented by phoneA. Justification Supporting Statement Request for OMB Review (SF83I)

12 interviews as needed . With permission from subjects, audio will be recorded and transcribed to en sure accuracy in data anal ys is and recording. Site visits will also be used to conduct direct observations of the RTS curriculum being implemented. Observations allow researchers to see programs “in action,” and provide data to cross
- check and validate in formation collected during interviews and focus groups. Observations will provide direct evidence of behavior and processes to enhance data on perceptions, attitudes, and beliefs collected through interviews. Documentation regarding the implementation of RTS will be essential to understanding individual cases. The study team will collect program descriptions, lesson plans, planning documents and additional relevant information associated with RTS and the students participating in the curriculum. Site visit protocols for each of these modes of data collection are described in detail in a ppendix C of this OMB submission. **Administrative data.** The study will incorporate data from North Dakota’s statewide longitudinal data sy stem (SLDS) to provide additional info rmation on the background of students, their academic achievement, their CTE status, and their attendance habits. Administrative data agreements for these data will be negotiated with ND state SLDS staff. The study will also include school
- level FAFSA comp letion data reported biweekly

from studentaid.gov: https://studentaid.ed.gov/about/data
center/student/application
- volume/fafsa
- completion
- h igh
- school . These data provide important contextual information about overall levels of student planning for higher education. **A.1. h**  **Legislative Authorization** This study

is authorized by

section 114(d)( 4)(A) of the Carl D. Perkins Career and Technical Education Act (20 U.S.C. § 2324(d)(4(A)). **A.1.i Prior and Related Studies**  As noted previousl y, from 2007 to 2009, Mathematica Policy Research conducted a two
- ye ar evaluation of the portion of RTS delivered to 7 th and 8 th graders (Duncan, Bleeker, and Booker 2010). Conducted in rural, predominately white schools in New York and West Virginia, the study randomly assigned schools within sampling blocks (defined by free and reduced
- price lunch eligibility rates and average test scores) to either a treatment (13 schools) or control (12 schools) condition. Using baseline and end
- of

- intervention student survey s combined with attendance and mobility data, the study

found that “RTS does not have clearly positive short
- term impacts” on school and class engagement measures (motivation in school; learning and study habits; and attendance and negative behaviors) (p. 43). RTS wasSupporting Statement Request for OMB Review (SF83I) A. Justification

13 found to positively

impact some individual measures of attendance and behavior (specificall y sc hool absences and detention), but to also negativel y impact classroom attendance. The study did find impacts for several career outcomes related to the typ e of informational deficits cited above: career exploration with school staff, knowing how to find jo bs that fit their needs, and knowing what is required for different careers. Effect sizes for these measures ranged from 0.15 to 0.50. Duncan, Bleeker, and Booker’s study is the only known evaluation of RTS. **A.2 Purpose and Use of Information Collection**  Th e data for the evaluation of 11 th and 12 th

- grade RTS curriculum will be obtained from student participants and non
- participants, RTS instructors in treatment schools, principals, and other school staff. **Baseline student survey** . Data from the baseline stud ent survey will be used in conjunction with the end
- of

- ye ar student survey

to obtain estimates of the increase in frequency of concrete college and career planning activities outlined in the research questions (such as completing FAFSA or appl yi ng for a jo b after high school). Data from the baseline student survey will also be used to compare differences in treatment effects across student subgroups such as gender or race/ethnicity. **End -**  **of -** **year student survey.** Data from the end
- of
- year student survey will be used in conjunction with the baseline student survey to obtain estimates of the increase in frequency of concrete college and career planning activities outlined in the research questions (such as completing FAFSA or applying for a job after high schoo l). Data from the end
- of
- year student survey will also be used to assess fidelity of implementation for RTS instruction by anal yz ing variance in concrete activities RTS instructors were expected to require of students (e.g., creating a resume). **First sem ester instructor survey.**  The first semester instructor survey will be used to assess fidelity of implementation in the first half of the school year b y asking questions about concrete instructional activities taken during the time period covered and overal l time spent on RTS instruction. This survey

also gathers background information on instructors to enable analy sis of impact by instructor ty pe (teacher, counselor, or other RTS facilitator). **Second semester instructor survey.** The second semester instruct or survey will be used to assess fidelity of implementation in the second half of the school year b y asking the same questions as in the first instructor survey

about concrete instructional activities taken during the time period covered and overall time s pent on RTS instruction. Conducting two survey s covering shorter blocks of time will enable better recall of activities undertaken b y the instructor than a comprehensive ye ar - end survey . This survey willA. Justification Supporting Statement Request for OMB Review (SF83I)

14 also ask instructors to indicate the given or expect ed pass/fail grade each student in the class earned, which will be used in analy ses relating student performance in RTS to key outcomes. **Principal survey.** The principal survey will be used to sy stematicall y document variations in school support for RTS instruction and obstacles to instructors’ implementation of RTS, as well as the extent to which control schools provide alternative college and career readin ess training. Data from this survey therefore will be used in anal ys is of contextual effects on RTS impact and multivariate impact models controlling for school support and control school activities that may substitute for and thereby blunt the impact of RTS. **Site visits.** Data from site visits will be used to construct a cross
- site anal ysis report documenting how RTS has been received in staff and students’ own words. The cross
- site report will develop a thematic understanding of the challenges and opportun ities for students and schools as a whole, in order to draw example lessons about the constraints and considerations schools and sites face in implementing a college and career planning curriculum. **Administrative data.** Data from administrative records held b y ND will be used to identify , where possible, students involved in a CTE program (of special interest to OCTAE), as well as the academic background of all students, for use in models accounting for student characteri stics in estimating overall impact. **A.2.a Content Justifications**  This section contains justifications for RTS
-ND survey and site visit instrument s , reflecting the main research objectives of the study overall
. The survey

instruments are included in a ppendix B , and site visit protocols in a

ppendix C . **Student surveys.** As noted previousl y, many questions in the student questionnaire derive from a 12 th

- grade survey

found in the RTS Program Manual that asks about program involvement, student plans, and attitude s toward RTS. The first section of both the baseline and end
- of
- ye ar survey

solicits information on student background, including confirmation of their school of attendance (to ensure that the web questionnaire is functioning properl y), a question to identify any s tudents that have experienced RTS previously , and a question about general grades that help classify the student academically for purposes of subsequent analy sis. It also incorporates two scales to focus on career readiness through two career readiness concepts: career readiness concern and career adaptability self
- efficacy
. Career readiness concern derives from the Career Maturity Inventory

—Form C (CMI
- C) , which is designed to assess four factors relating to career readiness: concern, curiosity, confidence and consultationSupporting Statement Request for OMB Review (SF83I) A. Justification

15 (Savickas and Porfeli, 2011). “Concern” refers to the extent that respondents give consideration to finding a career , and maps to the short
- term RTS outcome of understanding yourself and the intermediate outcome of developing career awareness. A hierarchical factor analy sis (Savickas and Porfeli, 2011) showed that the concern subscale of CMI
- C has a loading of about .51 onto the higher factor of career readiness. Career adaptability self
- efficacy derives from the Career Adapt
- Abi lities Scale (CAAS) , which is designed to assess factors relating to career adaptability : concern, control, curiosity and confidence. A lthough the sub
- factors of the CMI
- C and CAAS two scales are similarly named, they

and their measurement scales are disti nct. Adaptability is distinct from readiness, as it refers to an “individual’ s resources for coping with current and anticipated tasks, transitions, traumas in their occupational roles that, to some degree large or small, alter their social integration” (S avickas and Porfeli, 2012, p. 662). This measure maps to the RTS short
- term outcomes of work skills and solving real
- world problems. The second section of the survey s asks about postsecondary planning, and includes items adopted from external sources such as the National Center for Education Statistics’ (NCES) High School Longitudinal Study

of 2009 (e.g., question 12). The postsecondary planning questions are tailored to the grade and time of year during which the questions will be posed in order to elicit information about baseline and outcomes on key measures of college preparedness. These questions all relate to the pipeline of college and career readiness that RTS is designed to influence, including high school coursetaking, college or university

visits , college or university application, financial aid applications and receipt, career preparation tests, resume preparation, and immediate postsecondary plans. The third section of the student survey s asks further detail about plans for higher education, in cluding measures that relate to critical informational deficits that students may have (college costs) and concrete aspirations for postsecondary study (majors the students may be interested in). For students with plans to work after high school, this section is followed by information solicited on work plans, with questions about current job in high school, planned jobs for after high school, jobs applied to, and work intensity . Detailed information on interested or planned majors or jobs will be gathered through a web
based coder that allows students to search for an choose majors or job titles that correspond to existing major and occupational code frames. Collectively , data on immediate postsecondary plans permits evaluation not just of the activities RT S curriculum is designed to teach (such as creating a resume or applying for college), but of the immediate effects such instruction may have on actual behavior of students. The fourth section of the student surveys (and the last section of the end
- of
- ye ar versions) concerns distal educational and occupational plans, again drawing on the existing survey and similarA. Justification Supporting Statement Request for OMB Review (SF83I)

16 items employed in NCES’s high school longitudinal studies. Interest in specific careers, expectations fo r educational and occupational attainment, and a more recent item developed and employed in HSL S:09 that asks about the relationship between expected occupation and required education, gauge the student’s overall academic and occupational orientation and l ong - term expectations. These measures will permit another layer of anal ys is relating RTS exposure to student outcomes that are less proximate to the instructional activities of RTS but conceptually linked to RTS’s goals (as, for example, illustrated in the logic model in figure 1). Consistent with best survey practice placing more sensitive questions at the end of questionnaires, the last section of the baseline versions of the student questionnaires asks more specific information about student background, specifically demographic and socioeconomic information about birthdate, gender, race/ethnicity , parent’s education, and language background. Each of these questions are consistent with NCES question and item response wording, and provide key measures enab ling disaggregation of results by well
- recognized background characteristics. A final set of measures asks about student economic background using an approach seen on international assessments (specifically , the Program for International Student Assessment [PI SA] 2006 administration) : to wit, given that students may not be the best source of information about parental income or occupation, and that there is no corresponding parent survey planned as part of this study , the questionnaire uses household estimates as a foundation for a scale of wealth or income. Exploratory and confirmatory

factor anal ys es have shown these and similar scales to be reliable and valid, with items loading on a single wealth factor in representative nations (Tray nor and Ray kov, 2013 ). **Instructor surveys.** The first and second semester instructor survey s derives it items from NCES studies and from the organization of the RTS grade 11 and grade 12 instructional modules themselves. The first part of the first semester survey asks standard background questions of instructors, including gender, race/ethnicity , and educational attainment, in order to enable comparisons during anal ys is of RTS impacts by

instructor characteristics. The second section of both survey s primarily consists of a lis t of RTS instructional modules that instructors will be asked to indicate they have taught

—this is a primary measure of fidelity

of implementation for the study , and, for modules taught, will include solicited information about challenges encountered durin g instruction. This section also asks about time on RTS tasks in an average week, the regularity with which instructors have taught material, grading practices, and specific college and career preparation activities taught. Collectivel y, these questions wi ll provide insight into the extent to which teachers have regularly and thoroughl y followed the RTS curriculum. The list of specific activities can be correlated with student responses to help validate each respondent’s claims.Supporting Statement Request for OMB Review (SF83I) A. Justification

17 The third and final section of the first semester survey asks more general questions about whether the instructor has prior experience with a college and career readiness curriculum and how they perceive RTS training, materials, and support from other school staff such as principals and counselors. These questions provide a gauge of instructor preparation to teach RTS. The second and final section of the second semester survey

asks similarly

general questions, but about students’ reaction to RTS, likelihood of using RTS in the future replacing questions about prior experience teaching and reactions to any training they

received. These questions in these sections of the respective survey s provide summary measures of attitudes towards RTS which can be unpacked via more in
- depth explorati on through the site visits. **Principal survey .**  The principal survey in both control and treatment schools provides a brief, high
- level examination of college and career planning activities and supports either around RTS or in its absence , depending on treatment status. Measures for the principal survey draw from NCES’s HSLS:09 base
- ye ar school administrator survey and NCES’s Education L ongitudinal Study of 2002’s (ELS:2002) base
- ye ar school administrator survey . The first section asks general information ab out the school, including school type, attendance, typical (estimated) outcomes of 12 th graders, and school climate. These measures provide key

information on the context of college and career readiness for the school’s student. The second section poses qu estions about key college and career readiness activities, the approximate percentage of 11 th and 12 th graders who participate in them, and the reasons for not offering such programs or activities; these align with activities covered in the teacher and stu dent surveys. The third and final section is for principals in treatment schools, and asks broad questions about their perception of students’ reactions to RTS and the usefulness and quality of RTS materials and training. The content of the principal surve y is designed to complement the teacher and student surveys, on the one hand, as well as the planned interviews at site visit schools in which open
- ended exploratory items can provide further information about college and career preparation offered b y the school. **Site visits.** The site visits are designed to help researchers understand how and why RTS may

have impacted 11 th and 12 th graders’ college and career preparedness and their plans after high school. In order to do so, researchers will conduct intervi ews with RTS teachers, academic teachers, counselors, and principals, as well as focus groups with 11 th and 12 th grade students and observations of RTS lessons. These strategies will allow study team members to collect information on various perspectives o n RTS and triangulate across these data sources to develop a rich, detailed picture of RTS implementation in schools and its effectiveness. Site visit protocols, including que stion prompts, are included in a ppendix C . They cover interviews, focus groups, and document collection that will be used to address several sets of questions.A. Justification Supporting Statement Request for OMB Review (SF83I)

18 First, the site visits will seek to understand the college and career planning assistance received prior to the 2015
- 16 academic year. Understanding what was available prior to the intervention will be crucial to understanding how RTS has changed the content available in treatment schools and how that content is delivered and received from the perspectives of various groups impacted b y these changes: administrators, counselors, teachers, and students. Q uestions about prior college and career planning will give study participants an opportunity to reflect on change over time in how, if at all, college and career planning activities are delivered in schools , and how these changes i mpact student outcomes. Second, the site visits will investigate challenges and successes instructors and students have experienced in implementing the program
. This information is important to the goals of the research because it will help researchers understand how implementation of college and career planning curricula fits int o broader school contexts. Questions on challenges and successes will also address participants’ perceptions of and attitudes toward RTS, and how these attitudes might have affe cted implementation. These questions will help researchers identify factors that both facilitated and impeded successful RTS implementation and reflect on how factors that mitigated its implementation might be addressed. Third, site visits will help to und erstand how and why RTS is or is not implemented with fidelity to its model. This will allow researchers to understand how RTS implementation actually unfurls in schools, rather than the ideal picture perhaps presented in program documentation. Understanding fidelity of implementation, or lack thereof, will help give context to study findings from other data sources such as student and instructor survey s. Deviations from the model might influence program outcomes; understanding what these deviations are, an d how and why they might have occurred, will help researchers explore how to promote greater fidelity of implementation and, ideally , improved outcomes. Fourth, and finally , the site visits will provide insight into how college and career preparation activ ities differ across the control and treatment schools, beyond merel y the presence or absence of RTS. Questions pertaining to this issue will allow the study team to explore how the intervention differs from the control context in practice. Moreover, collec ting these data will help researchers understand how school contexts and practical factors can influence the rollout of college and career planning curricula across a state. This information, in turn, will provide context for any potential differences in p rogram outcomes. While other data sources will help researchers document outcomes, site visit data will allow the study team to reflect on how and wh y those differences might occur , with practical lessons for future RTS and college/career preparation curri cula implementation
. **Administrative data.** Administrative data will be key in providing s ys tematic data on the prior academic achievements and levels of school engagement of students in the study . This will enable parsingSupporting Statement Request for OMB Review (SF83I) A. Justification

19 of RTS impacts by

CTE program statu s, academic success, and measures of student behavior such as attendance and disciplinary incidences. The data elements requested will include coursetaking data including courses taken, credits earned, and grades earned; CTE, academic/gifted, or other program status or membership; absences; and suspensions. **A.3 Use of Information Technology**  The data collection plan is designed to obtain information in an efficient and reliable fashion while minimizing burden to respondent s. All surveys will be delivered as online modules using existing school computer equipment to record and secure student responses. Skip patterns and complex routing can be employed as necessary , but a simple questionnaire is planned to reduce costs. RTI ’s Hatteras instrument development and deplo yment sy stem will be used to create and distribute (i.e., make available online) the survey s, allowing responses to be immediately uploaded into a secure database. Reliance on a web
- based survey will reduce the risk that paper copies of questionnaires are lost, copied, or stolen and respondent answers thereby

disclosed. In addition, s ome items, particularl y those relating to college, college majors, and occupations, will be designed so that students can self
- code into existing codeframes such as the National Center for Education Statistics’ Integrated Postsecondary Education Data Sy stems’ ID, the Classification of Instructional Programs (CIP), and O*NET occupational categories. Although the study team has confirmed with ND state staff that all high schools should be capable of hosting the online survey with existing computer equipment, we will have available paper copies for any schools that require them. Administrative data records will be delivered electronically as w ell through a secure FTP server to which only

authorized ND staff and study leadership at RTIhave access. These records will be immediately

downloaded and stored on secure RTI servers, and removed from the FTP server. Electronic exchange of administrative data lowers the risk of disclosure compared to secure faxes or hard
- cop y document delivery , in addition to reducing burden on state staff compared to hard
- cop y preparation, faxing, and/or mailing. **A.4 Efforts to Identify Duplication and Use of Similar In**  **formation** As part of planning for this study , researchers reviewed existing college and career curriculum studies and evaluations, particularl y those related to RTS, and found only one prior data collection of lower grades conducted previously in northeast ern states. There are no known datasets or ongoing collections gathering information about RTS curriculum in the 11 th and 12 th grade . North Dakota has notA. Justification Supporting Statement Request for OMB Review (SF83I)

20 implemented any data collection or evaluation of RTS in the small number of schools that have so far adopted it. **A.5 Impact on Small Businesses or Other Small Entities**  Target respondents for RTS
-ND survey s and site visits are individuals , and data collection activities for this study will not burden small businesses or entities. The study will minimize burden for all respondents b y including onl y the questions and items necessary to address the research questions and how impacts differ b y s chool, instructor, and student characteristics. **A.6 Consequences of**  **Not Collecting Data** The implementation of RTS in ND represents a rare opportunity to study a statewide curriculum initiative with concrete, clearl y delineated components and goals in a rigorous manner. Large
- scale randomized studies are difficult to implement because of the n ecessity to gain buy
- in from a number of stakeholders and the challenges of operating in many varied sites. In ND, the existing state commitment to RTS means both a willing partner, a base of familiarity with RTS content and its implementation, and a somewhat uniform but large
- scale state environment in which to assess the impact of college and career readiness. Not collecting the data deprives Federal program staff at OCTAE and in other offices of the U.S. Department of Education of the chance to reliably and rigorousl y test a significant, coherent college and career preparation curriculum that can have important implications for policy design, grant programs, and other efforts to identify and promote effective postsecondary readiness training mechanisms. Specificall y, this study will provide direct, rigorous evidence of the utility of an intensive, coherent college
- and
- career readiness curriculum at the upper grades of high school. In contrast to other prominent readiness curricula, RTS serves both disadvantaged and non
- disadvantaged students, making documentation about its effectiveness important for a wide array of students, educators, and policy
makers. RTS also concretely targets information deficits and practical stepping stones that lead to college an d career preparedness

— needs which are acute at the end of high school where postsecondary options are diverse and pathway s to and through them are often unclear for students. Determining the strengths and weaknesses of RTS will help improve RTS directl y an d can provide guidance to other programs with similar goals. Furthermore, the North Dakota setting allows us to examine the implementation and impact of such a program in small
- school, rural settings that are often not rigorously and widel y studied, in ad dition to city

and suburban environments within the state.Supporting Statement Request for OMB Review (SF83I) A. Justification

21 **A.7 Special Circumstances**  No special circumstances of data collection are anticipated. **A.8 Federal Register Announcements and**  **Consultations A.8.a**  **Federal Register Announcement** The 60
- day

notice to solicit public comments was published on XX. The 30
- day notice will be published to solicit additional public comments. **A.8.b Consultations Outside the Agency**  Officials from the North Dakota Department of Career and Technical Education were consulte d extensively

in planning the design of this study and its associated information collections. **A.9 Provision**  **of Payment or Gift to Respondents** In centives will be given to RTS instructors, school coordinators, and principals to motivate participation in data collection activities. For teachers, counselors, or career advisors who will be delivering RTS, we will offer between $50
- $100 for participation, depending on the number of classes they are instructing. The specific level of incentive for RTS instruct ors will depend on their level of involvement in the study , as described in table 3. This size of incentive is commensurate with the burden placed on instructors for completing two surveys for each class ($25 per survey). Principals at both treatment and c ontrol schools will receive a small incentive ($25) for their participation in the principal survey . School coordinators will receive up to a $150 incentive for managing consent forms, scheduling the student survey s, and coordinating with researchers to pl an site visit logistics where applicable. In addition, consistent with research experience conducting evaluations, there will be a school
- level incentive for control schools which otherwise have little motivation to participate in the study . The incentive plan is summarized in table 2. **Table 2. School and Instructor Incentives** RTS instructors (teacher, counselor, or career adviser) One class Two or more classes $50 $100 School coordinators Schools with no site visits: $75 Schools with site visits: $150 Principals All principals: $25 Control schools All control schools: $1,000A. Justification Supporting Statement Request for OMB Review (SF83I)

22 **A.10 Assurance of Confidentiality Provided to Respondents**  OCTAE assures participating individuals that all identifiable information collected as part of the RTS
-ND study may

be used only

for statistical purposes and may not be disclosed, or used, in identifiable form for any other purpose except as required b y l aw. RTS
-ND data security and confidentiality protection procedures are in place to ensure that RTI and its subco ntractors comply with all privacy requirements, including:

 The approved Revised Proposal submitted by RTI and on file with OCTAE ;  Privacy

Act of 1974 5 U.S.C. § 552(a);  The U.S. Department of Education Incident Handling Procedures (February 2009);  The U.S. Department of Education General Handbook for Information Technology Security General Support Sy stems and Major Applications Inventory Procedures (March 2005);  The U.S. Department of Education, ACS Directive OM: 5
- 101, Contractor Employee Personnel Securi ty

Screenings.  Family

Educational and Privacy Act of 1974, 20 U.S.C. § 1232(g);  ESRA 2002 , 20 U.S.C. § 9573; and  All new legislation that impacts the data collect ed through this contract. Additionally , RTI will comply with the Department’s I T security poli cy requirements as set forth in the Handbook for Information Assurance Security Policy and related procedures and guidance as well as IT security

requirements in the Federal Information Security Management Act (FI SMA), OMB Circulars, and the National Insti tute of Standards and Technology (NI ST) standards and guidance. All data products and publications will also adhere to the NCES Statistical Standards, as described at the website: http://nces.ed.go v/statprog/standards.asp
. The RTS
-ND procedures for maintaining confidentiality include notarized nondisclosure affidavits obtained from all personnel who will have access to individual identifiers; personnel training regarding the meaning of confidentiali ty ; controlled and protected access to computer files; built
- in safeguards concerning status monitoring and receipt control systems; and a secure, staffed, in
- house computing facility
. RTS
-ND follows detailed guidelines for securing sensitive project data, including, but not limited to: phy sical/environment protections, building access controls, sy stem access controls, sy stem login restrictions, user identification and authorization procedures, encry ption, and project file storage/archiving/destruction
. Furthermore, the Department has established a policy regarding the personnel security screening requirements for all contractor emplo ye es and their subcontractors. The contractor must comply with these personnel security screening requirements throughout the life of the contract. The DepartmentSupporting Statement Request for OMB Review (SF83I) A. Justification

23 directive that contractors must comply with is OM:5
- 101, which was last updated on 7/16/2010. There are several requirements that the contractor must meet for each emplo ye e working on the contract for 30 day s or more. Among these requirements are that each person working on the contract must be assigned a position risk level. The risk levels are high, moderate, and low based upon the level of harm that a person in the position can cause to the Department’s interests. Eac h person working on the contract must complete the requirements for a “Contractor Security Screening.” Depending on the risk level assigned to each person’s position, a follow
- up background investigation b y the Department will occur
. Study notification mat erials sent to sample members will describe the voluntary nature of RTS
ND and convey

the extent to which respondent identifiers and all responses will be kept confidential. Contacting materials are presented in appendix A, including confidentiality language provided in a study brochure. **A.11 Justification for Sensitive Questions**  There are no sensitive questions to be asked as part of this data collection. **A.12 Estimates of H**  **ours Burden**  Estimates of response burden for the RTS
-ND study are shown in table 3
. Estimates of survey response burden are based on estimates developed from experience with similar items and interview subjects in conducting NCES survey s of high school students (EL S:2002 and HSL S:09). The number of respondents (targeted and expected given the response rates) overlap across some of the data collection efforts; that is, some of the students taking the survey s will also be part of the focus groups during the site visits. Therefore, the totals for those columns are less than the sum of the column values. The total estimated cost to school and state (administrative records) staff respondents is $ 4,224 , which represents the ir combined 164 burden hours times vari ous hourly rates depending on staff title. Estimates of hourly

rates for North Dakota principals ($35) and teachers ($21) in public schools was obtained from published tables of the 2011
- 12 Schools and Staffing Survey (hourly rates were calculated as estim ated annual salaries divided by 2080 hours). Counselors were estimated to have slightl y higher hourly rates ($25), as was administrative records staff person ($35)
.A. Justification Supporting Statement Request for OMB Review (SF83I)

24 **Table 3. Estimated Burden for RTS -**  **ND Data Collection Data collection activity**  **Number of Targeted**  **Respondents Expected** 

**Response Rate (%**  **) Number of** 

**Respondents Unit** 

**Response Time**  **(Hours) Total** 

**Response Time**  **(Hours/Year) Total** 

**Burden Time**  **(Hours)** 11 th

- grade student survey s 4,220 95 4,009 0.58 2325 2325 12 th

- grade student survey s 4,188 95 3,979 0.50 1990 1990 I nstructor survey s 120 90 108 0.50 54 54 Principal survey 95 90 86 0.25 22 22 S ite visit: f

ocus groups 32 0 100 32 0 0.83 266 266 Site visit: instructor interviews 40 100 40 1.00 40 40 Site visit: principal interview 20 100 20 1.00 20 20 Site visit: counselor interview 20 100 20 1.00 20 20 Administrative records match 1 100 1 8 8 8 **Total 8,7**  **60 8,**  **318 4**  **, 745**  **4 ,**  **745** Note: Because of overlap of respondents for the student surveys and focus groups, the instructor surveys and interviews, and the principal surveys and interviews, the totals for targeted and expected respondents is lower than the sum of the column values. **A.13 Estimates of Cost Burden to Respondents**  There are no capital, startup, or operating costs to respondent s for participation in the project. No equipment, printing, or postage charges will be incurred. **A.14 Annualized Cost to the Federal Government**  The e stimated annual cost to the federal government for RTS
-ND is $ 643 , 841 , which includes total OCTAE staff costs for managing NCICTE work ( $ 62,660 ) and total contract costs ($1,225,022) associated with this specific study
. The estimate includes costs for study recruitment, materials and instruments preparation, administration of data collection, processing and analyzing data, and producing reports. The contract costs were generated through budgeting specific contractor staff for express numbers of hours and direct costs associated with conducting the study (e.g., mailouts, incentives). **A.15 Reasons for Program**  **Changes** This is an original submission and no changes are being requested. **A.16 Publication Plans and Project Schedule**  Under the approved Revised Proposal submitted to OCTAE as part of the NCI CTE contract, this study will produce two products for public use: a summary report detailing research background,Supporting Statement Request for OMB Review (SF83I) A. Justification

25 methods, data collection results, descriptive findings, and impact estimates; and a res tricted
- use data file able to be licensed to researchers for additional analy sis. The anal ys is approach will be to compare changes in outcomes for students in treatment schools to those in control schools. Outcomes will include changes in percentages (suc h as of students who have applied for a job) as well as changes in averages for specific measures (e.g., number of postsecondary institutions visited). Tables 4 and 5 link key outcomes (by

grade) to the indicators measured through data collection, the source of data, and the type of outcome anal ys is to be conducted. At each grade level, one ye ar of the RTS curriculum is expected to be associated with a statistically significant increase in each outcome. **Table 4. Outcomes, measures, and analy sis for 11**  **th -**  **gr ade RTS intervention**  **Outcome Measure**  **Data source Anal**  **ys is (treatment**  **versus control)** Specific senior- year coursetaking plans Number of specific courses listed by student Student survey item Compare mean number of courses listed College or university visits Number of specific sites listed by student Student survey item Compare mean number of sites listed Having taken or registered for workforce development assessments (i.e., Test of Adult Basic Education (TABE), the Armed Services Vocational Aptitude Battery (ASVAB), the ACT Wo rkKeys Foundational and Personal Assessments) Number of specific assessments identified by student as registered or completed Student survey checklist items Compare (for registration, completion, and combined) mean number of val id assessments listed Resume preparation Self - rating of resume creation status (e.g., “I am planning to create it,” I have an early draft,” “I have a finished version,” “I haven’t given it much thought”) Student survey Likert scale item Compare mean rating of resume status; compare proportion of student resumes in each category Career exploration activities Number of relevant activities (e.g., job shadowing, attendance at career fairs, internships, apprenticeships) students report Student survey ite ms Compare mean number of exploration activities

Interest in specific careers Level of interest in careers Student survey Likert scale items Compare mean rating of interestA. Justification Supporting Statement Request for OMB Review (SF83I)

26 **Table 5 . Outcomes, measures, and analy** **sis for 12 th**  **- grade RTS intervention**  **Outcome Measure**  **Data Source Anal**  **ys is (treatment**  **versus control)** Applying to post - secondary education institutions (i.e., technical or trade school, two - year community college, or four- year college) Type and number of institutions students report having s ubmitted applications to attend Student survey items Compare mean number of applications; compare number of applications in each category Completing the Free Application for Federal Student Aid (FAFSA) Wh ether the student reports FAFSA completion Student survey item Compare mean number of students completing FAFSA Acceptance to post - secondary institution(s) for fall semester Number of institutions student reports receiving notice of acceptance Student survey item Compare mean number of accepting instituti ons; compare proportion accepted to at least one institution Registration at a post - secondary institution Wh ether student reports being registered Student survey item Compare proportion of students registered Resume preparation Self - rating of resume creation status (e.g., “I am planning to create it,” I have an early draft,” “I have a finished version,” “I haven’t given it much thought”) Student survey Likert scale item Compare mean rating of resume status; compare proportion o f student resumes in each category Applying for a job* Type and number of employers students report having submitted applications to work Student survey items Compare mean number of applications; compare number of applications by job type Receiving a job offer* Number of offers student reports receiving Student survey item Compare mean number of offers; compare proportion receiving at least one offer Interest in specific careers Level of interest in careers Student survey Likert scale items Compare mean rating of interest Planning to major in an area related to specific career interests Student self - report Student survey items Compare proportion of students with careerrelated major * Including full
- time employment or service in the armed forces Analy sis will be done both descriptivel y and through the use of linear regression models which improve precision of impact estimates b y controlling for additional sources of variation in outcomes. Models may

include multilevel models (a.k.a., hierarchical linear models, HLM) that model each outcome based on a set of covariates that include treatment status; baseline survey measures; administrative data including pre
- intervention test scores and attendance; student demographics and background characteristics ; instructor characteristics; school
- level contextual variables; and measures of the fidelity of implementation. The schedule for the RTS
-ND study is shown in table 6
.Supporting Statement Request for OMB Review (SF83I) A. Justification

27 **T able 6**  **. Schedule for**  **RTS -**  **ND Study** Task No. Deliverables Due date 1.3 IRB approvals 3/1/2015 1.4 State SLDS research agreement/MOU 7/17/2015 2.1 Draft sample of schools 1/16/2015 2.2 Final sample of schools 5/22/2015 3.1 Pre - recruitment documentation 3/18/2015 3.2 District research agreements 6/23/2015 3.3 School recruitment summary 8/12/2015 4.1 Draft online (student, instructor, principal) instruments 2/6/2015 4.2 Final online (student, instructor, principal) instruments 6/5/2015 4.3 Baseline student survey results memo 10/2/2015 4.4 Instructor survey results memo 5/18/2016 4.5 End - of- ye ar student survey results memo 6/3/2016 5.1 Principal interview results memo 6/3/2016 5.2 Case study report 7/8/2016 6.1 Draft restricted - use data file 7/15/2016 6.2 Final restricted - use data file 8/5/2016 6.3 Draft summary report 7/22/2016 6.4 Final summary report 8/19/2016
**A.17 Reason(s) Display of OMB Expiration Date Is Inappropriate**  The expiration date for OMB approval of the information collection will be displayed on data collection instruments and materials. No special exception to this requirement is requested. **A.18 Exceptions to Certification for Paperwork Reduction Act Statement**  There are no exceptions to the certification statement identified in the Certification for Paperwork Reduction Act Submissions of OMB Form 83
- I.A. Justification Supporting Statement Request for OMB Review (SF83I)

28 **References** Bowers, Judy , and Hatch, Trish. 2005. The ASCA National Model: A Framework for School Counseling Programs . Alexandria, VA: American School Counselor Association. Duncan, Chaplin; Bleeker, Mary ; and Booker, Kevin. 2010. Roads to Success: Estimated Impacts of an Education and Career Planning Program During Middle School: Final Report . Princeton, NJ: Mathematica Policy

Research. Grodsky , Eric and Riegle
- Crumb, Catherine. 2010. “Those Who Choose and Those Who Don’t: Social Background and Coll ege Orientation." The Annals of the American Academy of Political and Social Science, 627(2): 14
- 35. Hoxby , Caroline and Avery , Christopher. 2012. “The Missing ‘One
- Offs’: The Hidden Supply of High
Achieving, Low Income Students.” NBER Working Paper No. 18 586. Radford, Alexandria W. 2013. Top Student, Top School?: How Social Class Shapes Where Valedictorians Go to College . Chicago, IL: University of Chicago Press. Rosenbaum, James E. 2001. Beyond College for All: Career Paths for the Forgotten Half . American Sociological Association’s Rose Series in Sociology . New York: Russell Sage Foundation. Savickas, M. L., & Porfeli, E. J. (2011). Revision of the Career Maturity Inventory The Adaptability Form. Journal of Career Assessment , 19 (4), 355
- 374. Savickas, M. L., & Porfeli, E. J. (2012). Career Adapt
- Abilities Scale: Construction, reliability , and measurement equivalence across 13 countries. Journal of Vocational Behavior , 80 (3), 661
- 673. Tray nor, A., & Ray

kov , T. (2013). Household possessions indices as wealth measures: a validity evaluation. Comparative Education Review , 57(4), 662
- 688.