Appendix C

Site Visit Procedures and Protocols
C-2

Site Visit Procedures and Protocols

Overview

This appendix offers an overview of protocols and procedures for planning and conducting case study site 

visits to treatment and control schools. The case study site visits will provide rich background and context for 

survey findings and opportunities to gauge the fidelity of implementation of the intervention. Site visit data 

will be collected from schools in both the treatment and control groups to allow the study team to examine 

how RTS in context might differ from other college and career planning services provided in schools in the 

state. The site visits are designed to help researchers understand how and why RTS may have impacted 11th-

and 12th-graders’ perceptions of their college and career preparedness and their plans after high school. The 

following overarching research questions will guide case study data collection and analysis: 

 What college and career planning assistance did students receive prior to their current year? 

 What challenges and successes have instructors and students experienced in implementing the 

program? 

 To what extent was the program implemented with fidelity to the model (e.g., how many of the 

components were introduced, for how long, to what cross-section of students, and in what ways)? 

 How did college and career development strategies differ in the treatment and control schools?

Because we anticipate incorporating up to 47 treatment schools as part of the study, visiting each 

school would be cost-prohibitive; however, researchers plan to visit 14 of the treatment schools and 6 of the 

control schools to enhance the generalizability of site visit findings. Researchers will visit each of the selected 

sites once in the spring of 2016 (April or May). While the high schools will be the cases, several populations 

associated with RTS will serve as secondary cases or data sources in the treatment schools, including: 11th-

and 12th-grade students, teachers, academic and career counselors, and administrators. The high schools will 

serve as the context in which collaboration between these groups around RTS occurs.

Information to be Collected

Researchers will conduct interviews with RTS teachers, academic teachers, counselors, and principals, as well 

as focus groups with 11th- and 12th-grade students and observations of RTS lessons. These strategies will allow 

study team members to collect information on various perspectives on RTS and triangulate across these data 

sources to develop a full, detailed picture of RTS implementation in schools and its effectiveness. Openended, exploratory questions will be presented to respondents and focus groups in order to prompt reflection 

and allow for elaboration beyond what is allowable within a structured, closed-ended survey design. 

Questions to be used as part of the site visits can be grouped into the following four areas pertaining to (1) 
C-3

prior college and career planning assistance; (2) challenges and successes with RTS; (3) fidelity of 

implementation; and (4) other college and career planning activities: 

1. What college and career planning assistance did students receive prior to their current year? 

 What college and career planning assistance did students receive prior to RTS, if any? How helpful 

was it? What aspects were most valuable?

 Did RTS participation change senior year course taking patterns? Why? How?

 Did RTS participation change students’ plans for applying to college and careers? Why? How?

 Did RTS participation change students’ college course taking plans and/or plans for college major?

Why? How?

 How has RTS impacted teachers’ instructional practices or strategies?

 How has RTS impacted counselors’ practices or strategies? How?

 Has RTS impacted students’ preparedness for college and/or career in comparison to previous years? 

How?

2. What challenges and successes have instructors and students experienced in implementing the 

program? 

 What assistance was provided in registering and preparing for workforce development assessments? 

How helpful was it? Have students’ plans changed as a result?

 What assistance was provided in identifying and applying for employment? How helpful was it? Have 

students’ career plans changed as a result? How?

 What assistance was provided in preparing a resume? How helpful was it? Have students’ career plans 

changed as a result? How?

 What assistance was provided in identifying and applying for sources of financial aid (FAFSA, 

scholarships, etc.)? How helpful was it? Have students’ postsecondary plans changed as a result? 

How?

 What assistance was provided in identifying colleges and preparing and submitting college 

applications? How helpful was it? Have students’ postsecondary plans changed as a result? How?

 What 11th and 12th grade RTS topic areas have students found most and least useful? Why?

 What 11th and 12th grade RTS topic areas have teachers found most and least effective? Why?

 How do principals, counselors, and teachers assess students’ level of preparedness for college and 

career?

3. To what extent was the program implemented with fidelity to the model? 

 What college and career readiness topics were covered during RTS training?
C-4

 How many RTS components were introduced over the course of the academic year? For how long? 

To what cross-section of students? In what ways?

 How, if at all, does the RTS model advocate integrating college and career planning instruction with 

academic and CTE topics?

 What ongoing supports for RTS implementation, if any, do teachers receive during the academic year?

 What aspects of RTS training were most and least useful? Why?

4. How did college and career development strategies differ in the treatment and control schools?

 What college and career planning assistance do students receive? How helpful is it?

 What assistance is provided in registering and preparing for workforce development assessments? 

 What assistance is provided in preparing a resume? 

 What career exploration activities are provided? 

 What assistance is provided in enrolling or planning to enroll in college? 

 What assistance is provided in identifying and applying for sources of financial aid (FAFSA, 

scholarships, etc.)?

 What assistance is provided in identifying and applying for employment? 

 What impact, if any, has participation in college and career planning activities had on students? How 

do participants know that these changes have occurred?

Site Visit Procedures and Protocols

Before the Site Visit

Site Selection: Researchers plan to visit 14 of the treatment schools and six of the control schools to enhance 

the generalizability of study findings. The study team will randomly select treatment schools from each of the 

four largest districts in the state (Bismarck, Minot, Fargo, and Williston), and randomly select the remainder 

of treatment schools from each of seven regional education associations (REAs1) (1-2 additional treatment 

schools per REA). We plan to randomly select two control schools from among the four largest districts and 

the remaining four from other REAs. This will ensure geographic diversity and diversity of school size across 

our sites. Researchers will visit each of 14 treatment schools and six control schools once in the spring of the 

2015-2016 academic year.

Site Visit Planning: A school coordinator to serve as main point of contact will be identified for each of the site 

visit schools. One member of the research team will serve as site visit lead for planning and conducting each 

 

1 As noted in our sampling plan, two of the smallest REAs would be combined and treated as one REA as part of the study. 
C-5

site visit. Researchers will seek to provide ample advance planning time to school coordinators before the site 

visit. To facilitate site planning, the research team will share with the school coordinator several documents in 

advance of the site visit, including:

 A list of suggested interview and focus group participants and guidance for recruiting each.

 A one-page overview of the study, research questions, and information that will be sought during the 

site visit from each group of participants.

 A list of background documentation requested.

 A sample site visit agenda.

The site visit lead and school coordinator will collaborate to finalize the list of participants and schedule 

for each one-day site visit.

Site Visit Training: Prior to data collection, senior staff at FHI 360 will conduct a one-day training session to 

promote consistency in data collection, analysis, and reporting of qualitative data. The training will be offered 

in-person and via webcast. The schedule for site visits and all interview, focus group, and observation 

protocols will be made available to all participants two weeks prior to the one-day training session.

Notebooks prepared for the case study training will include background on RTS; the research design, 

including major research questions and other tasks; the project organization chart and staffing/role of site 

researchers; project abstracts and contact information for all sites to be visited; interview, focus group, and 

observation protocols; a checklist for site visit planning; outlines for individual case descriptions; and a college 

and career planning literature review. 

Researchers will receive training on effective moderating practices for interviews and focus groups, 

such as establishing a comfortable environment, remaining neutral, using pauses and probes effectively, 

transitioning between major topics, and maintaining participants’ interest throughout the interview or focus 

group. Team members will also be trained on collecting observational data.

Documentation: Prior to the site visit, the site visit lead will work with the school coordinator to identify and 

collect appropriate background information on college and career curricula in the school. This documentation 

might include program descriptions, lesson plans, planning documents and additional relevant information 

associated with RTS and the students participating in the curriculum. Site visit team members will review 

background documentation prior to the site visit and come prepared with clarifying questions and requests 

for further information, as appropriate. Information gleaned from documentation will help researchers 

understand the school context for college and career planning implementation and inform analysis of data 

collected during the site visit.
C-6

During the Site Visit

Teacher and Staff Interviews: During each site visit, researchers will conduct standardized, open-ended interviews 

with RTS and academic teachers, principals, and school counselors. Because of the small size of most North 

Dakota high schools, in most instances there will be only one representative from each category. The site visit 

lead will work with the site coordinator in advance of the visit to identify up to three academic teachers with 

appropriate perspectives on 11th and 12th graders’ college and career readiness. Interview guides will ask for 

participants’ perspectives on behaviors and opinions related to how RTS is implemented, its impact on 

students, and why or why not it has been successful in their schools.

For each interview, researchers will follow generally accepted best practices for preparing and 

conducting interviews, including: selecting interview settings with few distractions; explaining the purpose of 

the interview to participants; addressing the terms of confidentiality; explaining the format of the interview 

and encouraging participants to offer candid answers and ask clarifying questions; indicating that the interview 

will take approximately one hour; providing contact information for the research team; asking neutral, openended questions one at a time; providing appropriate probes and prompts to encourage the participant; 

providing transitions between major topics; remaining neutral to responses; carefully checking the time to 

cover all questions in the allotted one hour; and thanking all participants. The last question will be to allow 

participants’ to provide any additional information.

With participant permission, all interviews will be audio recorded and transcribed to facilitate data 

analysis.

Student Focus Groups: The site visit lead will serve as focus group moderator, working with the school 

coordinator to recruit participants and establish a comfortable environment. Six to eight carefully selected 

students will be recruited for each of two 90-minute focus groups during site visits; focus groups of 11th and 

12th grade students will be conducted separately. The site visit lead will ask the school coordinator in advance 

to identify and recruit students who will likely be able to share varied and comprehensive perspectives on 

RTS. 

Researchers will follow generally accepted best practices for preparing and conducting focus groups, 

including: reducing barriers to participation by scheduling around participants’ previously-scheduled 

commitments; clearly explaining the purpose of the research; following a prepared script to welcome 

participants and establish ground rules; asking short, focused, open-ended, and unambiguous questions; 

encouraging participation by all members; using probes and prompts to clarify responses and elicit additional 

information; summarizing long or complex comments to check for understanding; and thanking all 
C-7

participants. The moderator will seek out varied opinions and ideas from all members of the group related to 

their participation in the college and career planning curriculum and the curriculum’s impact.

With participant permission, all focus groups will be audio recorded and transcribed to facilitate data 

analysis.

Classroom Observations: Observations allow researchers to see programs “in action,” and provide data to crosscheck and validate information collected during interviews and focus groups. Teacher-student interactions are 

crucial to student outcomes. Observations will provide direct evidence of behavior and processes to enhance 

data on perceptions, attitudes, and beliefs collected through interviews. Observation protocols developed for 

this study will be descriptive rather than evaluative. Drawing on the RTS Program Manual, researchers will 

seek information on tools and materials shared with students through RTS as well as any supplemental 

information and how RTS lessons are carried out. The observation tool will ask for documentation of, for 

example, the lesson topic, instructional setting, instructor practice, student time on task, and evidence for 

students’ active engagement in the lesson. 

Researchers will follow best practices for conducting classroom observations, including: providing 

adequate training on the observation tool; observing in consistent ways across classrooms; focusing on rich 

description of observable behaviors; and limiting inferences to the scope of the tool. Observers will also 

conduct a short debrief with teachers to gauge their perceptions of the lesson.

After the Site Visit

Data Analysis: Initial site-specific data analysis will begin during the data collection phase. All audio transcripts, 

observation notes, and documentation will be immediately uploaded to NVivo for initial coding to identify 

themes within and across cases. The initial themes derived from early analysis will inform the development of 

a code book to guide subsequent analyses. Early data analysis will also provide the research team with the 

opportunity to collect additional data that could inform study findings. Once all of the data has been coded, 

researchers will conduct a within-case analysis to compare informant responses at each high school to 

understand implementation of RTS, fidelity to the model, participant perceptions on implementation and 

outcomes, and program outcomes. 

Reporting: A cross-case analysis will serve as the report for the case study phase of documentation. Researchers 

will conduct the cross-case analysis following completion of data analysis for individual sites. Analyzing the 

data across cases allows the research team to compare high schools in different contexts to identify common 

elements, themes, and patterns emerging across cases. The cross-site report will develop a thematic 

understanding of the challenges and opportunities for students and schools as a whole, in order to draw 
C-8

broader lessons about the constraints and considerations schools and sites face in implementing a college and 

career planning curriculum. Each researcher conducting a site visit will contribute to the cross-case report. All 

members of the research team will be expected to review and comment on the first draft of the cross-case 

report. The authors will elicit feedback on future drafts based on staff expertise.