Virtual Reality Job Interview Training: An Enhancement to Supported Employment in Severe Mental Illness
Phase: NA
Conditions: Schizophrenia, Schizoaffective Disorder, Bipolar Disorder, Depressive Disorder
Sponsor: University of Michigan

Page 1 of 34
PROTOCOL TITLE:
Virtual Reality Job Interview Training: An Enhancement to Supported Employment for Individuals with Severe 
Mental Illness
PRINCIPAL INVESTIGATOR:
Name: Neil Jordan, Ph.D.
Department: Psychiatry and Behavioral Sciences 
Telephone Number: (312) 503-6137
Email Address: neil-jordan@northwestern.edu 
Clinicaltrials.gov Identification Number: NCT03049813 
VERSION DATE: February 7, 2019
IRB #: STU00202936-MOD0048 Approved by NU IRB for use on or after 6/27/2019

Page 2 of 34
1.0 Purpose of the Study:
Approximately 10-15% of the 9 million adults with severe mental illnesses (SMI) are employed1-3, which is 
associated with reduced psychological distress4. However, the employment rate increases to ~30% in adults with 
SMI enrolled in Supported Employment (SE) (i.e., the gold standard for vocational training)5, 6. A recent model 
of SE shows that active job-seeking behavior (e.g., completing job interviews) is an essential treatment target for 
employment, and finding a job is moderated by self-efficacy and anxiety about interviewing7. Moreover, these 
targets are underdeveloped as adults with SMI have poor interviewing skills and want training to alleviate their 
fears and poor self-efficacy related to interviewing7-14. Currently, SE relies on role-play training led by 
“Employment Specialist” counselors to enhance interview skills, but SE clients reported that this method did not 
improve their skills15, 16. In response, NIMH funded the development of Virtual Reality Job Interview Training 
(VR) and evaluation of VR efficacy in a series of randomized controlled trials (RCTs, R44 MH080496). The 
results suggest that VR hit the targets by improving interview skills and self-efficacy in several cohorts with SMI17- 
21 and demonstrated efficacy by increasing the likelihood of getting job offers18, 21-23. Of note, the integration of 
new technologies into SE is a top research priority for SE developers24. Thus, VR is an ideal intervention to 
evaluate as an effective enhancement to SE. Thus, there are a series of critical next steps to test if VR is effective 
at improving vocational outcomes for SE clients with SMI. A first step will test if replacing SE interview training 
with VR enhances person-level (e.g., higher employment rate) and b) system-level (e.g., staff efficiency and cost- 
effectiveness) SE outcomes. A second step will evaluate the initial implementation of VR within SE, while a third 
step will evaluate potential mechanisms for enhanced SE outcomes.
Our long-term goals are to develop, evaluate, and implement virtual reality interventions that enhance vocational 
recovery from SMI. The objectives of this application are to: 1) evaluate if VR enhances SE outcomes and if the 
benefits of VR generalize to improved social skills and reduced psychological distress; and 2) identify facilitators 
and barriers associated with large-scale implementation of VR. Our central hypothesis is that VR will enhance 
vocational outcomes in a SE program. This hypothesis is based on our research showing that VR performance 
predicted greater odds of receiving a job offer and a shorter duration until receiving a job offer16, 19-21, and 
recommendations from a community advisory board of SE leaders and staff. We also recognize that VR needs 
research to maximize the effectiveness of VR in real-world settings.
In this application we propose to conduct a single-blinded confirmatory effectiveness RCT (and a process 
evaluation25) of VR by comparing subjects receiving SE with VR (instead of clinician role-plays) (SE+VR) 
to subjects receiving SE as usual (SE Only). Subjects will include adults with SMI who are newly enrolled in 
SE at Thresholds, the largest community mental health service provider in Chicago. We are well prepared to 
conduct this study because our team has extensive experience with community-based vocational and 
implementation research, and in leading prior studies at Thresholds. The Specific Aims are:
1. Evaluate if SE+VR, compared to SE Only, enhances SE outcomes for adults with SMI. At a person- 
level, we hypothesize (H) that SE+VR subjects, compared to SE Only subjects, will have higher employment rates 
(H1), greater improvement in job interview skills (H2), get jobs sooner (H3), and have reduced psychological 
distress (H4) by 6-month follow-up. At a system-level, we hypothesize that SE+VR will be more cost-effective 
than SE Only (H5). Exploratory Subaims: 1. Explore if VR generalizes to social skills by testing if VR training 
relates to enhanced social skills and longer job tenure for clients (person-level) and if SE staff spend fewer hours 
coaching communication at work (systems-level); 2. Explore if SE+VR staff have increased efficiency (e.g., time 
spent helping clients with applications and networking with employers (system-level)).
IRB #: STU00202936-MOD0048 Approved by NU IRB for use on or after 6/27/2019

Page 3 of 34
2. Evaluate mechanisms of employment outcomes and psychological distress. Based on Corbiere’s 
model7, we hypothesize is that interview skills, measured independently during role-play, will mediate the effect 
of interview training on employment outcomes (H6). Exploratory Subaims: 1. Explore if post-training levels of 
interviewing anxiety/self-efficacy mediate the relationship between VR training and interviewing skills and if this 
mediation is moderated by pre-training levels of anxiety and self-efficacy (Fig 1b); 2. Explore if employment 
outcomes mediate the relationship between interviewing skills and psychological distress at 6-month follow-up.
3. Conduct a multilevel, multidisciplinary, and mixed-method process evaluation of VR adoption and 
implementation to assess the acceptability, scalability, generalizability, and affordability of VR. We will use 
focus groups and interviews (with clients, staff, and leaders) to identify facilitators and barriers to implementing 
VR. We will use budget impact analysis to estimate the cost of implementing VR at Thresholds.
2.0  Background / Literature Review / Rationale for the study:
More than 9 million Americans have severe mental illnesses (e.g., major depression, bipolar disorder, and 
schizophrenia) with long periods of clinical stability when they can work26, 27. A primary aim for adults 
recovering from SMI is getting a job as 70% of this population wants to work43, 44, and employment increases 
quality of life28-31. Thus, adults with SMI, service providers, and policy makers view a return to work as a 
critical milestone for recovery32-34. However, only 10-15% of this group is employed1, 2 and struggling to find 
jobs increases their psychological distress4, 35, 36.
The field’s gold standard vocational program for adults with SMI is Supported Employment (SE)5, 6, 37, which uses 
staff-based role-plays to train interview skills. However, despite their desire to improve interview skills only 30% 
of SE clients reported this method was helpful16. Also, SE clients were more likely to get a job if SE staff were 
present during interviews10. Thus, SE job interview training methods may be limited, which is a major barrier for 
this group to get a competitive job offer38.
Based on the need to enhance interview skills, NIMH funded the development and evaluation of Virtual Reality 
Job Interview Training (VR) in a series of RCT efficacy studies (R44 MH080496). Briefly, VR overcomes the 
limitations of SE interview training by using behavioral learning principles89, 90 that facilitate sustainable changes 
in behavior39, 40. VR improved interview skills in a lab setting and demonstrated efficacy (increased likelihood of 
attaining a job offer and getting job offers sooner) for four independent groups of adults with SMI17-23. Moving 
forward we will evaluate VR in the community at Thresholds, which is the largest SE provider in Chicago and 
offers a standardized version of SE called Individual Placement and Support (IPS)48. Although IPS strives for 
rapid job placement, it devotes much less attention to job interview training, which is a limitation to IPS given 
that “competitive” jobs require interviews.
Replacing typical IPS staff-led interview role-play training with VR could enhance IPS person-level and system- 
level employment outcomes. Moreover, VR could generalize to better person-level (e.g., social interactions, job 
tenure) and system-level (e.g., fewer hours spent coaching communication at work) employment outcomes, 
which we will explore. The improvement in these outcomes would demonstrate that VR has more than an 
incremental impact on employment outcomes. The effectiveness of IPS interview training has never been 
evaluated, and as such, this study will gain critical knowledge. We will test if VR enhances interview skills 
beyond the effects of IPS, and test if VR contributes to person-level and system-level employment outcomes.
IPS and SE are synonymous within this protocol.
IRB #: STU00202936-MOD0048 Approved by NU IRB for use on or after 6/27/2019

Page 4 of 34
Lastly, prior studies evaluated the implementation of IPS41-45; however, the implementation of enhancements to 
IPS (e.g., VR) have yet to be studied. Thus, we propose to use the Consolidated Framework for Implementation 
Research (CFIR) to guide a process evaluation of VR46, 47. The CFIR organizes implementation into several 
domains, and we will focus on the CFIR’s process domain (i.e., is an intervention being used and delivered as 
designed) to evaluate the 1) agency’s plan to implement VR, 2) scalability of VR, and 3) client, staff, and leader 
views on a) barriers and facilitators to implementing VR, b) whether VR is used as designed, and c) suggested 
refinements. Hence, this study will gain critical knowledge by evaluating the cost effectiveness of VR and how 
VR fits into the system-level workflow, service structure, and billing of Thresholds to provide essential data for 
agencies that might consider adopting VR.
3.0  Inclusion and exclusion criteria:
We will recruit 180 IPS clients at Thresholds diagnosed with a mental disorder with or without psychotic 
features into the randomized controlled trial. Inclusion/exclusion criteria are outlined below:
Inclusion Criteria:
1) Over 18 years old;
2a) all chronic schizophrenia spectrum disorders, including schizophrenia, schizoaffective disorder, delusional 
disorder, brief psychotic disorder, schizophreniform disorder, other specified schizophrenia spectrum and related 
disorder, and unspecified schizophrenia spectrum and related disorders;
2b) all bipolar spectrum disorders beyond bipolar type I and II to include cyclothymia, other specified bipolar 
and related disorder, and unspecified bipolar and related disorders;
2c) all depressive disorders, including disruptive mood dysregulation disorder, major depressive disorder, 
persistent depressive disorder, other specified depressive disorder, and unspecified depressive disorder; 
2d) post traumatic stress disorder
3) fluency in English; 4) at least a 4th grade reading level; 5) enrolled in IPS; 6) active IPS enrollee as indicated 
by at least 1 contacts with his/her Employment Specialist in the past 30 days; 7) currently unemployed or 
underemployed; 8) Planning to go on interviews within 4 weeks of enrolling in the study; 9) willing to be video 
recorded; and 10) willing and able to provide informed consent. We will review each participant’s medical 
record to confirm his/her mental health history.
Exclusion Criteria: 1) documented developmental or learning disability; 2) medical condition that may 
compromise cognition (e.g., Parkinson’s Disease, Alzheimer’s Disease, Huntington’s Chorea, Moderate or 
Greater TBI); 3) uncorrected vision or hearing problems; 4) active suicidal ideation within the last 30 days with 
at least some intent to act, with or without a specific plan; this would be reflected in a score of 4 or 5 on the 
Suicidal Ideation section of the Columbia Suicide Severity Rating Scale (CSSRS); and 5) a past suicide attempt 
within the past 30 days that did not include preparatory acts or behavior, but was defined by a potentially self- 
injurious act committed with at least some wish to die as a result of the act.
We will not include any special populations: adults unable to consent; minors; infants; childrens; teenagers (ages 
17 and younger); or prisoners.
Notably, clients will self-report their intention to go on interviews. We will over-select female clients so that 
they represent ~40% (n= 60) of our sample.
IRB #: STU00202936-MOD0048 Approved by NU IRB for use on or after 6/27/2019

Page 5 of 34
We will recruit 60 IPS clients from the RCT and 20 Thresholds staff, managers, and leaders into the 
implementation evaluation of VR. Inclusion/exclusion criteria are outlined below:
Inclusion Criteria: 1) participated in RCT (IPS clients only); 2) employed by Thresholds as an Employment 
Specialist, Team Leader, or IPS Manager; or 3) employed by Thresholds at the level of Director or higher.
Exclusion Criteria: None.
4.0  Procedures Involved:
A. Study Settings
This study will collect human subjects data at several approved locations throughout the city of Chicago. First, 
subjects will be enrolled and evaluated by research team members at an approved Thresholds location (see 
Appendix A). Then subjects will travel to either the Northwestern University Feinberg School of Medicine (NU) 
campus Department of Psychiatry and Behavioral Sciences’ clinical research program (680 Lake Shore Dr.
Suite 1520, Chicago, IL, 60611) or an approved Thresholds site where the NU and Thresholds research teams 
will collect additional human subjects data. The decision on the location will be flexible due to scheduling of 
participant, actor, and staff availability. Subjects who are randomized to the VR training will complete the 
training at either NU or one of the approved Thresholds locations.
B. Study Design
This is a two arm randomized, single blinded, intent-to-treat, controlled trial (RCT) with baseline and pre-test 
assessments, up to 10 hours of VR training, post-test assessment, and follow-up at 6 months and at 9 months. 
We will evaluate both person-level and system-level outcomes associated with implementing VR at Thresholds. 
At the person-level, we propose to evaluate if IPS+VR, as compared to IPS Only, is associated with better 
employment outcomes (higher employment rate, shorter time-to-employment), job interviewing skills, and 
greater reductions in psychological distress. Interviewing anxiety and self-efficacy will be explored as potential 
mechanisms in the pathway between IPS and employment. At the system-level, we propose to conduct a cost- 
effectiveness analysis and budget impact analysis between IPS+VR and IPS Only as well as explore whether 
Employment Specialists (Thresholds Staff that work one-on-one with clients) working at Thresholds spending 
fewer hours leading interview training is related to spending more hours completing job applications with clients 
and building a network of employers. We will also explore if VR generalizes to social skills at both the person- 
level (via job retention measures) and the system-level (using a time log completed by Employment Specialists 
regarding time spent coaching communication skills at work).
C. Study Intervention
IRB #: STU00202936-MOD0048 Approved by NU IRB for use on or after 6/27/2019

Page 6 of 34
Figure A.
SIMmersion, LLC designed VR to enhance job interview skills for adults with SMI. The job-related interview 
content objectives target: 1) conveying oneself as
dependable, 2) negotiating skills (e.g., asking for a 
day off), 3) team work, and 4) honesty (e.g., 
following company policy). The Interviewee 
performance objectives target: 1) comfort level 
during the interview, 2) sharing information in a 
positive way, 3) sounding interested in the 
position, 4) sounding professional, and 5) 
establishing a rapport with the interviewer.
SIMmersion designed VR with the following 
components to help learning:
C.1.a Electronic Learning (e-learning): Interactive e-learning screens (Figure A.) display critical information 
needed to prepare for a job interview such as creating a resume, researching a position, types of questions to ask, 
advice on how to disclose a disability, and an emphasis on effective skills for interacting with the interviewer 
(e.g., dealing with emotionally provocative questions).
C.1.b VR Interface and “Molly Porter”: After using the e-learning module, trainees will navigate the program to 
begin the simulated interviews with a virtual human resource staff at a department store named “Molly Porter.” 
She is sitting in her office and the trainee has joined her for an interview. When “Molly” speaks, the program 
enters Full Screen mode (Figure B) so that she is speaking directly to the trainee without any distractions on 
screen. When she finishes asking her question, the program returns to the interface in Figure C where trainees 
speak a prescripted response of their choosing (see C.1.d).
“Molly” has three difficulty levels: easy (e.g., friendly), medium (e.g., direct), and hard (e.g., stern or asks 
illegal questions). She uses “memory” and her behavior is driven by an advanced “emotional model.” For
example, if a trainee responds inappropriately to several questions, “Molly” 
may become dismissive and end the interview, but if the trainee continually 
answers appropriately, “Molly” may become more friendly and 
encouraging. Based on the trainee’s statement, the software must then 
choose a reply for “Molly.” Selection is based on three factors: 1) 
difficulty, 2) the history of the conversation, and 3) “Molly’s” evolving 
relationship with the trainee, driven by trainee responses. Each factor will 
be used to compute conditional probabilities associated with each
Figure B.
possible reply and one is selected. “Molly” stays true to her character 
and the emotional state created during the play, and thus, may behave 
differently each time the system is used.
Figure C.
Figure C displays the layout of the interview interface. Buttons at the 
top of the screen control the interface. The yellow text displays various 
statements that can be spoken to respond to “Molly’s” questions. The 
interview transcript can be accessed through a tab in the middle of the
screen. The trainee may change the topic during the conversation using buttons to the left of the yellow text. An 
on-screen coach appears in the bottom right of the interface. The trainees can take notes and review prior 
questions (white and gray boxes).
IRB #: STU00202936-MOD0048 Approved by NU IRB for use on or after 6/27/2019

Page 7 of 34
C.1.c Video Clips: A professional actor portrayed “Molly” during the video-recorded sessions. The videos were 
separated into >1,000 question and response clips that are played during the simulated interview. Trainees talk 
with “Molly” using a microphone and voice recognition software. This method exposes trainees to an interactive 
environment and helps them learn to react to an interviewer’s social cues.
C.1.d Pre-scripted Statements and Voice Recognition: A panel of vocational experts supervised hundreds of 
responses to standard interview questions written by professional scriptwriters during the initial SBIR grant 
award. The scriptwriters worked with the scientific team to compose realistic dialogue that provides trainees 
with opportunities to practice the skills identified in the learning objectives. Trainee statements included a wide 
variety of natural choices with 5-15 potential responses that vary in appropriateness of their content. This 
method allows trainees to choose and learn from their responses. VR uses voice recognition technology so 
trainees can practice speaking the pre-scripted responses to difficult questions in a stress-free environment.
Then, trainees can use the rehearsed answers in real interviews.
C.1.e Non-branching Logic: SIMmersion’s PeopleSIMTM technology uses non-branching logic; which allows 
trainees to behave and speak freely within the confines of a safe simulation. Most social simulations, in contrast, 
use branching logic where trainees select a response from a list of options, which terminate when all options are 
exhausted. This approach minimizes repeated use to a few trials. SIMmersion’s technology integrates the video 
clips and non-branching logic to enable “Molly” to vary her memory, emotion, and personality. This variation 
supports hours of unique repeated practice and naturalistic conversations with Molly.
C.1.f Job Coach and Help Buttons: Trainees receive in-the-moment feedback from an on-screen coach (Figure 
D.) who provides nonverbal cues regarding the trainee’s choice of questions and statements. If further 
clarification is needed, the trainee can click “help” buttons that provide additional detail to clarify the interview 
question or the trainee’s response statement. For example, the coach gives the trainee a “thumbs down” sign if 
an inappropriate response is selected (Figure D.). If the trainee is unclear about the negative feedback, he or she 
can click the help button to get a more detailed written and verbal explanation about why the statement was 
inappropriate (e.g., “This statement focuses on a negative character trait; try focusing on your strengths”).
C.1.g Individualized Customization: Prior to starting VR interviews, trainees complete an online job application, 
which includes questions about employment history and work skills. Trainees will specify one of eight different 
jobs (cashier, stock clerk, customer service, maintenance/grounds, janitorial, food service, inventory, or 
security). This “on-line” application procedure is based on applications currently in use by employers such as 
Target and Home Depot. This practice prepares trainees to accurately complete online applications.
Additionally, the application data will populate the list of questions from which “Molly” will draw.  For 
example, a trainee may apply for a customer service position on the application, yet identify previous experience 
in inventory; Molly may ask, “I see from your resume that you have experience in inventory and are applying 
for a customer service position. Why are you looking to make that change?” This innovative feature allows 
trainees to customize their interview experiences to better prepare themselves for future interviews.
C.1.h VR Transcripts: During or after the interview, trainees can view a transcript 
that replays individual exchanges or the entire conversation. If trainees are using 
the speech recognition feature, the transcript will also replay his or her recorded 
voice. This feature lets the trainee hear (and reflect upon) the tone of voice used 
to ask questions and make statements. The transcripts are color coded to reflect
Figure D. 
Coach
IRB #: STU00202936-MOD0048 Approved by NU IRB for use on or after 6/27/2019

Page 8 of 34
helpful, unhelpful, and neutral responses. Participants can click on the response to receive detailed information 
about why a response was helpful for hurtful to their job interview and learn how “Molly” understood their 
responses.
C.1.i Scoring and Summary Feedback: After each virtual interview, trainees receive scores in eight categories 
(based on the learning objectives). The scores are scaled from 0-100 and are computed via an algorithm, which 
accounts for the types of responses provided by the trainee throughout the course of the interview. Participants 
view this scoring feedback and if they score 80 or higher they are informed, “You got the job! You made a great 
impression on “Molly” and she decided to offer you the job. Congratulations!” (See VR Training Manual in 
Supporting Documents for Screen Shots). Trainees also receive feedback summaries of what they did well and 
where they need improvement. This feedback helps trainees understand the subtleties of their interview skills.
D. Study Procedures for Community Effectiveness Randomized Controlled Trial
D. 1.a. Thresholds research staff will prescreen members using Electronic Health Record (EHR) 
information. Thresholds research staff will sit in on weekly case coordination visits with Employment 
Specialist (ES) to discuss members who may be a good fit for the study. If the member agrees, the ES will 
talk directly with the member to explain the study and see if he/she is interested. The ES may use a 
recruitment video at this time. Once interest is established, Thresholds research staff will reach out to the 
client directly via phone. The Thresholds research staff member will use the study screening form to 
determine the client’s initial eligibility for study participation, and then schedule a baseline visit. 
Screening data collected from individuals who are ineligible or decline participation will be kept on file 
during the duration of the recruitment period in case eligibility criteria changes then this information will 
be destroyed after recruitment is completed.
D.1.b. The Thresholds Employment Specialist identifies members who are eligible for research participation base 
on member’s enrollment and employment status. The Employment Specialist give members’ information about the 
research study using the kickoff meeting recruitment script and kickoff meeting recruitment handout, and lets the 
member know that they may be eligible for the study. The Employment Specialist asks members if they would be 
interested in learning more about the study at a kickoff meeting with the research team. The kickoff meeting will be 
a two-hour period at a Thresholds location in which the interested member will have the option to talk to the 
Thresholds research team. The member will also have the option to talk to the Thresholds’ research team via phone 
instead of attending the kickoff meeting. The Thresholds research team may use the Molly information video 
during the kickoff meeting. Once interest has been established in joining the study, the Thresholds research staff 
member will use the study screening form to determine the client’s initial eligibility for the study participation, and 
then schedule a baseline visit. Screening data collected from the individuals who are ineligible or decline 
participation will be kept on file during the duration of the recruitment period in case eligibility criteria changes. 
Then this information will be destroyed after recruitment is complete.
D. 2. The Baseline Visit for Thresholds’ Clients (completed at Thresholds) will begin with Wide Range 
Achievement Test (4th Edition) to assess current reading level (lasting approximately 10 minutes). If the 
client is found eligible based on reading level (6th grade or above), they will complete the informed consent 
process and provide contact information (lasting approximately 20 minutes). Thereafter, research staff 
will administer:
1. Structured Clinical Interview for DSM-V Axis I Disorders (SCID-5) (lasting approximately 1 hour) 48. The 
SCID will provide duration of illness, number of psychiatric hospitalizations, current medication status, and 
educational attainment. A medication log will be used to capture the medication information.
In the SCID, if a patient endorses any items regarding suicidality (item OP8, "Suicidal ideation lifetime" 
[if within last 30 days], item OP9, "Suicidal ideation past week," item OP13, "Suicide attempt lifetime" 
IRB #: STU00202936-MOD0048 Approved by NU IRB for use on or after 6/27/2019

Page 9 of 34
[if within last 30 days], or item OP14, "Suicide attempt past week"), research personnel will administer 
the Columbia-Suicide Severity Rating Scale (C-SSRS)49.  A total score of 4 or higher on the 
“Intensity of Ideation” section of this measure will indicate whether or not the participant has a level of 
suicidal risk that requires further consideration. Example questions from the C-SSRS are listed below),
Preparatory Acts or Behavior:
Acts or preparation towards imminently making a suicide attempt. This can include 
anything beyond a verbalization or thought, such as assembling a specific method 
(e.g., buying pills, purchasing a gun) or preparing for one’s death by suicide (e.g., 
giving things away, writing a suicide note).
Yes
□
No
□
IRB #: STU00202936-MOD0048 Approved by NU IRB for use on or after 6/27/2019

Page 10 of 
Have you taken any steps towards making a suicide attempt or preparing to kill 
yourself (such as collecting pills, getting a gun, giving valuables away or writing a 
suicide note)?
If yes, describe:
The procedures outlined in section 10 (Risks to Participants) will be followed if a participant scores 4 or 
higher.
2. Participants will receive a 5 minute break.
3. Alcohol Use Disorders Identification Test Self Report (AUDIT-SR) (lasting approximately 5-10 minutes). 
The AUDIT-SR will provide self-reported alcohol use information.
4. Drug Abuse Screening Test 10 item (DAST-10) (lasting approximately 5-10 minutes). The DAST-10 will 
provide self-reported drug use information.
5. Job Interview Self-Efficacy is a 9-item self-report (approximately <5 minutes)
6. Personal Report of Job Interview Anxiety is a 34 item self-report to assess one’s level of anxiety regarding the 
job interview process (approximately 2 minutes)
7. Intrinsic Motivation Inventory – Employment Research is a 30 item self-report (approximately 10 minutes)
8. Technology Comfort Level Questionnaire is a 6-item measure that assesses one's comfort level with computer 
technology (approximately 5 minutes).
9. Research staff will complete the inclusion / exclusion checklist. If participant is eligible to proceed, they will 
be scheduled to return to Thresholds for Baseline Visit 2/3.
The Baseline Visit 1 will last approximately 210 minutes or 3 hours and 30 minutes. Participants will be paid
$70. This amount will be prorated on the number of assessments completed if the session ends early. For 
example, if a participant is excluded based on reading level, he or she will be paid $15.
D. 3. Combined Baseline Visit 2/3 for Thresholds' Clients (completed at Northwestern University or at 
Threshold’s location) will require participants to complete:
1. Employment History Interview (EHI) The EHI will assess competitive work history data: length of time since 
last full-time (or any) employment, number of days employed in the past 5 years, history of prior interview 
training, and number of job interviews completed in the past three months. The EHI will also assess current 
housing status, income, and criminal justice history.
2. Brief Psychiatric Rating Scale (BPRS) (lasting approximately 30 minutes). The BPRS will assess psychiatric 
symptoms including somatic concern, anxiety, emotional withdrawal, conceptual disorganization, guilt feelings, 
tension, mannerisms and posturing, grandiosity, depressive mood, hostility, suspiciousness, hallucinatory 
behavior, motor retardation, uncooperativeness, unusual thought content, blunted affect, excitement, and 
disorientation.
IRB #: STU00202936-MOD0048 Approved by NU IRB for use on or after 6/27/2019

Page 10 of 34
3. The Beck Depression Inventory (approximately 15 minutes) is a self-report of depression symptoms.
A rating of" 2 ("I would like to kill myself," or 3 ("I would kill myself if I had the chance" on item 9 on 
the Beck Depression Inventory indicates presence of suicidal ideation.
When a participant meets one of the above cutoff scores, the research staff will administer the 
Columbia-Suicide Severity Rating Scale (C-SSRS). A total score of 4 or higher on the “Intensity of 
Ideation” section of this measure will indicate whether or not the participant has a level of suicidal risk 
that requires further consideration. The procedures outlined in section 10 (Risks to Participants) will be 
followed if a participant scores 4 or higher.
4. MATRICS Consensus Cognitive Battery (MCCB) (approximately 1 hour) 50. The MCCB is a standardized 
cognitive battery for use with adults with schizophrenia and related disorders.
5. Participants will receive a break and be provided a healthy snack.
6. WHO Quality of Life-BREF (WHOQOL-BREF) (approximately 10 minutes) 51. The WHOQOL-BREF is a 
brief quality of life survey.
7. The 12-item Short Form Health Survey (SF-12) (approximately 10 minutes) 53. This assessment is a brief 
survey about the participant's physical health.
8. Social Skills Performance Assessment (SSPA) (approximately 15 minutes) to assess social competence. 
Participants will complete two video-recorded roleplays with a research assistant 54.
9. Baseline Job Interview Role-Play Assessments (approximately 90 minutes): The participant will complete a 
typical job application, select two fictional job openings (listed below), and participate in two job interview role- 
plays with two trained role-play interviewers. The interviews will be videorecorded and the videos will be 
scored by student raters trained to a high standard of reliability and who are blinded to condition. Participants 
will select two jobs from the following scenarios:
1) Scenario A: The department of Public Health at the University of Illinois at Chicago is running a 
nation-wide study examining the effects of a newly implemented youth exercise program in elementary 
schools across the country. We are hiring a team of research assistants to collect and manage our study 
data. We look forward to building a talented team and have position openings in the following 
categories: Data Entry Technician, Research Assistant, and Study Coordinator. Position to which they 
are applying is Data Entry Technician.
2) Scenario B: Goldberg & Jones is a new law firm opening in downtown Chicago. G&J is a promising 
new law firm founded by two law students who recently graduated from Kent Law School. G&J is 
looking for a dedicated staff to ensure the new law firm’s success. Goldberg & Jones are seeking to hire 
individuals for the following positions: Administrative Assistants, Mail Clerks, and Paralegals.
Position to which they are applying is Mail Clerk.
IRB #: STU00202936-MOD0048 Approved by NU IRB for use on or after 6/27/2019

Page 11 of 34
3) Scenario C: Lakeview Hospital has provided quality care and treatment to Chicago for the past 50 
years. Lakeview Hospital is currently seeking individuals to add to their dedicated team of employees 
for the following positions: Medical Records Clerk and Medical Receptionist. Position to which they 
are applying is Medical Records Clerk.
4) Scenario D: Goldberg & Jones is a new law firm opening in downtown Chicago. G&J is a 
promising law firm founded by two law students who recently graduated from Kent Law School. G&J 
is looking for a dedicated staff to ensure the new law firm’s success. Goldberg & Jones are seeking to 
hire individuals for the following positions: Administrative Assistants, Mail Clerks, and Paralegals. 
Position to which they are applying is Paralegal.
5) Scenario E: Stop & Shop is the second largest, nation-wide supermarket chain. We are opening a 
new store to sell food and household goods. We look forward to building a talented team and have 
position openings in the following categories: Customer Service, Cashier, Stock Clerk and Inventory. 
Position to which they are applying is Inventory Manager.
6) Scenario F: At T&C Home Goods, we are committed to quality - both in the value of our product 
line and in the talent employ. We offer a personalized work environment to hundreds of dedicated 
associates, with all the perks of a multi-million-dollar company. We offer an extensive selection of 
home décor and building supplies. Position to which they are applying is Sales Associate.
7) Scenario G: Stop & Shop is the second largest, nation-wide supermarket chain. We are opening a 
new store in Park Ridge, IL to sell food and household goods. We look forward to building a talented 
team and have position openings in the following categories: Customer Service, Cashier, Stock Clerk 
and Inventory. Position to which they are applying is Stock Clerk.
8) Scenario H: The Cook County Public Library enables researchers, students, teachers, and others to 
have access to the records of Illinois’ past while preserving irreplaceable historical materials for future 
generations. The sizeable collection of books, journals, manuscripts, newspapers, maps, and prints, 
available to the public makes it the most frequented library in all of Illinois. Position to which they are 
applying is Reference Librarian.
Participants will be given an instruction card that corresponds to their selected job scenarios, which is 
presented below.
Instructions: [fill in company name and details] is interviewing for part-time workers, 25 to 30 hours 
per week.  Starting pay is $15.00/hour. You are interviewing for part-time work, particularly because 
you need to have Thursdays off for personal reasons. You will need to negotiate for a schedule that will 
accommodate for Thursdays off.
10. Specific Levels of Functioning (SLOF) Assessment is a 30-item measure that assess one’s functioning in the 
community (approximately 10 minutes) 55.
IRB #: STU00202936-MOD0048 Approved by NU IRB for use on or after 6/27/2019

Page 12 of 34
Combined Visit 2/3 (Pre-test visit) will last approximately 210 minutes or 3 hours and 30 minutes. Participants 
will be paid $70.
D. 4. Randomization for Thresholds’ Clients. After the video-recorded pre-training role-play interviews are 
completed, participants will be randomized into the intervention group or the wait list control group. Block 
randomization tables will be created prior to the study based on computer generated blocks of six. Blocks of six 
will ensure nearly equal groups. Sealed envelopes containing the randomization by sequential subject entry 
number will be created and stored, and then opened when a participant has completed pre-randomization 
procedures.
D. 5. Visits 4-9, Virtual Reality Job Interview Training Delivery. Participants randomized to the VR training 
group will complete their VR training at local Thresholds location (see Appendix A), or if requested by the 
member (at their home). After members have completed their VR-JIT orientation and have demonstrated 
independence at using VR-JIT with minimal support from staff and completed at least 10 virtual interviews in 
lab, then study team will support the member to use VR-JIT at home if home use is requested by the member. 
To provide this support, the research team will show the member how to access the tool in the lab. Moreover, 
the member will be instructed to call the research team so that the research team can walk the member through 
accessing the tool once they are at home. In addition, the research team will provide the member with a 
instructional document so that they can access the tool from home. 
The Thresholds Research Coordinator will schedule the study participants to complete 6 two-hour sessions of 
VR. When participants show up for the VR training, the visit will be run the Thresholds research staff who will 
provide the trainee with an initial orientation to VR and then monitor the trainee’s access and performance 
scores during the sessions. After completing the initial 6 visits, trainees will move on to complete their post-test 
visit. Trainees will be invited to use VR as-needed after completing the post-test visit, which will be monitored 
by the Team Leaders. Those participants assigned to the SE as usual group will have the option to use the VR 
training in the context of their regular use of Thresholds services (i.e., they will not have access to the study 
resources to use VR) following the completion of their participation in the study (after their 9-month follow-up 
phone call). At each training session, any subject who has had to travel to use the intervention will receive $10 
to reimburse them for the cost of transportation.
A note on VR delivery procedures (1/29/18): When the study initially began, Employment Specialists and Team 
Leaders were asked to run the VR orientation and training sessions. Due to workload expectations and 
coordination barriers, this responsibility was moved to the Thresholds Research Team.
D. 6. Visit 10 (Post-test Visit completed at Northwestern University, at a Threshold’s Odgen location, or 
via a phone call or video-conference) will be a repeat of the following measures (see D.2. and D.3. for 
more details):
1.
Brief Psychiatric Rating Scale (BPRS)
2.
Beck Depression Inventory (BDI)
a.
Columbia-Suicide Severity Rating Scale (C-SSRS) if triggered by BDI scores
3.
Job-Interview Role Plays
4.
Social Skills Performance Assessment (SSPA)
5.
Employment History Interview (follow-up version)
6.
Job Interview Self-Efficacy
7.
Personal Report of Job Interview Anxiety
8.
Intrinsic Motivation Inventory
9.
Technology Comfort Level Questionnaire is a 6-item measure that assesses one's comfort level with 
computer technology (approximately 5 minutes).
In addition, VR trainees will complete the 5 item Training Experience Questionnaire (TEQ; < 5 minutes). The 
IRB #: STU00202936-MOD0048 Approved by NU IRB for use on or after 6/27/2019

Page 13 of 34
Posttest Visit 10 will last approximately 150 minutes or 2 hours and 30 minutes. Participants will be paid
$50.
D. 7. Monthly Check-in Telephone Calls for Thresholds Clients (completed via telephone) will involve a 
brief (<10minutes) telephone survey asking about participants’ potential interview or job-related activity 
in the past month. These calls will be scheduled at monthly intervals after completion of post-test until the 6- 
Month Follow-Up (see D. 9.), for a total of 5 calls.
D. 8. Visit 11 (6-Month Follow-Up Visit completed at Northwestern University, a Threshold’s Ogden 
location or via a phone call or video-conference) will be a repeat of the following measures (see D.2. and 
D.3. for more details):
1.
Brief Psychiatric Rating Scale (BPRS)
2.
AUDIT-SR
3.
DAST-10
4.
WHOQOL-BREF 
5.   SF-36
6.
Beck Depression Inventory
a.
Columbia-Suicide Severity Rating Scale (C-SSRS) if triggered by BDI scores
7.
Employment History Interview (follow-up version)
8.
Job Interview Self-Efficacy
9.
Personal Report of Job Interview Anxiety
10. Intrinsic Motivation Inventory
Participants will also complete the SF-12 health questionnaire about their physical health and the Employment 
History Follow-up interview that will assess how many job interviews were completed, total hours worked, how 
many jobs were offered (and accepted), hourly wage, type of job, jobs terminated (and why).
The 6-Month Follow-Up Visit 11 will last approximately 90 minutes or 1 hour and 30 minutes. Participants will 
be paid $40. If the participant completed at least 3 of the 5 calls before the 6-Month Follow-Up visit, they will 
receive a $10 bonus.
Follow-Up Visit 10 and Visit 11 scheduling protocol. 
1.       Study Participants’ 6-month Follow-up visit (V11) will be scheduled 2-4 weeks before their due date. 
Scheduling will depend on the following:
a.       If a participant is engaging in the monthly employment calls, their V11 will be scheduled at the time 
of their 5-month employment call.
b.       If a participant is NOT engaging in the monthly employment calls, staff will contact the participant 4 
weeks before the V11 due date. The contact process will be to use the phone to call the participant up to 
three times per week (only one time per day). Given that this population is difficult to reach, we will 
follow-up this initial phone call with an email and text message (for those participants who identified on 
their consent form that we can contact them by phone or emai.
2.       If the study participant’s V11 has not been scheduled by 2 weeks before their due date, staff will reach out 
to the participant’s employment specialist and primary clinician for support contacting the participant.
3.       If the study participant’s V11 has not been scheduled by the time of their due date, staff will send a letter to 
the participant. The letter will explain the nature of the visit, the length of time the visit will take, and the staff 
person they should contact to schedule.
4.       If the study participant does not respond to the letter or if their V11 is not scheduled by 6 weeks past their 
IRB #: STU00202936-MOD0048 Approved by NU IRB for use on or after 6/27/2019

Page 14 of 34
due-date, staff will pull any available information from the electronic health record, and the member’s study 
participation will end.
The same process will be used for scheduling Visit 10 as well. 
Our primary method to collect Visit 10 and Visit 11 study data will be to invite participants to complete the visits 
at their local Thresholds location or Northwestern (as is the same for all in person study visits). 
Alternatively, if participants are unable to attend visit 10 and/or 11 in person, the study team will schedule a 
series of brief phone calls in order to collect the self-reported study data by phone. In addition, the study team 
will offer participants the opportunity to complete their follow-up role-play assessment by means of a 
videoconference through BlueJeans. The BlueJeans videoconference will then be recorded by the University of 
Michigan research team that is approved for this study and the recordings will be stored using the previously 
approved methods for storing audiovisual data.
Alternatively, if the participants do not want to complete the videoconference then the participants will be offered 
the opportunity to complete their follow-up role-play assessment by means of a teleconference through 
BlueJeans. The BlueJeans teleconference will then be recorded by the University of Michigan research team that 
is approved for this study and the recordings will be stored using the previously approved methods for storing 
audiovisual data.
D. 9. Monthly Check-in Telephone Calls for Thresholds Clients (completed via telephone) will be a repeat 
of procedures of the Monthly Check-in Telephone Calls for Thresholds Clients (see D. 8.). These calls will 
be scheduled at monthly intervals after completion of the 6-month follow-up until the 9-month follow-up (see D. 
11.), for a total of 2 calls.
D. 10. The 9-Month Follow-Up for Thresholds Clients (completed via telephone) will include only the 
Follow-up Employment History interview that will assess how many job interviews were completed, total 
hours worked, how many jobs were offered (and accepted), hourly wage, type of job, jobs terminated 
(and why).
The 9-Month Follow-Up Telephone Call will last approximately 10 minutes. If participants complete at least 
one of the two check-in calls between the 6-Month and 9-Month Follow-Up calls, they will be paid $10 by 
Thresholds after the completion of the 9-Month Follow-Up call.
D. 11. Participation Complete. If a Thresholds client becomes employed prior to the 6-month follow-up call, 
they will still participate in monthly check-in phone calls and the 6-month follow-up. Their participation in the 
study will then be complete after completing the 6 month visit.
If the participant becomes employed between the 6- and 9-month follow-up calls, the participant will be asked 
additional questions on their next check-in call, which will be their last check-in.
For the Thresholds clients who do not find employment during before this time, participation is complete for this 
study after finishing their 9-Month Follow-Up Telephone Call.
E. Study Procedures for a Mixed-Method Multi-Level Process Evaluation. 
IRB #: STU00202936-MOD0048 Approved by NU IRB for use on or after 6/27/2019

Page 15 of 34
E. 1. Level 1. The first level for the process evaluation of VR includes two pre-study implementation 
evaluations. The first is a 12-item Implementation Leadership Scale (<5 min) to evaluate the perspective of 
implementing evidence-based practice. The Thresholds Supported Employment Leadership team will complete 
this brief survey prior to conducting the RCT.
E. 2. Level 2. The second level for the process evaluation of VR will take place after approximately 30 VR 
trainees completed their work with VR and will include a series of 2-4 focus groups (consisting of 4-6 research 
participants per group) with existing research participants using VR. The groups will assess perceptions of initial 
VR implementation, with results guiding immediate modifications to VR delivery. The focus groups will be 
video-recorded and last approximately 60 minutes. Video-recording is required for participation in the focus 
groups. Each participant will be paid $20 for their time. The focus groups will be led by research staff and 
conducted at Thresholds. The participation in focus groups is a one-time experience. See attached list of 
questions guiding the focus group.
In addition, Level 2 will consist of recruiting Team Leaders and Employment Specialists to complete the 26- 
item Implementation Survey for Providers (<10 min).
E. 2. Level 3. The third level for the process evaluation of VR will take place after approximately 60 VR 
trainees completed their work with VR and will include conducting focus groups with Employment Specialists 
and Team Leaders involved with the study to complete 1-3 focus groups (consisting of 4-6 Thresholds Staff). 
The groups will target a discussion around client, service, and implementation outcomes. The focus groups will 
be video-recorded and last approximately 60 minutes. Video-recording is required for participation in the focus 
groups.. The focus groups will be led by research staff and conducted at Thresholds. The participation in focus 
groups is a one-time experience. See attached list of questioning guiding the focus group.
E. 3. Level 4. After 6 months after VR delivery is completed, the Thresholds Supported Employment 
Leadership will be invited to complete the 31-item Annual Survey of Evidence-Based Practice (<15 min). This 
survey will assess the likelihood that Thresholds can sustain their use of VR after the study has ended.
E. 3. Level 5. Thresholds requires Employment Specialists (staff working one on one with clients on vocational 
rehabilitation) and Team Leaders (staff that orient clients to use VR) to document their weekly hourly activity 
logs, which will be copied and provided to the research team for the duration of the Thresholds’ involvement in 
the study. Thresholds will provide the research team with activity logs for the 8 weeks prior to the beginning of 
the study. This data will help us evaluate how the allocation of person hours changes as a result of using VR, 
and in turn, will inform the cost effectiveness of VR. The activity log is attached to the IRB application. The 
logs will be anonymous to the study team, but will included a personal identification number so that the logs for 
the 8 weeks prior to the study can be compared to the logs collected once the study begins. The research team at 
Thresholds and Northwestern will not have access to the personal information of the Thresholds staff connected 
to these logs.
F. Participant Timelines
F. 1. Thresholds Clients. Individuals receiving mental health services at Thresholds who are enrolled in the 
effectiveness trial will participate in the study for a period of 9 months. Over the first month, they will complete 
a series of study measures and if randomized to the intervention group, this participation will include up to 10 
hours of virtual reality training. After the first month, the participants randomized to the training group are
IRB #: STU00202936-MOD0048 Approved by NU IRB for use on or after 6/27/2019

Page 16 of 34
welcome to use the training as-needed, which will be monitored using their ID number. Then approximately 6 
months after participants complete their post-test visit, they will be invited to complete their 6-month follow-up 
visit. Approximately 3 months after the 6-month visit, participants will be contacted via phone for a 9-month 
follow-up interview. Their participation in the study will end at this time.
After 1/3 of the effective trial participants have completed their intervention training (and before the completion 
of the 6 month follow-up) 20 VR trainees will be recruited to participate in a series of focus groups.
5.0 Multiple sites:
This is a multi-site project funded by NIMH where the overall Principal Investigator is Dr. Matthew Smith 
located at University of Michigan with a subcontract to Thresholds to recruit participants, collect data, and 
provide the intervention, and a subcontract to NU to collect additional data as outlined in the procedures and 
manage all coordination activities of the study. After the study data is collected, entered, cleaned, and de- 
identified (except for video data) at Northwestern University, the electronic data will be shared with the overall 
PI on the project. Dr. Smith and his team at UM (his team is To Be Determined) will process and analyze the 
study data at UM.
The Northwestern PI (Dr. Jordan) will be working closely with the Thresholds Site PI who is also the Research 
Director at Thresholds. In order to ensure successful coordination of research activities among sites, the 
Northwestern PI and Thresholds’ Site PI will meet weekly and as-needed during the first 6 months of the study 
(and monthly and as-needed) to prepare and sustain a collaborative and agreed-upon approach. In addition, the 
Northwestern PI and Thresholds’ Site PI will receive input from a community advisory board and an expert 
panel to review and discuss the approach for recruitment and data collection as well as to review and approve 
any modifications to the study protocol regarding recruitment or data collection methods. This panel will not 
have access to individual participant information or data. Recommendations to modify the protocol or consent 
form will be reviewed by the NU IRB and approvals will be immediately communicated via email to the 
Northwestern PI and Thresholds’ Site PI, and both the NU and Thresholds’ research coordinators as well as all 
project staff involved in the consent and data collection processes. As a safeguard, all IRB modifications will be 
reviewed at the next weekly research team meeting. All non-compliance with the research protocol will be 
reported to the Site PI and PI who will discuss and the PI will submit any deviations to the NU IRB for review.
A signed IAA will be uploaded in the supporting documents in IRB modification #4 and before Thresholds staff 
engage in study activities with human subjects.
6.0  Incomplete Disclosure or Deception:
This study will not be using deception.
7.0 Recruitment:
7.1 Research staff located at Thresholds (4423 N. Ravenswood, Chicago, IL 60613) will sit in on weekly case 
coordination visits to educate Thresholds IPS staff about the study and discuss clients who may be a good fit for 
the study. The Employment Specialist will speak with the identified members to explain the study and see if they
IRB #: STU00202936-MOD0048 Approved by NU IRB for use on or after 6/27/2019

Page 20 of 34
are interested. The Employment or Research staff may also show the clients a promotional video about the Molly 
intervention to help them understand the study. If interested, the Thresholds research staff will reach out to the 
client directly via phone. The adults with SMI will complete a phone screener with the Thresholds Research Team 
to determine if they are eligibile for the study. If they are found eligible from the screener, the Research Team 
will schedule their first visit.
To recruit participants to evaluate the process of implementing VR within Thresholds, we will recruit the existing 
study participants using VR as well as Thresholds staff and leadership involved with the project. Each participant 
using VR, staff, and leader involved with the study will receive a study pamphlet summarizing the purpose of 
evaluating the implementation of VR. Individuals interested in completing this aspect of the study will be directed 
by the pamphlet to call the Research Coordinator at Thresholds to begin enrolling in this part of the study.
7.2 The Thresholds Employment Specialist identifies members who are eligible for research participation base on 
member’s enrollment and employment status. The Employment Specialist give members’ information about the 
research study using the kickoff meeting recruitment script and kickoff meeting recruitment handout, and lets the 
member know that they may be eligible for the study. The Employment Specialist asks members if they would be 
interested in learning more about the study at a kickoff meeting with the research team. The kickoff meeting will 
be a two-hour period at a Thresholds location in which the interested member will have the option to talk to the 
Thresholds research team. The member will also have the option to talk to the Thresholds’ research team via 
phone instead of attending the kickoff meeting. The Thresholds research team may use the Molly information 
video during the kickoff meeting. Once interest has been established in joining the study, the Thresholds research 
staff member will use the study screening form to determine the client’s initial eligibility for the study 
participation, and then schedule a baseline visit. Screening data collected from the individuals who are ineligible 
or decline participation will be kept on file during the duration of the recruitment period in case eligibility criteria 
changes. Then this information will be destroyed after recruitment is complete. 
8.0  Consent Process
A. Consent Location. Participants with severe mental illness will be consented for the randomized controlled 
trial and their participation in the focus groups in a private research office located at the participants local 
Thresholds location (one of five locations: 4423 N. Ravenswood Ave, Chicago, IL 60613; 2240 W. Ogden 
Ave, Chicago, IL 60612; 734 W. 47th St., Chicago, IL 60609; 5000 W. Roosevelt, Chicago, IL 60644; 3638 S. 
Kedzie, Chicago, IL 60632). Participants with severe mental illness will be consented for their participation in 
the study during the initial consent process conducted at Thresholds. Thresholds staff and leadership will be 
consented at the above address on Ravenswood in either the research office or the staff or leader’s own personal 
office.
B. Consent Team. The consent process taking place at Thresholds will be led by either the Study PI, Site 
PI, research coordinator, or research assistants who all have up-to-date CITI training.
C. Consent Timeline. The consent process will require approximately 10 minutes to read and up to 15 minutes 
to discuss to make sure that the participants fully understand the extent of their involvement in the study.
The details of the consent process including:
i.
Steps that will be taken to minimize the possibility of coercion or undue influence.
ii.
Steps that will be taken to ensure the participants’ understanding.
D. Consent Addendum for Visit 10 and 11. For participants who are unable to attend Visit 10 and/or Visit 11 in 
person, we are providing these participants with the opportunity to complete abbreviated versions of the visit over 
the telephone or through videoconference (if available). In order to complete this process, the study team will call 
IRB #: STU00202936-MOD0048 Approved by NU IRB for use on or after 6/27/2019

Page 20 of 34
the participants and read the consent addendum to them. When the participant provides their verbal consent (or 
not) then the study team will document the participant’s decision and ID number on a hard copy of the 
addendum. Then the hard copy will be added to the participant’s file. If the participant agrees to the consent 
addendum, then the study team will work with the participant to set up a time to complete the visit using the 
preferred method of communication.
9.0 Process to Document Consent: 
Procedure for Consent

The person obtaining informed consent will:
o
Introduce themselves to the potential subject
o
Provide an explanation of a) what clinical research is and why you are approaching them; b) why 
people generally volunteer to participate in research; c) what they can expect if they choose to 
participate in clinical research; d) how participating in the study will not affect their regular care or 
relationships with their care providers.
o
Explain that research staff will look into their medical records to confirm their past/current mental 
health diagnosis and that HIPPA Authorization is being obtained.

After explaining the principles above, the person obtaining informed consent will:
o
Ask the potential subject if they want to hear the details about the particular research and answer 
any questions that may arise.
o
Acknowledge that the potential subject verbalized understanding of the research and research 
related procedures.
o
Explain to the potential subject that signing ICF is entirely voluntary.
o
Explain to the potential subject risks and benefits of participating in the particular protocol.

The potential subject, or legally authorized representative, will;
o
be given ample time to completely read and/or listen to the consent form being read to them and ask 
any questions.

The subject and the person obtaining informed consent will sign and date the last page of the consent form if 
they wish to participate in the study.

A witness signature will be obtained in order to be in compliance with Illinois law pertaining to the 
utilization of mental health information from a HIPAA covered record.

A copy of the fully signed consent form will be given to the subject.

Copies will be filed in the subject’s research file.

The ICF process will be documented in the study source document by the person who is obtaining the 
consent.

If during the course of the research trial, the informed consent form is revised, subjects will be re-consented 
using the revised IRB approved consent form, if applicable subjects will be re-consented after each IRB 
Continual Review with a newly IRB approved ICF even though it does not reflect any changes in the study 
protocol.

Those who were consented before the study was determined to meet the definition of an NIH-funded 
clinical trial will be given a document at their next study visit that contains information about what this 
means and includes the clinicaltrials.gov identification number.
10.0  Risks to Participants:
Potential Risks and Protections in Place Against Foreseeable Risks. The potential risks of this project are 
minimal. The possibility exists that participants may become bored or fatigued during: 1) pre-test and post-test 
role-plays; 2) VR training; 3) cognitive assessments; 4) vocational history interview; or 5) mental health history 
IRB #: STU00202936-MOD0048 Approved by NU IRB for use on or after 6/27/2019

Page 20 of 34
interview. In response, we will include scheduled and unscheduled breaks to protect against these risks. To 
minimize the risk of possible fatigue and/or boredom, breaks will be scheduled between assessments in long 
sessions and between role-plays, and participants will be reminded that they can request a break at any time 
during. Should a research staff member notice signs of fatigue and/or boredom from a participant, an 
unscheduled break will be taken. The research team has used these procedures for other studies, and they have 
been effective at reducing risks of fatigue and/or boredom for study participants.
There is a very low risk that participants could be distressed by the mental health interview or job interview role 
plays. If this happens, participants will be asked to take a break and be offered the chance to drop out of the 
study if these aspects of the study are too distressing for them. Of the more than 150 participants who completed 
the mental health interview and four role-plays each during the efficacy studies (600 total role-plays), no 
participants reported feeling distressed by the interview or role-plays.
These risks will be reviewed during the discussion regarding informed consent, and researchers will take all 
possible precautions to minimize these potential risks. Moreover, only personnel trained in interviewing and 
testing protocols with individuals 18 year and older will conduct these procedures.
Although there are no known adverse or severe adverse events that have occurred as a result of using the 
training, it is possible that a participant could arrive to a research visit (or escalate during a research visit) in a 
way that is symptomatic or suicidal during the course of the study. Per Thresholds and NU policies, both sites 
will have clinical staff available (including the PI, Dr. Smith) to assess the mental health and the risk of harm 
to self or others. If trained research staff observe a participant to be highly symptomatic or suicidal then they 
will follow the following policies at NU and Thresholds:
If SCID-V has been administered:

If a participant endorses suicidal ideation and behavior within past 30 days by any of the following items:
o
item OP8, "Suicidal ideation lifetime" [if within last 30 days],
o
item OP9, "Suicidal ideation past week,"
o
item OP13, "Suicide attempt lifetime" [if within last 30 days], OR
o
item OP14, "Suicide attempt past week"),
proceed to administer the Columbia Suicide Severity Rating Scale (C-SSRS) as instructed below.

If participant does not meet any of the criteria above on the SCID-V, continue with study procedures per 
protocol.
If Beck Depression Inventory has been completed:

For item 9 of the Beck Depression Inventory (self-report), if a participant endorses a rating a rating of 2 "I 
would like to kill myself," or 3 "I would kill myself if I had the chance," proceed to administer the Columbia 
Suicide Severity Rating Scale (C-SSRS) as instructed below.

If participant does not meet the cutoff score on item 9 of the BDI, continue with study procedures per 
protocol.
If participant endorses distress during other procedures (i.e., during job-interview roleplays):

Ask the participant if he/she would like to take a break and if the participant has any concerns about 
completing the [i.e., roleplay, survey].

If participant indicates willingness to proceed, continue with study procedures per protocol.
o
Participant will be given option to reschedule remaining study procedures for another date, if 
preferred by participant.

If participant indicates distress, proceed to administer Brief Psychiatric Rating Scale (BPRS) and/or 
IRB #: STU00202936-MOD0048 Approved by NU IRB for use on or after 6/27/2019

Page 20 of 34
Columbia Suicide Severity Rating Scale (C-SSRS) as appropriate, as instructed below.
If suicidal ideation is detected (based on Beck Depression Inventory item 9 cut-off score of 2 or 3, OR other 
indication):

Administer the Columbia Suicide Severity Rating Scale (C-SSRS). If participant scores 4 or higher on the 
"intensity of Ideation" section, proceed with emergency procedures.
o
If participant scores below 4, continue with study procedures per protocol and notify primary 
Thresholds research staff point of contact (i.e., Nikki Pashka, Karley Nelson) of C-SSRS outcome. 
Threshold’s research staff should follow up with participant’s clinical team.
Emergency procedures

NU staff should alert the Research Manager and primary Thresholds research contact about the concern 
(provide with demographics and relevant medical information such as suicidal and/or homicidal 
ideation/intent, diagnosis, medical condition).

Thresholds research staff should contact participant’s clinical team and obtain recommended next steps (i.e. 
go immediately to ED, wait for clinical staff to arrive, or schedule an appointment for the participant with 
clinical provider).

Pending recommendation from Thresholds clinical provider (if recommended to make appointment with 
provider or go immediately to Emergency Department), discuss the issue with all relevant research staff 
and participant.

If recommended to make appointment with provider, ensure that appointment is secure promptly and 
that date, time, location, and name of provider appointment are documented before participant leaves the 
research premises.

If recommended to go immediately to ED:
o
Call 911.
o
In the case of an involuntary hospitalization/ED evaluation: call Police; a research team member 
should go with police since “Petition” is required.
o
In the case of a voluntary hospitalization/ED evaluation: call Ambulance to escort the patient.
o
In the case of suicidal ideation/intent, a research team member must watch (i.e., must have eyes 
on) the participant until Police/Security/Ambulance arrives.

If participant is going to ED from Northwestern Location (680 N Lake Shore Drive):
o
Call the building security at 312-951-1844 and inform them that Police and/or Ambulance was 
contacted. The security officer from the building will come to NU-CRP facility, if needed. Security 
should arrive to facilitate within 10 minutes and help the research personnel until Police and/or 
Ambulance arrive.
o
A research team member must provide documentation (i.e., progress note) from the investigator to 
give to the police/security regarding the investigator recommendation.
o
Call ED (direct crisis line is 312.926.1878) to alert that patient is on the way and to provide with all 
necessary information.
o
Follow up with crisis team/on call resident to give information about the patient.
o
Email Principal Investigators and Thresholds Site PI to notify them that the emergency procedures 
took place.
o
Contact Thresholds research team member so that they can reach out to the participant’s clinical 
team for follow up.
o
Coordinate with Thresholds research staff contact to follow-up with participant within 24 hours to 
verify they were able to access the help they needed, or follow-up with ED to verify if participant 
IRB #: STU00202936-MOD0048 Approved by NU IRB for use on or after 6/27/2019

Page 20 of 34
was admitted to hospital.
o
Document all activities for the study file and for correspondence with clinicians/IRB.

If at Threshold’s Ogden location (2240 W. Ogden Ave):
o
Inform Nijha Maybon (773-572-5236) and/or Repsie Royster (773-572-5218) that the police and/or 
an ambulance are on the way. Ensure that they know where you are in the building so that they can 
escort emergency personnel to you when they arrive.
o
A research team member must provide documentation (i.e., progress note) from the investigator to 
give to the police/security regarding the investigator recommendation.
o
Document which hospital participant is being taken to then contact that hospital to provide the ED 
staff which all necessary information.
o
Follow up with crisis team/on call resident at hospital to give information about the participant
o
Email Principal Investigators and Thresholds Site PI to notify them that emergency procedures 
took place.
o
Follow up with participant’s clinical team to let them know what emergency procedures took place.
o
Follow-up with participant within 24 hours to verify they were able to access the help they needed, 
or follow-up with ED to verify if participant was admitted to hospital.
o
Document all activities for the study file and for correspondence with clinicians/IRB.

If at other Thresholds Location
o
Inform relevant building staff/front desk staff that the police and/or an ambulance are on the way. 
Ensure that they know where you are in the building so that they can escort emergency personnel to 
you when they arrive.
o
A research team member must provide documentation (i.e., progress note) from the investigator to 
give to the police/emergency personnel regarding the investigator recommendation.
o
Document which hospital participant is being taken to then contact that hospital to provide the ED 
staff which all necessary information.
o
Follow up with crisis team/on call resident at hospital to give information about the participant
o
Email Principal Investigators and Thresholds Site PI to notify them that emergency procedures took 
place.
o
Follow up with participant’s clinical team to let them know what emergency procedures took place.
o
Follow-up with participant within 24 hours to verify they were able to access the help they needed, 
or follow-up with ED to verify if participant was admitted to hospital.
o
Document all activities for the study file and for correspondence with clinicians/IRB.
Withdrawal of Participants
The above noted cases regarding suicidality and symptom exacerbation are the anticipated situations when 
the participant will be withdrawn without their consent.
Participants withdrawn from research will:

not be called to participate in future research visits for the current study;

have their data included in all analyses unless there is a formal, written request by the participant to have 
their data removed from analyses (as noted in the consent form).
11.0  Potential Benefits to Participants:
Participants randomized to the VR training group may experience an increased likelihood of receiving a job 
offer. This effect was observed in five efficacy studies. Across those 5 efficacy studies, approximately 50% of 
IRB #: STU00202936-MOD0048 Approved by NU IRB for use on or after 6/27/2019

Page 20 of 34
VR trainees received a job offer compared to approximately 25% of study participants who received services-as- 
usual. Participants randomized to the control group are not likely to receive a direct benefit from participating in 
this study.
Participants completing the focus groups or semi-structured interviews for the implementation study are not 
likely to receive a direct benefit from participating in this study.
12.0  Financial Compensation:
All participants will receive compensation per visit where research assessments are collected. Potential 
transportation costs are included in the planned compensation payments. If a visit is not completed in full, the 
participant will be paid for the portions completed. These payments are approximately $70 for Baseline Visit 1 
and Pretest Visit 2/3; $50 for Posttest Visit 10; and $40 for 6-Month Follow-up Visit. If a participant completes 
3 out of the 5 scheduled montly calls before the 6-Month Follow-up Visit, he/she will receive an additional $10.
IRB #: STU00202936-MOD0048 Approved by NU IRB for use on or after 6/27/2019

Page 21 of 34
If the participant completes the two additional check-in calls before the 9-month follow-up telephone call, 
he/she will receive an additional bonus of $10 after the completion of the 9-month call, Thus, the total amount 
that can be received for the study is approximately $250 per participant for completing research assessments. 
Study participants will not be paid for completing research visits 4, 5, 6, 7, 8, and 9 because they are receiving a 
free intervention.
13.0 Provisions to Protect the Privacy Interests of Participants:
All methods of research data collection (interviews, self-report, role-plays) will be conducted in a private office. 
All methods of correspondence will be secure, including telephone, voicemails, text messages, or email. 
Participants will provide informed consent with respect to their preferred method of communication. In addition, 
will will use a personal identification number (PIN) to protect participant privacy by labeling all study documents 
with the PIN rather than any identifying information. The use of the PIN will help keep participant enrollment 
confidential.
Access to protected information will be limited to the Northwestern PI, Thresholds’ Site PI, NU research team 
members, and the Thresholds’ research team members. There will be a document located on the NU servers 
(password protected behind firewalls) that links PINs to study participant names. The PI, Site PI, and research 
teams will have access to this file in order to coordinate study visits and track participant progress to complete the 
study.
14.0  Confidentiality and Data Management:
Description of Security measure to Protect Data Sources. Paper research records will be created at Thresholds, 
and labeled with the PIN, where participants will be consented and complete baseline measures. These records 
will be stored in a locked cabinet within a locked office. These records will serve as the source for electronic 
data entry. The first entry of the data will occur at Thresholds. Upon completion of data entry at Thresholds, the 
paper research records will be transferred via a locked box to the NU research team where the data will be 
cleaned, re-entered into the electronic database, and stored in a locked drawer in a locked room. Paper research 
records will also be created at NU to cover role-play and research staff for the role-play and self-report data 
collected at NU. The NU and Thresholds records will be merged prior to long term storage. All records will be 
de-identified after study completion. All paper research data data will be stored on password protected servers 
for a period of 7 years.
Video recordings will stored for primary access on Northwestern University password protected servers behind 
firewalls. The video recordings will be stored using the study PIN as a filename: IDXXXX_video_a.xxx. The 
file name designation will also be randomized from a,b,c,d, e,f to maintain blinding as to whether the video was 
pre-test, post-test, or follow-up.
All participant documents at Thresholds and NU will be labeled with the participant’s PIN. Each page of each 
document will be labeled with the PIN. Documents will be organized into a study binder that is also labeled with 
the participant PIN. All Binders and Study Documents will be stored in locked file drawers within a locked 
room.
As an adjunct faculty at Northwestern University, overall PI Dr. Smith still maintains his NU netid and 
password. He will be able to access REDCap to download the de-identified research data in order to review the
IRB #: STU00202936-MOD0048 Approved by NU IRB for use on or after 6/27/2019

Page 22 of 34
integrity of the data. The audio/video data that is collected will be downloaded from the video camera to a NU 
workstation where it will be uploaded to the server. Due to his adjunct appointment at NU, Dr. Smith will still 
have access to the NU server and can simply move the file from the NU server to a server at UM. Both servers 
will be password protected with firewall protection.
Quality Assurance Measures. All participants will be recruited using strategies, documents, and text approved 
by the Northwestern University Feinberg School of Medicine Institutional Review Board (IRB). The research 
team will regularly hold meetings to discuss the effectiveness of approved recruitment strategies and if new 
strategies should be reviewed by the IRB and then implemented. In addition, randomly selected measures will 
be selected for audio-taped review. Only those subjects who have given written consent to have measures audio- 
taped for quality control purposes will be part of the random audio-taped reviews. The study coordinator will 
screen participants for eligibility using formal study forms. The PI and Site PI will regularly audit accrual to 
ensure that participants meet eligibility criteria and that the study enrollment is consistent with the projected 
enrollment targets agreed upon with NIMH. In addition, the study coordinator will audit all study files to ensure 
that all required study data is completed on each form, and that there is no missing data. Data will be double 
entered on electronic REDCap data entry forms to maintain the validity and integrity of the collected data. The 
first entry will take place at Thresholds by subcontracted Thresholds staff (for Thresholds-based visits) or at NU 
by NU project staff (for NU-based visits), while the second entry will take place at NU by NU project staff (for 
all visits). The REDCap system is accessible via a virtual private network that is password protected and behind 
firewalls.The data will be transported from Threholds to NU in a locked storage box. To protect confidentiality, 
all data will be numerically coded using a personal identification number (PIN), and information linking the PIN 
to the subject’s name will be kept in a secured file cabinet and office. All audio and video files will be stored 
electronically on password-protected computers and communication among the staff will use PINs, not names. 
No information concerning data will be presented with participant names. The biostatistician and project 
coordinator will also perform all necessary checks and controls to ensure the reliability and validity of the data, 
including monitoring data collection and collection procedures, data storage, data management, and data 
analysis.
Data Analysis Plan for Aims 1 & 2. In this intent-to-treat study, we will collect data on 160 subjects. The 
biostatistician will monitor the data for distribution normality and transform abnormally distributed variables if 
needed. Correction for Multiple Comparisons: To minimize spurious findings, we will adjust the 3 primary 
tests using a false discovery rate at 0.0557 (see below). Covariates: Although randomization should protect against 
group differences, we will covary for any between-group differences that are observed regarding demographic, 
cognitive, clinical and vocational history, booster sessions, and the number of completed interviews. Missing 
Data: We will follow the intent-to-treat principle58 and in longitudinal analyses we will use multiple imputation 
models to analyze the data in the presence of missing data59-62. Data processing and cleaning will occur during the 
study, including checking for missing data and inconsistencies. All videos will be stored on the project’s secure 
server. A Data Safety and Monitoring Board will monitor for adverse events (see 15.0 below).
Hypothesis 1: IPS+VR trainees will have higher employment rates than IPS Only by T3. To test H1, we will 
use multiple logistic regression and a Wald Chi-square test of the coefficient to its standard error, to compare the 
adjusted employment proportions in the two conditions (attained a job=1 vs. failed to attain a job or censored with 
a job=0, between T1 and T3). We calculated power using simulation in R, correcting for small samples and 
multiple testing with false discovery rate for the 4 primary tests (our power analysis is conservative, relying on 
Bonferroni corrections for all tests). Accounting for 10% missing, our trial that recruits 80 subjects in each arm 
has 80% power for a two-sided 0.05/4=.0125 level test when the OR = 3.2, with IPS+VR effectively doubling the
IRB #: STU00202936-MOD0048 Approved by NU IRB for use on or after 6/27/2019

Page 23 of 34
rate from 25% to 53% employment. Based on our earlier study where we found an OR of 8.7, or effectively 
tripling the employment rate from 25% to 75%, and a power of 0.99 at this magnitude of effect, we feel confident 
that we will have sufficient power in this study.
Hypothesis 2: IPS+VR trainees will have greater improvement in job interview skills than IPS Only by T2. 
To test H2, we will conduct a repeated measures analysis of variance (RM-ANOVA) with pre and post interview 
scores as the repeated measures and treatment group as the fixed factor. Based on our pilot data, we expect r=.7 
between T1 and T2 scores, and an effect size of d=.67 between pre and post interview role-play scores using VR. 
Assuming a small-to-moderate effect within IPS Only (e.g., d=.25), our best power estimate assumes a medium 
effect size contrasting IPS Only with IPS+VR (d=.67-.25=.42). This effect size and correlation imply an expected 
6.9% reduction in within subject error for the treatment–x-time effect. Thus, 80 subjects at .0125 level will yield 
82% power to detect this effect. For the ANOVA model, 160 subjects will be sufficient to detect a reduction of 
within subject error of 4.7% with 80% power. Assuming a full sample effect size of d=.46 (.67+.25)/2), this 
reduction in error corresponds to an effect size of d=.62 for the IPS+VR group and d=.30 in the IPS only group (a 
difference of d=.32).
Hypothesis 3: IPS+VR will get jobs sooner than IPS only by T3. To test H3, we will use a Cox proportional 
(or non-proportional) hazards regression model63 to assess the adjusted hazard rate on time-to-employment for 
IPS+VR vs. IPS Only, adjusting for potential covariates (e.g., completed training trials). For a power analysis for 
H3, we assume a constant hazard rate for the distribution of time-to-employment in both groups. We also assume 
that a proportion of 0.20 of the subjects in the IPS Only group will find a job by T3 (that is, a hazard rate of 0.0085 
for IPS Only). This analysis is estimated based on limited pilot data and log-rank power tables64. Thus, 160 
subjects provide 80% power for a 0.0125-level test comparing time-to-employment in the two arms to detect a 
hazard ratio of 2.0 for IPS+VR vs. IPS Only.
Hypothesis 4: IPS+VR trainees will have greater reductions in psychological distress than IPS Only between 
T1 and T3. To test H4, groups will be contrasted on change via RM-ANOVA with time as a repeated measure 
and group as the main effect. Power analyses for RM-ANOVA focused on contrasts between T1 and T3. A 
correlation of r=.5 between T1 and T3 was assumed. Power was estimated for a range of relative effect sizes (d=.2 
to d=.8; difference between IPS Only and IPS+VR) for RM-ANOVA time-by-treatment interaction effects. 
Assuming an overall reduction in distress of d=.5 (across both groups), all relative effect sizes were centered on 
this average (e.g., a relative difference in d=.8, corresponds to d=.9 for IPS+VR and d=.1 for IPS, averaging out 
to d=.5 overall). For ANOVA change contrasts of IPS only vs. IPS+VR between T1 and T3, 160 subjects will be 
sufficient to detect d=.50 (d=.75 vs .25) with 80% power for 0.0125-level test. ANCOVA will test for differences 
in T3 distress while covarying for T1 distress. Baseline differences (e.g., cognition) will be tested as covariates.
Hypothesis 5: IPS+VR will be more cost-effective than IPS Only. To test H5, we will conduct a cost- 
effectiveness analysis (CEA) to assess the short-term cost-effectiveness of IPS+VR relative to IPS Only using a 
societal perspective, which includes intervention costs and client costs provided by the IPS program director65. 
Intervention costs include variable costs (e.g., time spent by ECRs) and fixed costs (e.g., costs supporting hiring, 
training, and coordination). Client costs will include time using VR and travel costs. We will use standard 
approaches to identifying and assigning unit costs for each cost component65. We will use job attainment rate as 
our measure of effectiveness. The main step is to calculate an incremental cost-effectiveness ratio (ICER), which 
is defined as the difference in total costs between the IPS+VR and IPS Only groups, divided by the between-group 
difference in job attainment rate. Confidence intervals will be calculated around the ICER using bootstrapping 
and Fieller’s theorem66, 67. We will conduct sensitivity analysis by deriving cost-effectiveness acceptability curves 
that display the probability of IPS+VR being cost-effective at various threshold values68.
IRB #: STU00202936-MOD0048 Approved by NU IRB for use on or after 6/27/2019

Page 24 of 34
Exploratory (Person-Level): To study the generalizability of VR to social skills, we will evaluate if more 
VR training (i.e., number of trials) is associated with a longer job tenure (number of weeks) and enhanced social 
skills using linear regression; covariates will include demographic, cognitive, vocational, and training 
characteristics that differentiate groups. Exploratory (System-Level): We will explore if, as a result of VR, ESPs 
reallocate their interviewing hours towards building larger employer networks, completing more job applications 
with clients, and engaging in more hours of coaching work communication with clients.
Hypothesis 6: Improved interviewing skills will mediate the relationship between VR (number of completed 
VR trials and employment outcomes (obtaining employment). To test H6, we will test first for a significant 
IPS+VR impact on role-play scores compared to IPS alone, then check for treatment by mediator interaction69, 
then on the product of the two coefficients70 with bootstrapped confidence intervals71. We simulated power for 
this text, finding 80% power when the effect size for skills is small (effect size=0.2) and the odds for skills leading 
to a job is 1.8; thus we expect to have sufficient power. These older mediational models are informative but 
incomplete, and these will be followed by computing the causally interpretable average natural indirect effect72 
under assumptions i – iv therein. We plan to conduct sensitivity analyses to the assumption of “no exposure 
induced mediation-outcome confounding”. For the exploratory moderated mediation model, we plan to use 
Approach 1 of Vanderweele et al.72 to account for the combined mediated (indirect) effect of both the 
psychological changes in self-efficacy, anxiety, and role-play scores, as suggested in Corbiere et al.7.
Exploratory: To test the moderated mediation (mediated effect varies as a function of a baseline variable), 
we will first conduct a standard model of post-intervention anxiety/self-efficacy serving as a mediator of the 
relationship between VR and interviewing skills using the “product of coefficients method”73. Mediation tests will 
be based on whether the confidence interval for the product of coefficients includes zero. This approach addresses 
the known non-normality of the test statistic73. To this standard mediation model we will include an interaction of 
baseline (pre-intervention) anxiety/self-efficacy to interact with VR training on post-intervention anxiety/self- 
efficacy. Similarly, we will include a baseline-by-intervention interaction on interviewing skills. The Johnson- 
Neyman approach will assess where mediation occurs as a function of baseline levels74. We will test if employment 
outcomes mediate the relationship between interviewing skills and psychological distress at 6-month follow-up, 
using the same approach outlined for H6.
Data Analysis Plan for Aim 3
Overview: We will recruit three levels of subjects for the VR process evaluation. ECRs will recruit IPS clients, 
while the PI, Site PI, and Thresholds IPS Director will recruit multidisciplinary staff (e.g., ECRs), and leaders 
(e.g., CEO). We will use both quantitative and qualitative methods of data collection and analysis.
Study Procedures, Measures, and Data Analysis
The mixed-method process evaluation will entail conducting focus groups and semi-structured interviews:
Focus groups will be conducted with clients to assess: barriers (e.g., using computers) and facilitators (e.g., 
accessibility) to the implementation (i.e., delivery) of VR. The focus group with clients will assess perceptions 
of initial VR implementation, with results guiding immediate modifications to VR delivery. The focus groups 
with Team Leaders and Employment Specialists will assess perceptions of changes made to VR delivery and 
elicit new barriers/facilitators (e.g., changes in VR accessibility), with results guiding new modifications to VR 
delivery. We will track which subjects receive a modified delivery of VR, to be evaluated as a possible 
covariate. In total, 2-4 video-recorded focus groups (60 minutes, 4-6 participants each) will be conducted until 
reaching saturation (i.e., no new topics emerge)75.
IRB #: STU00202936-MOD0048 Approved by NU IRB for use on or after 6/27/2019

Page 25 of 34
Qualitative Analysis: transcribe focus group data, analyze data iteratively using thematic analysis and the 
constant comparative approach76, 77 to identify emergent themes regarding the barriers and facilitators of 
implementing VR. Two research staff will analyze the data using The Ethnograph, a qualitative data analysis 
package. Staff will independently analyze a subset of transcripts to iteratively develop codes inductively as they 
emerge, and deductively based on initial topics (e.g., barriers, available resources). After the team agrees on a 
final codebook and inter-coder reliability is achieved, the codes will be applied to all transcripts78, 79. We will 
use framework analysis to compare client, staff, and leaders’ perceptions of barriers and facilitators to VR 
implementation80. To facilitate comparison, a matrix of themes will be developed: subject type (x-axis) vs. 
barriers and facilitators (y-axis). Matrices will identify y-axis themes common to all groups, and features 
specific to particular subgroups81. For instance, the experience of implementing VR for clients may be related 
to organizational themes not evident among staff or leaders.
Overall, Aim 3 will optimize VR delivery and inform VR scalability, sustainability, and generalizability.
Budget impact analysis (BIA) will assess the costs of implementing VR. BIA will yield two critical data: (1) an 
estimate of the cost of implementing VR at Thresholds, and (2) a spreadsheet model that other community 
mental health agencies can use to input site-specific parameters in order for them to estimate the costs of 
implementing VR. Following current best practices for BIA82, we will use the perspective of the implementing 
organization and enumerate costs to Thresholds of implementing VR. Beyond the VR costs mentioned in the 
CEA in H5, we will track time spent by Team Leaders and Employment Specialists training, time spent by staff 
maintaining the VR hardware, and software costs. All project staff will be provided an Excel-based template to 
record time spent on each VR-related activity83. Total costs for each arm will be aggregated and compared. 
Sensitivity analysis will be used to vary different cost component input values (e.g., number of trials per study 
participant) to determine the range of estimated total costs for each arm.
15.0 Data Monitoring Plan to Ensure the Safety of Participants:
We will use a protocol-specific Data Safety Monitoring Board (DSMB). The operation of this board will adhere 
to the guidelines for DSMBs outlined by the NIH and the U.S. Department of Human Services. The specific 
aspects of the DSMB for this study are as follows:
1.
The DSMB will consist of 3 members: a biostatistician (Borko Jovanovic, Ph.D., Department of 
Preventive Medicine), who will serve as DSMB Chair, a physician (Cindy Nowinsky, M.D., Ph.D., 
Department of Neurology), and a behavioral scientist (Benjamin Schalet, Ph.D., Department of Medical 
Social Sciences).
2.
The DSMB will meet twice per year in Year 1 and annually in Years 2-5 to review study data 
concerning recruitment, randomization, retention, compliance, form completion, gender and minority 
inclusion, intervention effects, and safety. In addition, the DSMB will: 1) identify specific safety 
concerns for participants and communicate these to the study PI; 2) consider the need for additional data 
concerning participant safety; 3) consider the rationale for the continuation of the study; 4) provide a 
written report concerning the protocol to the IRB and to the study PI; and 5) review manuscripts 
reporting study results prior to submission.
3.
Each meeting will consist of three parts. First, an open session will occur in which Dr. Smith and the 
DSMB will review the conduct of the trial (e.g., accrual, protocol compliance). Next, to maintain the
IRB #: STU00202936-MOD0048 Approved by NU IRB for use on or after 6/27/2019

Page 26 of 34
blind of the study, a closed session involving only the DSMB and the statistician will be held during 
which the statistician will present preliminary study data and any reported adverse events or serious 
adverse events. Lastly, an executive session involving only DSMB members will be held to allow the 
DSMB the opportunity to discuss the conduct of the trial and outcomes, including adverse events, 
develop recommendations, and take votes as needed.
4.
The DSMB written recommendations will be provided to the NU PI and to the IRB. The DSMB will 
summarize AE reports for the PI and the IRB Chair, and the PI must implement any DSMB 
recommendations expeditiously. All DSMB recommendations will also be forwarded to the NIH and the 
FDA (where necessary).
16.0 Qualifications to Conduct Research and Resources Available: 
Thresholds Inc.
Founded in 1959, Thresholds serves more than 6,700 people with psychiatric disabilities annually through 25 
agency programs at more than 70 sites (including 60+ staffed residences) throughout Chicago and surrounding 
suburbs. Thresholds works with individuals experiencing most severe psychiatric disabilities, namely 
schizophrenia, bipolar disorder and major depression; these disabilities affect between eight to 15 million 
Americans annually. Individuals with psychiatric disabilities at Thresholds also comprise are a diverse population 
of people who enter into the system of care not only affected by mental health issues and chronic poverty, but 
typically with co-occurring physical health conditions and disabilities including substance abuse, diabetes, 
respiratory illnesses related to smoking, and other advancing health problems. Many of individuals also live with 
co-occurring disabilities (e.g., people who are deaf or hard of hearing), are individuals with psychiatric disabilities 
who are exiting inpatient psychiatric facilities, jails and prisons, or other institutional settings, experience chronic 
homelessness, are youth and or Wards of State, pregnant and parenting teens, and veterans. Thresholds takes pride 
in serving some of society’s highest-need, highest-barrier populations that other service providers are ill-equipped 
to serve. The agency has received multiple awards, including those from the American Psychiatric Association, 
the Substance Abuse and Mental Health Services Administration (SAMHSA), Psychiatric Rehabilitation 
Association, the National Alliance on Mental Illness (NAMI), and the National Association for Business 
Resources, among others.
Based on the diverse and complex needs and challenges that face individuals with psychiatric disabilities, 
Thresholds supports comprehensive and adaptable approaches to mental and physical health recovery and 
wellness to ensure each individual is on a positive path towards self-sufficiency and community participation. In 
order to meet the needs of individuals with psychiatric disabilities, their families, and other stakeholders, 
Thresholds supports multi-layered, integrated systems of care, including specialized programs for individuals with 
other disabilities, e.g., are individuals who are deaf. Within the Research Department and Evaluation Department, 
Thresholds has been a partner in developing and evaluating model programs with academic centers (e.g., UIC, 
Northwestern University, the University of Chicago, Dartmouth College, Illinois Institute of Technology, and 
Boston University), among others. These programs include the implementation of recognized evidence-based 
practices (EBPs) in PSR, as well as other demonstration programs specifically designed, tested, and supported at 
the agency, such as case management, educational advancement, housing, independent living skills development, 
health education and assessment (including diabetes management, smoking reduction, HIV risk assessment and 
testing), and employment services, as part of its holistic approach to treatment. These model programs include 
intensive, team-based approaches that address the full spectrum of health and social service needs. To date, the 
agency also maintains active research and evaluation of innovative services and programs with the UIC Colleges
IRB #: STU00202936-MOD0048 Approved by NU IRB for use on or after 6/27/2019

Page 27 of 34
of Medicine and Nursing (including those related to supported employment and IPS), Dartmouth College, and 
Boston University, as well as other academic and community partners.
Thresholds uses the Gold Standard ‘Individual Placement and Support (IPS)’ version of ‘Supported Employment.’ 
Currently, more than 700 Thresholds members receive Supported Employment and they receive IPS at 4101 N. 
Ravenswood Ave, Chicago, IL 60613. This location is just a few miles from Dr. Jordan’s office in the Department 
of Psychiatry and Behavioral Sciences at Northwestern University Feinberg School of Medicine.
Thresholds will allocate shared space at their main center, 4423 N. Ravenswood Ave, Chicago, IL 60613. The 
space will include three offices to be shared by the study with existing services provided at Thresholds. The first 
two rooms are already outfitted with the necessary technology to administer the intervention. Since these rooms 
are already being used to facilitate vocational services, the addition of the study intervention in this environment 
will maintain the ecological validity of an effectiveness trial. Although the rooms are shared with other 
vocational services, time spent on the study will be coordinated by a scheduling calendar and time spent in the 
rooms will be private (no one else in the room during the simulated interviews). To demonstrate the scalability 
and generalizability of the intervention, Thresholds will use existing computers that include a microphone and 
speakers so that trainees can effectively interact with the intervention.
Northwestern University Feinberg School of Medicine
Northwestern University Clinical Research Program (NU-CRP):
NU-CRP is a collaborative, interdisciplinary research program within the Department of Psychiatry and 
Behavioral Sciences at Northwestern University Feinberg School of Medicine that conducts translational 
research projects that aim to determine the disease processes that cause severe mental illness or promote its 
progression as well as develop treatmens. NU-CRP uses a multidisciplinary approach with PIs trained in 
psychiatry, radiology, chemistry, social work, epidemiology, and computer engineering.
NU-CRP Clinical Research Core. The clinical research core is located on the 15th floor at 680 North Lake Shore 
Drive, Chicago, IL 606011. The pretest-posttest video-recorded role-plays will be conducted at this location. The 
Department of Psychiatry and Behavioral Sciences has allocated dedicated clinical research space. This space is 
located at 680 Lake Shore Dr. Suite 1520; Chicago, IL 60611 and occupies 3776 square feet. It is in an access 
restricted suite that is connected to the administrative and faculty offices via a locked doorway. In addition to 
office and workstation space for clinic staff, the clinical research space includes 5 interview rooms that will be 
used for this study. There are also several other rooms available in this suite that will not be used for this study 
including 2 cognitive testing rooms, an eye-tracking room, a modern laboratory for processing biological 
specimens, a phlebotomy area with 1 station for drawing blood (including one reclining chair), and a fully 
equipped exam room for physical exams. The interview rooms range in size from 100 square feet to 200 square 
feet and four public restrooms located in the hallway outside of the suite.
Center for Prevention Implementation Methodology (Ce-PIM):
Ce-PIM is a 5-year NIDA funded P30 Center of Excellence that is housed at Northwestern University and directed 
by Dr. C. Hendricks Brown. Its mission is threefold: to integrate and extend systems science methods to address 
critical research challenges in federally funded implementation research, to facilitate the seamless integration 
of methodology into the next generation of prevention research, and to integrate the methods into wide scale 
practice of implementation. The wide-scale implementation of such programs that target large portions of the 
population has the potential for reducing drug, alcohol, and tobacco abuse, improving mental health, and reducing
IRB #: STU00202936-MOD0048 Approved by NU IRB for use on or after 6/27/2019

Page 28 of 34
the incidence of HIV, HPV, and other sexually transmitted diseases. We focus on systems science methods to 
advance the conduct of implementation research. The term system science refers to a transdiciplinary approach to 
understanding how interactions between elementary units produce complex patterns. The structure of Ce-PIM 
involves an administrative core and a Methods Core. We work closely with 12 qualified grantees and other 
colleagues, who are already funded to conduct research related to implementation. We have focused on 
innovations that address all three major stages of the implementation process, from adoption, implementation 
with fidelity, and sustainability or moving to scale. These innovative methods all emerge from a systems 
orientation, and they include systems engineering to characterize and advance the implementation model; social 
network analysis; agent based modeling; intelligent data analysis and machine learning, and design of randomized 
implementation trials that can be used in large systems as programs are being rolled into communities.
The Center’s Administrative structure includes a committee on communication with both practitioners and 
scientists and partnership formation; a committee on pilot funding and mentoring of early stage investigators into 
the prevention implementation field. In additions, Ce-PIM is developing tools for use in both implementation 
research and in practice to support measurement of the implementation process and decision making by service 
organization and policy makers. Ce-PIM works closely with both researchers and policy makers to facilitate the 
development of more effective implementation strategies, to design tests and refinement procedures of these 
strategies, and to enhance training in implementation research methods.
The following Ce-PIM resources are fully available to assist Dr. Smith to assess the implementation of VR. First, 
Dr. Brown and his colleagues have experience in guiding early career investigators in the field of implementation 
science, and we will provide Dr. Smith access to material from their national trainings that they have conducted 
as well as linkages to experts around the country, a Ce-PIM expert in mixed methods who will serve as a consultant 
on this proposal. Secondly, Ce-PIM sponsors virtual grand rounds each week; this virtual network has a 
membership of over 300 researchers representing diverse areas of expertise in mental health and drug abuse. These 
grand rounds provide not only access to the newest research on implementation being conducted but also provide 
an effective way to obtain feedback from experts on this project as Dr. Smith will be asked to present his work at 
a point sufficiently early in the project where advice will be most beneficial. Thirdly, there will be local support 
from Ce-PIM that goes beyond the effort provided by Dr. Brown. Key to this project is the premise that the virtual 
reality based system will provide the user with practice and quality feedback around job interview skills. Fidelity 
monitoring and feedback is of central importance to Ce-PIM, and we have Ce-PIM experts in Dr. Smith’s and Dr. 
Brown’s department who are conducting groundbreaking research in automating such fidelity systems. The 
Center’s mission to synergize the field will be both informed by and provide innovation to this project.
Both research locations (Thresholds and NU) are located within close proximity to mental health treatment 
providers. The resources in place to support participants in case of anticipated consequences are listed in 10.0 
Risk to Participants
Adequate Training for Research Staff. The PI (Dr. Smith) or appropriate designate will conduct training 
sessions and direct research staff to run a series of mock participants in all study procedures prior to beginning 
the studies. All research staff will be cross-trained to administer all aspects of the research study. Prior to being 
authorized to provide an assessment, study personnel must observe 3 sessions of the PI or qualified research 
staff administering the assessment. Then the trainee must perform a series of 3 administrations with the 
assessment while observed by the PI or qualified research staff. After administering the assessments 3 times, the 
ability of the research staff will be reviewed with him or her and they will be authorized or asked to fine tune 
their performance until rated as acceptable.
IRB #: STU00202936-MOD0048 Approved by NU IRB for use on or after 6/27/2019

Page 30 of 34
References
1.
Rosenheck R, Leslie D, Keefe R, McEvoy J, Swartz M, Perkins D, Stroup S, Hsiao JK, Lieberman J. Barriers 
to employment for people with schizophrenia. Am J Psychiatry. 2006;163(3):411-417.
2.
Salkever DS, Karakus MC, Slade EP, Harding CM, Hough RL, Rosenheck RA, Swartz MS, Barrio C, 
Yamada AM. Measures and predictors of community-based employment and earnings of persons with 
schizophrenia in a multisite study. Psychiatr Serv. 2007;58(3):315-324.
3.
Administration SAaMHS. Results from the 2012 National Survey on Drug Use and Health: Mental 
Health Findings. In: Services DoHaH, ed. NSDUH Series H-47 ed. Rockville, MD: Substance Abuse and 
Mental Health Services Administration; 2012.
4.
Murphy GC, Athanasou JA. The effect of unemployment on mental health. Journal of Occupational and 
Organizational Psychology. 1999;72(1):83-99.
5.
Drake RE, Bond GR. The future of supported employment for people with severe mental illness.
Psychiatr Rehabil J. 2008;31(4):367-376.
6.
Drake RE, Bond GR. Supported employment: 1998 to 2008. Psychiatr Rehabil J. 2008;31(4):274-276.
7.
Corbiere M, Zaniboni S, Lecomte T, Bond G, Gilles PY, Lesage A, Goldner E. Job acquisition for people 
with severe mental illness enrolled in supported employment programs: a theoretically grounded 
empirical study. J Occup Rehabil. 2011;21(3):342-354.
8.
Solinski S, Jackson HJ, Bell RC. Prediction of employability in schizophrenic patients. Schizophr Res.
1992;7(2):141-148.
9.
Marwaha S, Johnson S. View and experiences of employment among people with psychosis: a 
qualitative descriptive study. International Journal of Social Psychiatry. 2006;51(4):302-316.
10.
Gervey R, Kowal H. The job developer's presence in the job interview: is it helpful or harmful to 
persons with psychiatric disabilities seeking employment? Psychiatr Rehabil J. 2005;29(2):128-131.
11.
Charisiou J, Jackson HJ, Boyle GJ, Burgess P, Minas IH, Joshua SD. Are employment-interview skills a 
correlate of subtypes of schizophrenia? Psychol Rep. 1989;65(3 Pt 1):951-960.
12.
Charisiou J, Jackson HJ, Boyle GJ, Burgess PM, Minas IH, Joshua SD. Which employment interview skills 
best predict the employability of schizophrenic patients? Psychol Rep. 1989;64(3 Pt 1):683-694.
13.
Corbiere M, Mercier C, Lesage A. Perceptions of barriers to employment, coping efficacy, and career 
search efficacy in people with mental illness. Journal of Career Assessment. 2004;12:460-478.
14.
Jacobs HE, Collier R, Wissusik D. The job-finding module: training skills for seeking competitive 
community employment. New Dir Ment Health Serv. 1992(53):105-115.
15.
Bell MD, Weinstein A. Simulated job interview skill training for people with psychiatric disability: 
feasibility and tolerability of virtual reality training. Schizophrenia Bulletin. 2011;37 Suppl 2:S91-97.
16.
Salyers MP, Becker DR, Drake RE, Torrey WC, Wyzik PF. A ten-year follow-up of a supported 
employment program. Psychiatr Serv. 2004;55(3):302-308.
17.
Smith MJ, Boteler Humm L, Fleming MF, Jordan N, Wright MA, Ginger EJ, Wright K, Olsen D, Bell MD. 
Virtual Reality Job Interview Training For Veterans with Posttraumatic Stress Disorder. Journal of 
Vocational Rehabilitation. 2015;42:271-279.
18.
Smith MJ, Fleming MF, Wright MA, Roberts AG, Humm LB, Olsen D, Bell MD. Virtual reality job 
interview training and 6-month employment outcomes for individuals with schizophrenia seeking 
employment. Schizophr Res. 2015;166(1-3):86-91.
19.
Smith MJ, Ginger EJ, Wright M, Wright K, Boteler Humm L, Olsen D, Bell MD, Fleming MF. Virtual 
reality job interview training for individuals with psychiatric disabilities. J Nerv Ment Dis. 
2014;202(9):659-667.
20.
Smith MJ, Ginger EJ, Wright K, Wright MA, Taylor JL, Humm LB, Olsen DE, Bell MD, Fleming MF. Virtual 
reality job interview training in adults with autism spectrum disorder. J Autism Dev Disord. 
2014;44(10):2450-2463.
IRB #: STU00202936-MOD0048 Approved by NU IRB for use on or after 6/27/2019

Page 31 of 34
21.
Smith MJ, Bell MD, Wright MA, Humm LB, Olsen D, Fleming MF. Virtual reality job interview training 
and 6-month employment outcomes for individuals with substance use disorders seeking employment. 
Journal of Vocational Rehabilitation. 2016;44:323-332.
22.
Smith MJ, Fleming MF, Wright MA, Jordan N, Humm LB, Olsen D, Bell MD. Job Offers to Individuals 
With Severe Mental Illness After Participation in Virtual Reality Job Interview Training. Psychiatr Serv. 
2015:appips201400504.
23.
Smith MJ, Fleming MF, Wright MA, Losh M, Humm LB, Olsen D, Bell MD. Brief Report: Vocational 
Outcomes for Young Adults with Autism Spectrum Disorders at Six Months After Virtual Reality Job 
Interview Training. J Autism Dev Disord. 2015.
24.
Lord SE, McGurk SR, Nicholson J, Carpenter-Song EA, Tauscher JS, Becker DR, Swanson SJ, Drake RE, 
Bond GR. The potential of technology for enhancing individual placement and support supported 
employment. Psychiatr Rehabil J. 2014;37(2):99-106.
25.
Curran GM, Bauer M, Mittman B, Pyne JM, Stetler C. Effectiveness-implementation hybrid designs: 
combining elements of clinical effectiveness and implementation research to enhance public health 
impact. Med Care. 2012;50(3):217-226.
26.
Association AP. American Psychiatric Association practice guidelines for the treatment of psychiatric 
disorders: compendium. Washington, D. C.: American Psychiatric Association; 2006.
27.
Mueser KT, Bond GR, Drake RE, Resnick SG. Models of community care for severe mental illness: a 
review of research on case management. Schizophr Bull. 1998;24:37-74.
28.
Dunn EC, Wewiorski NJ, Rogers ES. The meaning and importance of employment to people in recovery 
from serious mental illness: results of a qualitative study. Psychiatr Rehabil J. 2008;32(1):59-62.
29.
Corrigan PW, Mueser KT, Bond GR, Drake RE, Solomon P. Principles and practices of psychiatric 
rehabilitation: An empirical approach. New York: Guilford Press; 2008.
30.
Bell MD, Ryan ER. Integrating psychosocial rehabilitation into the hospital psychiatric service. Hosp 
Community Psychiatry. 1984;35(10):1017-1023.
31.
Warner R. Recovery from schizophrenia and the recovery model. Curr Opin Psychiatry. 2009;22(4):374- 
380.
32.
Shumway M, Saunders T, Shern D, Pines E, Downs A, Burbine T, Beller J. Preferences for schizophrenia 
treatment outcomes among public policy makers, consumers, families, and providers. Psychiatr Serv. 
2003;54(8):1124-1128.
33.
Provencher HL, Gregg R, Mead S, Mueser KT. The role of work in the recovery of persons with 
psychiatric disabilities. Psychiatr Rehabil J. 2002;26(2):132-144.
34.
Rogers JA. Work is key to recovery. Psychosocial Rehabilitation Journal. 1995;18(4):5-10.
35.
Paul KI, Moser K. Unemployment impairs mental health. Journal of Vocational Behavior.
2009;74(3):264-282.
36.
Vinokur AD, Schul Y, Vuori J, Price RH. Two years after a job loss: Long-term impact of the JOBS 
program on reemployment and mental health. Journal of Occupational Health Psychology. 
2000;5(1):32-47.
37.
Bond GR, Drake RE, Becker DR. An update on randomized controlled trials of evidence-based 
supported employment. Psychiatr Rehabil J. 2008;31(4):280-290.
38.
Huffcutt AI. An empirical review of the employment interview construct literature. International 
Journal of Selection and Assessment. 2011;19(1):62-81.
39.
Roelfsema PR, van Ooyen A, Watanabe T. Perceptual learning rules based on reinforcers and attention.
Trends Cogn Sci. 2010;14(2):64-71.
40.
Vinogradov S, Fisher M, de Villers-Sidani E. Cognitive training for impaired neural systems in 
neuropsychiatric illness. Neuropsychopharmacology. 2012;37(1):43-76.
41.
Becker DR, Lynde D, Swanson SJ. Strategies for state-wide implementation of supported employment: 
the Johnson & Johnson-Dartmouth Community Mental Health Program. Psychiatr Rehabil J. 
2008;31(4):296-299.
IRB #: STU00202936-MOD0048 Approved by NU IRB for use on or after 6/27/2019

Page 32 of 34
42.
Corbiere M, Bond GR, Goldner EM, Ptasinski T. Brief reports: the fidelity of supported employment 
implementation in Canada and the United States. Psychiatr Serv. 2005;56(11):1444-1447.
43.
Mank D, Cioffi A, Yovanoff P. Supported employment outcomes across a decade: is there evidence of 
improvement in the quality of implementation? Ment Retard. 2003;41(3):188-197.
44.
McGrew JH, Griss ME. Concurrent and predictive validity of two scales to assess the fidelity of 
implementation of supported employment. Psychiatr Rehabil J. 2005;29(1):41-47.
45.
Rosenheck RA, Mares AS. Implementation of supported employment for homeless veterans with 
psychiatric or addiction disorders: two-year outcomes. Psychiatr Serv. 2007;58(3):325-333.
46.
Damschroder LJ, Hagedorn HJ. A guiding framework and approach for implementation research in 
substance use disorders treatment. Psychol Addict Behav. 2011;25(2):194-205.
47.
Damschroder LJ, Aron DC, Keith RE, Kirsh SR, Alexander JA, Lowery JC. Fostering implementation of 
health services research findings into practice: a consolidated framework for advancing 
implementation science. Implement Sci. 2009;4:50.
48.
First MB, Williams, J.B.W., Karg, R.S., Spitzer, R.L. Structured Clinical Interview for DSM-5, Research 
Version. Arlington, VA: American Psychiatric Association; 2015.
49.
Posner K BG, Stanley B, Brent DA, Yershova KV, Oquendo MA, Currier GW, Melvin G, Greenhill L, Shen 
S, Mann JJ. The Columbia-Suicide Severity Rating Scale: Initial Validity and Internal Consistency 
Findings from Three Multisite Studies with Adolescents and Adults. American Journal of Psychiatry. 
2011;68:1266-1277.
50.
Marder SR, Fenton W. Measurement and Treatment Research to Improve Cognition in Schizophrenia: 
NIMH MATRICS initiative to support the development of agents for improving cognition in 
schizophrenia. Schizophr Res. 2004;72(1):5-9.
51.
Development of the World Health Organization WHOQOL-BREF quality of life assessment. The 
WHOQOL Group. Psychol Med. 1998;28(3):551-558.
52.
Hogarty GE, Goldberg SC, Schooler NR. Drug and sociotherapy in the aftercare of schizophrenic 
patients. III. Adjustment of nonrelapsed patients. Arch Gen Psychiatry. 1974;31(5):609-618.
53.
Hays RD, Sherbourne CD, Mazel RM. The RAND 36-Item Health Survey 1.0. Health Econ. 1993;2(3):217- 
227.
54.
Patterson TL, Moscona S, McKibbin CL, Davidson K, Jeste DV. Social skills performance assessment 
among older patients with schizophrenia. Schizophrenia Research. 2001;48(2-3):351-360.
55.
Schneider LC, Struening EL. SLOF: a behavioral rating scale for assessing the mentally ill. Social Work 
Research Abstracts. 1983;19(3):9-21.
56.
Patterson TL, Goldman S, McKibbin CL, Hughs T, Jeste DV. UCSD Performance-Based Skills Assessment: 
development of a new measure of everyday functioning for severely mentally ill adults. Schizophr Bull. 
2001;27(2):235-245.
57.
Benjamini Y, Drai D, Elmer G, Kafkafi N, Golani I. Controlling the false discovery rate in behavior 
genetics research. Behav Brain Res. 2001;125(1-2):279-284.
58.
Lachin JM. Statistical considerations in the intent-to-treat principle. Control Clin Trials. 2000;21(3):167- 
189.
59.
Eekhout I, de Vet HC, Twisk JW, Brand JP, de Boer MR, Heymans MW. Missing data in a multi-item 
instrument were best handled by multiple imputation at the item score level. J Clin Epidemiol. 
2014;67(3):335-342.
60.
Rubin DB. Multiple Imputation for Nonresponse in Surveys. Hoboken, NJ: John Wiley & Sons, Inc.; 1987.
61.
Sterne JA, White IR, Carlin JB, Spratt M, Royston P, Kenward MG, Wood AM, Carpenter JR. Multiple 
imputation for missing data in epidemiological and clinical research: potential and pitfalls. BMJ. 
2009;338:b2393.
62.
Mazumdar S, Liu KS, Houck PR, Reynolds CF, 3rd. Intent-to-treat analysis for longitudinal clinical trials: 
coping with the challenge of missing values. J Psychiatr Res. 1999;33(2):87-95.
63.
Cox DR, Oakes D. Analysis of Survival Data. New York: Chapman & Hall; 1984.
IRB #: STU00202936-MOD0048 Approved by NU IRB for use on or after 6/27/2019

Page 33 of 34
64.
Freedman LS. Tables of the number of patients required in clinical trials using log-rank test. Statistics in 
Medicine. 1982;1:121-129.
65.
Siegel JE, Weinstein MC, Russell LB, Gold MR. Recommendations for reporting cost-effectiveness 
analyses. Panel on Cost-Effectiveness in Health and Medicine. JAMA. 1996;276(16):1339-1341.
66.
Chaudhary MA, Stearns SC. Estimating confidence intervals for cost-effectiveness ratios: an example 
from a randomized trial. Stat Med. 1996;15(13):1447-1458.
67.
Willan AR, O'Brien BJ. Confidence intervals for cost-effectiveness ratios: an application of Fieller's 
theorem. Health Econ. 1996;5(4):297-305.
68.
Fenwick E, O'Brien BJ, Briggs A. Cost-effectiveness acceptability curves--facts, fallacies and frequently 
asked questions. Health Econ. 2004;13(5):405-415.
69.
Kraemer HC, Gibbons RD. Why does the randomized clinical trial methodology so often mislead clinical 
decision making? Focus on moderators and mediators of treatment. Psychiatric Annals. 
2009;39(7):736-745.
70.
MacKinnon DP, Fairchild AJ, Fritz MS. Mediation analysis. Annu Rev Psychol. 2007;58:593-614.
71.
Preacher KJ, Hayes AF. SPSS and SAS procedures for estimating indirect effects in simple mediation 
models. Behav Res Methods Instrum Comput. 2004;36(4):717-731.
72.
Vanderweele TJ, Vansteelandt S, Robins JM. Effect decomposition in the presence of an exposure- 
induced mediator-outcome confounder. Epidemiology. 2014;25(2):300-306.
73.
MacKinnon DP. Introduction to statistical mediation analysis. New York: Erlbaum; 2008.
74.
Preacher KJ, Rucker DD, Hayes AF. Addressing moderated mediation hypotheses: Theory, methods, 
and prescriptions. Multivariate Behavioral Research. 2007;42:185-227.
75.
Krueger RA. Focus Groups: A Practical Guide for Applied Research. 4th ed. Thousand Oaks, CA: Sage 
Publications; 2009.
76.
Guest G, MacQueen KM, Namey EE. Applied Thematic Analysis. Thousand Oaks, CA: Sage Publications, 
Inc.; 2012.
77.
Lincoln Y, Guba E. Processing Naturally Obtained Data. Naturalistic Inquiry. London: Sage Publications; 
1985:332-356.
78.
Bernard HR, Ryan GW. Text analysis: qualitative and quantitative methods. In: Bernard HR, ed.
Handbook of Methods in Cultural Anthropology. Walnut Creek, CA: Altamira Press; 1998.
79.
Kurasaki K. Intercoder reliability for validating conclusions drawn from open-ended interview data.
Field Methods. 2000;12:179-194.
80.
Green J, Thorogood N. Qualitative Methods for Health Research. 3rd ed. Los Angeles, CA: Sage 
Publications; 2014.
81.
Crabtree BF, Miller WL. A template approach to text analysis: Developing and using codebooks. . In: 
Crabtree BF, Miller ML, eds. Doing Qualitative Research. Vol 3. Thousand Oaks, CA: Sage Publications; 
1992:93-109.
82.
Sullivan SD, Mauskopf JA, Augustovski F, Jaime Caro J, Lee KM, Minchin M, Orlewska E, Penna P, 
Rodriguez Barrios JM, Shau WY. Budget impact analysis-principles of good practice: report of the ISPOR 
2012 Budget Impact Analysis Good Practice II Task Force. Value Health. 2014;17(1):5-14.
83.
Ritzwoller DP, Sukhanova A, Gaglio B, Glasgow RE. Costing behavioral interventions: a practical guide 
to enhance translation. Ann Behav Med. 2009;37(2):218-227.
IRB #: STU00202936-MOD0048 Approved by NU IRB for use on or after 6/27/2019

Page 34 of 34
Appendix A: Thresholds Locations for Study Visits and Molly Delivery
12145 S. Western Ave., Blue Island IL 60406- “South Suburbs”
12151 S. Western Ave., Blue Island 60643- “New Freedom Center, South Suburbs” 
4101 N. Ravenswood Ave., Chicago IL 60613- “4101”
4423 N. Ravenswood Ave., Chicago IL 60613- “Research Team Office” 
5357 N. Broadway, Chicago IL 60640- “Peer Success”
734 W. 47th Street, Chicago IL 60609- “Bridge South”
4219 N. Lincoln Ave, Chicago IL 60618- “Young Adult Program” 
3638 S. Kedzie Ave., Chicago IL 60632- “Bridge Southwest”
2240 W. Ogden Ave., Chicago IL 60612- “Supported Employment & Veterans Program” 
5000 W. Roosevelt Rd., Chicago IL 60608- “Bridge West”
716 W. 47th Street, Chicago IL 60606- “Freedom Center, South”
2045 W. Grand Ave, Chicago IL 60612- “New Freedom Center, North” 
120 S. LaSalle St., Chicago IL 60603- “HQ”
202 N. Schuyler Ave., Kankakee IL 60901- “Kankakee”
444 Frontage Rd., Northfield IL 60093- “New Foundations Center” 
7400 Skokie Blvd., Skokie IL 60077- “North Suburbs”
777 Oak Lane, Westmont IL 60559- “Westmont”
109 S. Jefferson St., Woodstock IL 60098- “McHenry”
IRB #: STU00202936-MOD0048 Approved by NU IRB for use on or after 6/27/2019