IMPRINT Randomized Evaluation
Conditions: Virtual Reality
Sponsor: Charles River Analytics

 
Page 1 of 22 
v08062019 
 
STUDY TITLE: 
IMPRINT Randomized Evaluation 
 
PRINCIPAL INVESTIGATOR: 
Name: Dr. E. Vincent Cross II  (Contact), Mr. Dan Duggan, Dr. Dave Turcotte 
(Contact) 
 
CO-INVESTIGATORS: 
Name: Mr. David Coffey  
 
VERSION/DATE: 
4/24/2024 
 
Check any applicable boxes in the table below – you will be asked for 
further detail on these topics later in the protocol: 
 
Indicate Vulnerable 
Population(s) to be Enrolled 
☐ Children  
☐ Cognitively Impaired Adults   
☐ Pregnant Women (IF the research activities will affect   
the pregnancy or the fetus) 
☐ Prisoners (or other detained/paroled individuals)  
Research has U.S. Federal 
government funding (e.g., 
NIH, NSF, other federal 
agencies/departments) 
 
☒ 
 
 
NOTE: The protocol should describe your research procedures and participant population in detail.  
Your consent documents, data collection instruments (surveys, questionnaires, interview guides, 
etc.), and recruitment materials need to be submitted for review as well and should not be attached or 
incorporated into this protocol document. 
 
TIPS ON COMPLETING THE PROTOCOL FORM: 
 
• If any sections are not applicable to your research, mark that section as N/A (for not 
applicable) 
• Keep an electronic copy of your protocol.  If you submit modifications to your study at a 
later time, you will need to include tracked changes to all affected study documents, 
including the protocol. 
• As you write this protocol, remove the text boxes and all instructional text contained inside 
the text boxes in each section. There should be no text boxes or instructional text (including 
these instructions) in the final version of your protocol. 

 
Page 2 of 22 
v08062019 
 
 

 
Page 3 of 22 
v08062019 
 
1.0 
Purpose of the study: 
 
Charles River Analytics and our collaborators at The New England Consortium 
propose to develop and evaluate an Immersive Modular Preparedness 
Intelligent Tutor (IMPRINT).  IMPRINT aims to enhance trainee performance 
and set operational expectations outside of the hot zone in hyper-realistic 
exercises that challenge physical, psychological, and cognitive capabilities 
through adaptive behavioral science enhanced virtual reality (VR) modules and 
seamlessly integrate with HAZWOPER training. The overall objective of 
IMPRINT Phase II is to conduct a 1-year randomized-controlled trial among 
regular HAZWOPER Trainees comparing protocol adherence during required 
hands-on field exercises for trainees who used the IMPRINT behavioral science 
VR training with trainees who received standard PowerPoint and video training. 
 
We anticipate trainees and classes assigned to the intervention will perform 
better than those who use standard paper and PowerPoint based activities. We 
believe their comprehension of topics will be greater, and it is likely that they 
will have a greater ability to recall HAZWOPER guidelines in the field. We 
also anticipate after class survey results will indicate a better HAZWOPER 
training experience and increased levels of satisfaction with the course. 
 
Additionally, we anticipate that trainers using the IMPRINT VR modules will 
find the tool more effective and engaging compared to standard PowerPoint and 
video-based training methods. We expect that trainers will report greater ease in 
delivering complex HAZWOPER concepts and that the VR modules will 
enhance their ability to facilitate trainee comprehension and recall of 
HAZWOPER guidelines. We also anticipate that trainer feedback will reflect a 
more positive teaching experience and higher levels of satisfaction with the 
training process. 
 
Long-term benefits of participation include improvement in retention of 
HAZWOPER guidelines when responding to Hazardous Material (HAZMAT) 
disasters. We anticipate trainees and classes assigned to the intervention will 
perform better than those who use standard paper and PowerPoint based 
activities. We believe their comprehension of topics will be greater, and it is 
likely that they will have a greater ability to recall HAZWOPER guidelines in 
the field. We also anticipate after class survey results will indicate a better 
HAZWOPER training experience and increased levels of satisfaction with the 
course. 
 
2.0 
Background / Literature Review / Rationale for the study: 
 
 
To successfully transfer skills and develop trainees’ procedural cognitive maps 
VR training systems must enable full immersion into training exercises by 

 
Page 4 of 22 
v08062019 
integrating PPE, enabling 3D object haptic interaction and natural walking to 
support spatial skill development, and provide trainer support (Mossel et al., 
2017). Research has shown significant performance benefits from VR delivered 
targeted mission rehearsal (Xie et al, 2021). To realize these benefits before 
facing real emergencies, trainees must have the opportunity to hone their skills 
in a controlled setting that mimics real-life scenarios as closely as possible 
through fully immersive exercises (Mossel et al., 2017). This immersive 
context driven approach is critical for skill transfer and well understood in 
parallel domains (Mossel, 2020). Immersion is best achieved through a 
combination of sensations, including haptic (tactile touch, force feedback, 
temperature, etc.), audio, and visual (simulated detector reads of simulated 
radiation). Several recent empirical assessments of VR systems demonstrate 
superior skill transfer when haptic immersion is integrated in a domain 
appropriate way (e.g., heat exposure fire training, or stress inducing stimuli for 
disaster response). 
 
It is not known whether this skill transfer occurs within the HAZMAT domain, 
or how to best operationalize these findings within training.  
 
3.0 
Inclusion and Exclusion Criteria: 
 
We will seek out participants who self-select to participate in the New England 
Consortium (TNEC) training offerings through open enrollment, including 
populations of any racial or ethnic background. 
 
We will also seek out participants who are experienced trainers involved in 
delivering the New England Consortium (TNEC) training offerings, including 
trainers from any racial or ethnic background. 
 
Participants must meet the eligibility requirements for the class. Given the class 
requires students to complete standardized HAZMAT training in fully suited 
PPE with a self-contained breathing apparatus, students must be able to speak 
and understand English, and not be pregnant. 
 
The HAZWOPER certification course falls under TNEC’s open enrollment 
courses. TNEC offers on an ongoing basis at their training center in Lowell, 
Massachusetts. The open enrollment courses generally consist of 15-25 students 
from various employers or organizations with a variety of backgrounds. Prior to 
each open enrollment course, TNEC trainers review the student enrollment's 
business types, chemical handling information and hazard exposures. Upon this 
review the trainers will personalize the course to best meet the needs of the 
attendees. 
 
Age Limits  
Min Age: 18 Years Max Age: N/A (No limit) 
 

 
Page 5 of 22 
v08062019 
We will be targeting participants who participate in TNEC training offerings. 
Given the training takes place in Massachusetts we will use Massachusetts 
demographics to set our inclusion targets. 
 
 
American 
Indian or 
Alaskan 
Native 
Asian  
Black or 
African 
American 
Hispanic 
or 
Latinx 
Native 
Hawaiian 
or other 
Islander 
Multiple 
Race, 
Non-
Hispanic 
White 
Unknown 
Percent  
.01 
9 
9.9 
20.5 
0 
0 
56.7 
0 
 
 
 
4.0 
Sample Size: 
 
According to The New England Consortium, during the pandemic they have 
trained approximately 15-25 people per class once a month.  
 
Our study follows two stages. First, we will conduct a Pilot consisting of 2 
months with one month serving as a case and the other month serving as the 
control. The Pilot will be used to indicate effect size. (30-50 participants) 
 
Our study will also include the main evaluation, consisting of 6 randomly 
selected courses that receive the IMPRINT VR intervention and 6 randomly 
selected control courses. (180-300 participants) 
 
This creates a sample of approximately 300 participants. 
5.0 
Recruitment and Screening Methods: 
 
Recruitment process: Given participants opt into training we will use a 
convenience sampling frame. Recruitment will consist of including the term, 
“Virtual Reality Immersive Activities”, to the existing course title and course 
link. The current open enrollment advertisement is available through this link: 
https://tnectraining.org/training/schedule-register/ 
 
Screening process: N/A 
 
 
6.0 
Research Locations: 
 
The evaluation will take place on site at The New England Consortium (TNEC) 
training center at Wannalancit Mills, as part of the University Massachusetts 
Lowell Campus. The facility is well equipped with the space necessary and 
standard equipment provided for training. The site is located at the heart of 
Lowell, offers shuttle bus service, state-of-the-art fiber optic cable and HVAC, 
and 7 elevators and 4 loading docks enabling easy access for trainees of any 

 
Page 6 of 22 
v08062019 
background, disability status, or socioeconomic status, and has the ability to 
host and maintain a VR exercises for evaluation. 
 
In addition to the Wannalancit Mills location, the evaluation will also take 
place a various offsite locations where TNEC has been contracted to deliver the 
40-hour HAZWOPER training. 
 
All required permissions and approvals are already obtained to conduct this 
research at TNEC. 
 
 
 
7.0 
Multi-site Research (research that involves external collaborating 
institutions and individuals):  
 
N/A 
 
 
8.0 
Procedures Involved: 
 
The standard TNEC 40-hour workbook will be used in both the intervention 
and control classes.  
 
In the intervention, trainees will be provided with an Oculus VR headset to 
complete virtual reality modules based on activities covered in the TNEC 40 
hour course workbook. In control classes, trainees will complete all of the 
activities following standard practice which includes filling out sheets, 
reviewing 2D photographs, and reviewing slides. VR modules will last no 
longer than 1-5 minutes in length, which is a comparable length to the time 
allowed for students in control classes to complete their assigned tasks. There 
will be no differences in content provided between cases and controls. We are 
examining the effect of a difference in delivery mechanism and interaction with 
the content to enhance retention, understanding, and performance. 
 
 
VR modules will be provided intermittently throughout the 40 hour course. 
Each course is required to be 40 hours in length by the OSHA standard. 
 
All evaluations and assessments of trainees will take place at the same sequence 
between cases and controls. 
 
Outcomes measured include student accuracy on the TNEC HAZWOPER 
performance instruments including; 1) the 40 Hr HazWoper Course Workbook, 
2) the Toxic Properties Activity, 3) The New England Consortium Health and 
Safety Training Survey, 4) System Usability Scale (SUS), 5) Occupational Self-

 
Page 7 of 22 
v08062019 
Efficacy Scale - Short Form (OSS-6), 6) iGroup Presence Questionnaire (IPQ), 
7) Flow State Scale (FSS), 8) UX for VR Questionnaire, and 9) NASA-TLX 
 
We will first run a two-month pilot to estimate ICC, the intracluster correlation 
coefficient. We will use ICC to determine effect size for the following 12-
month study (see Figure 10). Using this calculation, we will determine expected 
effect size from the more extensive 12-month portion of the evaluation effort. 
 
Prior to analysis of the 12-month effort, we will use matching techniques based 
on trainee and trainer characteristics. We will use pretest matching criteria 
based on trainee cohort population characteristics and prior knowledge 
ascertained through pre-HAZWOPER training registration information.  
 
We will match cases and controls using Euclidian distance between 
intervention and control samples rather than propensity score matching to find 
the best match to mitigate risk of confounding due to low sample size following 
the guidance of Dr. Gary King (King, G. and Nielsen, R., 2019. Why 
propensity scores should not be used for matching. Political Analysis, 27(4), 
pp.435-454.). We will compare how these modules enhance student 
performance across TNEC HAZWOPER performance instruments including; 
1) the 40 Hour HAZWOPER Course Workbook, 2) the Toxic Properties 
Activity, 3) The New England Consortium Health and Safety Training Survey, 
4) System Usability Scale (SUS), 5) Occupational Self-Efficacy Scale - Short 
Form (OSS-6), 6) iGroup Presence Questionnaire (IPQ), 7) Flow State Scale 
(FSS), 8) UX for VR Questionnaire, and 9) NASA-TLX 
 
We will compare how these modules enhance performance through a t-test. 
 
Intervention VR modules will be based on the existing activities within the 40 
Hour HAZWOPER Course Workbook and the field exercise including: 
 
Module 1 
Duration: 5 
Minutes 
Activity 3-A What’s on the Site? 

 
Page 8 of 22 
v08062019 
Instrument 
IMPRINT 
Intervention 
(Virtual 
Environment) 

 
Page 9 of 22 
v08062019 
Control Part A 
Control Part B 
Module 2 
Duration 5 
Minutes 
Activity 11-A Controlling the Site 

 
Page 10 of 22 
v08062019 
Instrument 
IMPRINT 
Intervention 
(Virtual 
Environment) 

 
Page 11 of 22 
v08062019 
Control 
Module 3 
Duration 3 
Minutes 
Decontamination Line Ordering and Field Exercise 

 
Page 12 of 22 
v08062019 
Instrument 
(field training 
exercise/ 
evaluation) 
 
IMPRINT 
Intervention 
(Virtual 
Environment) 

 
Page 13 of 22 
v08062019 
Control  
 
 
 
 
 
Module 4 
Visual Inspection Checklist 

 
Page 14 of 22 
v08062019 
Duration 5 
Minutes 
Instrument 
IMPRINT 
Intervention 
(Virtual 
Environment) 
Control 
Module 5 
Duration 5 
Minutes 
Chemical Properties 

 
Page 15 of 22 
v08062019 
Instrument 
 
 
 
 
 

 
Page 16 of 22 
v08062019 
IMPRINT 
Intervention 
(Virtual 
Environment) 
Control 
 
 
 

 
Page 17 of 22 
v08062019 
Materials for 
both 
Intervention 
and Control 
 
  
 
 
9.0 
Research with Vulnerable Populations 
 
N/A  
 
 
10.0 Incomplete Disclosure or Deception: 
 
N/A 
 
11.0 Consent Process:  
 
Prior to enrollment in the study students and trainers will be presented with 
informed consent documentation. Informed consent will take place online or in 
the classroom. Storage of informed consent will be saved on a secure server 
used to host sensitive information where a password is required for entry and/or 
decryption or in locked cabinet file. 
 
 

 
Page 18 of 22 
v08062019 
Students will have the opportunity to contact TNEC or the researchers directly 
with any questions prior to registering for the course and consenting to the 
study. 
 
  
 
 
12.0 Research with Children – Parental Permission, Child Assent, and 
Other Considerations: 
 
N/A 
 
 
13.0 Waiver of Participant Signature on Consent Form: 
 
N/A 
 
14.0 Waivers and Alterations of Consent Information: 
 
N/A 
15.0 Financial Compensation: 
 
Trainees will have the opportunity to be entered into a $50 gift card raffle with the 
completion of the post course survey. 
 
They will NOT receive any type of rights for discoveries, patents or products 
developed from this research. 
 
There is no more than Minimal Risk to participants. 
 
16.0 Audio/Video Recording/Photography 
 
N/A 
 
 
17.0 Potential Benefits of this Research:  
 
No benefits are promised. Some benefits might include an improved training 
experience and improved retention of training concepts.  
 
We anticipate trainees and classes assigned to the intervention will perform 
better than those who use standard paper and PowerPoint based activities. We 
believe their comprehension of topics will be greater, and it is likely that they 
will have a greater ability to recall HAZWOPER guidelines in the field. We 

 
Page 19 of 22 
v08062019 
also anticipate after class survey results will indicate a better HAZWOPER 
training experience and increased levels of satisfaction with the course. 
 
Long-term benefits of participation include improvement in retention of 
HAZWOPER guidelines when responding to Hazardous Material (HAZMAT) 
disasters. We anticipate trainees and classes assigned to the intervention will 
perform better than those who use standard paper and PowerPoint based 
activities. We believe their comprehension of topics will be greater, and it is 
likely that they will have a greater ability to recall HAZWOPER guidelines in 
the field. We also anticipate after class survey results will indicate a better 
HAZWOPER training experience and increased levels of satisfaction with the 
course. 
 
 
 
18.0 Risks to Participants: 
There is a slight chance of injury through falling when walking or moving through the 
room during the tasks performed. There are additional risks when using virtual reality 
equipment or engaging in a virtual environment. These risks are infrequent, but may 
include motion sickness, disorientation, and dizziness. The experimenter will explain 
how you can take breaks during the experiment to prevent or relieve these potential 
risks, and you will be allowed to stop a task, and if necessary, exit the virtual 
environment by removing the virtual reality goggles at any time if you begin to feel 
discomfort or other undesirable sensations. 
 
This research is no more than minimal risk. The level of risk is expected to about the 
same as risks of daily life or a working with protective equipment used throughout 
training. 
 
19.0 Provisions to Protect Participant Privacy and Data 
Confidentiality: 
 
Data collection during training sessions will include written notes of observed 
activities, with a follow-up interview to gain greater context. A demographic 
survey will be administered at the end of each data collection session for all 
participants to collect information, such as age, sex, ethnicity, and race.  
 
We will make every attempt to not collect identifiable data, but we will also 
review and de-identify all data. Participants will be assigned a unique, 
anonymous identifier linking them to their data. To protect participant 
confidentiality, all paperwork that could be directly linked to the participant 
personally (such as a signed consent form or payment documentation forms) 
will be stored in the investigators’ locked filing cabinets. All other data will be 
de-identified (i.e., identified by code rather than with personal information) and 
stored in a locked cabinet (different from the cabinet where data that could be 

 
Page 20 of 22 
v08062019 
linked personally to the participant is stored) or on a computer where a 
password is required for entry and/or decryption. The study data will be stored 
on a password protected and encrypted network storage drive that only the 
Principle Investigator and research team can access. This system is currently in 
use and has been used in previous studies for storage of human subjects study 
files. Only properly trained study staff who require data access for the purposes 
of completing the study will receive a password or key. 
 
Data will be stored for 5 years and transferred through VPN secured server.  
 
20.0 Data Monitoring Plan to Ensure the Safety of Participants: 
Data will be monitored monthly by the PI(s) given the frequency of the course. 
At that time the safety of equipment will be evaluated and VR Headsets will be 
cleaned.   
 
 
21.0 Long-term Data and Specimen Storage and Sharing:  
We will make every attempt to not collect identifiable data, but we will also 
review and de-identify all data. Participants will be assigned a unique, 
anonymous identifier linking them to their data. To protect participant 
confidentiality, all paperwork that could be directly linked to the participant 
personally (such as a signed consent form or payment documentation forms) 
will be stored in the investigators’ locked filing cabinets. All other data will be 
de-identified (i.e., identified by code rather than with personal information) and 
stored in a locked cabinet (different from the cabinet where data that could be 
linked personally to the participant is stored) or on a computer where a 
password is required for entry and/or decryption. The study data will be stored 
on a password protected and encrypted network storage drive that only the 
Principal Investigator and research team can access. This system is currently in 
use and has been used in previous studies for storage of human subjects study 
files. Only properly trained study staff who require data access for the purposes 
of completing the study will receive a password or key. 
 
Data will be stored for 5 years and transferred through VPN secured server 
 
 
 
 
22.0 Qualifications of Research Team to Conduct the Research: 
 
Dr. Cross completed his Ph.D. at Auburn University in the Human-Centered 
Computing lab with a focus on understanding how the design of user interfaces 

 
Page 21 of 22 
v08062019 
impact situational awareness and mental workload for operators attempting to 
control a team of unmanned ground vehicles. He is a Senior Research Scientist 
at Charles River Analytics and is a proven user experience (UX) leader with 
research interest in Human-Computer Interaction (HCI) and Human-Artificial 
Intelligence and Teaming (HAIT). He was previously the NASA Human 
Research Program (HRP) Risk of Inadequate Human-Computer Interaction 
Discipline Scientist. In this role Dr. Cross was responsible for facilitating 
research that would address gaps crewmembers being to safely and efficiently 
use technology to perform missions’ tasks associated with long-duration 
missions. In this role Dr. Cross was also a SME for the Commercial Crew 
Program ensuring requirements were properly addressing standards in the 
NASA-STD-3001 VOL 2. While at NASA, his research explored how the 
design of user interfaces impacted crew’s ability to recognize critical 
information such as faults or low consumable information.  
 
Mr. Duggan is a Software Engineer and serves as the XR Development Lead 
for the Human Center Intelligent Systems (HCIS) Division at Charles River 
Analytics, Inc. He has a background in Computer Science and Software 
Engineering with a focus on Virtual and Augmented Reality development, 
collectively known as Extended Reality (XR). As Technical Lead on a number 
of XR development projects, he has created XR applications for fields as varied 
as industrial design, medical training, and distributed mission planning; he was 
responsible for overseeing the software development lifecycle, including 
requirements analysis, software design, implementation, testing, and 
deployment. Many of these applications were built using the VIRTUOSO SDK 
(VSDK), a free and open source solution for simplifying XR development that 
he created. He will utilize VSDK to accelerate the development of IMPRINT, 
providing a basis for naturalistic interactions and haptics as well as enabling 
IMPRINT to run on a variety of hardware solutions for a more flexible product. 
As the XR Development Lead for the HCIS Division, he recommends best 
practices, aid in device selection, and maintain expertise across XR technology 
and domains. 
 
Thomas Eastbrook is employed by the New England Consortium (TNEC), 
based at the University of Massachusetts – Lowell, as the   project director with 
TNEC responsible for the operations of TNEC’s Hazardous Materials and 
Emergency Response training program in New England. He has more than 23 
years of worker health and safety training experience during which he has made 
an important impact on thousands of students by providing accurate, clear, and 
comprehensive information about workplace hazards and the best practices for 
protecting workers from those hazards. As a trainer he has used the evaluation 
process – including course evaluation sheets, program evaluation, and trainer 
evaluation, including his own – to continuously improve his training methods. 
He has followed closely best practices in adult and popular education and has 
constantly adjust his approach with each class of workers that he trains. He 
keeps up on latest developments in occupational health and safety, 

 
Page 22 of 22 
v08062019 
environmental, and public health practice and policy, attend seminars, 
professional trainings, and conferences, and bring back ideas and concepts to 
enhance and stimulate my training. Most significantly, he enjoys being a part of 
a community of trainers who create a positive environment of mutual learning 
from one another in order to provide the best possible training experience for 
students. 
 
Dr. Turcotte has collaborated on many environmental health projects and in-
home intervention research projects within low-income diverse communities. 
Dr. Turcotte is currently Principal Investigator (PI) for an $700,000 U.S. 
Department of Housing and Urban Development (HUD) Healthy Homes 
Technical Studies Program grant that gathers environmental and health data in 
low-income, homes of asthmatic children and older adults to assess the health 
outcome impacts of comprehensive interventions and the sustainability of these 
interventions. Dr. Turcotte is also Co-Principal Investigator on the 
Massachusetts Department of Public Health’s Reducing Older Adult Asthma 
Disparities (ROADD) study which overall goal is to improve asthma control 
and reduce urgent care utilization for older adults with asthma through asthma 
home-visiting interventions that integrates the clinical care with the home-
based intervention using a team-based approach that includes a home visiting 
community health worker, visiting nurse and primary care team to study 
participants. Dr. Turcotte was also PI on three U.S. Department of Housing and 
Urban Development (HUD) funded projects, including a $750,000 Healthy 
Homes Technical Studies grant to conduct multifaceted interventions with 
CHWs in the homes of older adults with asthma and COPD to evaluate their 
effectiveness in improving health outcome and reducing asthma triggers, as 
well as $875,000 Healthy Homes Demonstration Program and $425,000 
Asthma Intervention in Public and Assisted Housing grants that gathered 
environmental and health data in low-income, homes with asthmatic children 
and assess the health outcome impacts of comprehensive interventions. These 
projects have also evaluated cost and technical effectiveness, sustainability, 
impact of remediation and educational interventions on health, behavior, 
exposure, and safety. Dr. Turcotte was also a co-Investigator on and managed 
several EPA and HUD funded projects focusing on mitigating negative 
environmental health impacts facing low-income populations. Dr. Turcotte’s 
interdisciplinary background and experience working with multidisciplinary 
research team on environmental health projects targeting low-income older 
adults with asthma and COPD make me well suited to make positive 
contributions to the proposed study. In addition, Dr. Turcotte’s past success 
managing complex, multifaceted projects and familiarity with the target 
community will be an asset to the research team. 
 
 
23.0 References