A Mixed Methods Study to Assess the Clinical Effectiveness and Acceptability of qER Artificial Intelligence Software to Prioritise CT Head Interpretation.
Conditions: Trauma, Brain, Mass Lesion, Skull Fractures, Hemorrhage, Intracranial, Midline Deviation, Stroke, Ischemic, Encephalomalacia
Sponsor: Guy's and St Thomas' NHS Foundation Trust

 
 
 
GSTFT – Observational Research Protocol template_v1 24.11.2022 
 
[ACCEPT] [IRAS 313507] [Version 1.0] [05/APRIL/2023]                                                                   Page 1 of 54 
 
 
FULL PROTOCOL TITLE OF THE STUDY 
A mixed methods study to assess the clinical effectiveness and acceptability of qER artificial 
intelligence software to prioritise CT Head interpretation. 
 
 
SHORT STUDY TITLE and ACRONYM  
 
Assess the Clinical Effectiveness in Prioritising CT Heads (ACCEPT) 
 
 
Chief Investigator: 
Haris Shuaib 
Consultant Physicist, Head of Clinical Scientific Computing 
Guy’s and St Thomas’ NHS Foundation Trust (GSTT) 
 
 
Sponsored by: 
Guy’s and St Thomas’ NHS Foundation Trust (GSTT) 
 
Funded by: 
National Institute for Health Research 
 
 
Protocol version number and date: 
1.0, 05.04.2023 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 

 
 
 
GSTFT – Observational Research Protocol template_v1 24.11.2022 
 
[ACCEPT] [IRAS 313507] [Version 1.0] [05/APRIL/2023]                                                                   Page 2 of 54 
 
Name and address of Co-Investigator(s), Trial Manager and key study 
contacts, Funder, Statistician, Laboratories etc.  
 
 
 
Name: 
Professor David Lowe (Lead Collaborator & Co-Investigator) 
Address: 
Consultant Emergency Medicine,  
Queen Elizabeth University Hospital,  
Glasgow 
Telephone: 
 
Email: 
david.lowe@ggc.scot.nhs.uk  
 
 
Name: 
Rachel Fay (Sponsor Representative) 
Address: 
R&D Department, 16 Floor  
Guy’s Hospital  
Great Maze Pond  
London SE1 9RT  
Telephone: 
020 7188 7188  
Email: 
R&D@gstt.nhs.uk 
 
 
Name: 
Sandra Nwokeoha (Funder Representative) 
Address: 
Innovation Manager, Medtech & Digital  
NHS England   
Telephone: 
 
Email: 
sandra.nwokeoha1@nhs.net   
 
 
Name: 
Mariusz Grzeda 
Address: 
Department of Biomedical Engineering, School of Biomedical Engineering 
and Imaging Sciences, King’s College London 
Telephone: 
 
Email: 
Mariusz.grzeda@kcl.ac.uk  
 
 
 
 
 

 
 
 
GSTFT – Observational Research Protocol template_v1 24.11.2022 
 
[ACCEPT] [IRAS 313507] [Version 1.0] [05/APRIL/2023]                                                                   Page 3 of 54 
 
 
 
PROTOCOL VERSION NUMBER AND DATE 
 
Version 
Stage 
Versions No 
Version Date 
Protocol updated & 
finalised by; 
Detail the key protocol 
update 
First 
0.2 
11/01/2023 
Mr Haris Shuaib 
Template update 
Second 
0.3 
29/01/2023 
Mr Haris Shuaib 
Edits 
in 
response 
to 
sponsor review 
Third 
0.4 
08/02/2023 
Mr Haris Shuaib 
Edits 
in 
response 
of 
sponsor review – addition 
of data flow diagram 
Final 
1.0 
05/04/2023 
Mr Haris Shuaib 
Finalisation of Protocol 
with accepted changes. 
 
 
 
 
 
 
 
 
 
 
 
 
 
 

 
 
 
GSTFT – Observational Research Protocol template_v1 24.11.2022 
 
[ACCEPT] [IRAS 313507] [Version 1.0] [05/APRIL/2023]                                                                   Page 4 of 54 
 
 
 
 
SIGNATURE PAGE  
 
The Chief Investigator and the R&D (sponsor office) have reviewed this protocol. The investigators 
agree to perform the investigations and to abide by this protocol.  
The investigator agrees to conduct the trial in compliance with the approved protocol, EU GCP, the UK 
Data Protection Act (2018), the Trust Information Governance Policy (or other local equivalent), the 
UK policy Framework for Health and Social Care research, the Sponsor’s SOPs, and other regulatory 
requirements as amended. 
 
Chief investigator 
Haris Shuaib 
 
 
 
 
 
12/01/2023 
 
Signature 
Date 
 
 
 
 
 
 
 
 
 
 
This Protocol template is intended for use with UK sites only. 
 
 
 
 
 
 
 
 
 

 
 
 
GSTFT – Observational Research Protocol template_v1 24.11.2022 
 
[ACCEPT] [IRAS 313507] [Version 1.0] [05/APRIL/2023]                                                                   Page 5 of 54 
 
 
 
 
 
 
CONTENTS 
 
SIGNATURE PAGE .................................................................................................................................... 4 
CONTENTS ............................................................................................................................................... 5 
1 
LIST OF ABBREVIATIONS AND DEFINITIONS ................................................................................... 8 
2 
SUMMARY/SYNOPSIS .................................................................................................................... 10 
3 
INTRODUCTION ............................................................................................................................. 11 
3.1 
Rationale ............................................................................................................................... 11 
3.2 
Study hypothesis ................................................................................................................... 12 
4 
PATIENT AND PUBLIC INVOLVEMENT ........................................................................................... 12 
5 
TRIAL OBJECTIVES AND PURPOSE ................................................................................................. 12 
5.1 
Definition of Terms ............................................................................................................... 12 
5.2 
Primary Objective.................................................................................................................. 13 
5.3 
Secondary Objectives ............................................................................................................ 13 
6 
STUDY DESIGN & FLOWCHART ..................................................................................................... 14 
6.1 
Study Design.......................................................................................................................... 14 
6.1.1 
Summary ....................................................................................................................... 14 
6.1.2 
Hospital Sites ................................................................................................................. 16 
6.2 
Primary Outcome Measure ................................................................................................... 17 
6.3 
Secondary Outcome Measures ............................................................................................. 17 
6.4 
Technical Evaluation ............................................................................................................. 17 
6.4.1 
Objective ....................................................................................................................... 18 
6.4.2 
Outcome measures ....................................................................................................... 18 
6.5 
Health Economic Assessment ............................................................................................... 18 

 
 
 
GSTFT – Observational Research Protocol template_v1 24.11.2022 
 
[ACCEPT] [IRAS 313507] [Version 1.0] [05/APRIL/2023]                                                                   Page 6 of 54 
 
6.5.1 
Summary ....................................................................................................................... 18 
6.5.2 
Objective ....................................................................................................................... 19 
6.5.3 
Health Economic Data ................................................................................................... 19 
6.6 
Qualitative Evaluation .............................................................. Error! Bookmark not defined. 
6.6.1 
Summary .......................................................................... Error! Bookmark not defined. 
6.6.2 
Objective .......................................................................... Error! Bookmark not defined. 
6.7 
Ground Truth ........................................................................................................................ 20 
7 
PARTICIPANT SELECTION .............................................................................................................. 22 
7.1 
Study Population ................................................................................................................... 22 
7.2 
Participant inclusion criteria ................................................................................................. 22 
7.3 
Participant exclusion criteria ................................................................................................ 22 
8 
STUDY PROCEDURES ..................................................................................................................... 23 
8.1 
Consent ................................................................................................................................. 23 
8.2 
Sampling ................................................................................................................................ 23 
8.3 
Data Collection ...................................................................................................................... 23 
8.4 
National Data opt-out ........................................................................................................... 24 
8.5 
Follow up Procedures ........................................................................................................... 24 
9 
MEDICAL EQUIPMENT (DEVICES) .................................................................................................. 24 
10 
END OF STUDY DEFINITION ....................................................................................................... 25 
11 
ASSESSMENT OF SAFETY  .......................................................................................................... 25 
11.1 
Definitions ............................................................................................................................. 25 
11.2 
Ethics Safety Reporting ......................................................................................................... 27 
11.3 
Study Management ............................................................................................................... 27 
11.3.1 
Trial Steering Committee .............................................................................................. 27 
11.3.2 
Trial Management Group .............................................................................................. 27 
11.3.3 
Data Monitoring Committee ......................................................................................... 27 
11.4 
Ethics & Regulatory Approvals .............................................................................................. 28 

 
 
 
GSTFT – Observational Research Protocol template_v1 24.11.2022 
 
[ACCEPT] [IRAS 313507] [Version 1.0] [05/APRIL/2023]                                                                   Page 7 of 54 
 
11.5 
Protocol Amendments .......................................................................................................... 28 
12 
COMPLIANCE AND WITHDRAWAL ............................................................................................ 28 
12.1 
Participant compliance ......................................................................................................... 28 
12.2 
Withdrawal / dropout of participants ................................................................................... 28 
12.3 
Protocol Compliance ............................................................................................................. 28 
13 
DATA ......................................................................................................................................... 29 
13.1 
Data to be collected .............................................................................................................. 29 
13.2 
Data handling and record keeping ........................................................................................ 29 
13.3 
Data sharing .......................................................................................................................... 29 
13.4 
Information Governance and Security .................................................................................. 29 
13.5 
Personal Data Breaches ........................................................................................................ 30 
14 
MONITORING AND AUDITING................................................................................................... 30 
15 
STATISTICAL CONSIDERATIONS ................................................................................................. 30 
15.1 
Sample Size ........................................................................................................................... 30 
15.2 
Statistical Analysis Plan ......................................................................................................... 31 
16 
PEER REVIEW ............................................................................................................................. 32 
17 
FINANCING ................................................................................................................................ 32 
18 
INSURANCE AND INDEMNITY ................................................................................................... 32 
19 
DATA CONTROLLER ................................................................................................................... 32 
20 
REPORTING AND DISSEMINATION ............................................................................................ 32 
21 
REFERENCES .............................................................................................................................. 35 
APPENDIX 1 ........................................................................................................................................... 36 
APPENDIX 2 ........................................................................................................................................... 37 
APPENDIX 3 ........................................................................................................................................... 39 
APPENDIX 4 ........................................................................................................................................... 40 
 

 
 
 
GSTFT – Observational Research Protocol template_v1 24.11.2022 
 
[ACCEPT] [IRAS 313507] [Version 1.0] [05/APRIL/2023]                                                                   Page 8 of 54 
 
1 LIST OF ABBREVIATIONS AND DEFINITIONS 
AI  
Artificial intelligence  
AUC  
Area under the receiver operating characteristics curve  
CE  
Conformité Européenne  
CI  
Chief investigator  
CIP  
Clinical investigation plan  
CNN  
Convolutional neural network  
CNORIS  
Clinical negligence and other risks indemnity scheme  
CRF  
Case report form  
CRN  
Clinical research network  
CT  
Computed tomography  
CTU  
Clinical trials unit  
CQC  
Care Quality Commission  
DICOM  
Digital imaging and communications in medicine  
DMC  
Data management committee  
DPIA  
Data protection impact assessment  
ED  
Emergency department  
EPR  
Electronic patient record  
FDA  
(US) Food and Drug Administration  
GCP  
Good clinical practice  
GDPR  
General data protection regulation  
GSTT  
Guy’s and St Thomas’ Trust  
HL7  
Health level seven  
HSCN  
Health and social care network  
ICF  
Informed consent form  
IMP  
Investigational medical product  
ISO  
International Organisation for Standardisation  

 
 
 
GSTFT – Observational Research Protocol template_v1 24.11.2022 
 
[ACCEPT] [IRAS 313507] [Version 1.0] [05/APRIL/2023]                                                                   Page 9 of 54 
 
ISRCTN  
International standard randomised control trial number  
LPAC  
Local privacy advisory committee  
MHRA  
Medicines and Healthcare Regulatory Authority  
NCCT  
Non-Contrast Computed Tomography  
NHSGGC  
National Health Service Greater Glasgow and Clyde  
NICE  
The National Institute for Health and Care Excellence  
PACS  
Picture archiving and communication system  
PPI  
Patient and Public Involvement  
QALY  
Quality-adjusted life-year  
QEUH  
Queen Elizabeth University Hospital  
REC  
Research ethics committee  
RIS  
Radiology information system  
SAE  
Serious adverse event  
SADE  
Serious adverse device effect  
SaMD  
Software as a medical device  
SMG  
Study management group  
SOP  
Standard operating procedure  
SSP  
System security policy  
SWAN  
Scottish wide area network  
TFA  
Theoretical framework of acceptability  
TMG  
Trial management group  
TSC  
Trial steering committee  
VPN  
Virtual private network  
 
 

 
 
 
GSTFT – Observational Research Protocol template_v1 24.11.2022 
 
[ACCEPT] [IRAS 313507] [Version 1.0] [05/APRIL/2023]                                                                   Page 10 of 54 
 
2 SUMMARY/SYNOPSIS  
 
Title  
A mixed methods study to assess the clinical effectiveness and 
acceptability of qER artificial intelligence software to prioritise CT Head 
interpretation 
Protocol Short Title/Acronym 
Assess the Clinical Effectiveness in Prioritising CT Heads (ACCEPT) 
IRAS Number 
313507 
REC Reference 
TO BE COMPLETED BEFORE SUBMISSION 
EDGE reference  
141791 
Study Duration 
14 months  
Health condition(s) or problem(s) 
studied 
Radiological Turn-around-time (TAT) of Head-NCCT scan reporting 
Primary objective 
To assess if qER (an artificial intelligence-based prioritisation software 
for non-contrast CT (NCCT) head scans) tool-based reporting and triage 
significantly reduces report turnaround time (TAT) of prioritised NCCT 
head scans for patients attending the emergency department. 
Secondary objective (s) 
• 
To assess utility of qER to support emergency department 
pathways for patients requiring NCCT head and radiology reporting 
workflow. 
• 
To assess the safety of qER at identifying patients with critical 
findings on NCCT heads. 
• 
To evaluate the technical performance of qER.   
• 
To conduct a Heath Economic, cost utility analysis of qER.  
End of study definition  
The end of the study is defined as the end of the 28 days follow-up of 
the last patient undergoing CT within study period. Final data analysis 
shall follow this.   
Number of Participants 
About 16,800 Head-NCCT scans from patients presenting to ED 
department during the study period. 
Study Type 
Observational 
Human Tissue Samples (if applicable) 
Not applicable 
Data collected/storage (if applicable) 
Data will be collected and stored only within the internal participating 
sites 
 
 
 

 
 
 
GSTFT – Observational Research Protocol template_v1 24.11.2022 
 
[ACCEPT] [IRAS 313507] [Version 1.0] [05/APRIL/2023]                                                                   Page 11 of 54 
 
3 INTRODUCTION 
3.1 Rationale 
Emergency Department CT Head Pathway – meeting a need.  
A non-contrast head CT (NCCT) is the first line imaging investigation for patients presenting to 
the emergency department (ED) for a variety of indications including, head injuries, non-
traumatic intracranial haemorrhage and strokes. These scans are predominantly interpreted by 
a radiologist to guide management. Prompt diagnosis results in earlier treatment, reducing brain 
injury, mortality and illness1. Rising ED attendances and concurrent radiologist shortage have 
resulted in increased wait times and workload2,3. This has inadvertently led to higher demands for 
shorter report turnaround time in an effort to streamline throughput and decrease healthcare 
expenditures. Furthermore, for time-critical diagnoses like head injuries and strokes, an artificial 
intelligence (AI) tool which prioritises certain patients’ NCCTs for earlier attention, could improve 
triage and improve patient outcome. Therefore, integrating such a tool in the radiologist 
workflow could allow for more rapid diagnosis and reporting of these critical conditions, 
enhancing the triage of patient the appropriate level of care.   
The pressures on radiology departments  
Diagnostic imaging plays a critical role in the management of ED patients and delays related to 
imaging are associated with longer hospital stays. As demand for acute care has risen and imaging 
equipment becomes more readily available, there has been a sustained rise in the demand for 
acute imaging. An independent review of the NHS England diagnostic service conducted by Sir 
Mike Richards has recommended doubling the number of scanners to reduce delays4.   
Currently, most scans are interpreted by radiologists, but the Royal College of Radiologists (RCR) 
are predicting a shortage of 2000 NHS radiologists by 20235. Image reporting turnaround times 
are now a major bottleneck for EDs. An RCR national audit showed <50% of ED patients receive 
their scan reports within the recommended time6 and only 2% of radiology departments can fulfil 
their reporting requirements within contracted hours. 
AI – the opportunity  
The potential applications of AI in radiology go well beyond image analysis for diagnostic and 
prognostic opportunities. It is becoming increasingly clear that AI algorithms have the potential 
to improve productivity, operational efficiency, and accuracy in diagnostic radiology. AI tools are 
being developed to aide diagnosis and enhance processes at multiple point in the radiology 
workflow including: (a) protocolling the prioritised scan, (b) clinical decision support systems for 
detection of critical findings, (c) worklist priority adjustment via AI results, and (d) reducing 
turnaround time through worklist prioritisation and semiautomated structures reporting. The 
adoption of AI tools is dependent on the demonstration of a tangible effect on patient care and 
improvement in radiologist workflow. Thus, in this study, we aim to assess whether real-world 
implementation of an AI tool which augments (b), (c) and (d) of the imaging life cycle would affect 
turnaround times.   
  

 
 
 
GSTFT – Observational Research Protocol template_v1 24.11.2022 
 
[ACCEPT] [IRAS 313507] [Version 1.0] [05/APRIL/2023]                                                                   Page 12 of 54 
 
qER  
qER, a CE Class II approved medical software device, detects, and localizes the presence of six 
target abnormalities - intracranial haemorrhage, cranial fracture, midline shift, mass effect, 
atrophy and hypodensities suggestive of infarcts in non-contrast Head-CT scans7. A priority status 
is assigned if any one of the target abnormalities (intracranial haemorrhage, cranial fracture, 
midline shift or mass effect) is detected by the software, and the user will be able to view a single 
summary slice listing all the target abnormalities found by qER on the CT scan followed by all 
slices in scan with the overlay of above abnormalities localization. Alternately, if none of the 
target abnormalities are detected, the output will indicate that the software has analysed the 
image and identified no critical findings. qER reports are intended to support certified radiologists 
and/or licensed medical practitioners for clinical decision making. It is a support tool and, when 
used with original scans, can assist the clinician to improve efficiency, accuracy, and turnaround 
time in reading head CTs. It is not to be used to provide medical advice, determine treatment 
plan, or recommend a course of action to the patient.   
3.2 Study hypothesis 
Implementation of the qER product will reduce time to reporting of prioritised NCCT head 
findings requested through the Emergency Department and improve radiology reporting 
workflow, enabling improved Emergency Department clinical pathways for patients requiring 
NCCT imaging.  
4 PATIENT AND PUBLIC INVOLVEMENT  
 
Patient representatives and groups have contributed to protocol writing and will continue to be 
integral members of the team for wider dissemination. PPI leads will also be part of the Trial 
Management Group (TMG). 
5 TRIAL OBJECTIVES AND PURPOSE 
The purpose of this research is to test the hypothesis that implementation of the qER product in 
Emergency Department will reduce time to reporting of critical NCCT head findings requested 
through the Emergency Department and improve radiology reporting workflow, enabling 
improved Emergency Department clinical pathways for patients requiring NCCT imaging.  
5.1 Definition of Terms 
For the purpose of this study, there are custom terms specific to the qER device which are used 
throughout this document. The table below lists these terms and the corresponding definitions.  
 
 

 
 
 
GSTFT – Observational Research Protocol template_v1 24.11.2022 
 
[ACCEPT] [IRAS 313507] [Version 1.0] [05/APRIL/2023]                                                                   Page 13 of 54 
 
Term  
Definition  
Target abnormality  
This refers to the list of all target abnormalities that qER can 
detect in a Head NCCT scan. These are: 
• 
Intracranial haemorrhage  
• 
Midline shift  
• 
Mass effect  
• 
Cranial fracture  
• 
Atrophy  
• 
Hypodensities suggestive of infarct  
qER prioritised 
findings  
A subset of target abnormalities which when identified in a 
Head NCCT scan by qER will lead to the prioritisation of such 
scans in the radiology worklist.   
• 
Intracranial haemorrhage  
• 
Midline shift  
• 
Mass effect  
• 
Cranial fracture  
qER non-prioritised 
findings  
A subset of target abnormalities which are detected by qER but 
not in the list of qER-prioritized findings: 
• 
Atrophy  
• 
Hypodensities suggestive of infarct  
These scans will not get prioritized in the radiology worklist for 
interpretation by the Trust radiologist but will still be available 
for interpretation in the worklist in a non-prioritized manner.  
qER no findings   
  
Any Head NCCT scans where none of the target abnormalities 
are identified by qER will be classified as scans where no qER 
findings are identified  
qER not interpreted  
Head NCCT scans which were not processed by qER and thus 
have no AI outputs  
 
5.2 Primary Objective 
To assess if qER (an artificial intelligence-based prioritisation software for Head-NCCT scan 
findings) tool-based reporting and triage significantly reduces report turnaround time (TAT) 
of prioritised NCCT head scans for patients attending the emergency department. 
5.3 Secondary Objectives 
• 
To assess utility of qER to support emergency department pathways for patients requiring 
NCCT head and radiology reporting workflow. 
• 
To assess the safety of qER at identifying patients with critical findings on NCCT heads  
• 
To evaluate the technical performance of qER   
• 
To conduct a Heath Economic, cost utility analysis of qER.  

 
 
 
GSTFT – Observational Research Protocol template_v1 24.11.2022 
 
[ACCEPT] [IRAS 313507] [Version 1.0] [05/APRIL/2023]                                                                   Page 14 of 54 
 
6 STUDY DESIGN & FLOWCHART 
6.1 Study Design 
6.1.1 
Summary  
A multi-centre stepped wedged cluster randomised study (Figure 1) will be conducted in 4 
NHS hospitals over a 13-month period. Hospitals will be identified and initiated into the 
qER solution with a 30-day implementation period. The order in which sites will receive the 
qER intervention will be determined by computer-based randomisation. The stepped 
wedge design allows delivery of the intervention at an organisational level with evaluation 
of outcome measures at a patient level. Structuring the implementation through a staged 
activation in a random order provides important methodological advantages for both 
qualitative and quantitative elements of the study. The design allows control of adoption 
bias and adjust for time-based changes in the background patient characteristics at a 
patient level.  
 
Figure 1: illustration of site initiation within the stepped-wedge design. Each square block represents a duration of 1 month. 
6.1.1.1 
Pre-Implementation 
During the pre-implementation phase, data will be collected to support the primary and 
secondary outcomes as this will constitute the baseline data for analysis. 
6.1.1.2 
Implementation 
The thresholds for detecting target abnormalities will be adjusted, if necessary, during the 
implementation phase as per the deployment SOP for qER. Data from this implementation 
phase will not be used for analysis of primary and secondary outcomes pertaining to the 
utility of qER.  
6.1.1.3 
Post-Implementation 
The implementation of new healthcare practices and the adoption of AI technologies is not 
linear. Study design and the elements of the study will enable evaluation of broadly the 
performance of the AI technology, acceptability, and the economic costs of adoption.   
This randomised stepped wedge healthcare service delivery study to assess the clinical 
effectiveness of qER to prioritise patients reporting of NCCT scans that have prioritised 
findings (identified from AI analysis of a CT). The study will have an unaided (without qER 

 
 
 
GSTFT – Observational Research Protocol template_v1 24.11.2022 
 
[ACCEPT] [IRAS 313507] [Version 1.0] [05/APRIL/2023]                                                                   Page 15 of 54 
 
implemented) and aided phase (with qER implemented). During the unaided phase the 
reporting of CT will follow the same workflow as in the current standard of care (i.e., the 
images/cases will appear in the RIS chronologically and the radiologist either follows this 
order or prioritise some cases based on communication from ED). When the radiologist clicks 
a case in RIS, a secondary capture of qER along with the original images will be available in 
PACS.  
During the post-implementation phase there will be a notification (prioritised flag) in RIS. 
The order of the cases in RIS will not be altered. When the radiologist clicks a case in RIS, a 
secondary capture of qER along with the original images will be available in PACS. This 
secondary capture will have contour showing the algorithm’s attention point. The radiologist 
can choose to agree with qER findings as it or modify or ignore it according to their clinical 
judgement, writing and signing off the report. For scans which were not processed by qER 
the radiologist can prioritise and report as per the standard of care.  
For the purposes of technical evaluation of qER, the scans taken during the aided phase will 
be divided into four categories: 
• 
Scans with qER prioritised findings (prioritised pathway)  
• 
Scans with qER non-prioritised findings (qER non-prioritised pathway)   
• 
Scans not flagged as having no qER findings (qER no finding pathway), and  
• 
Not interpreted (scans not processed by qER, Not interpreted pathway)  
 
In the qER prioritised, qER non-prioritised and qER no finding pathway, all cases where the 
radiologist did not agree with qER findings will be sent for ground truthing (2+1 method, Figure 
2). A random sample of 500 scans will also be sent for ground-truthing process for the purpose 
of technical evaluation. 
 
Figure 2: Post-implementation workflow 
 

 
 
 
GSTFT – Observational Research Protocol template_v1 24.11.2022 
 
[ACCEPT] [IRAS 313507] [Version 1.0] [05/APRIL/2023]                                                                   Page 16 of 54 
 
 
Figure 3: Example of current and proposed pathways for patients undergoing Head-NCCT scans from the Emergency 
Department.  
 
“qER finding” workflow.  
This will involve scans where qER identifies one or more of the target abnormalities which 
includes prioritised and non-prioritised findings. qER offers the opportunity to optimise NCCT 
finding detection. The study will integrate qER into ED clinical pathways (Figure 3). Images 
from ED pathway will be interpreted by qER and those identified to having the presence of 
any one or more of the prioritised findings will be flagged to radiologist for ‘prioritised’ 
reporting (allocated to duty radiologist). Along with the prioritised flag, radiologists can be 
provided with a secondary DICOM series with a contour that outlines the abnormality 
identified by qER. Following review, a radiology report will be generated and prioritised 
findings communicated to the ED clinician through existing escalation pathways. For scans 
where only non-prioritised finding have been detected by qER, they will also be flagged 
appropriately so that the reporting radiologist is informed of the same. 
            “No qER findings” workflow  
qER also supports the workflow by ruling out target abnormalities. All images identified as 
absence of finding will be reviewed and reports verified.   
6.1.2 
 Hospital Sites   
Participating hospitals must undertake a significant number (3000) of NCCT Head within the 
Emergency Department per annum. Specialty leads from radiology and Emergency 
Department will be identified with support from their NHS Trust/Board to participate in the 
deployment of qER. Hospitals that have already deployed qER or similar AI-enhanced NCCT 
head will be excluded. Sites were identified that provide good representation of potential 
heterogeneity (geographical, population, ethnicity, population age).  
An independent statistician will perform a computerised randomisation of the clusters and 
keep randomisation records prior to the beginning of the data collection. Simple 
randomisation will be used to randomise one trust to receive the intervention in each of the 

 
 
 
GSTFT – Observational Research Protocol template_v1 24.11.2022 
 
[ACCEPT] [IRAS 313507] [Version 1.0] [05/APRIL/2023]                                                                   Page 17 of 54 
 
time periods. Local investigators will be notified ~6 weeks in advance of activation qER tool at 
their hospital.  
6.2 Primary Outcome Measure 
Time taken to report NCCT head from acquisition for patients with prioritised findings in 
Emergency Department compared to standard of care.   
6.3 Secondary Outcome Measures 
Utility of qER compared to standard of care:  
• 
Time taken from acquisition to report NCCT head for patients without prioritised 
findings in Emergency Department  
• 
Time taken from acquisition to report completed for patients with absence of findings 
on NCCT head scans in Emergency Department  
• 
To assess impact of qER on radiology reporting workflow on other requests for CT 
scans  
• 
To assess impact of teleradiology on qER supported reporting    
• 
Time to diagnosis from CT acquisition.  
• 
Time to referral from CT acquisition.  
• 
Time to initiation of treatment from CT acquisition for prioritised scans.  
• 
Death within 28 days of NCCT head acquisition  
• 
Percentage of NCCT heads that qER classifies as prioritised and absence of findings.  
• 
Percentage of qER non prioritised scans that radiologist identified as having prioritised 
finding   
• 
Percentage of qER non prioritised but identified by radiologist as absence of finding.   
 
Technical Evaluation of product performance:  
• 
Sensitivity, specificity, positive and negative predictive values of qER in detecting 
scans with prioritized findings overall and also stratified by all 6 target abnormalities.  
• 
Percentage of CT scans that could not be processed by qER due to technical factors.  
 
Safety of qER:  
• 
To assess the safety of qER of identifying patients with no prioritised findings 
identified on NCCT head.  
• 
Percentage of patient NCCT Head reported incorrectly as qER non prioritised.   
• 
Time from acquisition to reporting of CT Chest Abdo Pelvis (CAP) and Pulmonary 
Angiograms (CTPA) within the trust.  
 
Health economic assessment:  
• 
Compare costs and health benefits between pre and post implementation of qER, 
including cost evaluation of fully automatic diagnosis of high confidence normal 
triage.  
6.4 Technical Evaluation 
The technical retrospective study will determine performance of the algorithm in each of 
the 4 sites. 500 NCCT scans will be identified for analysis from each site for technical 
evaluation. Sampling will be stratified by age, sex, month of year and target abnormalities.  

 
 
 
GSTFT – Observational Research Protocol template_v1 24.11.2022 
 
[ACCEPT] [IRAS 313507] [Version 1.0] [05/APRIL/2023]                                                                   Page 18 of 54 
 
 
The use of retrospective images and data has been approved by each site’s Caldicott 
Guardian. In addition, approval for the release of anonymised patient data for research 
purposes forms part of the ethics application through IRAS.   
6.4.1 
Objective 
• 
Evaluate performance of qER  
6.4.2 
Outcome measures   
• 
Sensitivity, specificity, positive and negative predictive values of qER in detecting 
scans with prioritized findings overall and also stratified by all 6 target 
abnormalities.  
• 
AUC of qER in detecting presence of 6 target abnormalities  
6.5 Health Economic Assessment 
6.5.1 
Summary 
An economic evaluation will be conducted comparing costs and outcomes of a current 
provision of care in the NHS and one inclusive of qER. qER potentially impacts costs and 
benefits via two mechanisms: (1) the identification of critical findings can improve 
radiology reporting times; and (2) the identification of critical findings can support 
optimisation of patients clinical care, accelerating the treatment and referral pathways, 
and ultimately better outcomes.  
The proposed health economic approach is a cost effectiveness analysis where both the 
clinical and financial impact of qER will be captured. This approach is chosen in 
accordance with the guidance in NICE’s Evidence Standards Framework (ESF) for digital 
health interventions (reference). The ESF recommends the development of a cost 
effectiveness analysis to demonstrate the value of digital health interventions that are 
deemed to be higher financial risk. qER is categorised as a tier C intervention and is 
considered to be a high financial risk since the aim is to commission qER on large scale 
or potential nationwide adoption.   
A cost utility model will be developed that allows extrapolation from clinical evaluation 
endpoints to estimates of incremental costs and quality adjusted life years (QALYs), 
comparing the use of qER with standard of care. It is likely the model will adopt a 
lifetime time horizon to capture the potential benefits qER may have on outcomes such 
as patient survival, surrogates of patient survival and/or long-term disability.   
In order to capture such value components in the model, the model requires additional 
parameter estimates describing prognosis, and the relationship between ‘time to 
treatment initiation’ and treatment outcomes, to enable the extrapolation from the 
endpoints to overall survival (or a surrogate of overall survival), treatment costs and 
QALYs. The model will only include direct medical costs associated with qER and 
standard of care as well as implementation and maintenance costs necessary for the 
integration of qER as outlined in NICE’s ESF.  

 
 
 
GSTFT – Observational Research Protocol template_v1 24.11.2022 
 
[ACCEPT] [IRAS 313507] [Version 1.0] [05/APRIL/2023]                                                                   Page 19 of 54 
 
The results of the model will include scenario and sensitivity analysis (both probabilistic 
and deterministic). These will be founded on alternative variations of parameter 
utilisation and care pathway design identified through engagement with the clinical 
advisory team during external validation. For example, a comparison of trusts with the 
presence of stroke centres and those not.   
The following steps will be undertaken to develop the model:  
A targeted literature review of previous published health economic studies evaluating 
head CT.  
Using the findings from the evidence review, the appropriate model framework will be 
chosen for the cost utility model e.g., a Markov model, or decision tree. The model 
pathway will replicate the current standard of care and capture key therapy area 
characteristics. The identified model pathway will be tested with key opinion leaders 
(both clinical and economic).  
 A further evidence review will be completed to identify any required data that will not 
be collected in the prospective or retrospective clinical study e.g., health utilities.   
A prototype of the economic model will be built in Microsoft Excel, incorporating the 
agreed data points with a focus upon key outputs of interest. The prototype will be 
completed in line with the interim results of the prospective and retrospective study to 
produce early cost effectiveness results. Updates to the model structure may be made 
to refine the economic value story.  
A complete version of the economic model will be produced using results of the clinical 
trials. Results will be tested with clinical experts as well as an external health economist 
to the primary modelling team.   
The results and version of the model will be documented along with a model quality 
assurance report.  
Methods and results to be presented in a peer-reviewed manuscript suitable for 
submission to a peer reviewed journal.  
6.5.2 
Objective 
Compare costs and health benefits between pre and post implementation of qER and 
the associated ratios between the two to examine whether the ratio meets willingness 
to pay thresholds observed in England (i.e. £20,000 to £30,000 per QALY gained).  
6.5.3 
Health Economic Data 
The data elements that will be used for the analysis of health economic assessment are 
listed below: 
Intervention and comparator   
• 
Numbers of ground truth positive and ground negative cases  
• 
Al false positives, true positives, false negatives, true negatives  

 
 
 
GSTFT – Observational Research Protocol template_v1 24.11.2022 
 
[ACCEPT] [IRAS 313507] [Version 1.0] [05/APRIL/2023]                                                                   Page 20 of 54 
 
• 
AI output (+/- per disease)  
• 
Radiologist output and/or agreement with AI decision per scan  
• 
Time of admission to ED  
• 
Time (with and without qER) of CT scan  
• 
Time (with and without qER) time to ER physician viewing scan  
• 
Time (with and without qER) to radiologist open scan  
• 
Time (with and without qER) to radiologist report / close scan  
• 
Time (with and without qER) ER physician opens report  
• 
Time (with and without qER) to onwards referral  
• 
Time (with and without qER) to treatment (need specific treatment per speciality 
pathway e.g. time to embolisation, time to surgery etc)   
• 
Time to AI output delivered to PACS  
  
Resource Use  
• 
Number of head CTs performed out of hours  
• 
Number of head CTs performed in hours  
• 
Definition of in and out of hours per trust  
• 
Reason for the scan (primary suspected diagnosis and/or main symptom)  
• 
Death  
• 
Reattendance to hospital with same presentation  
• 
Transfer to other hospital  
• 
Admission to same hospital, including care pathway  
• 
Clinical diagnosis, including sub-type and severity of (e.g ASPECTS score for stroke, 
any relevant head trauma scales)  
• 
False negative outcomes (e.g. missed diagnoses and severity)  
• 
False positive outcomes (e.g. misdiagnosis and severity)  
• 
Length of stay and wait in the ED  
• 
Length of stay in hospital / wait to transfer  
 
Costs  
• 
AI implementation, maintenance, training, per scan costs  
• 
Treatment cost per pathway/speciality (to be sourced from published literature)  
• 
ED visit cost (to be sourced from published literature)  
• 
Cost of hospital admission (to be sourced from published literature)  
• 
Cost of hospital transfer (to be sourced from published literature)  
• 
Cost by ward type (to be sourced from published literature)  
• 
Cost of outsourcing CT head reading in and out of hours (to be sourced from 
published literature)  
 
Utilities  
• 
Will be applied to patient outcomes (values to be sourced from published 
literature)  
 
6.6 Ground Truth 
Both the prospective (stepped wedge) and retrospective (technical evaluation) 
elements of this study compare the performance of qER against ground truth and 

 
 
 
GSTFT – Observational Research Protocol template_v1 24.11.2022 
 
[ACCEPT] [IRAS 313507] [Version 1.0] [05/APRIL/2023]                                                                   Page 21 of 54 
 
ground truthing is required for confirming which scans contain a prioritised finding and 
not.  
The following prospective scenarios (i.e. from the stepped-wedge study) will require 
independent generation of ground truth by two post-fellowship radiologists (and an 
arbitrating senior radiologist, where there is interobserver disagreement):  
• 
All images where radiologist disagrees with the qER assessment. 
  
Head-NCCT scans where original reporting radiologist agreed with Qer assessment will 
not be sent for ground truthing and the original report will be considered as the ground 
truth for such scans. 
All NCCT head images will be reviewed by trust radiologist. Those that the trust 
radiologist disagrees will require independent generation of ground truth by two post-
fellowship radiologists (and an arbitrating senior radiologist, where there is 
interobserver disagreement). 
An image will be presented to ground truth radiologists only after the result has been 
generated by qER. Images from all sources will be presented in a single list for ground 
truth generation, so that radiologists cannot identify the source (i.e., whether the 
ground truth is necessary because there was a discrepancy, or not). All images will be 
deidentified to ensure the radiologist cannot identify the patient and thereby view the 
corresponding AI result or seek additional supporting clinical data to make a ground 
truth decision.  
 
Figure 4: Ground truth for retrospective data. 

 
 
 
GSTFT – Observational Research Protocol template_v1 24.11.2022 
 
[ACCEPT] [IRAS 313507] [Version 1.0] [05/APRIL/2023]                                                                   Page 22 of 54 
 
 
 
Figure 5: Ground truth for prospective data. 
 
7 PARTICIPANT SELECTION 
 
7.1 Study Population  
At each of the four participating sites, we will identify all patients referred through the 
Emergency Department NCCT requests. Expected number of eligible participants and 
rationale behind sample sizing are detailed in the ‘STATISTICAL CONSIDERATIONS’ section. 
7.2 Participant inclusion criteria 
• 
Individuals undergoing Head CT scan at the ED / A&E (Accident and Emergency 
Services). 
• 
Age ≥ 18 years.  
• 
Non-contrast axial CT scan series with consistently spaced axial slices.  
• 
Soft reconstruction kernel covering the complete Brain.   
• 
Maximum slice thickness of 6mm.  
7.3 Participant exclusion criteria 
• 
There are no explicit exclusion criteria for qER as all scans in inclusion criteria will 
be processed by qER.  Exclusion criteria are implicit within the inclusion criteria 
listed above. 

 
 
 
GSTFT – Observational Research Protocol template_v1 24.11.2022 
 
[ACCEPT] [IRAS 313507] [Version 1.0] [05/APRIL/2023]                                                                   Page 23 of 54 
 
8 STUDY PROCEDURES 
8.1  Consent 
The study intervention is randomised and applied at an institutional level, and therefore 
individual patient consent is not required. We will not seek individual patient consent.   
Data will be captured by the direct care team and anonymised before transfer to central 
research team at GSTT. Approval for the release of anonymised patient data for research 
purposes is integral to the ethics application through IRAS.  
Under UK GDPR a patient may object to their data being used under certain circumstances. 
Should a patient involved in this study object to their data being used, then this will be 
processed by the site’s information governance team using local standard operating 
procedures. If the objection is upheld, then the patient’s data will be removed from the study.  
qER solution is deployed through individual site agreements with qER, which include a local 
data processing agreement. Data sharing between sites and GSTT is governed by individual 
site data sharing agreements.  
GP practices in the trust will be notified of the change in pathway through communication 
with each practice within the trust (GP letter in the Appendix).   
Posters will be displayed within each radiology and emergency department detailing the 
deployment of qER to support radiology reporting (NHS service user/patient poster). Leaflets 
providing additional detail of the tool will be available on request (supporting patient 
information leaflet).   
A patient co-investigator has reviewed early drafts of this protocol and we are planning on 
working with PPI groups in London and Oxford to provide further feedback on patient 
experience.  
8.2 Sampling 
During the pre-implementation period, data will be gathered on the current clinical pathway. 
Each site will provide data on volume of images and time periods from referral to acquisition 
and reporting of NCCT heads. This data collected during pre-implementation period will serve 
as the baseline data for final study analysis. 
8.3 Data Collection 
Evaluation sites will periodically collect, clean, link, de-identify and supply data in an agreed 
format, beginning with baseline data from the pre-implementation phase until the end of the 
post-implementation phase. The frequency of data collection is subject to change though 
depending on the feasibility of sites. 
 
A data dictionary will be supplied (example included in the Appendix) that will support fulfilling 
these requirements from:  
        Extraction of routinely collected hospital data:  
• 
Referral Data: Age, Sex, date of referral for CT, date of appointment. Where 
possible ethnicity and deprivation will be captured.  

 
 
 
GSTFT – Observational Research Protocol template_v1 24.11.2022 
 
[ACCEPT] [IRAS 313507] [Version 1.0] [05/APRIL/2023]                                                                   Page 24 of 54 
 
• 
Radiology Data: Number of referrals, Datetime CT acquired, Datetime CT 
reported, presence or absence of discordance between qER and reporting 
radiologist, CT device manufacturer/ type, Usual Care Report  
• 
Clinical Data: Clinical Diagnosis, treatment initiation, referral, including date of 
death if applicable.  
• 
Follow-up Data: Length of stay ED, Length of stay Hospital, Specialty pathway.  
• 
qER data outputs from qER interpretation  
• 
Changes in treatment pathways after implementation    
• 
Time to diagnosis  
         
Data linkage for follow-up (death within 28 days of Head-NCCT acquisition date) against 
outcomes will be collected for 28 days. 
8.4 National Data opt-out.  
 
 If patient has selected National Data opt-out, their data will be excluded from final analysis. 
8.5 Follow up Procedures. 
 
Data linkage for follow-up (death within 28 days of Head-NCCT acquisition date) against 
outcomes will be collected 28 days after the Head-NCCT acquisition date. 
 
9 MEDICAL EQUIPMENT (DEVICES) 
 
qER, a CE Class II approved medical software device, detects, and localizes the presence of six 
target abnormalities - intracranial haemorrhage, cranial fracture, midline shift, mass effect, 
atrophy and hypodensities suggestive of infarcts in non-contrast Head-CT scans7. It uses pre-
trained artificial intelligence algorithms to detect the presence or absence of the target 
abnormalities in Head-NCCT scans. This device has not been modified for the purpose of this 
research and will be used within its intended use for this study. The device will be deployed in all 
the four participating sites for the purpose of this study.  
A priority status is assigned if any one of the target abnormalities (intracranial haemorrhage, 
cranial fracture, midline shift or mass effect) is detected by the software, and the user will be able 
to view a single summary slice listing all the target abnormalities found by qER on the CT scan 
followed by all slices in scan with the overlay of above abnormalities localization. Alternately, if 
none of the target abnormalities are detected, the output will indicate that the software has 
analysed the image and identified no critical findings. qER reports are intended to support 
certified radiologists and/or licensed medical practitioners for clinical decision making. It is a 
support tool and, when used with original scans, can assist the clinician to improve efficiency, 
accuracy, and turnaround time in reading head CTs. It is not to be used to provide medical advice, 
determine treatment plan, or recommend a course of action to the patient.   
This study has been awarded funding from NHSx AI Award (Award reference: AI_Award02354).  

 
 
 
GSTFT – Observational Research Protocol template_v1 24.11.2022 
 
[ACCEPT] [IRAS 313507] [Version 1.0] [05/APRIL/2023]                                                                   Page 25 of 54 
 
Funding for the study team to deploy and use qER for the study purpose, to perform its analysis 
of the data gathered from sites, and to produce a final report, is confirmed under the research 
contract between Qure.ai (manufacturer of qER) and Guys and St Thomas’ NHS Foundation Trust. 
 
10 END OF STUDY DEFINITION 
The end of the study is defined as the end of the 13th month following the study start date. 
Final data analysis shall follow this.   
11 ASSESSMENT OF SAFETY  
11.1 Definitions 
Adverse events (AE)  
An adverse event is any untoward medical 
occurrence, unintended disease or injury, or 
untoward clinical signs (including abnormal 
laboratory findings) in subjects, users, or other 
persons, whether related to the investigational 
medical device or not.  
   
Note 1 to entry: This definition includes events 
related to the investigational medical device or the 
comparator.  
Note 2 to entry: This definition includes events 
related to the procedures involved.  
Note 3 to entry: For users or other persons, this 
definition is restricted to events related to the use of 
investigational medical devices.  
   
Serious Adverse Event  
An adverse event that led to any of the following:  
a. 
death,  
b. 
serious deterioration in the health of the 
subject, users or other persons as defined by one or 
more of the following:  
 
1. 
a life-threatening illness or injury, or  
2. 
a permanent impairment of a body 
structure or a body function including 
chronic diseases, or  
3. 
in-patient or prolonged 
hospitalisation, or  
4. 
medical or surgical intervention to 
prevent life-threatening illness or injury or 
permanent impairment to a body structure 
or a body function,  
c. 
foetal distress, foetal death or a congenital 
abnormality or birth defect including physical or 
mental impairment.  
  

 
 
 
GSTFT – Observational Research Protocol template_v1 24.11.2022 
 
[ACCEPT] [IRAS 313507] [Version 1.0] [05/APRIL/2023]                                                                   Page 26 of 54 
 
Note 1 to entry: Planned hospitalisation for a pre-
existing condition, or a procedure required by the 
protocol, without serious deterioration in health, is 
not considered a serious adverse event.  
   
Serious adverse device event (SADE)  
A serious adverse device effect (SADE) is any adverse 
device effect that has resulted in any of the 
consequences characteristic of a serious adverse 
event. A SADE may be anticipated or unanticipated.  
Unanticipated Serious Adverse Device 
Effect (USADE)  
Unanticipated serious adverse device effect (USADE) 
is a serious adverse device effect which by its nature, 
incidence, severity or outcome has not been identified 
in the current version of the risk analysis report.  
Recording and Reporting of Adverse Events 
  
AEs must be recorded, assessed, reported, analysed, and managed in accordance with the 
Research Governance Framework for Health and Community Care and the study protocol. All 
AEs must be assessed for seriousness. AEs for this study will be recorded within the patient 
notes but are not considered reportable.  
 
Recording and Reporting of Serious Adverse Events 
  
Where an SAE requires recording; full details including the nature of the event, start and stop 
dates, severity, causal relationship to the device and/or trial procedures, and the outcome of 
the event will be recorded in the patient’s medical notes. These events will be monitored and 
followed up until satisfactory resolution and stabilisation.   
 
Assessment of causality  
I.e., does the event have a “reasonable causal relationship” with any trial specific procedures 
or the investigational device?  
The assessment of causality must be carried out by the PI or other medically qualified local 
investigator.  
 
Assessment of expectedness  
If the SAE is considered related to any trial specific procedures or the investigational medical 
device an assessment should be made of the expectedness i.e., is the SAE a recognised adverse 
effect of the investigational medical device or a trial procedure.   
 
Reporting to Sponsor  
The following events defined as USADEs are considered reportable to the sponsor.  
 
• 
Any Serious Adverse Event that is considered related to the trial specific 
procedures, or the medical device(s) that is considered unexpected by the 
Chief Investigator or their delegate.  
• 
Any device related serious adverse event that led or may have led to one of 
the following outcomes:  
• 
The death of a patient  
• 
A serious deterioration in the health of a patient.   
 
A serious deterioration in health may include (non-exhaustive):  
 
• 
Life threatening illness  

 
 
 
GSTFT – Observational Research Protocol template_v1 24.11.2022 
 
[ACCEPT] [IRAS 313507] [Version 1.0] [05/APRIL/2023]                                                                   Page 27 of 54 
 
• 
Permanent impairment of a body function or permanent damage to a body 
structure  
• 
A condition necessitating medical/surgical intervention to prevent a) or b)  
• 
Foetal distress or death, or any congenital anomalies or birth defects.  
 
SAEs meeting the above criteria must be reported to the Pharmacovigilance (PV) Office 
immediately (within 24 hours) using the SAE form for a non CTIMP. The SAE form should be 
completed and signed by appropriately delegated staff. The form should be e-mailed to the 
PV Office and a copy placed in the Study Site File.  
 
If all the required information is not available at the time of initial reporting, the CI (or 
designee) must ensure that any missing information is forwarded to the PV Office as soon as 
this becomes available. The report should indicate that this information is follow-up 
information for a previously reported event.   
11.2 Ethics Safety Reporting 
Reporting of USADEs to the REC and IRIC   
The Sponsor will report all USADEs to the ethics committee within 15 days of the PV office 
becoming aware of the event, via the ‘report of serious adverse event form’ for non-CTIMPs 
published on the Health Research Authority web site.  
 
The form will be completed in typescript and signed by the Chief Investigator.  The CI or 
delegate will submit reports relating to the medical device that meets the above criteria to 
the Incident Reporting and Investigation Centre (IRIC – part of NHS National Services 
Scotland).  
 
Annual reports to REC  
Annual progress reports will be made on the anniversary of the original REC approval. 
 
Once the study has ended, we will declare the end of study to the Research Ethics committee 
(REC) that gave a favourable opinion within 90 days of the study ending using the appropriate 
method. This will be followed up with the final report within 12 months of the end of the 
study.  
11.3 Study Management 
11.3.1 Trial Steering Committee 
No Trial Steering Committee is required for this study. 
11.3.2 Trial Management Group 
A Trial Management Group (TMG) will convene consisting of the representatives from 
the Sponsor (GSTT) including the CI, co-investigators, statisticians, PPI leads and the 
Study Manager (including project managers from each participating Trust). The role of 
the group is to monitor all aspects of the conduct and progress of the trial, ensure that 
the protocol is adhered to and take appropriate action to safeguard participants and 
the quality of the trial itself.  
11.3.3 Data Monitoring Committee 
No Data Monitoring Committee is required for this study. 

 
 
 
GSTFT – Observational Research Protocol template_v1 24.11.2022 
 
[ACCEPT] [IRAS 313507] [Version 1.0] [05/APRIL/2023]                                                                   Page 28 of 54 
 
11.4 Ethics & Regulatory Approvals 
Prior to commencing the study, approval will be sought from an appropriate REC. The research 
will be undertaken in accordance with the principles of Good Clinical Practice and the 
Declaration of Helsinki. The CI will be responsible for updating the Ethics committee of any 
new information related to the study.  
11.5 Protocol Amendments 
Any change in the study protocol will require an amendment. Any proposed protocol 
amendments will be initiated by the CI following discussion with the sponsor and any required 
amendment forms will be submitted to the ethics committee and sponsor.   
 
The CI will liaise with study sponsor to determine whether an amendment is non-substantial 
or substantial. All amended versions of the protocol will be signed by the CI and Sponsor 
representative.   
 
Before the amended protocol can be implemented favourable opinion/approval must be 
sought from the original reviewing research ethics committee (REC) and Research and 
Development (R&D) office.  
 
12 COMPLIANCE AND WITHDRAWAL 
12.1 Participant compliance 
Not applicable for this study as data is collected only from the existing data records. Specific 
study participant interviews are not required for the study data collection and there will be 
no change in standard of care.  
12.2 Withdrawal / dropout of participants 
 
If patient has selected National Data opt-out, their data will be excluded from final analysis. 
Patients who wish to withdraw their data from being used in this research will be removed 
from analysis prior to anonymisation. Data extracted will be destroyed and the patient 
withdrawn from this part of the study if requested. The contact information for the research 
team will be made available to patients through posters and other publications within the 
department to allow them to express their wish to withdraw their data from the research. 
12.3  Protocol Compliance  
The CI will monitor any protocol deviations and list them in a deviation log /include a file note 
in the TMF/Site file where applicable. Significant deviations to the protocol or deviations which 
are found to frequently recur will be assessed by the CI to see if an amendment to the Protocol 
is required. These will be reported to the sponsor and action taken through Corrective and 
preventative Actions (CAPA). The CI and sponsor will be notified immediately of a serious 
breach, where applicable. The Breach will be reported to the REC Committee with the Sponsor 
in copy within 7 calendar days of the breach being confirmed as serious.  

 
 
 
GSTFT – Observational Research Protocol template_v1 24.11.2022 
 
[ACCEPT] [IRAS 313507] [Version 1.0] [05/APRIL/2023]                                                                   Page 29 of 54 
 
13 DATA 
13.1 Data to be collected: 
Evaluation sites will periodically collect, clean, link, de-identify and supply data in an agreed 
format, beginning with baseline data from the pre-implementation phase until the end of the 
post-implementation phase. The frequency of data collection is subject to change though 
depending on the feasibility of sites. 
 
A data dictionary will be supplied (example included in the Appendix) that will support fulfilling 
these requirements from:  
 
        Extraction of routinely collected hospital data:  
 
• 
Referral Data: Age, Sex, date of referral for CT, date of appointment. Where 
possible ethnicity and deprivation will be captured.  
• 
Radiology Data: Number of referrals, Datetime CT acquired, Datetime CT 
reported, presence or absence of discordance between qER and reporting 
radiologist, CT device manufacturer/ type, Usual Care Report  
• 
Clinical Data: Clinical Diagnosis, treatment initiation, referral, including date of 
death if applicable.  
• 
Follow-up Data: Length of stay ED, Length of stay Hospital, Specialty pathway. 
• 
qER data outputs from qER interpretation  
• 
Changes in treatment pathways after implementation    
• 
Time to diagnosis  
         
Data linkage for follow-up (death within 28 days of Head-NCCT acquisition date) against 
outcomes will be collected for 28 days. 
 
Data dictionary is attached as a separate document (Appendix 5). 
13.2 Data handling and record keeping 
Raw medical data from clinical information systems shall be anonymized using native 
anonymizing export functions that are available using applications. Where a native 
anonymizing facility is not available to the researcher, other bespoke methods will be 
provided depending on the datatype (e.g. DICOM images, JSON files etc.).  
13.3 Data sharing 
The qER algorithm only analyses the NCCT Images and is done on site. GSTFT will act as the 
central research team. A clinical research fellow will collect anonymized data from the other 
sites for the final analysis. This data will be shared by other research sites through a secure 
KCL repository for the duration of the study in a password-protected database on an 
encrypted hard drive. This data will not be added to a public/private data repository at the 
end of the study period.  
13.4 Information Governance and Security 
Each site will have a data processing agreement (DPA) or equivalent with qER to cover the 
interpretation of NCCT. Each site, with the assistance of qER, GSTT and other internal 
stakeholders as appropriate, will produce a Data Protection Impact Assessment (DPIA) and a 

 
 
 
GSTFT – Observational Research Protocol template_v1 24.11.2022 
 
[ACCEPT] [IRAS 313507] [Version 1.0] [05/APRIL/2023]                                                                   Page 30 of 54 
 
System Security Policy (SSP), or local alternatives. These will be substantively similar to the 
templates in Appendix.  A site may require further information governance and security 
documentation (e.g., imaging governance assessments, etc.), depending upon local operating 
procedures. We will produce a single ‘master’ DPIA and separate SSP for the project, which 
can be amended as necessary for each site.  
As Qure.ai are compliant with the ISO27001 information security standard and use a 
commonly accepted approach to processing patient and personal data within a healthcare 
environment, we expect that sites will accept their information handling and systems 
approach.   
13.5 Personal Data Breaches 
Personal data breaches will be immediately reported to the Sponsor/data controllers and to the 
Data Protection Officer/IG Department of the site that incurred the breach. The following 
information will be provided to assess the full risk/impact of the breach: full details as to the 
nature of the breach, an indication as to the volume of material involved, and the sensitivity of 
the breach (and any timeframes that apply), steps that have been taken to mitigate the risk 
(trying to retrieve the data asking third parties to delete information that was sent to them in 
error).  
Sites will additionally follow their Trust incident reporting mechanisms and will document this 
within their TMF/ISFs in the form of a file note provided by the sponsor with corrective and 
preventative measures addressed. 
The sponsor/data controller will determine whether the breach meets the definition of a 
serious breach and warrants reporting to the regulators including the ICO. 
14 MONITORING AND AUDITING  
 
The Chief Investigator will be responsible for the ongoing management of the study.  The Sponsor 
will monitor and conduct audits on a selection of studies in its clinical research portfolio. 
Monitoring and auditing will be conducted in accordance with the UK Policy Framework for Health 
and Social Care and in accordance with the Sponsor’s monitoring and audit procedures. 
15  STATISTICAL CONSIDERATIONS 
15.1 Sample Size  
The sample size available for analysis is determined by the number of hospitals taking part in 
the study and the number of patients coming through the Emergency Department requiring a 
NCCT during the study period.  
The proposed study is a stepped-wedge trial, in which all participating sites will receive the 
intervention, in a random order, throughout the study period. With 4 participating sites, within 
a 12-month study period, study sites will receive the intervention in a staggered manner (Figure 
1). During the study, each site will have received the intervention at a certain time-point. The 
specific time point for when the site is will be switched from current standard pathway group 
to intervention group will be allocated based on randomisation procedure.  

 
 
 
GSTFT – Observational Research Protocol template_v1 24.11.2022 
 
[ACCEPT] [IRAS 313507] [Version 1.0] [05/APRIL/2023]                                                                   Page 31 of 54 
 
Baseline data for all sites prior to the start of the study will be available from electronic records. 
Follow-up data will be available throughout the study period, including at least 3 months post-
intervention at all sites. It is likely that changes in treatment pathways after implementation 
will take time to reach a steady state, and one of the objectives of any analysis will be to model 
the transition from the pre-intervention to post-intervention state.   
This design has elements of a cluster-randomised trial, and of a before-and-after study. Within 
each site, there will be sufficient statistical power to detect a 20% change in average outcomes 
(TAT measure) achieved before and after the implementation of the intervention.   
The power estimations were conducted via a simulation approach. The range of conditions for 
sample size and intraclass correlation coefficient (ICC) were tested. The values of sample sizes 
tested ranged from 120 to 600 abnormal scans per site per year, while the range of ICC ranged 
from 0.05 to 0.30.  Assuming type I statistical error at the α=0.05 level and the total duration of 
the study 12months the generalized linear mixed-effect regression model (GLMM) with 
expected distribution of the outcome variable Gamma the total sample size of 1,680 abnormal 
scans (35 abnormal per site/month; total yearly per site 420) ensures satisfactory level of 
statistical power exceeding 80% for detecting reduction in TAT by 20%. The models were 
estimated for abnormal cases. Assuming that abnormal cases are only ~10% of all CT scans 
performed at sites it can be expected that the total number of available scans in this study will 
be 16,800.  Simulations were performed in R version v.4.1.0. If the study was conducted in a 
single hospital, this would not provide definitive evidence. However, if a similar impact is 
observed across four sites, especially if the time course of changes in multiple outcomes track 
the implementation of the intervention within each site, then the study will provide compelling 
evidence of the impact of the intervention. Given the expected amount of data that will be 
generated as part of the study, then it is expected that there will be good power to detect 
intervention effects within subgroups of patients, defined by age, sex, deprivation, and final 
diagnosis.  
15.2 Statistical Analysis Plan 
To investigate the reduction in TAT (the outcome variable) the multistage data analysis process 
will be applied in the study. Initial statistical analyses will consist of basic descriptive analyses, 
and within-site descriptions of changes in outcomes relating to the patient pathway. As a first 
step also the graphical inspection of the distributions of variables at each site and time points 
will be conducted. At this stage the shape of the distribution of outcome variable will be 
assessed. All outlying observations will be detected and checked. In the next step exploratory 
univariate analyses will be applied. Since the outcome variable is expected to be continuous 
and highly right-skewed, we are going to apply non-parametric tests Wilcoxon tests to compare 
the outcome variables between pre- and post-intervention at each site.   
In the second step generalized linear mixed-effect regression model (GLMM) will be applied to 
the data. The assumed structure of the data will be multilevel with measures nested within 
sites. The specific model to be applied is going to be GLM rather than normal linear regression 
model since the distribution of the outcome variable is expected to be Gamma (a non-negative 
continuous values, right skewed). Within our modelling process we will be controlling for study 
centre, intervention, calendar month, after-hours reporting, individual patient diagnosis 

 
 
 
GSTFT – Observational Research Protocol template_v1 24.11.2022 
 
[ACCEPT] [IRAS 313507] [Version 1.0] [05/APRIL/2023]                                                                   Page 32 of 54 
 
(ground truth indicating bleed, infarct, presence/ absence of midline-shift, mass-effect, and 
cranial fracture).   
The regression coefficients obtained after the model fit will be investigated to find if there is 
statistically significant reduction in TAT with qER based reporting and triage. Specifically, the 
applied models will be used to explore the time course of outcomes within sites, in relation to 
the timing of the implementation of the intervention, allowing for differences between sites, 
and over time across all sites. Results will be adjusted for individual patient characteristics.   
16 PEER REVIEW 
The protocol has been reviewed by all investigators including principal investigators from each 
site, clinical research science team at Qure.ai (manufacturer of qER), study statistician, members 
of the PPI group and members of trial management group. 
17 FINANCING  
This study has been awarded funding from NHSx AI Award. Funding for the study team to deploy 
and use qER for the study purpose, to perform its analysis of the data gathered from sites, and to 
produce a final report, is confirmed under the research contract between Qure.ai (manufacturer 
of qER) and Guys and St Thomas’ NHS Foundation Trust. Funding for evaluation sites (those 
contributing data to the study) is allocated from the AI award to the technology provider 
(Qure.ai).  
Award reference number: AI_Award02354. 
18 INSURANCE AND INDEMNITY  
This study is sponsored by Guy’s and St Thomas’ NHS Foundation Trust (GSTFT) and indemnity is 
provided through NHS Resolution’s Clinical Negligence Scheme for Trusts (CNST) which provides 
indemnity for clinical negligence.  In the case of negligent harm, health care professionals 
undertaking clinical trials or studies on volunteers, whether healthy or patients, during their NHS 
employment are covered by NHS Resolution. In the case of non-negligent harm, legal liability 
does not arise where a person is harmed but no one has acted negligently. In exceptional 
circumstances NHS bodies may consider whether an ex-gratia payment could be offered. 
19 DATA CONTROLLER 
Guy’s and St Thomas’ NHS Foundation Trust is the Data Controller as defined by UK general data 
protection legislation (UK GDPR) for this study and as such agrees to comply with the obligations 
placed on a Data Controller by the UK GDPR. This is not limited to, but includes, being responsible 
for and able to demonstrate compliance with the principles relating to Processing of Personal 
Data (Article 5 UK GDPR).  
20 REPORTING AND DISSEMINATION 
 
The results from this study will be presented at AI, radiology, and emergency medicine meetings, 
and published in high-impact peer-reviewed journals in collaboration with all investigators.   
 

 
 
 
GSTFT – Observational Research Protocol template_v1 24.11.2022 
 
[ACCEPT] [IRAS 313507] [Version 1.0] [05/APRIL/2023]                                                                   Page 33 of 54 
 
 
 
 

 
 
 
GSTFT – Observational Research Protocol template_v1 24.11.2022 
 
[ACCEPT] [IRAS 313507] [Version 1.0] [05/APRIL/2023]                                                                   Page 34 of 54 
 
Useful reading/websites 
 
Integrated Research Application System (IRAS) 
https://www.myresearchproject.org.uk/  
 
Health Research Authority (HRA) 
www.hra.nhs.uk  
 
HRA Guidance for Patient Information Sheet and Informed Consent  
http://www.hra.nhs.uk/research-community/before-you-apply/participant-information-sheets-and-informed-
consent/  
 
CONSORT statement ICH Harmonised Tripartite Guidelines for Good Clinical Practice (1996) 
http://www.ich.org/fileadmin/Public_Web_Site/ICH_Products/Guidelines/Efficacy/E6/E6_R1_Guideline.pdf  
 
Martin Bland et al, Statistical guide for research grant applications 
http://www-users.york.ac.uk/~mb55/guide/guide.htm  
Includes detailed information and definitions of many aspects required for a research protocol.  
 
Declaration of Helsinki  
(http://www.wma.net/en/30publications/10policies/b3/index.html) 
 
 
 
 

 
 
 
GSTFT – Observational Research Protocol template_v1 24.11.2022 
 
[ACCEPT] [IRAS 313507] [Version 1.0] [05/APRIL/2023]                                                                   Page 35 of 54 
 
 
 
21 REFERENCES 
1. 
Saver, J. L. Time is brain--quantified. Stroke 37, 263–6 (2006).  
2. 
Jones, S. et al. Association between delays to patient admission from the emergency 
department and all-cause 30-day mortality. Emergency Medicine Journal 39, 168–173 
(2022).  
3. 
Rimmer, A. Radiologist shortage leaves patient care at risk, warns royal college. BMJ 359, 
j4683 (2017).  
4. 
Sir Mike Ric & hards. Diagnostics: Recovery and Renewal – Report of the Independent 
Review 
of 
Diagnostic 
Services 
for 
NHS 
England. 
https://www.england.nhs.uk/publication/diagnostics-recovery-and-renewal-report-of-
the-independent-review-of-diagnostic-services-for-nhs-england/ (2020).  
5. 
Royal College of Radiologists. New RCR census shows the NHS needs nearly 2,000 more 
radiologists. (2021).  
6. 
Greenhalgh, R., Howlett, D. C. & Drinkwater, K. J. Royal College of Radiologists national 
audit evaluating the provision of imaging in the severely injured patient and compliance 
with national guidelines. Clin Radiol 75, 224–231 (2020).  
7. 
Chilamkurthy, S. et al. Deep learning algorithms for detection of critical findings in head 
CT scans: a retrospective study. Lancet 392, 2388–2396 (2018).  
 

 
 
 
GSTFT – Observational Research Protocol template_v1 24.11.2022 
 
[ACCEPT] [IRAS 313507] [Version 1.0] [05/APRIL/2023]                                                                   Page 36 of 54 
 
APPENDIX 1 
DATA FLOW DIAGRAM FOR DATA LEAVING TRUST/SITE PREMISES. 
  
 
 
Site 
collected 
Formats 
Data items  
Stored 
Transfer 
Methods  
Locations 
Return of data to 
Sponsor at the 
end of trial  
Hospital   
Electronic 
anonymous: 
CRF 
data, 
radiologist 
ground truth 
assessment 
Data will either be 
entered 
into 
eCRF 
database or sent via 
encrypted 
and 
password-protected 
Excel files 
Web, 
Email 
  
Data sharing 
with Guy’s & 
St Thomas’ 
NHS 
Foundation 
Trust 
for 
analysis.  
Compiled/analyse
d data will be 
transferred 
to 
GSTFT 
in 
accordance 
with 
their data transfer 
protocols. 
 
 
 

 
 
 
GSTFT – Observational Research Protocol template_v1 24.11.2022 
 
[ACCEPT] [IRAS 313507] [Version 1.0] [05/APRIL/2023]                                                                   Page 37 of 54 
 
 
 
 
 
 

 
 
 
GSTFT – Observational Research Protocol template_v1 24.11.2022 
 
[ACCEPT] [IRAS 313507] [Version 1.0] [05/APRIL/2023]                                                                   Page 38 of 54 
 
APPENDIX 2 
SAE Reporting Flow Diagram-Non CTIMPs 
 
 
 
 
 
 
 
 
 

 
 
 
GSTFT – Observational Research Protocol template_v1 24.11.2022 
 
[ACCEPT] [IRAS 313507] [Version 1.0] [05/APRIL/2023]                                                                   Page 39 of 54 
 
APPENDIX 3  
Information with regards to Safety Reporting in Non-CTIMP Research 
 
Who 
When 
How 
To Whom 
SAE 
(related 
and 
unexpected) 
Chief 
Investigator 
-Report to Sponsor within 
24 hours of learning of 
the event 
 
-Report to the MREC 
within 15 days of learning 
of the event 
 
SAE Report form for non-
CTIMPs, available from NRES 
website. 
Sponsor and MREC 
Urgent 
Safety 
Measures  
Chief 
Investigator  
Contact the Sponsor and 
MREC Immediately 
 
Within 3 days  
By phone 
 
 
 
 
Substantial 
amendment 
form giving notice in writing 
setting out the reasons for 
the urgent safety measures 
and the plan for future 
action. 
Main 
REC 
and 
Sponsor  
 
 
 
Main REC with a 
copy also sent to the 
sponsor. The MREC 
will 
acknowledge 
this within 30 days 
of receipt.  
Progress 
Reports  
Chief 
Investigator  
Annually 
(starting 
12 
months after the date of 
favourable opinion) 
Annual 
Progress 
Report 
Form (non-CTIMPs) available 
from the NRES website 
Main REC with a 
copy to be sent to 
the Sponsor 
Declaration 
of 
the conclusion 
or 
early 
termination of 
the study 
Chief 
Investigator  
Within 
90 
days 
(conclusion) 
 
Within 15 days (early 
termination) 
 
The end of study should 
be defined in the protocol 
End of Study Declaration 
form available from the 
NRES website 
Main REC with a 
copy to be sent to 
the sponsor  
Summary 
of 
final Report  
Chief 
Investigator 
Within 
one 
year 
of 
conclusion 
of 
the 
Research 
No Standard Format 
However, 
the 
following 
Information 
should 
be 
included:  
Where the study has met its 
objectives, the main findings 
and 
arrangements 
for 
publication or dissemination 
including 
feedback 
to 
participants 
Main REC with a 
copy to be sent to 
the sponsor 
 
 
 
 
 

 
 
 
GSTFT – Observational Research Protocol template_v1 24.11.2022 
 
[ACCEPT] [IRAS 313507] [Version 1.0] [05/APRIL/2023]                                                                   Page 40 of 54 
 
APPENDIX 4 
Study Flowchart 
 
 
 
 

 
 
 
GSTFT – Observational Research Protocol template_v1 24.11.2022 
 
[ACCEPT] [IRAS 313507] [Version 1.0] [05/APRIL/2023]                                                                   Page 41 of 54 
 
APPENDIX 5 
Data Dictionary 
 5.1. Summary 
A data dictionary of a study explains exactly what data will be collected by each site and in what 
format. It explains the record type level (e.g., pre-scan, scan, post-scan, etc.) and attribute level (e.g., 
patient identifier, age, sex, date of referral, etc.) requirements. It can only be produced after an 
analysis of the data available at each site, to ensure study-wide consistency. 
It will incorporate the following: 
• 
Cohort and data selection criteria 
• 
Source data format 
• Data schema (record types along with their attributes, attribute formats, attribute capture 
requirement and attribute description) 
 
5.2 Cohort and data selection criteria 
A common set of cohort and data selection criteria will be developed once the study team has selected 
the sites and discussed the detailed format and content of their clinical records. The data in each site 
is likely to differ, especially across regional boundaries (e.g., CDS datasets in England versus SMR 
datasets in Scotland). The existing clinical protocols are also likely to differ between sites, which 
influences the way in which data is sequenced, modelled, and stored. 
5.3 Source data format 
Each site will provide a single file for each record, once every three months (to be defined in 
consultation with each site). Each file will be in CSV format, with a fixed number of commas separated 
values, each pertaining to an attribute as defined within the schema for that record type. 
The naming convention will be: 
<Site ID>_<Batch ID>_<dataset short name>_<version number>.csv 
Where: 
• 
‘Site ID’ is the number allocated to as site by the study team 
• 
‘Batch ID’ is the sequential number indicating the 3-monthly batch that the file relates to (e.g., 
the first batch is 1, after 3 months the next batch is 2, etc.) 
• 
‘Dataset short name’ is the short name for the dataset within the file (e.g., ‘patientcore’, 
‘referrals’, ‘labresults’, etc.) 
• 
‘Version number’ is used where the same data needs to be sent more than once, perhaps 
because of quality issues in the previous file. It increments each time a new version of a file is 
sent. 
An example record format for patient demography might be as follows: 
<Patient identifier, postcode sector, age, sex, ethnicity> 

 
 
 
GSTFT – Observational Research Protocol template_v1 24.11.2022 
 
[ACCEPT] [IRAS 313507] [Version 1.0] [05/APRIL/2023]                                                                   Page 42 of 54 
 
98740833,ML5 2,63,male,1A 
89837748,AB43 6,27,female,5B  
The exception will be AI results files and ground truth records, which will be linked and returned as 
individual json format files within compressed folders. These folders will be named as follows: 
<Site ID>_<Batch ID>_<dataset short name>_<version number>.zip 
Where: 
‘Site ID’ is the number allocated to a site by the study team 
‘Batch ID’ is the sequential number indicating the 3-monthly batch that the file relates to (e.g., the 
first batch is 1, after 3 months the next batch is 2, etc.) 
‘Dataset short name’ is the short name for the datasets within the file. It will be ‘airesults’ for the AI 
results files and ‘gtresults’ for the ground truth results files. 
‘Version number’ is used where the same data needs to be sent more than once, perhaps because of 
quality issues in the previous file. It increments each time a new version of a file is sent. 
5.4 Schema 
The following record types are to be extracted from routinely collected clinical data or data that is 
made readily available through the qER platform or from clinical fellow’s report after he/she reviews 
the NHS trust radiology reports. 
• 
Pre-scan 
• 
Scan 
• 
Post-scan 
• 
Ground truth 
Limitations of documented schema 
• 
The list of attributes in each of the record types is not a comprehensive list. Additional 
attributes for study analysis may be derived from attributes listed below for each of the four 
record types during the study analysis phase of the study as applicable. 
• 
The record types and attributes listed below are not the same as short datasets. A subset of 
attributes within each record type or a combination of attributes between multiple record 
types may constitute a short dataset depending on the feasibility of data capture. All record 
types will have one or more unique identifiers which can be used to merge multiple record 
types or multiple short datasets. 
5.4.1 Pre-scan 
This record type contains all attributes that are collected before a Head NCCT scan is completed for 
the patient in the study 
Attribute 
Data type 
Mandatory 
Description 
Source 
1 
Site identifier 
Varchar  
Yes 
The identifier 
allocated by the study 
Assigned 

 
 
 
GSTFT – Observational Research Protocol template_v1 24.11.2022 
 
[ACCEPT] [IRAS 313507] [Version 1.0] [05/APRIL/2023]                                                                   Page 43 of 54 
 
team to the site which 
this patient belongs.  
2 
Study patient 
identifier 
Varchar  
Yes 
Unique Identification 
number given to each 
patient whose data is 
being analysed for the 
study. 
Assigned 
3 
Patient identifier 
Varchar  
Yes 
Unique identifies for a 
patient (across all 
sites) - likely to be 
conflation of site ID 
and site patient ID.  
Assigned 
4 
Cohort 
Varchar  
Yes 
The reason for having 
this patient in the 
cohort  
Assigned 
5 
Date of Birth (DOB) 
Date   
 dd.mm.yyyy  
Yes 
This will determine the 
age of patient at time 
of scan  
DICOM 
Meta Data  
6 
Ethnicity  
Varchar  
No 
The likely 
race/ethnicity based 
on self-reported 
sources, public 
sources and 
surname/ethnicity 
tables. 
Site specific 
electronic 
patient 
record 
7 
Gender  
Numeric   
Yes 
Gender of the patient   
DICOM 
Meta Data  
8 
Time of arrival to 
Emergency 
Department (ED) 
Datetime 
dd.mm.yyyy  
hh:mm  
Yes 
The time of patient 
arrival to the 
Emergency 
Department.  
Site specific 
electronic 
patient 
record 
9 
Time of triage in ED 
Datetime 
dd.mm.yyyy  
hh:mm  
Yes 
The time of first 
digitally recorded 
clinical contact. 
Site specific 
electronic 
patient 
record 
1
0 
Initial location cared 
for 
Varchar  
No 
Where the patient was 
first seen.  
Site specific 
electronic 

 
 
 
GSTFT – Observational Research Protocol template_v1 24.11.2022 
 
[ACCEPT] [IRAS 313507] [Version 1.0] [05/APRIL/2023]                                                                   Page 44 of 54 
 
patient 
record 
1
1 
Indication for Head 
CT 
Varchar  
No 
What did the patient 
present with? 
  
List  
1 - Trauma; 2 - Altered 
focal neurology; 3 - 
Reduced GCS; 4 - 
Headache; 5 - 
Seizures; 6 - Other 
Radiology 
Report. RIS - 
free text. 
1
2 
Time of CT Request 
Datetime 
dd.mm.yyyy  
hh:mm  
Yes 
Time the clinical team 
requested the patient 
to have a head CT 
scan. 
RIS  
1
3 
Does NCCT head 
meet qER inclusion 
and exclusion 
criteria? 
Binary 
Yes 
Technical 
requirements of qER 
read met. 
DICOM 
Meta Data, 
qER Log 
1
4 
CT Scan Model 
Varchar  
No 
The model of the 
scanner used to 
generate images. 
DICOM tag 
1
5 
CT Scan 
Manufacturer 
Varchar  
No 
The manufacturer of 
the scanner uses to 
generate images. 
DICOM tag 
1
6 
CT Scan Software 
Version 
Varchar  
No 
Software version of 
the CT scan.  
DICOM 
Meta Data  
  
5.4.2 Scan 
This record type contains all attributes that are collected during the time when a patient is undergoing 
a Head NCCT scan and immediately after it to capture the necessary data elements for study analysis. 
For attributes which are of ‘binary’ data types, a value of 1 will indicate the presence of that attribute 
and 0 indicates absence. 
Attribute 
Data type 
Mandatory 
Description 
Source 
1 
Patient identifier 
Varchar  
Yes 
Unique identifies for a 
patient (across all 
sites) - likely to be 
Assigned 

 
 
 
GSTFT – Observational Research Protocol template_v1 24.11.2022 
 
[ACCEPT] [IRAS 313507] [Version 1.0] [05/APRIL/2023]                                                                   Page 45 of 54 
 
conflation of site ID 
and site patient ID.  
2 
SOP instance UID 
Varchar  
Yes 
The unique ID for the 
image to which the 
report relates 
DICOM 
Meta Data  
3 
Accession number 
Varchar 
Yes 
Unique identifier 
linking the report to a 
suite of imaging 
studies 
DICOM 
Meta Data 
4 
Time of CT Scan 
Acquisition 
Datetime 
dd.mm.yyyy  
hh:mm  
Yes 
The time at which CT 
scan was completed 
for the patient 
DICOM 
Meta Data  
5 
Out of hours 
scans? 
Binary 
Yes 
Whether the scan was 
sent for teleradiology 
reporting (value = 1) 
or not (value = 0) 
qER 
log/RIS/PA
CS 
6 
Time NCCT Brain 
was received 
Datetime 
dd.mm.yyyy  
hh:mm  
Yes 
The time NCCT was 
received by qER  
qER Log 
7 
Was CT image 
processed by qER? 
Binary 
Yes 
Did the image meet 
qER requirements for 
processing? 
qER Log 
8 
Time qER report  
 received   
Datetime  
 
dd.mm.yyyy  
hh:mm   
Yes 
The time the qER 
report left the 
gateway and available 
for the radiologists to 
review   
qER Log 
9 
qER probability 
score for ICH 
Numeric   
Yes 
Probability of the 
presence of 
intracranial 
haemorrhage 
qER Log 
10 
qER ICH 
Numeric   
Yes 
Presence or absence 
of ICH 
qER Log/ 
qER Report 
11 
qER probability 
score for EDH 
Numeric   
Yes 
Probability of the 
presence of extradural 
haemorrhage 
qER Log 

 
 
 
GSTFT – Observational Research Protocol template_v1 24.11.2022 
 
[ACCEPT] [IRAS 313507] [Version 1.0] [05/APRIL/2023]                                                                   Page 46 of 54 
 
12 
qER EDH 
Binary 
Yes 
Presence or absence 
of EDA 
qER Log 
13 
qER probability 
score for SDH 
Numeric   
Yes 
Probability of the 
presence of subdural 
haemorrhage 
qER Log 
14 
qER SDH 
Binary 
Yes 
Presence or absence 
of SDH 
qER Log 
15 
qER probability 
score for SAH 
Numeric   
Yes 
Probability of the 
presence of 
subarachnoid 
haemorrhage 
qER Log 
16 
qER SAH 
Binary 
Yes 
Presence or absence 
of SAH 
qER Log 
17 
qER probability 
score for IPH 
Numeric   
Yes 
Probability of the 
presence of 
intraparenchymal 
haemorrhage 
qER Log 
18 
qER IPH 
Binary 
Yes 
Presence or absence 
of IPH 
qER Log 
19 
qER probability 
score for IVH 
Numeric   
Yes 
Probability of the 
presence of 
intraventricular 
haemorrhage 
qER Log 
20 
qER IVH 
Binary 
Yes 
Presence or absence 
of IVH 
qER Log 
21 
qER probability 
score for MS 
Numeric   
Yes 
Probability of the 
presence of midline 
shift 
qER Log 
22 
qER MS 
Binary 
Yes 
Presence or absence 
of midline shift 
qER Log/ 
qER Report 
23 
qER probability 
score for mass 
effect 
Numeric   
Yes 
Probability of the 
presence of mass 
effect 
qER Log 
24 
qER ME 
Binary 
Yes 
Presence or absence 
of mass effect 
qER Log/ 
qER Report 

 
 
 
GSTFT – Observational Research Protocol template_v1 24.11.2022 
 
[ACCEPT] [IRAS 313507] [Version 1.0] [05/APRIL/2023]                                                                   Page 47 of 54 
 
25 
qER probability 
score for cranial 
fracture 
Numeric   
Yes 
Probability of the 
presence of cranial 
fracture 
qER Log 
26 
qER fracture 
Binary 
Yes 
Presence or absence 
of cranial fracture 
qER Log/ 
qER Report 
27 
qER probability 
score for atrophy 
Numeric   
Yes 
Probability of the 
presence of atrophy 
qER Log 
28 
qER atrophy 
Binary 
Yes 
Presence or absence 
of atrophy 
qER Log/ 
qER Report 
29 
qER probability 
score for infarct 
Numeric   
Yes 
Probability of the 
presence of 
hypodensities 
suggestive of infarct 
qER Log 
30 
qER Infarct 
Binary 
Yes 
Presence or absence 
of hypodensities 
suggestive of infarct 
qER Log/ 
qER Report 
31 
Time CT Report 
Generated by 
radiologist   
Datetime 
dd.mm.yyyy  
hh:mm    
Yes 
The time when the 
final CT report was 
generated by a 
radiologist. 
RIS/ PACs  
32 
qER full report of 
NCCT Brain 
Varchar  
Yes 
Published final qER 
report  
qER 
log/RIS/PA
CS 
33 
qER Scan Category 
Integer 
Yes 
Prioritised or No 
Finding or Other non-
prioritised finding? 
  
 
1= Prioritised 
 2 = No finding 
 3 = Other non-
prioritised finding 
 4 = Not processed 
Derived 
from qER 
Log/ qER 
Report 
  
5.4.3 Post-scan 
This record type will capture attributes that have occurred after the NHS radiologist final report was 
available for stakeholders to review including any patient follow-up data that is collected. For 

 
 
 
GSTFT – Observational Research Protocol template_v1 24.11.2022 
 
[ACCEPT] [IRAS 313507] [Version 1.0] [05/APRIL/2023]                                                                   Page 48 of 54 
 
attributes which are of ‘binary’ data types, a value of 1 will indicate the presence of that attribute and 
0 indicates absence.  
Attribute 
Data type 
Mandatory 
Description 
Source 
1 
Patient identifier 
Varchar  
Yes 
Unique identifies for a 
patient (across all sites) 
- likely to be conflation 
of site ID and site 
patient ID.  
Assigned 
2 
SOP instance UID 
Varchar  
Yes 
The unique ID for the 
image to which the 
report relates 
DICOM 
Meta Data  
3 
Accession number 
Varchar 
Yes 
Unique identifier 
linking the report to a 
suite of imaging studies 
DICOM 
Meta Data 
4 
Rad ICH 
Binary 
Yes 
Radiologist reported 
presence of  
Intracranial 
haemorrhage in the 
radiology report? 
Clinical 
fellow's 
report 
5 
Rad MS 
Binary 
Yes 
Radiologist reported 
presence   
midline shift in the 
radiology report? 
Clinical 
fellow's 
report 
6 
Rad ME 
Binary 
Yes 
Radiologist reported 
presence   
mass effect in the 
radiology report? 
Clinical 
fellow's 
report 
7 
Rad Fracture 
Binary 
Yes 
Radiologist reported 
presence   
cranial/skull fracture in 
the radiology report? 
Clinical 
fellow's 
report 
8 
Rad Atrophy 
Binary 
Yes 
Radiologist reported 
presence   
atrophy in the 
radiology report? 
Clinical 
fellow's 
report 

 
 
 
GSTFT – Observational Research Protocol template_v1 24.11.2022 
 
[ACCEPT] [IRAS 313507] [Version 1.0] [05/APRIL/2023]                                                                   Page 49 of 54 
 
9 
Rad Infarct 
Binary 
Yes 
Radiologist reported 
presence   
hypodensities 
suggestive of infarct in 
the radiology report? 
Clinical 
fellow's 
report 
10 
Rad Other 
Binary 
Yes 
Radiologist reported 
presence of 
 any other findings 
other than the six 
target abnormalities in 
the radiology report? 
Clinical 
fellow's 
report 
11 
Rad Scan Category 
Integer 
Yes 
Prioritised or No 
Finding or Other non-
prioritised finding? 
  
 
1= Prioritised 
 2 = No finding 
 3 = Other non-
prioritised finding 
Derived 
from 
clinical 
fellow's 
report 
12 
Discordance 
between qER 
report and NHS 
Trust radiologist 
report 
Binary 
Yes 
Is there a discordance 
between qER report 
findings and NHS Trust 
radiologist report? 
  
1 = Yes 
 0 = No 
Derived 
from 
clinical 
fellow's 
report and 
qER report 
13 
Discordance 
Review Decision: 
Which AI findings 
were accepted by 
the panel? 
Varchar 
Yes 
To capture the 
agreement between 
qER and trust 
radiologist based on 
specific findings. 
Derived 
from 
clinical 
fellow's 
report and 
qER report 
14 
Time of treatment 
related to head CT 
findings in ED  
  
Datetime 
  
 
dd.mm.yyyy  
hh:mm  
No 
Time of intervention to 
treat CT Head findings.  
  
Site 
specific 
electronic 
patient 
record 

 
 
 
GSTFT – Observational Research Protocol template_v1 24.11.2022 
 
[ACCEPT] [IRAS 313507] [Version 1.0] [05/APRIL/2023]                                                                   Page 50 of 54 
 
  
  
15 
Time of first 
intervention for 
critically identified 
scans 
Datetime 
  
 
dd.mm.yyyy  
hh:mm 
No 
Time recorded for 
initial treatment option 
and/or referral. 
Site 
specific 
electronic 
patient 
record 
  
16 
Time of discharge 
or referral  
Datetime 
  
 
dd.mm.yyyy  
hh:mm 
Yes 
Electronic discharge 
point from ED (to ward 
or home).  
Site 
specific 
electronic 
patient 
record 
17 
Length of stay in 
hospital  
Numeric 
Yes 
Length of stay in 
hospital related to that 
admission.   
Site 
specific 
electronic 
patient 
record 
18 
Was the patient 
enrolled in other 
clinical trial? 
Binary 
Yes 
Has the patient been 
recruited to another 
study? 
1 = Yes 
 0 = No 
Site 
specific 
electronic 
patient 
record 
19 
If 21 = Yes; Clinical 
trial number 
[PLACEHOLDER] 
Clinical trial 
code 
Yes 
The study ID if the 
patient is a subject? 
Site 
specific 
electronic 
patient 
record 
20 
Patient's National 
Data Opt Out 
Binary 
Yes 
If patient has selected 
National Data Opt out, 
their data will be 
excluded from final 
analysis. 
 
1= Patient data 
excluded from analysis 
 0 = Not excluded 
  

 
 
 
GSTFT – Observational Research Protocol template_v1 24.11.2022 
 
[ACCEPT] [IRAS 313507] [Version 1.0] [05/APRIL/2023]                                                                   Page 51 of 54 
 
  
 
21 
Death within 28 
days following 
discharge 
Binary 
Yes 
Had the patient died 
within 28 days 
following discharge 
from NHS Trust 
hospital? 
 1 = Yes 
 0 = No 
Site 
specific 
electronic 
patient 
record 
22 
Date of Death 
Date 
  
 
dd.mm.yyyy 
Yes 
If the patient had died 
within 28 days 
following discharge 
from NHS Trust 
hospital, what is the 
date of death 
Site 
specific 
electronic 
patient 
record 
23 
Immediate 
condition 
Varchar 
Yes 
ICD code relating to the 
disease or condition 
directly relating to 
death 
Site 
specific 
electronic 
patient 
record 
24 
Contributing 
condition 
Varchar 
No 
ICD code relating to 
other significant 
conditions contributing 
to the death but not 
related to the disease 
or condition causing it 
Site 
specific 
electronic 
patient 
record 
  
5.4.4 Ground truth 
This record type will contain attributes that captures the ground truth results for the Head NCCT scans 
which have findings that are discordant between qER report and NHS trust radiologist report. 
Attribute 
Data type 
Mandatory 
Description 
Source 
1 
Patient identifier 
Varchar  
Yes 
Unique identifies for a 
patient (across all 
sites) - likely to be 
conflation of site ID 
and site patient ID.  
Assigned 

 
 
 
GSTFT – Observational Research Protocol template_v1 24.11.2022 
 
[ACCEPT] [IRAS 313507] [Version 1.0] [05/APRIL/2023]                                                                   Page 52 of 54 
 
2 
SOP instance UID 
Varchar  
Yes 
The unique ID for the 
image to which the 
report relates 
DICOM 
Meta Data  
3 
Accession number 
Varchar 
Yes 
Unique identifier 
linking the report to a 
suite of imaging 
studies 
DICOM 
Meta Data 
4 
Ground truther ID 
Varchar 
Yes 
The unique ID of the 
person generating 
ground truth 
Assigned 
5 
Ground truther 
role 
Varchar 
Yes 
‘Adjudicator’ if the 
ground truther is 
adjudicating a 
disagreement between 
two ground truthers, 
otherwise ‘Default 
ground truther' 
Assigned 
6 
GT ICH 
Binary 
Yes 
Ground truther 
reported presence of  
intracranial 
haemorrhage in the 
radiology report? 
Ground 
truth 
report 
7 
GT MS 
Binary 
Yes 
Ground truther 
reported presence   
midline shift in the 
radiology report? 
Ground 
truth 
report 
8 
GT ME 
Binary 
Yes 
Ground truther 
reported presence   
mass effect in the 
radiology report? 
Ground 
truth 
report 
9 
GT Fracture 
Binary 
Yes 
Ground truther 
reported presence   
cranial/skull fracture in 
the radiology report? 
Ground 
truth 
report 
10 
GT Atrophy 
Binary 
Yes 
Ground truther 
reported presence   
atrophy in the 
radiology report? 
Ground 
truth 
report 

 
 
 
GSTFT – Observational Research Protocol template_v1 24.11.2022 
 
[ACCEPT] [IRAS 313507] [Version 1.0] [05/APRIL/2023]                                                                   Page 53 of 54 
 
11 
GT Infarct 
Binary 
Yes 
Ground truther 
reported presence   
hypodensities 
suggestive of infarct in 
the radiology report? 
Ground 
truth 
report 
12 
GT Other 
Binary 
Yes 
Ground truther 
reported presence of 
 any other findings 
other than the six 
target abnormalities in 
the radiology report? 
Ground 
truth 
report 
13 
GT Scan Category 
Integer 
Yes 
Prioritised or No 
Finding or Other non-
prioritised finding? 
  
 
1= Prioritised 
 2 = No finding 
 3 = Other non-
prioritised finding 
Derived 
from 
ground 
truth 
report 
 
 
 
  
 
 
 
 
 
 
 

 
 
 
GSTFT – Observational Research Protocol template_v1 24.11.2022 
 
[ACCEPT] [IRAS 313507] [Version 1.0] [05/APRIL/2023]                                                                   Page 54 of 54 
 
APPENDIX 6 
GP Letter