Real World Testing of an Artificial Intelligence-enabled App as an Early Intervention and Support Tool in the Mental Health Referral Care Pathway
Phase: NA
Conditions: Mental Health Issue, Anxiety, Depressive Symptoms
Sponsor: University of Plymouth

 
 
 
 
 
 
 
 
 
 
Real World Testing of an Artificial Intelligence-enabled App as an Early Intervention and 
Support Tool in the Mental Health Referral Care Pathway   
 
 
NCT05533190 
 
 
1 March 2023 

 
 
 
 
 
1 
 
Real world testing of an artificial intelligence-enabled 
app as an early intervention and support tool in the 
mental health referral care pathway 
 
Protocol Version 
1.2      
Date 
01 March      2023      
Sponsor 
The University of Plymouth 
IRAS Number 
310377 
 
 
Chief Investigator 
Rohit Shankar, MBBS FRCPsych 
Professor of Neuropsychiatry 
Peninsula Medical School, University of Plymouth 
 
Co-Investigators 
Ceire Costelloe, BA (Hons) MSc PhD CStat 
Professor of Health Informatics 
The Institute of Cancer Research 
 
Visiting Professor of Digital Health and Statistics 
Imperial College London 
 
Becky Inkster, BSc DPhil 
Neuroscientist & Digital Mental Health Advisor 
London 
Honorary Associate, University of Cambridge 
Honorary Associate, The Alan Turing Institute 
 
Edward Meinert, MA MSc MBA MPA PhD CEng FBCS EUR ING 
Associate Professor of eHealth 
Centre for Health Technology, University of Plymouth 
 
Honorary Senior Lecturer 
Imperial College London 
 
Felix Achana, BSc MSc PhD  
Associate Professor  
Warwick Medical School, University of Warwick  
 
Senior Researcher in Health Economics  

 
 
Version number 1.2      – 01 March      2023      
 
 
 
 
 
         2 
 
Nuffield Department of Primary Care, University of Oxford 
Contacts  
rohit.shankar@plymouth.ac.uk  
ciere.costelloe@icr.ac.uk  
edward.meinert@plymouth.ac.uk 
becky@beckyinkster.com 
felix.achana@phc.ox.ac.uk   
 
 

 
 
Version number 1.2      – 01 March      2023      
 
 
 
 
 
         3 
 
Table of Contents 
 
1 STUDY INFORMATION 
6 
1.1 Full/Long Title of the Study 
6 
1.2 Short Study Title / Acronym 
6 
1.3 Protocol Version Number(s) and Date(s) 
6 
1.4 Research Reference Numbers 
6 
2 SIGNATURE PAGE 
7 
3 KEY STUDY CONTACTS 
8 
4 ABBREVIATIONS AND KEY TERMS 
10 
5 STUDY SUMMARY 
12 
5.1 Funding and Support in Kind 
13 
5.2 Role of Study Sponsor and Funder 
13 
5.3 Roles and Responsibilities of Study Management Committees/Groups and Individuals 
13 
5.4 Protocol Contributors 
13 
5.5 Keywords 
14 
6 STUDY FLOW CHART 
15 
7 INTRODUCTION 
17 
7.1 Background 
17 
7.2 Rationale 
18 
7.2.1 Benefits for patients and public 
18 
7.2.2 Benefits for NHS and service provision 
18 
7.3.3 Benefits for environment 
19 
7.3 Theoretical Frameworks 
19 
7.4 Research Question 
19 
7.5 Aims and Objectives 
19 
8 METHODS 
21 
8.1 Trial Design 
21 
8.1.1 Timeline 
21 
8.2 Study Context 
21 
8.2.1 Participants 
21 
8.2.2 Setting 
21 
8.2.3 Intervention(s) 
21 
8.2.4 Comparator(s) 
22 
8.2.5 Outcomes 
22 
8.2.5.1 Primary Outcomes 
22 

 
 
Version number 1.2      – 01 March      2023      
 
 
 
 
 
         4 
 
8.2.5.2 Secondary Outcomes 
22 
8.3 Sample and Recruitment 
24 
8.3.1 Eligibility Criteria 
24 
8.3.1.1 Inclusion Criteria 
24 
8.3.1.2 Exclusion Criteria 
24 
8.3.2 Sampling 
25 
8.3.2.1 Sample Size 
25 
8.3.2.2 Sampling Technique 
25 
8.3.3 Recruitment 
25 
8.3.3.1 Participant Identification 
25 
8.3.3.2 Consent 
25 
8.3.4 Adherence 
26 
8.4 Data Collection 
26 
8.4.1 Randomisation 
26 
8.4.1.1 Sequence Generation 
26 
8.4.1.2 Allocation Concealment Mechanism 
26 
8.4.1.3 Implementation 
26 
8.4.2 Blinding 
26 
8.4.3.1 Quantitative data collection 
27 
8.4.3.2 Qualitative data collection 
27 
8.5 Data Analysis 
27 
8.5.1 Statistical Methods for Primary Outcome 
27 
8.5.2 Other Data Analyses 
27 
8.6 Limitations 
27 
8.7 Generalisability 
28 
9 ETHICAL AND REGULATORY CONSIDERATIONS 
29 
9.1 Assessment and Management of Risk and Other Ethical Issues 
29 
9.2 Research Ethics Approval 
30 
9.3 Peer Review 
30 
9.4 Patient and Public Involvement 
30 
9.4.1 Involving a lay representative on the study monitoring group 
30 
9.4.2 Involving a dedicated PPI group 
31 
9.4.3 Role of PPI group 
32 
9.5 Protocol Compliance 
32 
9.6 Consent 
32 
9.7 Data Protection and Patient Confidentiality 
33 

 
 
Version number 1.2      – 01 March      2023      
 
 
 
 
 
         5 
 
9.8 Stopping Guidelines 
34 
9.9 Indemnity 
34 
9.10 Access to code 
34 
9.11 Access to data 
34 
10 DISSEMINATION POLICY 
35 
10.1 Dissemination Strategy 
35 
10.2 Authorship Eligibility Guidelines and Use of Professional Writers 
35 
11 DECLARATIONS 
36 
11.1 Funding 
36 
11.2 Protocol Registration 
36 
11.3 Competing Interests 
36 
12 REFERENCES 
37 
13 APPENDICES 
42 
13.1 Appendix 1. Participant Information Sheet 
42 
13.2 Appendix 2. Informed Consent Form 
43 
13.3 Appendix 3. Sample Question Guide for Semi-Structured Interviews 
44 
13.4 Appendix 4. GP Letter 
45 
13.5 Appendix 5. SPIRIT-AI Checklist: Recommended items to address in a protocol and related 
documents for clinical trials evaluating AI interventions 
46 
 
 
 
 
 

 
 
Version number 1.2      – 01 March      2023      
 
 
 
 
 
         6 
 
1 STUDY INFORMATION 
 
1.1 Full/Long Title of the Study  
Real world testing of an artificial intelligence-enabled app as an early intervention and support tool 
in the mental health referral care pathway 
 
1.2 Short Study Title / Acronym  
Clinical investigation of Wysa 
 
1.3 Protocol Version Number(s) and Date(s)  
0.1 
Initial draft 
 
 
 
 
 
25 October 2021  
 
0.2 
Circulation drafts 
 
 
 
 
05 November 2021 
0.3 
Revisions following investigator review  
 
26 November 2021 
 
0.4 
Minor revisions  
 
 
 
 
01 December 2021 
0.5 
UoP Sponsor Review 
 
 
 
 
02 December 2021 
0.6 
Minor drafting amendments 
 
 
 
12 January 2022 
0.7 
Drafting amendments  
 
 
 
7 February 2022 
0.8 
Amendments following Sponsor/faculty review  
21 February 2022 
0.9 
Faculty approved version for HRA/REC review 
 
19 March 2022 
1.0 
Amendments following REC Provisional Opinion  
22 May 2022 
1.1 
Minor amendments following MHRA review 
 
29 September 2022 
 
1.4 Research Reference Numbers  
 
IRAS Number: 
310377 
SPONSORS Number: 
AM1000411, 3173 
NIHR Award Number: 
AI_AWARD02176 
 
 

 
 
Version number 1.2      – 01 March      2023      
 
 
 
 
 
         7 
 
2 SIGNATURE PAGE 
The undersigned confirm that the following protocol has been agreed and accepted and that the 
Chief Investigator agrees to conduct the study in compliance with the approved protocol and will 
adhere to the principles outlined in the Declaration of Helsinki, the Sponsor’s procedures, and other 
regulatory requirements. 
 
I agree to ensure that the confidential information contained in this document will not be used for 
any other purpose other than the evaluation or conduct of the investigation without the prior 
written consent of the Sponsor. 
 
I also confirm that I will make the findings of the study publicly available through publication or 
other dissemination tools without any unnecessary delay and that an honest accurate and 
transparent account of the study will be given; and that any discrepancies from the study as planned 
in this protocol will be explained. 
 
 
For and on behalf of the Study Sponsor: 
Signature:  
 
 
…................................................................................................... 
 
Date: 
....../....../...... 
Name (please print): 
 
...................................................................................................... 
 
 
Position:  
 
...................................................................................................... 
 
 
 
 
Chief Investigator:  
Signature:  
 
 
...................................................................................................... 
 
Date: 
....../....../...... 
Name: (please print): 
 
Professor Rohit Shankar  
 
 
 
 
 
 
 

 
 
Version number 1.2      – 01 March      2023      
 
 
 
 
 
         8 
 
3 KEY STUDY CONTACTS 
 
Table 1. Key Study Contacts 
Chief Investigator 
Rohit Shankar, MBBS, FRCPsych 
Professor of Neuropsychiatry, Peninsula Medical School 
University of Plymouth 
+44 (0)345 155 8109 
rohit.shankar@plymouth.ac.uk  
Study Coordinator 
Dr Edward Meinert, MA MSc MBA MPA Phd CEng FBCS EUR ING 
Associate Professor of eHealth 
Centre for Health Technology 
University of Plymouth 
Plymouth, PL4 6DN 
edward.meinert@plymouth.ac.uk  
Sponsor 
University of Plymouth 
Drake Circus, Plymouth, PL4 8AA 
01752 585858 
Funder(s) 
National Institute of Health Research (NIH–) - Funding body  
Artificial Intelligence in Health and Care Award 
Charlotte Westbrook, Programme Manager  
Programme Management Office 
Grange House 
15 Church Street 
Twickenham, TW1 3NL 
charlotte.westbrook@nhs.net  
 
Wysa Limit–d - Grant Holder 
Miss Emma Selby, Clinical Lead  
Wysa Limited 
c/o 1 London Street, Reading, 
Berkshire RG1 4PN 
emma@wysa.ai  
Key Protocol 
Contributors 
Felix Achana, BSc MSc PhD  
Associate Professor  
Warwick Medical School 
Coventry, CV4 7HL 
 
Senior Researcher in Health Economics  
University of Oxford  
 
Ceire Costelloe, BA(hons) MSc PhD CStat 
Professor of Health Informatics  
Institute of Cancer Research (ICR) 

 
 
Version number 1.2      – 01 March      2023      
 
 
 
 
 
         9 
 
Visiting Professor of Digital Health and Statistics 
Imperial College London 
 
Becky Inkster, BSc, DPhil 
Neuroscientist & Digital Mental Health Advisor 
21 Tufton Court 
Tufton Street 
London, SW1P 3QH 
 
Honorary Associate, University of Cambridge 
Honorary Associate, The Alan Turing Institute 
 
Edward Meinert, MA MSc MBA MPA PhD CEng FBCS EUR ING 
Associate Professor of eHealth 
Centre for Health Technology 
University of Plymouth 
Plymouth, PL4 6DN 
 
Honorary Senior Lecturer 
Imperial College London 
 
Madison Milne-Ives, BAS MSc  
Research Fellow in Digital Health  
Centre for Health Technology 
University of Plymouth 
Plymouth, PL4 6DN 
 
Emma Selby, BSc, PGDIP 
Wysa Limited 
c/o 1 London Street, Reading, 
Berkshire, London, RG1 4PN 
 
 

 
 
Version number 1.2      – 01 March      2023      
 
 
 
 
 
         10 
 
4 ABBREVIATIONS AND KEY TERMS 
Table 2. Abbreviations 
Abbreviation / 
Key Term 
Full Phrase / Definition 
A&E 
Accident and Emergency (Emergency department) 
AI 
Artificial Intelligence 
API 
Application Programming Interface  
CEO 
Chief Executive Officer  
CNWL  
Central North West London (NHS Foundation Trust) 
CONSORT 
Consolidated Standards of Reporting Trials 
DPA 2018 
Data Protection Act 2018 
EBCD workshops 
Experience-based co-design workshops 
EC 
European Commission 
EQ-5D-5L 
EuroQoL 5 Dimensions 
FREIC 
Faculty Research Ethics and Integrity Committees 
GAD-7 
General Anxiety Disorder questionnaire  
GDPR 
General Data Protection Regulation 
GP 
General Practitioner  
HRA 
Health Research Authority 
IAPT 
Improving Access to Psychological Therapies 
ICMJE  
International Committee of Medical Journal Editors 
ICO 
Information Commissioner's Office 
IRAS 
Integrated Research Application System 
MeSH 
Medical Subject Headings 
NASSS 
Non-adoption, Abandonment, Scale-up, Spread, and Sustainability framework  
NHS 
National Health Service 
NIHR 
National Institute for Health Research  
PERC 
Patient Experience Research Centre 
PHQ-9 
Patient Health Questionnaire  

 
 
Version number 1.2      – 01 March      2023      
 
 
 
 
 
         11 
 
PPI 
Patient and Public Involvement 
RCT 
Randomised controlled trial 
REC 
Research Ethics Committee 
SF-12 
Short Form 12 health survey 
SPIRIT-AI 
Standard Protocol Items: Recommendations for Interventional Trials - Artificial 
Intelligence 
SW AHSN 
South West Academic Health Science Network 
SWIFT  
Structured What-IF Technique (prospective hazards analysis method)   
TFA 
Theoretical Framework of Acceptability  
Wysa 
Wysa Limited (Company Number: 11220172) 
  
 

 
 
Version number 1.2      – 01 March      2023      
 
 
 
 
 
         12 
 
5 STUDY SUMMARY 
Table 3. Study Summary  
Study Title 
Real world testing of an artificial intelligence-enabled app as an 
early intervention and support tool in the mental health referral 
care pathway 
Short title 
Clinical investigation of Wysa 
Study Design 
The study proposed is a non-blinded randomised controlled trial 
and mixed-methods evaluation to inform the impact of Wysa on 
depression and/or anxiety symptoms and user engagement and 
perceptions.  
Study Participants 
The target population of the will include individuals over the age of 
18 years who are referred, or self-refer, to Improving Access to 
Psychological Therapies programme (IAPT) for mental health 
support.  
Planned Size of Sample  
The planned sample size is 480 individuals from the three provider 
sites of the London Community Living Well Service (run by the 
CNWL NHS Foundation Trust).  
Follow up duration  
The study implementation and follow-up period will last for nine 
months, with the primary outcome being measured at three 
months post-randomisation.  
Planned Study Period 
The study will last 17 months which includes five months of 
evaluation refinement, ethical approval, technical integration and 
testing, A further nine months of implementation and follow-up, 
ending with three months of post-evaluation analysis and 
preparation for dissemination. 
Research Question/Aim(s) 
 
The study aim is to establish real-world evidence of the Wysa 
impact on mental health outcomes (depression and/or anxiety 
symptoms) in individuals waiting for treatment through the NHS 
standard mental health care pathway.  
 
The main research question of the study is: how does an Artificial 
Intelligence (AI)-enabled self-help and triage app impact the 
mental well-being of patients who have been referred to the IAPT 
service for mental health support? 
 
 
 

 
 
Version number 1.2      – 01 March      2023      
 
 
 
 
 
         13 
 
5.1 Funding and Support in Kind 
Table 4. Funding Information  
FUNDER 
FINANCIAL AND NON-FINANCIAL SUPPORT GIVEN 
National Institute for Health Research (NIHR) 
and NHSX 
Charlotte Westbrook  
Programme Management Office 
Grange House, 15 Church Street 
Twickenham 
TW1 3NL 
charlotte.westbrook@nhs.net  
£997,986.00 
 
5.2 Role of Study Sponsor and Funder  
This study is funded by an NIHR Artificial Intelligence in Health and Care Award granted to Wysa. 
Wysa distributes on behalf of NIHR funding to the University of Plymouth, the Institute of Cancer 
Research, and Dr Becky Inkster for this study execution. This funding arrangement is governed by a 
collaboration agreement which defines the University of Plymouth as being responsible for the study 
design, conduct, data analysis and interpretation, manuscript writing, and dissemination of results in 
collaboration with the academic collaboration including the Institute of Cancer Research. Whilst 
Wysa has contributed and reviewed the study design and will support the trial in providing 
technology support, the academic collaboration, led by the University of Plymouth, is solely 
responsible for the study execution and dissemination of results to assure an independent 
assessment of Wysa technology. The academic collaboration maintains a free and independent 
unrestricted right of publication of the study's findings as to avoid any conflict of interest. 
5.3 Roles and Responsibilities of Study Management Committees/Groups and 
Individuals 
A study steering board composed of the Chief Investigator, Principal Investigators, Co-Investigators, 
a member of the public, Wysa’s CEO and Clinical Lead, and an external researcher (from the study) 
will meet every two months to review progress against the protocol to provide study governance 
and oversight. Reports shall be distributed for review by the study team for action. An independent 
Data Governance Advisory Board will be formed to help further steer the project. 
Wysa will appoint a Patient and Public Involvement (PPI) committee. With input from the Imperial 
College Patient Experience Research Centre (PERC) and the South West Academic Health Science 
Network (SW AHSN) and based on NIHR Involve guidance, a term of reference, confidentiality 
agreement and a background assessment form to Wysa to structure the aims of the PPI group. 
Membership of this Wysa PPI group is voluntary but requires members to be committed to attend 
meetings and to respond to emails/correspondence. The initial term of membership is for the 
duration of the study from early in the preparation phase (app. Dec 2021 / Jan 2022). Further details 
about PPI are included in the study protocol (see section 9.4). 
5.4 Protocol Contributors 
The sponsor, the University of Plymouth, controls the final decision regarding any aspect of the 
study.  

 
 
Version number 1.2      – 01 March      2023      
 
 
 
 
 
         14 
 
Professor Rohit Shankar: Prof Shankar will be the Chief Investigator, will oversee the clinical 
investigation to ensure it follows the approved ethics and protocol, and will arrange suitable on call 
arrangements for any emergencies or decision-making around safeguarding.  
Professor Ciere Costelloe: Prof Costelloe oversaw the drafting of the study protocol and statistical 
methods. She will lead the quantitative evaluation.  
Dr Edward Meinert: Dr Meinert oversaw the drafting of the study protocol and coordinated 
revisions. He will lead the mixed-methods evaluation and the preparation of study findings for 
dissemination. 
Miss Emma Selby: Miss Selby conceived of the study topic, contributed to the study design, and is 
the NIHR grant holder on behalf of Wysa in her role as Clinical Lead.  
Dr Becky Inkster: Dr Inkster contributed to the study design and drafting and revising the study 
protocol. 
Ms Madison Milne-Ives: Ms Milne-Ives contributed to drafting and revising the study protocol. 
Dr Felix Achana: Dr Achana contributed to the design of the cost-effectiveness evaluation. 
5.5 Keywords  
Artificial Intelligence (MeSH), Mental Health (MeSH), Telemedicine (MeSH) 
 
 

 
 
Version number 1.2      – 01 March      2023      
 
 
 
 
 
         15 
 
6 STUDY FLOW CHART 
This project will conduct a real-world evaluation of Wysa as an early intervention tool introduced 
into the mental health referral pathway for adult IAPT service. There is currently an average wait 
time for IAPT between referral and treatment start of 8+ weeks with 95% of people starting 
treatment within 18 weeks [1]. Wysa will be made available to patients upon referral to IAPT. The 
control group will continue with service as usual whilst the test group will interact with Wysa while 
waiting for treatment. 
Wysa will remain available to patients throughout their progression through the standard clinical 
pathway. Patients will be provided with information about the study at their initial referral point and 
will be contacted by our research assistants to collect informed consent and provide the product 
link. This will enable Wysa to offer immediate support tools and monitoring. While patients are 
waiting for treatment, Wysa will serve as a screening support tool for flagging assessment needs to 
the clinical team. After patients have received treatment, Wysa can be used as an extended 
monitoring tool. This is in line with the IAPT manual 2018’s future goal [2] of using an app to prompt 
patients to fill in key outcome measures at regular intervals during the follow-up year, providing 
patients with easy access to their relapse prevention plan, alerting the service if relapse has 
occurred, and facilitating the scheduling of booster sessions. Figure 1 demonstrates the current 
clinical IAPT pathway with the proposed study procedures in blue. 
 
Figure 1. Current clinical pathway for adult IAPT referral and proposed study procedures (blue) 
(adapted from [2])  
Based on the artificial intelligence (AI) recommendations generated by Wysa, patients may be 
passed to the IAPT for a full assessment using a combination of outcome measure scores and 
language used within the tool. A copy of the patients’ responses to outcome measures will be 

 
 
Version number 1.2      – 01 March      2023      
 
 
 
 
 
         16 
 
automatically pushed into their electronic patient record through the Mayden clinical information 
system and an automated email will be sent to the referrals coordinator to flag the need for onward 
referral. 
Wysa will have two pathways to crisis alert and response. The first pathway is for patients to trigger 
the crisis response pathway by pressing a manual override button on the app. The second pathway is 
for the app to trigger the crisis response because of an individual's response to an outcome measure 
or the language used in free conversation with the AI. Because this pathway can be triggered by the 
app’s natural language processing, the AI regularly checks in with the patient to ensure they are 
understanding them correctly. 
When Wysa or the user triggers the crisis response pathway, users will be provided with a stepped 
crisis plan including grounding exercises, local crisis helplines, or a recommendation to attend A&E 
where there is significant injury or risk to life. Data will be collected indicating the number of false 
positives identified by the AI. Coinciding with this, the app pushes an update to the patient’s linked 
electronic patient record through Mayden and sends an alert email to the ‘Clinician of the Day’ at 
each trust for human follow up. 
 
Figure 2. High-level study Gantt chart  
 
 

 
 
Version number 1.2      – 01 March      2023      
 
 
 
 
 
         17 
 
7 INTRODUCTION 
7.1 Background   
Mental health concerns can have significant negative impacts on health, well-being, and economies 
[3]. There is a growing recognition of the importance of mental health, which is reflected across the 
UK and international health sectors. Mental health concerns are projected to be the global leading 
cause of mortality and morbidity by 2030 [4]. Globally, depression and anxiety are estimated to cost 
US$1 trillion each year [5]. However, the impact of mental health conditions is not limited to those 
who have official diagnoses. It is estimated that 1-in-6 people will have experienced a common 
mental health concern in the past week below the threshold of a diagnosable mental health concern 
[6].  
Mental health conditions can have wide ranging negative impacts on individuals’ lives and well-
being. For instance, depression and anxiety disorders can lead to a range of adverse psychological, 
social, and employment outcomes [7]. Mental health conditions also affect physical health; anxiety, 
depression, and substance abuse disorders have been associated with an increased risk of non-
communicable diseases such as diabetes, cancer, and cardiovascular diseases [8–10]. Prolonged and 
severe mental illness is associated with reduced life expectancy; with people at risk of dying an 
average of 15 years sooner than those without severe mental illness. The cause of death in such 
cases is often avoidable, with two thirds of the deaths caused by heart disease and cancer, often 
because of excess smoking [11]. 
Common mental health concerns frequently go unsupported; an estimated 75% of people who 
would benefit from support for common mental health concerns do not receive it, resulting in 
approximately costs of £105.2 billion a year to the UK economy [12,13]. Lack of support at an early 
stage can lead to poorer physical health outcomes and the development of unhealthy coping 
strategies such as alcohol consumption or poorer diet choices [14]. Of those people with recognised 
mental health conditions, negative experiences with mental health services are commonly reported 
[15,16]. Nearly half of survey respondents (44%) who had received NHS therapies in 2019 reported 
that they had to wait too long to access them [15,16] and in 2017, half of the respondents reported 
only being offered pharmaceutical treatment [17]. Long wait times to access mental health services 
can have significant health and economic impacts; a recent study by the Royal College of 
Psychiatrists found that 38% of people waiting for treatment reach out to crisis services and 11% go 
to A&E [18].  
The economic cost of mental health conditions can occur from sick days, reduced quality of life, and 
reduction in productivity [3,4]. Depression, anxiety, or stress are some of the most common reasons 
for lost workdays [19]. Symptoms of common mental health conditions like depression and anxiety 
have increased since before the Covid-19 pandemic, referrals to mental health services decreased, 
and the number of people presenting in crisis increased [20]. It is estimated that the economic cost 
for the UK due to poor mental health is greater than NHS spending on mental health services [21]. 
Reducing the impact of common mental illness will increase our national income and productivity. 
In 2008, a programme for IAPT was launched by the NHS [22]. The number of referrals demonstrates 
the demand for mental health services in the UK; over 1.5 million referrals to adult IAPT services 
were made in 2018/2019 [23]. Over a quarter of these patients (n=441,775) were discharged 
without face-to-face contact following a telephone referral and another quarter (n=440,344) were 
discharged with a workbook [23]. With each contact costing an average of approximately £100 [24], 

 
 
Version number 1.2      – 01 March      2023      
 
 
 
 
 
         18 
 
the total cost to the NHS was over £90 million. This figure does not include additional care needs or 
A&E attendances for those individuals who were unable to wait up to 12 weeks for assessment. 
The NHS Long Term Plan aims to promote digitally enabled care, improve access to mental health 
support for 1.9 million people, and support NHS Trusts at meeting their referral waitlist key 
performance indicators (KPIs) [25]. Wysa’s solution is well-placed to support these goals by providing 
a digital early intervention approach that can be implemented at the point of self-referral, reducing 
the need for inappropriate assessments and escalating cases based on clinical outcome measures as 
needed. This is also aligned with NHSx’s missions, particularly the aims to use digital technology to 
improve health and care productivity and empower people to access information and services to 
manage their own health [26], and the Industrial Strategy’s grand challenge of embedding AI to 
improve productivity in the UK [27]. 
7.2 Rationale 
A key issue with the current mental health service provision is lack of accessibility. Our solution 
enables immediate service provision at the point of referral, to mitigate the impact of long wait 
times to access mental health services. Onboarding processes at the point of referral to IAPT involve 
collecting routine outcome measures (aligned to existing service referral measures). Users are asked 
to provide information and goals to personalise the guided self-help or escalation recommendations. 
These are provided using a combination of AI-enabled self help tools, natural language processing 
and conversation engines. Our Application Programming Interface (API) through the Mayden Prism 
system then pushes data directly into the iaptus electronic patient records and automates safety 
alerts. 
Our preliminary work suggests that implementing Wysa’s self help tools at the onboarding stage will 
result in improved outcomes for the service user, whilst automations in the referral process will 
result in less assessments that are then screened out of the service and a more comprehensive risk 
identification whilst awaiting assessment [28]. This is based upon our previous findings from our 
2018 study showing a significant decrease in depressive symptoms in users of Wysa versus non users 
[28]. 
7.2.1 Benefits for patients and public  
IAPT services used a stepped care model, with Steps 2-4 reflecting increasing intervention beyond 
the GP level; Step 2 is conducted by psychological wellbeing practitioners to manage mild to 
moderate mental health conditions, while Steps 3 and 4 involve CBT therapists and other specialised 
treatments for more severe or complex conditions [29]. Whilst computerised Cognitive Behavioural 
Therapy and guided self-help are recognised treatment options for step two low intensity support in 
IAPT, they are often offered later in the referral pathway resulting in a higher assessment cost and 
poorer user experience. The use of Wysa could help to improve patient experience of, and outcomes 
from, their use of mental health services, for example, by reducing their wait time to access 
evidence-based advice. Delivering timely and convenient evidence-based support could empower 
users to make choices about their health and coping strategies. The use of standard outcome 
measures also enables the app to escalate those patients in need of step three or higher intensity 
interventions. This solution also has the potential to reduce inequalities in service provision (e.g., 
those who might otherwise remain untreated [30] and is especially important for individuals from 
high risk groups (such as LGBTQ+ or Black and minority ethnic groups) who are least likely to receive 
support [31,32]).   

 
 
Version number 1.2      – 01 March      2023      
 
 
 
 
 
         19 
 
7.2.2 Benefits for NHS and service provision  
Wysa has the potential to benefit service provision by providing a more efficient care pathway. By 
providing mental health support and user outcome measures at the point of referral, rather than 
when patients are assessed and offered treatment, Wysa can help to identify users who have a more 
urgent need for treatment and escalate their care. Some users might find that Wysa sufficiently 
meets their needs for mental health support, which would reduce the number of patients who need 
to be assessed and treated in the standard care pathway and reduce the waiting times for patients 
who do need further care. Timely support provision and early identification of deteriorating mental 
health could enable earlier interventions before the patient’s condition becomes more costly to 
their well-being and to the healthcare system. 
7.3.3 Benefits for environment 
Implementing automated support at an early intervention stage has the potential to reduce the 
number of face-to-face assessments both at the point of referral to the IAPT pathway and at the 
point of full assessment. This reduction could potentially contribute to a reduction in the demand for 
travel to attend appointments whilst also reducing the demand of building spaces, contributing to 
the NHS’s Net Zero Carbon goal [33]. Early interventions also have the potential to prevent further ill 
health, which increases operational efficiency and reduces emissions [33]. If Wysa is proven effective 
and widely adopted, its early provision of self-help resources could help to prevent worsening 
mental health and identify people most at risk so that they can be treated before reaching a crisis. 
Evidence will be gathered as part of this project’s health economic assessment to examine this 
hypothesis. 
7.3 Theoretical Frameworks  
Two frameworks will form the theoretical basis for the study:  
1. Long-term adoption and suitability to further trials will be evaluated using the Non-
Adoption, Abandonment and Challenges to the Scale-up, Spread and Suitability (NASSS) 
framework [34].This framework is important to include because interventions can only add 
value if they are successfully adopted and used. This framework emphasises the 
consideration of the multiple levels - individuals and systems - that influence adoption and 
non-adoption. 
2. The Theoretical Framework of Acceptability (TFA) will be used to structure the semi-
structured interview guide [35]. This framework was chosen because it was developed 
specifically for health interventions, captures a multi-faceted view of acceptability (with 
seven components), and emphasises the importance of considering different time points 
when evaluating acceptability.  
7.4 Research Question  
The main research question of the study is: how does an AI-enabled self-help and triage app impact 
the mental well-being of patients who have been referred to the IAPT service for mental health 
support? 
7.5 Aims and Objectives 
The primary aim of this project is to establish real-world efficacy that patient use of Wysa while 
waiting for treatment through the NHS mental health care pathway positively impacts well-being by 
comparing system users with standard services users. 

 
 
Version number 1.2      – 01 March      2023      
 
 
 
 
 
         20 
 
The secondary aims of this project are: 
1. To examine the feasibility of using Wysa to identify users at high risk of a mental health crisis 
and escalate them to further care; 
2. To examine levels of user engagement with Wysa; 
3. To evaluate patient acceptability of an autonomous early intervention supporting mental 
health services; 
4. To complete a full health economic assessment of using Wysa for autonomous early 
intervention and triaging support; 
5. To assess the potential for sustainable adoption of Wysa in mental healthcare pathways. 
 
Figure 3. Wisconsin logic diagram 
 
 

 
 
Version number 1.2      – 01 March      2023      
 
 
 
 
 
         21 
 
8 METHODS  
8.1 Trial Design  
This study will use an efficacy non-blinded randomised controlled trial (RCT) and mixed methods 
(process evaluation) design. Quantitative methods will be used to evaluate Wysa’s efficacy at 
improving mental health outcomes as well as to examine engagement. Quantitative and qualitative 
methods, specifically semi-structured interviews, will also be used to gather a more in-depth 
understanding of users’ engagement with Wysa and their feedback regarding its acceptability (see 
Figure 4 for an overview). 
 
Figure 4. Participant flow diagram 
8.1.1 Timeline 
The study will last 17 months: 5 months of evaluation refinement and technical integration and 
testing, 9 months of implementation and follow-up, and 3 months of post-evaluation analysis and 
preparation for dissemination. 
 
8.2 Study Context  
8.2.1 Participants  
The target population of the study will include adult patients who are referred, or self-refer, to IAPT 
for mental health support.  
8.2.2 Setting  
Participants will be recruited from the Central North West London NHS Foundation Trust but the 
intervention (Wysa) is accessible to users through a mobile app available on Apple and Android 
smartphones.  
8.2.3 Intervention(s)  
Wysa is a guided self-help and triaging tool delivered to patients via an app or widget. It uses Natural 
Language Processing to understand individuals’ written inputs but not to generate responses. Wysa 
makes use of a wide range of clinically underpinned modules whilst gamification, clinical outcome 
measures and AI promotes engagement, improving efficacy and triage and reducing the cost of 
scale.  
 
Wysa is delivered as a standalone system which patients download directly onto their personal 
device. Upon self-referral or professional referral to IAPT services the individual is automatically sent 

 
 
Version number 1.2      – 01 March      2023      
 
 
 
 
 
         22 
 
a download link via sms or email. Once downloaded, data from within the app, including outcome 
measures, tools used and crisis alerts are automatically pushed to the electronic patient record 
iaptus using the Mayden Prism software. When considering the application of AI to mental 
wellbeing, the ability to feel listened to is the most influential factor in patient satisfaction and 
response. Users need to feel able to share their thoughts with the AI and our user testimonials show 
that the anonymous nature of this text-based interaction is imperative to them, even though they 
know they are communicating with an AI. Wysa has benefited from over 6 million conversations 
around the world to build up a complex listening algorithm, rules engine and content library. Clinical 
safety is applied to our rules engine and Content library to ensure that we regularly test for 
appropriateness, empathy, and presence of triggers in cases of misdirection. Each of Wysa’s AI 
models is validated on at least 10,000 records that are manually tagged and not a part of the training 
set. Overall, Wysa has 100 million conversations to draw upon, with over 2 million users globally, 
which helps us overcome the cold-start problem that most AI faces. 
 
Wysa can be used as a native app through android or IOS or as a web widget.  
8.2.4 Comparator(s) 
The intervention will be compared against a waitlist group. This comparator was selected to 
compare Wysa to the current standard of care for patients waiting for standard IAPT assessment and 
treatment. 
8.2.5 Outcomes  
8.2.5.1 Primary Outcomes  
The primary analysis will be the comparison of depression severity between users of the app and 
patients in the standard clinical pathway (see Table 5). The primary outcome will be measured at 
baseline and three months post randomisation. In line with the national recommendations laid out 
by the IAPT Manual, we will be using Wysa’s AI automation to encourage users to complete the 
Patient Health Questionnaire (PHQ-9 [36]) and General Anxiety Disorder (GAD-7 [37]) routine 
outcome measures [2]. Depression severity was chosen as the primary outcome because the PHQ-9, 
a depression symptom measure, is recommended for use in all patients presenting to IAPT services 
[2]. 
Table 5. Primary Outcome Measurement  
Primary Outcome  
Outcome Measure 
Depression severity 
● Score on the PHQ-9 [36] 
 
8.2.5.2 Secondary Outcomes 
There will be several secondary outcome measures to assess Wysa’s ability to improve anxiety 
severity and identify people who need urgent support, engagement, and the acceptability and cost-
effectiveness of the app (see Table 6). The secondary outcomes will also be measured at baseline 
and 3 months post randomisation.   
Anxiety severity 
As the GAD-7 [37], which measures anxiety symptoms, is recommended for the 6 most common 
presentations, it will also be used as an outcome measure for the study and will be presented to 
patients through the app in the same way as the PHQ-9 [36]. The remaining measures used by IAPT 

 
 
Version number 1.2      – 01 March      2023      
 
 
 
 
 
         23 
 
are linked to specialist presentations of pre-existing conditions which would be excluded as 
participants from the study due to non-eligibility to IAPT standard pathway. 
Crisis identification 
Wysa’s ability to identify users who are at high risk of a crisis and who need to be provided with 
additional support will be measured by comparing the number of patients that Wysa flags for 
escalation with the number of patients in the control group who contact the NHS Trust’s crisis 
helpline or attend A&E. This data will be collected from iaptus, a cloud-based electronic patient 
record for managing psychological therapy services [38], which flags patients on the waitlist who 
access these crisis services. 
Engagement 
User engagement with Wysa will be assessed using a combination of complementary measures. 
Examining time spent using a digital health intervention is not by itself a reliable measure of 
effective engagement, as it cannot differentiate between users who spend a long time on the app 
because they’re positively engaged compared to those who are frustrated and struggling to use 
it [39]. Therefore, objective measures of app usage will be supplemented with an assessment of 
uptake and dropout rates and a qualitative exploration of subjective user experience and 
engagement behaviour. 
Acceptability 
Wysa’s acceptability will be examined through qualitative semi-structured interviews in a subset of 
participants. These interviews will provide a means of collecting in-depth data about participants' 
experiences with Wysa, what they liked and disliked, and why. 
Health-related quality of life  
Two questionnaires will be used to assess health-related quality of life: the EQ-5D-5L [40] and the 
Short Form 12 (SF-12) health survey [41]. The EQ-5D-5L provides a descriptive self-assessment of 
health in 5 dimensions: mobility, self-care, usual activities, pain/discomfort and anxiety/depression 
[40]. It is commonly used as a quality of life measure to conduct cost effectiveness analyses [42]. The 
EQ-5D-5L has a greater emphasis on physical than mental health, but has demonstrated the ability 
to capture changing anxiety severity [43]. The SF-12 survey is a reliable measure of functional health 
and is well-suited for this study, as it includes mental health items and has been shown to be 
acceptable for use as a screening tool for depressive disorders [44].  
Cost-effectiveness  
To conduct a health economic analysis, the study will collect data on health and social care service 
use, medication use and treatment costs, the implementation costs of Wysa, out of pocket medical 
expenses borne by patient and carers and productivity related costs such as work absence / time off 
work due illness. This data will be combined with the quality-of-life indicators to produce an 
assessment of the costs of adopting Wysa from the perspective of both the UK health and social care 
services and the wider society [45].  This analysis will also produce a budget impact model for each 
site that will be used to provide a final overall implementation strategy for more rapid adoption at 
other NHS sites and dissemination across the AHSN network in partnership with the new Integrated 
Care Systems. The team will also incorporate the environmental sustainability calculation in line with 
the Greener NHS agenda. 
Table 6. Secondary Outcome Measurement  

 
 
Version number 1.2      – 01 March      2023      
 
 
 
 
 
         24 
 
Secondary Outcome  
Outcome Measure 
Anxiety severity  
● Score on the GAD-7 [37] 
Crisis identification  
● Number of users identified by the app for escalation of care compared to 
the number of patients in the control group who access A&E or out-of-
hours services while waiting for treatment 
Engagement 
● Uptake rates  
● Dropout / ‘did not attend’ rates 
● App usage data  
● Qualitative feedback from semi-structured interviews 
Acceptability 
● Qualitative feedback from semi-structured interviews 
● Automated review questions periodically requested during general use of 
the tool 
Health-related quality of 
life 
● EQ-5D-5L [40] 
● Short Form 12 (SF-12) health survey [41] 
Cost-analysis 
● Data on health and social care service use, medication use and treatment 
costs, patient-bourne costs, productivity related costs, and Wysa’s 
implementation costs  
 
8.3 Sample and Recruitment  
8.3.1 Eligibility Criteria  
8.3.1.1 Inclusion Criteria  
● Willing and able to provide informed consent; 
● Aged 18 years or older; 
● User is confident in their ability to speak and understand English at a proficient level; 
● Own a mobile device capable of supporting Wysa; 
● A valid email address; 
● Referred or self-referred to proceed through the standard IAPT care pathway.  
8.3.1.2 Exclusion Criteria  
● Patients ineligible for the standard IAPT care pathway; 
● Patients with previous and current known major mental illness such as Schizophrenia, severe 
depression, any co-morbid neurological or neuro-psychiatric condition such as epilepsy; 
● Patients with current psychosis or a history of psychotic symptoms within the last 6 months 
● Patients with suicidal ideation; 
● Patients scoring > 15 points on PHQ-     9; 
● Patients scoring > 15 points on GAD-     7; 
● Patients with significant cognitive disorders; 
● Patients with noted neurodevelopmental conditions such as autism or ADHD; 
● Patients previously diagnosed with a personality disorder; 
● Patients who been under the care of CMHT or a specialised mental health services in the last 
2 years; 
● Patients who failed IAPT previously; 

 
 
Version number 1.2      – 01 March      2023      
 
 
 
 
 
         25 
 
● Patients with referrals for specialist presentations of pre-existing, diagnosed conditions 
requiring a specialised assessment beyond the standard clinical pathway; 
● Incapable of self-consent; 
● In a dependent/unequal relationship with the research or care teams or any PPI 
representatives. 
8.3.2 Sampling  
8.3.2.1 Sample Size  
Approx 1100 eligible patients are seen per month at the IAPT in the CNWL NHS Foundation Trust. 
Preliminary work suggests that Wysa is associated with a mean clinical improvement in PHQ-9 of 
5.84 (SD6.66) [28]. This study is powered to detect a clinically meaningful effect  on the PHQ-9 scale 
[28]. Based on a SD of 6.6 and a 2:1 recruitment ratio (usual care: intervention) a sample size of 393 
would allow detection of a difference in PHQ-9 of at least 2 points, with 80% power. Augmenting the 
sample size to 480 would allow for 20% attrition, whilst still maintaining sufficient power to detect 
clinically and statistically meaningful differences with a Type 1 error rate of 0.05. This inflated 
sample size would also allow for adjustment for stratification variables, gender, age, ethnicity, self-
reported antidepressant use, and severity of depression. 
 
8.3.2.2 Sampling Technique 
Participants will be randomly allocated to intervention or control groups by a computer algorithm 
after they have provided informed consent and demographic information. Of this sample, a subset 
will be selected to participate in a semi-structured interview to collect qualitative data about 
patients’ perceptions of Wysa.  
To select this subset, a stratified random sampling technique will be used. Patients will be divided 
based on demographic characteristics (e.g. gender, age, socio-economic status, and ethnicity). From 
these subgroups, patients will be randomly selected so that the sample interviewed is representative 
of the UK population.  
8.3.3 Recruitment 
8.3.3.1 Participant Identification 
Participants who are referred, or refer themselves, to IAPT for mental health support (online or by 
phone) will be recruited from the three provider sites of the London Community Living Well Service, 
run by the Central North West London NHS Foundation Trust. The service receives on average 1100 
referrals a month. All referrals will be screened by the psychology assistant against exclusion criteria 
and those eligible for the trial will be provided with information regarding the trial via email (see 
13.1 Appendix 1). The psychology assistant will then contact participants to check further exclusion 
criteria not clear from service referral and to take informed consent. Screening will continue through 
the first 6 months of the intervention period, and we expect to be able to onboard significantly more 
individuals than our target sample size. All participants will have access to the Wysa app (including 
control participants who will receive access following the control period).    
8.3.3.2 Consent  
Current consent for referral to service is collected through a tick box acceptance of terms during 
online referral. This states: ‘Through completing this form you are consenting to have this 
information stored confidentially on a secure electronic system separate from your GP's system and 
for your GP to be informed of your contact with us. By providing these details you are giving consent 
for us to contact you regarding confidential information’ 

 
 
Version number 1.2      – 01 March      2023      
 
 
 
 
 
         26 
 
For the trial, explicit consent for participation will be collected (see 13.2 Appendix 2). When 
individuals self-refer or are referred to the IAPT, they will be screened against the eligibility criteria. 
Eligible individuals will be sent a study information sheet and then contacted by phone by a study 
team member to take informed consent and discuss any questions participants might have. Once the 
trial has started and users download the app as part of intervention they are asked during 
onboarding if they still consent to take part of the trial and informed of how to remove their data 
from the trial should they wish to do so.  
8.3.4 Adherence  
The study team will monitor the use of Wysa and assure the system is being used within designed 
parameters during the study. 
8.4 Data Collection  
8.4.1 Randomisation  
8.4.1.1 Sequence Generation 
Demographic data will be collected from all patients immediately after they complete the informed 
consent process, during the same phone call. Together with data from the IAPT referral, this 
information will be used in the randomisation algorithm to ensure that the intervention and control 
groups are balanced on key variables (gender, age, ethnicity, self-reported antidepressant use, and 
severity of depression).  
8.4.1.2 Allocation Concealment Mechanism 
Consenting participants will be inputted into a masked computer randomisation algorithm and the 
outcome will be recorded in an electronic record to conceal treatment allocation from study 
researchers. Participants will be sent an automatic email to inform them of their group assignment, 
and participants in the intervention group will receive the link to download Wysa.  
8.4.1.3 Implementation 
Psychology Assistant recruited specifically for the trial and covering the three London Community 
Living Well Service sites will enrol participants in the study.  A computer will generate the random 
allocation sequence and assign participants to interventions.  
8.4.2 Blinding  
Due to the nature of the intervention, no blinding of participants will be possible in the study, as all 
participants will know whether or not they are using the app. Clinical treatment teams will also not 
be blinded as they will know through the electronic patient record if a participant has received the 
intervention. The evaluation team will be blinded to treatment using the randomisation algorithm.  
 8.4.3 Methods of Data Collection 
The timeline for the data collection is detailed in Figure 5. 

 
 
Version number 1.2      – 01 March      2023      
 
 
 
 
 
         27 
 
 
Figure 5. Data collection sequence diagram 
8.4.3.1 Quantitative data collection  
The quantitative outcome measures will be delivered directly to patients through the Wysa app or 
by Qualtrics survey (e.g. for participants in the control group). Participants in both conditions will 
complete four questionnaires at baseline and 3-months post-randomisation (PHQ-9, GAD-7, EQ-5D-
5L, and SF-12). Outcome measures have been chosen that will not be a significant burden for 
participants to complete; none of the questionnaires are longer than 12 items or expected to take 
more than a few minutes each to complete. Together, the questionnaires should take participants 
from 10 to 20 minutes to complete. Patient responses will be automatically uploaded to the iaptus 
electronic patient records using the Mayden Prism API. App use data will be automatically stored 
through the app. 
8.4.3.2 Qualitative data collection 
A subset of study participants will also be invited to participate in a semi-structured interview, so 
that more in-depth, qualitative engagement and user experience feedback can be collected. A topic 
guide will be developed to provide a framework for the interviews. Interviews will be conducted 
using video or call conferencing software and recorded for later transcription (if the participants 
consent). If they do not want the interview recorded, notes will be taken by hand and shared with 
the participant following the interview for verification. See 13.3 Appendix 3 for sample interview 
questions.  
8.5 Data Analysis  
8.5.1 Statistical Methods for Primary Outcome  
Descriptive statistics will be used to characterise users, assess baseline comparability, and compare 
side effects. Primary comparisons between intervention and control groups will be conducted for 
primary and secondary outcomes. Primary analyses are designed with 80% power to detect a two-
sided α of 0.05. All analyses will report 95% CI around estimate of effect. Comparative analyses will 
be conducted on an intention to treat basis, using linear or logistic regression, with adjustment for 
stratification variables.  
We will explore missingness in our data, be that in relation to primary outcomes, follow-up data and 
in WYSA analytic information. The proportion of missing data will be explored and reported on 
descriptively. An hypothesised scenario, where patients remove themselves from the waiting list for 
treatment before the duration of follow-up may lead to missing data scenario. Currently, estimates 

 
 
Version number 1.2      – 01 March      2023      
 
 
 
 
 
         28 
 
suggest that xxx% of our cohort will ‘drop-off’ waiting list before our primary endpoint. We will 
conduct exploratory a sensitivity analyses in a complete case (‘per protocol’) cohort compared with 
the ITT analysis. 
8.5.2 Other Data Analyses  
Secondary analyses will include additional adjustment for factors showing possible imbalance at 
baseline and pre-planned exploratory analyses for differential effects of the intervention for 
different age groups, depression severity, and interaction (dose) of the Wysa app. Secondary 
analyses including adjustment for factors showing potential imbalance at baseline and pre-planned 
exploratory subgroup analyses for differential effects in a priori specified groups will be conducted 
using linear or logistic regression. For secondary analyses, where we have not specifically powered 
the study to detect statistical or clinical meaningful differences, we will report p-values but will focus 
on interpretation of 95% CI around estimates. 
The qualitative semi-structured interviews will be evaluated using a thematic analysis. 
8.6 Limitations 
There are a couple limitations with the protocol.  
Technical limitations of the system will affect the generalisability of results. At present, patients 
without access to any digital device would not be able to use the system and are therefore not 
eligible for inclusion in the study. As Wysa’s algorithm is currently only compatible with the English 
language, non-English speaking patients will be excluded from the study, also limiting the 
representativeness of the sample.  
Potential bias is introduced to the results by the use of qualitative interviews because of the 
influence of the interviewer on how participants respond (e.g. social desirability bias). This potential 
bias will be assessed by examining the balance of positive and negative feedback and by comparing 
the qualitative and quantitative data about Wysa’s usability, acceptability, and impact. 
8.7 Generalisability 
Limitations to the generalisability of the study results were noted in the previous section. However, 
it is expected that the results will be generalisable to most individuals seeking general mental health 
support. We will also make efforts to recruit a representative sample of the UK population and 
include a representative subset in the semi-structured interviews.  
 
 

 
 
Version number 1.2      – 01 March      2023      
 
 
 
 
 
         29 
 
9 ETHICAL AND REGULATORY CONSIDERATIONS 
9.1 Assessment and Management of Risk and Other Ethical Issues 
Standard ethical issues will be managed as follows:  
● Recruitment: Participant eligibility will be determined by the screening process they undergo 
upon referral to IAPT. Participants who are eligible will be randomly divided by the computer 
into intervention and control groups with no influence from the researchers. Interview 
participants will also be randomly selected (from demographic groups) by a computer script 
of intervention users. All participants will be included in the process evaluation’s 
quantitative analyses, thus avoiding recruitment bias. 
● Power relations and other potential biases: Pre-established inclusion and exclusion criteria 
will be used to avoid any potential bias from the researchers as well as any coercion from 
participants. To prevent participation bias, only participants who would not otherwise 
progress down the standard care pathway will be excluded. 
● Sensitivity: The outcomes measures that concern potentially sensitive topics (depression and 
anxiety) will be delivered to the patients’ phones via the app. This allows them to avoid 
discussing any sensitive topics with a researcher. The semi-structured interview questions 
will focus only on the participants’ experiences with the app (engagement and acceptability) 
and are intended to avoid any areas of cultural or psychological sensitivity.  
● Confidentiality: To control any potential perceived issues in this area, participant 
confidentiality will be protected using data protection procedures that are compliant with 
the Data Protection Act 2018 (DPA 2018) [46] (as described in section 9.7).  
● Timing: To mitigate any time concerns for participants, qualitative interviews will last for a 
maximum of 60 minutes. 
● Study Governance: To maintain the integrity of the study and avoid any commercial 
influence, a research contract will be established between Wysa, the University of Plymouth 
(Dr Meinert’s institution), and the Institute of Cancer Research (Dr Costelloe’s institution) 
that will ensure the independence of study design and execution and the unrestricted right 
to publish all results. 
● Acceptability of study: To ensure that there are no ethical concerns with the study that 
might have been overlooked by the researchers and that the design is participant-friendly, a 
local PPI group will be formed to collaborate with researchers throughout the study. For 
further details, please see the PPI section. 
A comprehensive risk register and Hazard log is maintained by Wysa across its development as part 
of its Clinical Safety Management file. This risk register will be regularly reviewed and updated as 
part of the project management governance. Key risks for this project include: 
● Clinical: A Structured What-IF Technique (SWIFT) analysis has been completed which 
highlighted that the main clinical risk of this project is unintended harm. This could be 
psychological or physical risk caused by unintended increase in psychological distress or the 
physical impact of increased reliance on screen time. We have mitigated the risk to patients 
by building a clinically underpinned crisis response pathway with handover capabilities to 
the clinical team at Central North West London NHS Foundation Trust. 
● Excessive screen time: Health risks associated with excessive screen time are mitigated by 
ensuring that many of the strategies and skills taught by the app encourage users to 
complete them away from the screen. All risks and mitigations are outlined in the clinical 
safety management file. 

 
 
Version number 1.2      – 01 March      2023      
 
 
 
 
 
         30 
 
● Technical: The main technical risk is that the system is not fully integrated with iaptus ready 
for our go live date, delaying the project. This is mitigated by completing planning works and 
scoping prior to the application for this award and for allowing contingency time in our 
project planning. 
● Delays: A second risk is that information governance permissions are delayed or withheld at 
pilot sites - mitigated by a technical, security and privacy architecture that has already been 
approved by the representatives of the trust who are project partners on this award. 
● Commercial: The need for good quality evidence combined with long sales cycle times in 
healthcare can produce a risk of limited cash flow - this is mitigated by Wysa’s maturity as a 
company and track record of securing investment funding as needed. As a real-world use 
study the evidence generated by the project will help to respond to commissioners' need for 
a high quality evidence base, removing a core barrier to implementation. 
9.2 Research Ethics Approval 
The study protocol will undergo institutional and regulatory review and approval prior to 
commencement. To ensure that our data reporting is in line with NHS standards we will be following 
the Department of Health and Social Cares guidelines for good practice for digital and data driven 
health technologies [47]. As per the guidelines, we will be applying the Caldicott Principles for the 
collection and evaluation of patient data and everyone dealing with patient information will be 
required to complete the online training prior to starting on the project to evidence familiarity with 
these. 
Ethical approval is being sought from the Health Research Authority and the relevant Research Ethics 
Committee with this submission [48]. As the IRAS system brings together these assessments, only 
one application is needed [48]. The protocol will therefore receive medical device review from the 
associated REC. 
Additional independent ethical approval will be sought from the University of Plymouth. At the 
University of Plymouth, an application for ethical approval will be submitted through the Plymouth 
Ethics Online System to the Health Faculty Research Ethics and Integrity Committees (FREIC) [49].  
The University of Plymouth, as the study sponsor, will ensure that the study has received ethics 
approval from a research ethics committee (REC) and has received Health Research Authority (HRA) 
approval. 
9.3 Peer Review 
The study will undergo peer review and ethical approval via the University of Plymouth research 
governance process.  
9.4 Patient and Public Involvement  
The CNWL NHS Foundation Trust, SW AHSN, and Institute of Cancer Research have robust PPI 
support infrastructures to ensure we are able to have the necessary support in place to establish our 
PPI framework for the research study. Partnerships with both of these institutions will ensure we 
align our PPI practices as we move through the research cycle. A PPI lead for the study has been 
recruited and is working with a number of PPI engagement groups at CNWL to support the trial.  
9.4.1 Involving a lay representative on the study monitoring group 
A lay representative will be appointed to the study monitoring group, additional the patient 
representative will be invited to key strategic and public meetings. The lay representative will be 
involved in all aspects of the study, specifically, these include, but are not restricted to review of 

 
 
Version number 1.2      – 01 March      2023      
 
 
 
 
 
         31 
 
each milestone and acceptance of deliverables - seeking their views around acceptability of patient-
facing materials, study procedures, and plans for translation to clinical practice.  
9.4.2 Involving a dedicated PPI group  
The broad aims of the PPI group are to support the governance of patient and public involvement 
and engagement approaches and activities within this study and ensure the public voice is present 
through all stages of the research cycle. Membership of the PPI group will be open to those both 
with a direct interest in telemedicine or mental health, but also any lay member with an interest in 
our broader aims. We will include patients and lay members both with and without previous 
experience of PPI to make sure we have a variation of skills and expertise.  
We will form a bespoke PPI group for the project including existing members of the Wysa PPI 
community as well as members of the CNWL PPI community who are interested in progressing the 
use of AI in population access to mental health. We will hold a number of experience-based co-
design (EBCD) workshops using storyboarding and community asset mapping to establish design and 
preference mapping and the group will also provide final sign off at milestone achievements in the 
work packages. 
With input from the Imperial College PERC, we have drafted, based on NIHR Involve guidance, a 
term of reference, confidentiality agreement and a background assessment form to allow us to 
realise the broad aims of our PPI group. Membership of this Wysa PPI group is voluntary but requires 
members to be committed to attend meetings and to respond to emails/correspondence. The initial 
term of membership is for the duration of the study from early in the preparation phase (app. Dec 
2021 / Jan 2022). We aim to have a minimum of four formal members at the inception of the panel 
and each subsequent meeting with representation from both London and the Southwest. We will 
seek to make participation as diverse as possible by selecting members based on 1) demographics 2) 
levels of PPI experience and 3) connection to telemedicine or mental health based on the results of 
our basic background form. We are aware that we have to ensure Wysa’s product is as accessible to 
as wide a range of possible demographics as possible, and the first phases of setting up our PPI 
group will involve a wide search using networks from SW AHSN and NIHR Involve’s “People in 
Research” platforms. 
The panel will aim to convene every four months for a 90-minute formal group discussion, given the 
evolving Covid-19 situation, we anticipate at least some of these group discussions to be conducted 
via video-conferencing, with more informal discussions happening via email or phone-calls. As the 
study spans two sites, we envision using teleconferencing to help enable maximal participation of 
our participants and to lower the barriers to involvement. At both sites, a dedicated location for 
teleconferencing will be designated to ensure we are not excluding participants less comfortable 
with or less willing to use technology. 
We anticipate that we will be having an ongoing dialogue with study participants both formally in 
the afore-described evaluation phase, and informally via ongoing feedback. As such, we anticipate 
that new members may be recruited to the PPI group if there are gaps in experience. The panel may 
choose to invite other patients/members of the public involved; other university members and/or 
representatives of voluntary or community organisations on a one/off or long-term basis. Observers, 
guests and presenters may also be invited on a one/off basis.  
The meetings will be co-chaired by a lay chairperson who is nominated by all attendees of the 
meeting, and a professional chairperson will be a senior Wysa team member who has received 
formal experience in chairing PPI meetings. We have drafted a “role” specification with Imperial 
PERC of a chairperson which may be a different individual between meetings depending on topics of 
discussion.  

 
 
Version number 1.2      – 01 March      2023      
 
 
 
 
 
         32 
 
Payments will be made in recognition of members’ time based on NIHR Involve guidance on 
payment of fees and expenses for our members actively involved [50]. Travel expenses will be 
reimbursed in accordance with this policy together with other expenses and travel costs. We have 
budgeted based on suggested reimbursement schedules and have produced our estimates following 
use of the Involvement Cost Calculator.  
9.4.3 Role of PPI group 
We will be seeking the group’s support on several key points: 
1. Recruitment - PPI representatives will be included in the interview panel for any outstanding 
staff positions to ensure we recruit empathic and relatable staff members who will best 
engage the community. Due to time constraints, this may involve either the Wysa PPI team 
or the CNWL participation group independently. 
2. Methodology - While PPI representatives have already been involved in the initial design 
process, we will also ask the group to review our final proposal as well as any iterations of 
outcome measures.  
3. Tool packs - Members of the groups will be invited to brainstorm ideas for tool packs and 
modules to be automated into guided self-help available on the clinical version of the app. 
4. Testing - Our PPI groups are an important part in our testing process prior to release of the 
clinical version of the app to the research cohort. 
5. Dissemination - A number of our PPI groups have previously been involved in the review of 
academic articles, the development of public facing content, and the presentation of 
findings at conferences and we plan to continue this during this project.  
To minimise impact on time, and in reflection of restrictions due to the Covid-19 pandemic, we run 
most of our engagement sessions online, making use of text- or video-based communication. We 
also collect feedback and input from our users through the app interface. For this award we have 
included a reimbursement budget to cover the time commitment of the PPI group as well as budget 
for The PHA Group to work with the PPI group in creating PR content. 
9.5 Protocol Compliance  
Bi-monthly study governance meetings will be held, alternating between monitoring and clinical 
reference group sessions. Monitoring meetings will include all study partners and NIHR/NHS 
representatives, and patient representatives. They will review progress against the study plan and 
budget, sign-off study deliverables and update the project’s risk register. Clinical reference group 
meetings will include senior representatives from both clinical sites who will provide oversight of 
progress, approval for go-live and ongoing monitoring of delivery. A group of external academics will 
be appointed as a trial monitoring committee who will also perform audits to ensure compliance to 
procedures. 
Planned deviations or waivers to the protocol are not allowed under the UK regulations on Clinical 
Trials and will not be used. Any accidental protocol deviations will be adequately documented on the 
relevant forms and reported to the Chief Investigator and Sponsor immediately should they happen. 
Immediate action will be taken towards any deviations to avoid serious breach. The Sponsor will be 
notified immediately should there be a serious breach of the safety or physical or mental integrity of 
the participants of the trial or the scientific value of the trial. The Sponsor will notify the licensing 
authority in writing of any serious breach.  

 
 
Version number 1.2      – 01 March      2023      
 
 
 
 
 
         33 
 
9.6 Consent  
BERA guidelines have been followed for voluntary informed consent, use of methods, and university 
policies in the event there are issues in delivery [51]. Prior to completing informed consent, 
participants will be given information that fully describes the process of the study (including the 
possibility of being randomly selected to be invited to participate in a semi-structured interview), 
including why their participation is necessary, how their data will be used, and who the results will 
be reported to. As many patients are understandably concerned about how their data will be used, 
data management will be explained in detail as part of the consent process. It will also make clear 
their right to withdraw from the study at any time and have their data destroyed. Patients will also 
be asked to separately consent for their data to be used to further studies. Declining to share this 
conversation data will not affect patients’ participation in the study or their clinical care. 
9.7 Data Protection and Patient Confidentiality  
The Wysa system stores patient identifiable data directly into iaptus electronic patient records using 
an API built through the Mayden Prism system. This solution complies with the General Data 
Protection Regulation (GDPR) and was built to meet the NHS DCB0129 safety standard and the code 
of conduct for the use of artificial intelligence [52]. 
The team has specific expertise in this area as the project lead is a qualified Clinical Safety Officer 
and has experience of overseeing the safe integration of solutions that share patient data into the 
NHS. Wysa also monitors compliance with the information governance requirements set out by the 
Department of Health and Social Care by using the Data Security and Protection Toolkit which is also 
a ‘live’ process. 
Wysa maintains a data recovery plan in case of emergency which ensures that loss of functionality 
does not directly impact on patient experience or safety. Wysa completes a regular internal audit of 
data protection and data safety as well as an annual external audit. We have clear reporting and 
escalation procedures in place and can provide copies of these policies as required. 
A Data Protection Impact Assessment will be conducted with the organisations involved in this 
study, and Information Sharing Agreements will be developed in collaboration with the CNWL NHS 
Foundation Trust’s Data Protection Officer. To mitigate patient concerns about data use, privacy, 
and security, these procedures will be clearly explained in the participant information sheet 
provided at the point of recruitment. If patients are unclear or unhappy with how their data will be 
protected, they will have the opportunity to refuse to consent to participate in the study without 
any consequences to their care. They will also be informed that they are free to withdraw their 
consent at any point in the study and will continue in the standard care pathway. 
Interactions with the AI component will create potentially identifiable patient data which will be 
stored as part of the medical record in Wysa’s role as data processor. Patient identifiable data will 
not be sold to any other party and will not be shared with any organisation unless they are a partner 
in the study and have an appropriate information sharing agreement in place. Strict access controls 
will be implemented to protect this data. 
Each of the study participants will be given a unique identifier. The primary key between unique ID 
and participant shall be stored securely for reference purposes and provided to the participant so 
that they can request that their data be withdrawn from the study. This can be requested at any 
point prior to data aggregation and will result in all of the patient’s files and data being destroyed. 
Data collected in the study will be analysed using the unique identifiers rather than patient names. 
Data subject consent forms and primary research data will be retained for 10 years following 

 
 
Version number 1.2      – 01 March      2023      
 
 
 
 
 
         34 
 
publication of the final study results. Study participants will have the option to retain their 
information within the WYSA app for their onward use or to reset their data at the completion of the 
study. 
Follow-up interview sessions shall be audio recorded by Dr Meinert and his research team. The 
interviews will be transcribed by a transcription service with only reference to the unique identifier 
provided in the audio file for transcription (the audio file will be reviewed by the CI to ensure no 
identifying information is in the audio recording; if any is provided it will be edited out); the risk of 
identification of the interview sessions shall be very low due to this measure being taken. The 
original audio recording will be destroyed following transcription. Only the Academic CIs and their 
research staff will have access to research data. The transcription service will have access only to 
interview audio, following the controls previously mentioned.  
 
9.8 Stopping Guidelines  
The study monitoring group will review the interim data for consideration of stopping the study 
early. Given that the estimated sample size is close to the minimum required for adequate power, 
the study will not be stopped early on the basis of efficacy or benefit. If the monitoring committee 
identifies any risks that have not been foreseen and mitigated against, the study will be paused 
while those risks are assessed.  
9.9 Indemnity  
The University of Plymouth holds research insurance policies which apply to this study. The Central 
North West London NHS Foundation Trust holds standard NHS Hospital Indemnity and insurance 
cover with NHS Resolution for NHS Trusts in England, which apply to this study. 
9.10 Access to code  
At the time of the close of the intervention refinement period, the Wysa app version number will be 
recorded. Changes to the platform required to maintain safety, security or 3rd party software 
dependencies will be approved by the trial steering committee and ratified by an external trial 
monitoring committee. Changes will be validated before and after the trial by an independent 
software engineer on Wysa’s source code change log. 
9.11 Access to data 
Audio recordings, transcriptions, and meta-data will be securely stored in UK data-centres with strict 
role-based access control. The transcription service will only have reference to the unique IDs and 
the audio recording will be reviewed by the CI to remove any identifying information before being 
shared. Patient identifiable data will not be sold to any other party and will not be shared with any 
organisation unless they are a partner in the study and have an appropriate information sharing 
agreement in place. Records of consent will be kept for ten years after the publication of final study 
results, but no other personally identifiable information will be stored beyond the end of the study.  

 
 
Version number 1.2      – 01 March      2023      
 
 
 
 
 
         35 
 
10 DISSEMINATION POLICY 
10.1 Dissemination Strategy  
Analysis of the data will be compiled into a published peer-reviewed paper. This data will also be 
used to support MHRA Class IIa regulatory approval and wider adoption practices. This WP will also 
produce a budget impact model for each site and provide a final overall strategy for rapid adoption 
at other NHS sites. 
The national framework for IAPT services outlines the referral and assessment pathway. WYSA has 
been designed to align with outcome measures stipulated in this framework whilst removing 
capacity restraints making it highly exploitable across the 942 IAPT service providers in England. 
Strong relationships across the AHSN network, central NHSX and Institute of Cancer Research will be 
used to disseminate the results and gain support for further deployments. 
Interoperability with digital systems is an important element of this real-world evaluation study. 
iaptus is used across the country for data capture and patient records relating to the IAPT service. 
The API integration with Mayden Prism software is an important part of our current and proposed 
work streams, as it supports easy integration with existing systems and facilitates the incorporation 
of Wysa into existing care pathways. iaptus is currently the leading patient management software 
for psychological therapies, covering over two thirds of services, which will allow for rapid technical 
deployment following the results of this study. 
10.2 Authorship Eligibility Guidelines and Use of Professional Writers  
The International Committee of Medical Journal Editors (ICMJE) guidelines will be used to determine 
the study authors [53]. The ICMJE stipulates four criteria that people involved in the study must 
meet to be considered authors for this paper:  
1. Having made a substantial contribution to the design or execution of the study; 
2. Having drafted or significantly contributed to the revision of the paper; 
3. Having final approval over submission for publication;  
4. Agreeing to be accountable for the published work.  
No professional writers will be employed.  
 
 

 
 
Version number 1.2      – 01 March      2023      
 
 
 
 
 
         36 
 
11 DECLARATIONS  
11.1 Funding  
This study is supported by the NIHR AI in Health and Care Award (AI_AWARD02176).  
11.2 Protocol Registration  
The protocol will be registered through this IRAS submission 310377 and additionally on the 
database clinicaltrials.gov (NCT05533190) [54].   
11.3 Competing Interests  
ES is an employee of Wysa; although she was involved in the drafting and revision of the protocol, 
the final decision on the evaluation design lies with the academic collaboration led by the University 
of Plymouth. 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 

 
 
Version number 1.2      – 01 March      2023      
 
 
 
 
 
         37 
 
12 REFERENCES 
1.  NHS performance and waiting times [Internet]. [cited 2021 Nov 17]. Available from: 
https://www.health.org.uk/publications/long-reads/nhs-performance-and-waiting-times 
2.  The Improving Access to Psychological Therapies Manual [Internet]. The National Collaborating 
Centre for Mental Health; 2019 Mar. Available from: https://www.rcpsych.ac.uk/docs/default-
source/improving-care/nccmh/iapt/the-iapt-manual--final--republished-7-3-
18.pdf?sfvrsn=a192d1af_0 
3.  Mental health [Internet]. World Health Organization. [cited 2021 Mar 2]. Available from: 
https://www.who.int/health-topics/mental-health#tab=tab_1 
4.  World Health Assembly. Global burden of mental disorders and the need for a comprehensive, 
coordinated response from health and social sectors at the country level: report by the 
Secretariat [Internet]. World Health Organization; 2012. Report No.: A65/10. Available from: 
https://apps.who.int/iris/handle/10665/78898 
5.  Mental health in the workplace [Internet]. World Health Organization. [cited 2021 Mar 2]. 
Available from: https://www.who.int/teams/mental-health-and-substance-use/mental-health-
in-the-workplace 
6.  McManus S, Bebbington P, Jenkins R, Brugha T, editors. Mental Health and Wellbeing in 
England: Adult Psychiatric Morbidity Survey 2014 [Internet]. Leeds: NHS Digital; 2016. Available 
from: 
https://files.digital.nhs.uk/pdf/q/3/mental_health_and_wellbeing_in_england_full_report.pdf 
7.  The Improving Access to Psychological Therapies (IAPT) Pathway for People with Long-Term 
Conditions and Medically Unexplained Symptoms: Full implementation guidance [Internet]. 
London: National Collaborating Centre for Mental Health; 2018. Available from: 
https://www.rcpsych.ac.uk/docs/default-source/improving-care/nccmh/iapt/nccmh-iapt-ltc-
full-implementation-guidance.pdf?sfvrsn=de824ea4_4 
8.  Cohen A. Addressing comorbidity between mental disorders and major noncommunicable 
diseases: Background technical report to support implementation of the WHO European Mental 
Health Action Plan 2013–2020 and the WHO European Action Plan for the Prevention and 
Control of Noncommunicable Diseases 2016–2025 [Internet]. World Health Organization; 2017. 
Available from: https://www.euro.who.int/__data/assets/pdf_file/0009/342297/Comorbidity-
report_E-web.pdf?ua=1 
9.  Payne RA, Abel GA, Guthrie B, Mercer SW. The effect of physical multimorbidity, mental health 
conditions and socioeconomic deprivation on unplanned admissions to hospital: a retrospective 
cohort study. Canadian Medical Association Journal 2013;185(5):E221–E228. 
10.  Naylor C, Parsonage M, McDaid D, Knapp M, Fossey M, Galea A. Long-term conditions and 
mental health: The cost of co-morbidities [Internet]. The King’s Fund; 2012 Feb. Available from: 
https://www.kingsfund.org.uk/sites/default/files/field/field_publication_file/long-term-
conditions-mental-health-cost-comorbidities-naylor-feb12.pdf 
11.  The Five Year Forward View for Mental Health [Internet]. The Mental Health Taskforce; 2016 
Feb. Available from: https://www.england.nhs.uk/wp-content/uploads/2016/02/Mental-
Health-Taskforce-FYFV-final.pdf 

 
 
Version number 1.2      – 01 March      2023      
 
 
 
 
 
         38 
 
12.  The Five Year Forward View for Mental Health [Internet]. The Mental Health Taskforce; 2016 
Feb. Available from: https://www.england.nhs.uk/wp-content/uploads/2016/02/Mental-
Health-Taskforce-FYFV-final.pdf 
13.  Alonso J, Liu Z, Evans-Lacko S, Sadikova E, Sampson N, Chatterji S, Abdulmalik J, Aguilar-Gaxiola 
S, Al-Hamzawi A, Andrade LH, Bruffaerts R, Cardoso G, Cia A, Florescu S, de Girolamo G, Gureje 
O, Haro JM, He Y, de Jonge P, Karam EG, Kawakami N, Kovess-Masfety V, Lee S, Levinson D, 
Medina-Mora ME, Navarro-Mateu F, Pennell B-E, Piazza M, Posada-Villa J, Ten Have M, Zarkov 
Z, Kessler RC, Thornicroft G, WHO World Mental Health Survey Collaborators. Treatment gap 
for anxiety disorders is global: Results of the World Mental Health Surveys in 21 countries. 
Depress Anxiety 2018 Mar;35(3):195–208. PMID:29356216 
14.  The Improving Access to Psychological Therapies Manual [Internet]. The National Collaborating 
Centre for Mental Health; 2019 Mar. Available from: https://www.rcpsych.ac.uk/docs/default-
source/improving-care/nccmh/iapt/the-iapt-manual--final--republished-7-3-
18.pdf?sfvrsn=a192d1af_0 
15.  Community mental health survey 2020 [Internet]. Care Quality Commission. 2020 [cited 2021 
Mar 2]. Available from: https://www.cqc.org.uk/publications/surveys/community-mental-
health-survey-2020 
16.  Community mental health survey 2020 [Internet]. [cited 2021 Mar 2]. Available from: 
https://www.cqc.org.uk/publications/surveys/community-mental-health-survey-2020 
17.  Mental health and the NHS [Internet]. [cited 2021 Mar 2]. Available from: 
https://indepth.nice.org.uk/mental-health-and-the-nhs/index.html 
18.  Two-fifths of patients waiting for mental health treatment forced to resort to emergency or 
crisis services [Internet]. Royal College of Psychiatrists. 2020 [cited 2021 Mar 2]. Available from: 
https://www.rcpsych.ac.uk/news-and-features/latest-news/detail/2020/10/06/two-fifths-of-
patients-waiting-for-mental-health-treatment-forced-to-resort-to-emergency-or-crisis-services 
19.  Working days lost in Great Britain [Internet]. Health and Safety Executive. 2020 [cited 2021 Mar 
2]. Available from: https://www.hse.gov.uk/statistics/dayslost.htm 
20.  Davies J. Mental health [Internet]. Nuffield Trust. 2020 [cited 2021 Mar 2]. Available from: 
https://www.nuffieldtrust.org.uk/news-item/mental-health-2 
21.  Parsonage M, Saini G. Mental health at work: The business costs ten years on [Internet]. Centre 
for Mental Health. 2017 [cited 2021 Mar 2]. Available from: 
https://www.centreformentalhealth.org.uk/publications/mental-health-work-business-costs-
ten-years 
22.  Clark DM. A guide to Improving Access to Psychological Therapies services [Internet]. NHS 
England. 2018 [cited 2021 Mar 2]. Available from: https://www.england.nhs.uk/blog/a-guide-
to-improving-access-to-psychological-therapies-services/ 
23.  Psychological Therapies: Annual report on the use of IAPT services, England 2018-19 [Internet]. 
NHS Digital; 2019 Jul. Available from: https://files.digital.nhs.uk/1C/538E29/psych-ther-2018-
19-ann-rep.pdf 
24.  Griffiths S, Steen S. Improving Access to Psychological Therapies (IAPT) Programme: Scrutinising 
IAPT Cost Estimates to Support Effective Commissioning. The Journal of Psychological Therapies 
in Primary Care 2013 Nov;2:142–156. 

 
 
Version number 1.2      – 01 March      2023      
 
 
 
 
 
         39 
 
25.  The NHS Long Term Plan [Internet]. NHS; 2019 Jan. Available from: 
https://www.longtermplan.nhs.uk 
26.  What we do [Internet]. NHSX. [cited 2021 Mar 2]. Available from: 
https://www.nhsx.nhs.uk/about-us/what-we-do/ 
27.  The Grand Challenges [Internet]. Department for Business, Energy, and Industrial Strategy. 2021 
[cited 2021 Mar 2]. Available from: https://www.gov.uk/government/publications/industrial-
strategy-the-grand-challenges/industrial-strategy-the-grand-challenges 
28.  Inkster B, Sarda S, Subramanian V. An Empathy-Driven, Conversational Artificial Intelligence 
Agent (Wysa) for Digital Mental Well-Being: Real-World Data Evaluation Mixed-Methods Study. 
JMIR mHealth and uHealth JMIR mHealth and uHealth; 2018 Nov 23;6(11):e12106. 
29.  Stepped Care [Internet]. Mental Health Matters. [cited 2022 Mar 10]. Available from: 
https://www.mhm.org.uk/pages/faqs/category/stepped-care 
30.  Andrews G, Cuijpers P, Craske MG, McEvoy P, Titov N. Computer therapy for the anxiety and 
depressive disorders is effective, acceptable and practical health care: a meta-analysis. PLoS 
One 2010 Oct 13;5(10):e13196. PMID:20967242 
31.  Black, Asian and Minority Ethnic (BAME) communities [Internet]. Mental Health Foundation. 
2015 [cited 2021 Mar 2]. Available from: https://www.mentalhealth.org.uk/a-to-z/b/black-
asian-and-minority-ethnic-bame-communities 
32.  Mental health statistics: LGBTIQ+ people [Internet]. 2018 [cited 2021 Mar 2]. Available from: 
https://www.mentalhealth.org.uk/statistics/mental-health-statistics-lgbtiq-people 
33.  Delivering a “Net Zero” National Health Service [Internet]. NHS; 2020 Oct. Available from: 
https://www.england.nhs.uk/greenernhs/publication/delivering-a-net-zero-national-health-
service/ 
34.  Greenhalgh T, Wherton J, Papoutsi C, Lynch J, Hughes G, A’Court C, Hinder S, Fahy N, Procter R, 
Shaw S. Beyond Adoption: A New Framework for Theorizing and Evaluating Nonadoption, 
Abandonment, and Challenges to the Scale-Up, Spread, and Sustainability of Health and Care 
Technologies. J Med Internet Res 2017 Nov 1;19(11):e367. PMID:29092808 
35.  Sekhon M, Cartwright M, Francis JJ. Acceptability of healthcare interventions: an overview of 
reviews and development of a theoretical framework. BMC Health Serv Res BioMed Central; 
2017 Jan 26;17(1):1–13. 
36.  Kroenke K, Spitzer RL, Williams JB. The PHQ-9: validity of a brief depression severity measure. J 
Gen Intern Med 2001 Sep;16(9):606–613. PMID:11556941 
37.  Spitzer RL, Kroenke K, Williams JBW, Löwe B. A brief measure for assessing generalized anxiety 
disorder: the GAD-7. Arch Intern Med 2006 May 22;166(10):1092–1097. PMID:16717171 
38.  About iaptus [Internet]. iaptus. 2020 [cited 2022 Mar 10]. Available from: https://iaptus.co.uk/ 
39.  Torous J, Michalak EE, O’Brien HL. Digital Health and Engagement—Looking Behind the 
Measures and Methods. JAMA Network Open 2020;3(7):e2010918. 
40.  Herdman M, Gudex C, Lloyd A, Janssen M, Kind P, Parkin D, Bonsel G, Badia X. Development and 
preliminary testing of the new five-level version of EQ-5D (EQ-5D-5L). Qual Life Res 2011 

 
 
Version number 1.2      – 01 March      2023      
 
 
 
 
 
         40 
 
Dec;20(10):1727–1736. PMID:21479777 
41.  SF-12v2® Health Survey [Internet]. Quality Metric | We Measure Health. 2021 [cited 2022 Feb 
22]. Available from: https://www.qualitymetric.com/health-surveys-old/the-sf-12v2-health-
survey/ 
42.  Brazier J. Is the EQ-5D fit for purpose in mental health? Br J Psychiatry. 2010. p. 348–349. 
PMID:21037210 
43.  Franklin M, Enrique A, Palacios J, Richards D. Psychometric assessment of EQ-5D-5L and ReQoL 
measures in patients with anxiety and depression: construct validity and responsiveness. Qual 
Life Res Springer; 2021 Apr 9;30(9):2633–2647. 
44.  Huo T, Guo Y, Shenkman E, Muller K. Assessing the reliability of the short form 12 (SF-12) health 
survey in adults with mental health conditions: a report from the wellness incentive and 
navigation (WIN) study. Health Qual Life Outcomes 2018 Feb 13;16(1):34. PMID:29439718 
45.  Introduction to health technology evaluation [Internet]. National Institute for Health and Care 
Excellence; 2022 Jan. Available from: 
https://www.nice.org.uk/process/pmg36/chapter/introduction-to-health-technology-
evaluation 
46.  Data Protection Act 2018 [Internet]. legislation.gov.uk. Queen’s Printer of Acts of Parliament; 
2018 [cited 2021 Jan 20]. Available from: 
https://www.legislation.gov.uk/ukpga/2018/12/contents/enacted 
47.  A guide to good practice for digital and data-driven health technologies [Internet]. Department 
of Health & Social Care. 2021 [cited 2021 Mar 2]. Available from: 
https://www.gov.uk/government/publications/code-of-conduct-for-data-driven-health-and-
care-technology/initial-code-of-conduct-for-data-driven-health-and-care-technology 
48.  HRA Approval [Internet]. NHS Health Research Authority. 2020 [cited 2021 Jan 20]. Available 
from: https://www.hra.nhs.uk/approvals-amendments/what-approvals-do-i-need/hra-
approval/ 
49.  Research ethics [Internet]. University of Plymouth. [cited 2021 Jan 20]. Available from: 
https://www.plymouth.ac.uk/research/governance/research-ethics-policy 
50.  Policy on payment of fees and expenses for members of the public actively involved with 
INVOLVE [Internet]. INVOLVE, National Institute of Health Research; Available from: 
https://www.invo.org.uk/wp-content/uploads/2016/05/INVOLVE-internal-payment-policy-
2016-final-1.pdf 
51.  Ethical Guidelines for Educational Research [Internet]. British Educational Research Association; 
2018. Available from: https://www.bera.ac.uk/publication/ethical-guidelines-for-educational-
research-2018-online 
52.  Department of Health and Social Care. New code of conduct for artificial intelligence (AI) 
systems used by the NHS [Internet]. GOVUK. 2019 [cited 2021 Mar 2]. Available from: 
https://www.gov.uk/government/news/new-code-of-conduct-for-artificial-intelligence-ai-
systems-used-by-the-nhs 
53.  Defining the Role of Authors and Contributors [Internet]. International Committee of Medical 
Journal Editors. [cited 2021 Jan 20]. Available from: 

 
 
Version number 1.2      – 01 March      2023      
 
 
 
 
 
         41 
 
http://www.icmje.org/recommendations/browse/roles-and-responsibilities/defining-the-role-
of-authors-and-contributors.html 
54.  Home [Internet]. ClinicalTrials.gov. [cited 2021 Jan 20]. Available from: 
https://www.clinicaltrials.gov/ 
 
 
 

 
 
Version number 1.2      – 01 March      2023      
 
 
 
 
 
         42 
 
13 APPENDICES  
13.1 Appendix 1. Participant Information Sheet  
 
Most up to date version incorporated into IRAS documents 
 
 
 
 

 
 
Version number 1.2      – 01 March      2023      
 
 
 
 
 
         43 
 
13.2 Appendix 2. Informed Consent Form   
 
 
Most up to date version incorporated into IRAS documents 
 
 

 
 
Version number 1.2      – 01 March      2023      
 
 
 
 
 
         44 
 
13.3 Appendix 3. Sample Question Guide for Semi-Structured Interviews 
The topic guide was developed based on the Theoretical Framework of Acceptability (TFA) [34], 
which was created to provide a framework for assessing the multiple facets of acceptability of health 
interventions. The TFA has seven components: “1) affective attitude, 2) burden, 3) ethicality, 4) 
intervention coherence, 5) opportunity costs, 6) perceived effectiveness, and 7) self-efficacy” [34].  
 
1. How did you feel about your experience interacting with Wysa? 
2. How much effort did interacting with Wysa take? 
3. What concerns, if any, did you have while interacting with Wysa? 
4. How well did you understand how Wysa worked? 
5. What benefits or losses did you experience while you were interacting with Wysa? 
6. How much confidence did you have in Wysa’s ability to provide mental health support? 
7. How comfortable and confident were you in your ability to interact with Wysa? 
 
 

 
 
Version number 1.2      – 01 March      2023      
 
 
 
 
 
         45 
 
13.4 Appendix 4. GP Letter 
[header: GP letter Participant Consented, v1.0, XX/XX/XXXX; study logo] 
[GP Name] 
[GP Practice] 
[Address] 
[Date] 
[Study Address] 
[Address] 
[Telephone/Email/Social media] 
 Our ref: XXXX 
Dear Dr [GP name] 
  
Re: [participant’s full name], [participant’s date of birth], [participant’s address] 
  
We would like to inform you that [participant’s full name] has consented to join the Wysa Study. This 
project is a collaboration between the University of Plymouth, the Institute for Cancer Research, Wysa 
Ltd, and the Central North West London NHS Foundation Trust. The project is funded by the NIHR AI 
in Health and Care Award (ref: AI_AWARD02176) and has full ethical approval from [Research Ethics 
Committee (ref: TBD)]. No action is needed on your part, but [participant’s full name] has provided 
permission that should anything of concern arise as a result of their participation in our study, we will 
discuss this with you.   
  
Our study aims to measure and characterise the impact of engaging with Wysa while waiting for their 
first appointment through the IAPT service on symptoms of depression and anxiety. Participants 
randomised into the intervention group will be invited to use the app while waiting for treatment; 
participants randomised into the control group will be given access to the app after 3 months. 
 
Participation is voluntary – [participant’s name] is free to withdraw from our study at any point.  If you 
would like any further information or if there is anything you would like to discuss, then please do not 
hesitate to contact us. 
 
  
Yours sincerely, 
  
[CI’s/PI’s name] Team Member’s name] 
 
 
 

 
 
Version number 1.2      – 01 March      2023      
 
 
 
 
 
         46 
 
13.5 Appendix 5. SPIRIT-AI Checklist: Recommended items to address in a protocol 
and related documents for clinical trials evaluating AI interventions 
Cruz Rivera S, Liu X, Chan A-W, Denniston AK, Calvert MJ, SPIRIT-AI and CONSORT-AI Working Group. 
Guidelines for clinical trial protocols for interventions involving artificial intelligence: the SPIRIT-AI extension. 
Lancet Digit Health 2020 Oct;2(10):e549–e560. PMID:33328049 
Section 
 
SPIRIT 2013 Itema 
SPIRIT-AI Item 
Section 
No 
Administrative Information 
 
Title 
 
1 
Descriptive title identifying the study design, 
population, interventions, and, if applicable, 
trial acronym 
SPIRIT-AI 
1(i) 
Elaboration 
Indicate that the 
intervention involves 
artificial intelligence / 
machine learning and 
specify the type of 
model. 
1 
SPIRIT-AI 
1(ii) 
Elaboration 
Specify the intended use 
of the AI intervention. 
1 
Trial 
registration 
2a 
Trial identifier and registry name. If not yet 
registered, name of intended registry 
 
 
11.2 
2b 
All items from the World Health Organization 
Trial Registration Data Set 
 
 
Through-
out 
protocol 
Protocol 
version 
3 
Date and version identifier 
 
 
1.3 
Funding 
4 
Sources and types of financial, material, and 
other support 
 
 
5.1 
Roles and 
responsibilit
ies 
5a 
Names, affiliations, and roles of protocol 
contributors 
 
 
3, 5.4 
5b 
Name and contact information for the trial 
sponsor 
 
 
3 
5c 
Role of study sponsor and funders, if any, in 
study design; collection, management, 
analysis, and interpretation of data; writing of 
the report; and the decision to submit the 
report for publication, including whether they 
will have ultimate authority over any of these 
activities 
 
 
5.2 
5d 
Composition, roles, and responsibilities of the 
coordinating centre, steering committee, 
endpoint adjudication committee, data 
management team, and other individuals or 
groups overseeing the trial, if applicable (see 
Item 21a for data monitoring committee) 
 
 
5.3 

 
 
Version number 1.2      – 01 March      2023      
 
 
 
 
 
         47 
 
Introduction 
Background 
and 
rationale 
6a 
Description of research question and 
justification for undertaking the trial, including 
summary of relevant studies (published and 
unpublished) examining benefits and harms 
for each intervention 
SPIRIT-AI 6a 
(i) Extension 
Explain the intended use 
of the AI intervention in 
the context of the clinical 
pathway, including its 
purpose and its intended 
users (e.g. healthcare 
professionals, patients, 
public). 
6, 7.1, 
7.2, 8.2.3 
SPIRIT-AI 6a 
(ii) 
Extension 
Describe any pre-existing 
evidence for the AI 
intervention. 
7.2 
6b 
Explanation for choice of comparators 
 
 
8.2.4 
Objectives 
7 
Specific objectives or hypotheses 
 
 
7.5 
Methods: Participants, Interventions and Outcomes 
Trial design 
8 
Description of trial design including type of 
trial (e.g., parallel group, crossover, factorial, 
single group), allocation ratio, and framework 
(e.g., superiority, equivalence, noninferiority, 
exploratory) 
 
 
8.1 
Study 
setting 
9 
Description of study settings (e.g., community 
clinic, academic hospital) and list of countries 
where data will be collected. Reference to 
where list of study sites can be obtained 
SPIRIT-AI 9 
Extension 
Describe the onsite and 
offsite requirements 
needed to integrate the 
AI intervention into the 
trial setting. 
8.2.2 
Eligibility 
criteria 
10 
Inclusion and exclusion criteria for 
participants. If applicable, eligibility criteria for 
study centres and individuals who will perform 
the interventions (e.g., surgeons, 
psychotherapists) 
SPIRIT-AI 10 
(i) 
Elaboration 
State the inclusion and 
exclusion criteria at the 
level of participants. 
8.3.1 
SPIRIT-AI 10 
(ii) 
Extension 
State the inclusion and 
exclusion criteria at the 
level of the input data. 
 
 
 
Interventio
ns 
11a 
Interventions for each group with sufficient 
detail to allow replication, including how and 
when they will be administered 
SPIRIT-AI 
11a (i) 
Extension 
State which version of 
the AI algorithm will be 
used. 
9.10 
SPIRIT-AI 
11a (ii) 
Extension 
Specify the procedure for 
acquiring and selecting 
the input data for the AI 
intervention. 
8.2.3 
SPIRIT-AI 
11a (iii) 
Extension 
Specify the procedure for 
assessing and handling 
poor quality or 
unavailable input data. 
8.2.3 

 
 
Version number 1.2      – 01 March      2023      
 
 
 
 
 
         48 
 
SPIRIT-AI 
11a (iv) 
Extension 
Specify whether there is 
human-AI interaction in 
the handling of the input 
data, and what level of 
expertise is required for 
users. 
8.2.3 
SPIRIT-AI 
11a (v) 
Extension 
Specify the output of the 
AI intervention. 
8.2.3 
SPIRIT-AI 
11a (vi) 
Extension 
Explain the procedure for 
how the AI intervention’s 
output will contribute to 
decision-making or other 
elements of clinical 
practice. 
8.2.3 
11b 
Criteria for discontinuing or modifying 
allocated interventions for a given trial 
participant (e.g., drug dose change in response 
to harms, participant request, or 
improving/worsening disease) 
 
 
N/A 
11c 
Strategies to improve adherence to 
intervention protocols, and any procedures for 
monitoring adherence (e.g., drug tablet return, 
laboratory tests) 
 
 
N/A 
11d 
Relevant concomitant care and interventions 
that are permitted or prohibited during the 
trial 
 
 
N/A 
Outcomes 
12 
Primary, secondary, and other outcomes, 
including the specific measurement variable 
(e.g., systolic blood pressure), analysis metric 
(e.g., change from baseline, final value, time to 
event), method of aggregation (eg, median, 
proportion), and time point for each outcome. 
Explanation of the clinical relevance of chosen 
efficacy and harm outcomes is strongly 
recommended 
 
 
8.2.5 
Participant 
timeline 
13 
Time schedule of enrolment, interventions 
(including any run-ins and washouts), 
assessments, and visits for participants. A 
schematic diagram is highly recommended 
(see Figure) 
 
 
8.1  
Sample size 
14 
Estimated number of participants needed to 
achieve study objectives and how it was 
determined, including clinical and statistical 
assumptions supporting any sample size 
calculations 
 
 
 
8.3.2.1 
Recruitmen
t 
15 
Strategies for achieving adequate participant 
enrolment to reach target sample size 
 
 
8.3.3 

 
 
Version number 1.2      – 01 March      2023      
 
 
 
 
 
         49 
 
Methods: Assignment of Interventions (For Controlled Trials) 
Sequence 
generation 
16A 
Method of generating the allocation sequence 
(eg, computer-generated random numbers), 
and list of any factors for stratification. To 
reduce predictability of a random sequence, 
details of any planned restriction (eg, blocking) 
should be provided in a separate document 
that is unavailable to those who enrol 
participants or assign interventions 
 
 
8.4.1.1 
Allocation 
concealmen
t 
mechanism 
16b 
Mechanism of implementing the allocation 
sequence (eg, central telephone; sequentially 
numbered, opaque, sealed envelopes), 
describing any steps to conceal the sequence 
until interventions are assigned 
 
 
8.4.1.2 
Implementa
tion 
16c 
Who will generate the allocation sequence, 
who will enrol participants, and who will 
assign participants to interventions 
 
 
8.4.1.3 
Blinding 
(masking) 
17a 
Who will be blinded after assignment to 
interventions (eg, trial participants, care 
providers, outcome assessors, data analysts), 
and how 
 
 
8.4.2 
17b 
If blinded, circumstances under which 
unblinding is permissible, and procedure for 
revealing a participant’s allocated intervention 
during the trial 
 
 
8.4.2 
Methods: Data Collection, Management, And Analysis 
Data 
collection 
methods 
18a 
Plans for assessment and collection of 
outcome, baseline, and other trial data, 
including any related processes to promote 
data quality (eg, duplicate measurements, 
training of assessors) and a description of 
study instruments (eg, questionnaires, 
laboratory tests) along with their reliability 
and validity, if known. Reference to where 
data collection forms can be found, if not in 
the protocol 
 
 
8.4.3 
18b 
Plans to promote participant retention and 
complete follow-up, including list of any 
outcome data to be collected for participants 
who discontinue or deviate from intervention 
protocols 
 
 
8.4.3.2 
Data 
manageme
nt 
19 
Plans for data entry, coding, security, and 
storage, including any related processes to 
promote data quality (eg, double data entry; 
range checks for data values). Reference to 
where details of data management procedures 
can be found, if not in the protocol 
 
 
9.7 

 
 
Version number 1.2      – 01 March      2023      
 
 
 
 
 
         50 
 
Statistical 
methods 
20a 
Statistical methods for analysing primary and 
secondary outcomes. Reference to where 
other details of the statistical analysis plan can 
be found, if not in the protocol 
 
 
8.5 
20b 
Methods for any additional analyses (eg, 
subgroup and adjusted analyses) 
 
 
8.5 
20c 
Definition of analysis population relating to 
protocol non-adherence (eg, as randomised 
analysis), and any statistical methods to 
handle missing data (eg, multiple imputation) 
 
 
8.5 
Methods: Monitoring 
Data 
monitoring 
21a 
Composition of data monitoring committee 
(DMC); summary of its role and reporting 
structure; statement of whether it is 
independent from the sponsor and competing 
interests; and reference to where further 
details about its charter can be found, if not in 
the protocol. Alternatively, an explanation of 
why a DMC is not needed 
 
 
9.4, 9.5 
21b 
Description of any interim analyses and 
stopping guidelines, including who will have 
access to these interim results and make the 
final decision to terminate the trial 
 
  
 
9.8 
Harms 
22 
Plans for collecting, assessing, reporting, and 
managing solicited and spontaneously 
reported adverse events and other 
unintended effects of trial interventions or 
trial conduct 
SPIRIT-AI 22 
Extension 
Specify any plans to 
identify and analyse 
performance errors. If 
there are no plans for 
this, explain why not. 
9.1 
Auditing 
23 
Frequency and procedures for auditing trial 
conduct, if any, and whether the process will 
be independent from investigators and the 
sponsor 
 
 
  
9.5 
Ethics and Dissemination 
Research 
ethics 
approval 
24 
Plans for seeking research ethics 
committee/institutional review board 
(REC/IRB) approval 
 
 
9.2 
Protocol 
amendment
s 
25 
Plans for communicating important protocol 
modifications (eg, changes to eligibility 
criteria, outcomes, analyses) to relevant 
parties (eg, investigators, REC/IRBs, trial 
participants, trial registries, journals, 
regulators) 
 
 
9.5 
Consent or 
assent 
26a 
Who will obtain informed consent or assent 
from potential trial participants or authorised 
surrogates, and how (see Item 32) 
 
 
8.3.3.2, 
9.6 

 
 
Version number 1.2      – 01 March      2023      
 
 
 
 
 
         51 
 
26b 
Additional consent provisions for collection 
and use of participant data and biological 
specimens in ancillary studies, if applicable 
 
 
9.6 
Confidential
ity 
27 
How personal information about potential and 
enrolled participants will be collected, shared, 
and maintained in order to protect 
confidentiality before, during, and after the 
trial 
 
 
9.7 
Declaration 
of interests 
28 
Financial and other competing interests for 
principal investigators for the overall trial and 
each study site 
 
 
11 
Access to 
data 
29 
Statement of who will have access to the final 
trial dataset, and disclosure of contractual 
agreements that limit such access for 
investigators 
SPIRIT-AI 29 
Extension 
State whether and how 
the AI intervention 
and/or its code can be 
accessed, including any 
restrictions to access or 
re-use. 
9.10, 9.11 
Ancillary 
and post-
trial care 
30 
Provisions, if any, for ancillary and post-trial 
care, and for compensation to those who 
suffer harm from trial participation 
 
 
N/A 
Disseminati
on policy 
31a 
Plans for investigators and sponsor to 
communicate trial results to participants, 
healthcare professionals, the public, and other 
relevant groups (eg, via publication, reporting 
in results databases, or other data sharing 
arrangements), including any publication 
restrictions 
 
 
10.1 
31b 
Authorship eligibility guidelines and any 
intended use of professional writers 
 
 
10.2 
31c 
Plans, if any, for granting public access to the 
full protocol, participant-level dataset, and 
statistical code 
 
 
N/A 
Appendices 
Informed 
consent 
materials 
32 
Model consent form and other related 
documentation given to participants and 
authorised surrogates 
 
  
 
13.1, 13.2  
Biological 
specimens 
33 
Plans for collection, laboratory evaluation, and 
storage of biological specimens for genetic or 
molecular analysis in the current trial and for 
future use in ancillary studies, if applicable 
 
 
N/A 
a It is strongly recommended that this checklist be read in conjunction with the SPIRIT 2013 
Explanation & Elaboration for important clarification on the items.  
b Indicates page numbers to be completed by authors during protocol development