URAISE: Ultrasound Regional Anaesthesia Interpretation Skill Evaluation
Phase: NA
Conditions: Anesthesia
Sponsor: Imperial College Healthcare NHS Trust

 
 
Research Governance 
 
           and Integrity Team 
 
 
1 
 
Version 1.0 / 07.07.2021 
 
URAISE: Ultrasound Regional Anaesthesia 
Interpretation Skill Evaluation  
 
 
Main Sponsor:  Imperial College Healthcare NHS Trust                                      
Funders: Unfunded 
 
Study coordination centre: St Mary’s Hospital, Paddington 
IRAS ID: 294856 
 
 

 
 
Research Governance 
 
           and Integrity Team 
 
 
2 
 
Study Management Group 
Chief Investigator: Dr Boyne Bellew 
Co-investigators: Dr FA Cook, Dr M Murali, Dr S Habib, Dr S Haddad, Dr J Brandreth, Dr Amit 
Pawa, Dr AJ MacFarlane, Dr Peter Merjavy, Dr Madan Narayanan, Dr Lloyd Turbitt, Dr James 
Bowness, Prof William Harrop-Griffiths 
Statistician Mr Roger A’Hern, 
Study Management: Dr Boyne Bellew 
 
Study Coordination Centre 
For general queries, supply of study documentation, and collection of data, please contact: 
Study Coordinator: Dr Boyne Bellew 
Address: Department of Anaesthesia & Critical Care, 
                 St Mary’s Hospital, Paddington 
                 Anaesthesia / Critical Care / Theatres, Specialist Services - CPG 3 
                 4th Floor, QEQM Building, St Mary’s Hospital, Praed Street,  
                 London W2 1NY 
Registration: GMC 7000811 
Tel: 07710787321 
 
 
 
 
E-mail:  boyne.bellew@nhs.net 
 
Clinical Queries 
Clinical queries should be directed to Dr Boyne Bellew who will direct the query to the 
appropriate person. 
 
Sponsor 
Imperial College Healthcare NHS Trust is the main research sponsor for this study. For 
further information regarding the sponsorship conditions, please contact the Head of 
Regulatory Compliance at: 
 
Research Governance and Integrity Team 
Imperial College London and Imperial College Healthcare NHS Trust 
Room 215, Level 2, Medical School Building 
Norfolk Place 
London, W2 1PG 
Tel: 0207 594 9459/ 0207 594 1862 
http://www3.imperial.ac.uk/clinicalresearchgovernanceoffice  
 
         
Funder 
Unfunded 
 

 
 
Research Governance 
 
           and Integrity Team 
 
 
3 
 
This protocol describes the URAISE: Ultrasound Regional Anaesthesia Interpretation Skill 
Evaluation study and provides information about procedures for entering participants.  Every 
care was taken in its drafting, but corrections or amendments may be necessary. These will 
be circulated to investigators in the study.  Problems relating to this study should be referred, 
in the first instance, to the Chief Investigator.  
This study will adhere to the principles outlined in the UK Policy Frame Work for Health and 
Social Care Research. It will be conducted in compliance with the protocol, the Data Protection 
Act and other regulatory requirements as appropriate.  
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 

 
 
Research Governance 
 
           and Integrity Team 
 
 
4 
 
Table of Contents                                                              
 
 
Page No 
1. 
INTRODUCTION 
7 
1.1. 
BACKGROUND 
7 
2. 
STUDY OBJECTIVES 
9 
3. 
STUDY DESIGN 
10 
4. 
STUDY OUTCOME MEASURES 
14 
5. 
PARTICIPANT ENTRY 
14 
5.1. 
PRE-REGISTRATION EVALUATIONS 
14 
5.2 
PARTICIPANT RECRUITMENT 
15 
5.3. 
INCLUSION CRITERIA 
15 
5.4. 
EXCLUSION CRITERIA 
15 
5.5.     PARTICIPANT WITHDRAWL 
15 
6.    ASSESSMENT 
15 
6.1.  DEFINITION OF THE END OF THE TRIAL 
15 
6.2.    LIKELY RATE OF LOSS TO FOLLOW-UP 
16 
7. 
STATISTICS AND DATA ANALYSIS 
16 
8. 
REGULATORY ISSUES 
17 
8.1. 
ETHICS APPROVAL 
17 
8.2. 
CONSENT 
17 
8.3. 
CONFIDENTIALITY 
18 
8.4. 
INDEMNITY 
18 
8.5. 
SPONSOR 
18 
8.6. 
FUNDING 
18 
8.7. 
AUDITS 
18 
9. 
STUDY MANAGEMENT 
18 
10. 
PUBLICATION POLICY 
18 
11. 
REFERENCES 
20 
12. 
APPENDIX (Sample questions) 
 
 
 

 
 
Research Governance 
 
           and Integrity Team 
 
 
5 
 
Glossary of Abbreviations 
RCoA 
Royal College of Anaesthetists  
CRQ 
Constructed response questions 
 
 
 
 
 
 
 
 
 
 
 
 
 
Keywords: Regional Anaesthesia, Sonoanatomy 
 
 

 
 
Research Governance 
 
           and Integrity Team 
 
 
6 
 
Study Summary 
Title 
URAISE: Ultrasound Regional Anaesthesia 
Interpretation Skill Evaluation 
Design 
Prospective study based on psychometric 
principles 
Aims 
To generate a validated test of ultrasound 
guided regional anaesthesia that can used to 
differentiate reliably between users of different 
skill levels.  
Outcomes 
Primary outcome:  
Development of a validated, reliable and 
defensible tool to test ultrasound guided 
regional anaesthesia image interpretation skills.  
Secondary outcomes: 
Development a validated, reliable and defensible 
assessment tool assessing anatomical, clinical 
knowledge and the performance of ultrasound guided 
regional anaesthesia: 
 
Assessment of anatomical knowledge relevant 
to performing a local anaesthetic injection 
around peripheral nerves 
Assessment of clinical knowledge relevant to 
regional anaesthesia required to perform a local 
anaesthetic injection around peripheral nerves 
Assessment of knowledge relating to the  
performance of an ultrasound-guided local 
anaesthetic injection around peripheral nerve 
Population 
UK anaesthetic trainees and consultants 
Eligibility 
Participants must be registered in an approved 
UK anaesthetic training programme, or 
anaesthetic consultants on the GMC register 
Duration 
12 months 

 
 
Research Governance 
 
           and Integrity Team 
 
 
7 
 
1. Introduction 
1.1 Background 
There is a drive for trainee anaesthetists to be competent in a number of common ultrasound 
guided blocks1 and proficiency is considered an important part of the anaesthetic curriculum 
laid out by the RCoA2. Ultrasound improves the safety, efficacy and efficiency of peripheral 
nerve blocks3-6. It is now the accepted standard of practice in most institutions. However, one 
challenge for trainers and trainees is that there is a lack of simple, readily available, evidence-
based and validated methods to assess the anaesthetists’ ability to identify structures using 
ultrasound. The development of a validated tool to evaluate proficiency in ultrasound-guided 
regional anaesthesia (UGRA) interpretation would improve the uniformity and consistency of 
assessment in the UK.  
Validated tools for assessment of ultrasound may be even more valuable with the recognised 
advantages and increased use of regional anaesthetic techniques during the COVID-19 
pandemic. Possible advantages to anaesthetists and hospitals include reduced risk of 
transmission to staff, and preservation of drugs and personal protective equipment required 
during critical care of COVID-19 patients. These patients may also benefit from avoiding the 
further cardiorespiratory strain of undergoing a general anaesthetic. All patients may also 
experience improved postoperative pain relief (thereby also minimising direct contact with 
care givers), reduced time in recovery and earlier hospital discharge7.  
Legal defensibility is a concept that is central to any assessment tool. There needs to be 
confidence in the processes used to construct the assessment tool and that the outcomes of 
the test reflect the participant’s ability. The application of rigorous psychometric principles is 
essential to achieving the high standard of validity and reliability of an assessment required 
by educational institutions. Anaesthetists seeking certification that is not based on a validated 
assessment method could give rise to the question of ethical injustice and possibly expose an 
organisation to legal challenges8. On the other hand, anaesthetists that are inappropriately 
certified may risk patient safety.  
The assessment tool must measure what it is supposed to measure. Firstly, face validity will 
be ensured by our participants judging the questions as easy to understand and the 
ultrasound video quality as representative of clinical practice. The content validity of the 
assessment tool will be verified by a panel of experts who will be asked to assess its relevance 
to regional anaesthesia by matching it to the RCoA curriculum. Next, the construct validity of 
the assessment will be evidenced by anaesthetists with more regional anaesthesia experience 
achieving higher ultrasound interpretation scores. The stages of training in our assessment 
tool will include consultants and will also be based on the new 2021 RCoA curriculum that is 

 
 
Research Governance 
 
           and Integrity Team 
 
 
8 
 
divided into Stage 1 (CT1-3), Stage 2 (ST4-5) and Stage 3 (ST6-ST7) in a spiral curriculum that 
is both competency and time-based. This is also a proxy outcome measure in that 
advancement through these competency-based stages of training already include forms of 
assessment of proficiency in UGRA. Demographic information will be gathered from the 
participants to demonstrate their prior regional anaesthesia experience. This will factor in 
additional courses, qualifications and fellowships in relevant areas to account for 
inconsistencies in clinical exposure to ultrasound throughout training and as a practising 
anaesthetist (specifically, training level, courses attended, regional anaesthesia fellowships 
completed, and ultrasound-based fellowships completed).  
Consistency of measurement or reliability is essential in the assessment tool. Cronbach’s 
alpha method is commonly used to assess internal consistency reliability. We will use also use 
a “split-half” method of comparison in the final version of the test wherein the performance 
of the same participant in the first half is compared to their performance in the second half. 
The optimal test length is a balance. Using more test items produces more measurements of 
the construct and therefore result in higher reliability. This must be set against potential 
cognitive fatigue and participant drop out with longer test times. Statistical analysis of the 
pilot testing will allow items with higher discrimination to be kept in the final version of the 
test and lower discrimination, or lower measurement value, to be removed. Inter-rater or 
marker variability can be minimised by a system of automatic computer marking. However, 
the marking will also be corroborated by manual checking by investigators blinded to the 
participant’s training level and experience.   
Several studies have conducted psychometric evaluations of assessment tools for the 
development of regional anaesthesia knowledge and skills9. These include measuring hand-
motion analysis10, 11, visuospatial and psychomotor screening12, 13, 14, checklists15 and global 
rating scales16. In particular, Woodworth et al showed the validity and reliability of a 47-item 
multiple-choice online test of ultrasound interpretation skills for regional anaesthesia that 
can be used as an assessment of competence milestone in anaesthesia training for a training 
programme in the USA17.  There has not been a study to create such an assessment tool in 
the UK, where the training programme differs from that in the USA. The UK has a seven year 
specialist anaesthetic training programme compared to the American four year residency in 
anaesthesiology. There is evidence that “constructed response questions” (CRQs) with free 
text answers require a higher degree of understanding and are considered higher on Bloom’s 
taxonomy15 (knows, comprehends, applies, analyses, synthesises, evaluates) in terms of the 
complexity of skills demonstrated by the learner.  This is also in line with the changing 
curriculum of the RCoA that has recently moved towards using CRQs in their Final Fellowship 
of the Royal College of Anaesthetists (FRCA) examinations. These CRQs challenge trainees to 

 
 
Research Governance 
 
           and Integrity Team 
 
 
9 
 
apply, justify or compare information to arrive at solutions to clinical questions, thereby 
better reflecting daily clinical practice18. We therefore aim to create a well-validated 
assessment tool that uses CRQs to assess ultrasound interpretation skills, which is grounded 
in the psychometric principles of validity, reliability and defensibility.  
2. Study Objectives 
The primary objective of the study is the development of a validated, reliable and defensible 
assessment tool to test ultrasound-guided regional anaesthesia image interpretation skills. 
The secondary objectives are development a validated, reliable and defensible assessment 
tool assessing anatomical and clinical knowledge relevant to regional anaesthesia; as  well as 
assessing knowledge pertaining to the performance of ultrasound guided regional 
anaesthesia: 
• Assessment of anatomical knowledge relevant to performing a local anaesthetic 
injection around peripheral nerves 
• Assessment of clinical regional anaesthesia knowledge relevant to performing a local 
anaesthetic injection around peripheral nerves 
• Assessment of the performance of an ultrasound-guided local anaesthetic injection 
around peripheral nerve 
This will support training and assessment in UGRA by UK anaesthetists by enabling trainers to 
track educational progress. This in turn will support performance of UGRA, improving patient 
experience, enhancing patient safety by reducing complications associated with these blocks, 
and increasing the pool of skilled UGRA practitioners. 
 
 
 
 
 
 
 
 

 
 
Research Governance 
 
           and Integrity Team 
 
 
10 
 
3. Study Design 
This is a prospective, multi-centre study leading to the identification of a validated, reliable 
and defensible assessment tool for assessing UGRA image interpretation. 
Study population 
 
Anaesthetists in the UK 
Inclusion 
Registered in an approved UK anaesthetic training programme 
Anaesthetic consultants on the GMC register 
Exclusion(s) 
Staff grade, associate specialist and specialty doctors not 
enrolled in a UK anaesthetic training programme 
Sample size  
Pilot phase: 35-50 
Study phase: 250  
Duration 
 
 
12 months 
Study centres 
 
Imperial College Healthcare NHS Trust, London 
 
Royal Gwent Hospital, Aneurin Bevan University Health Board, 
Wales 
Southern Health and Social Care Trust, Northern Ireland, UK  
Belfast Health and Social Care Trust, Northern Ireland, UK 
 
 
 
 
NHS Greater Glasgow & Clyde, Scotland, UK 
 
 
 
 
Guy’s & St Thomas NHS Foundation Trust, London, UK 
 
 
 
 
Frimley Health NHS Foundation Trust, UK 
Primary outcome 
A validated, reliable and defensible assessment tool to test 
ultrasound-guided regional anaesthesia image interpretation 
skills 
Secondary outcome(s) 
The secondary outcomes are development a validated, reliable 
and defensible assessment tool assessing anatomical, clinical knowledge and the 
performance of ultrasound guided regional anaesthesia: 
• Assessment of anatomical knowledge relevant to performing a local anaesthetic injection 
around peripheral nerves 
• Assessment of clinical regional anaesthesia knowledge relevant to performing a local 
anaesthetic injection around peripheral nerves• Assessment of 
the performance of an ultrasound-guided local anaesthetic 
injection around peripheral nerve 
 
 
 
This study will have 5 Phases: 

 
 
Research Governance 
 
           and Integrity Team 
 
 
11 
 
3.1 Phase 1 – Nerve block identification 
A panel of academic experts in regional anaesthesia will extensively review published 
evidence from UGRA international consensus guidelines. This first step will determine which 
UGRA nerve blocks are relevant to clinical practice and anaesthetic training in the UK, in 
particular their relevance to the new RCoA curriculum. The anatomical and clinical knowledge 
that is relevant to these nerve blocks will also be listed.  
3.2 Phase 2 – Content and question generation 
A panel of experts in regional anaesthesia will review UGRA nerve blocks and their relevant 
anatomical structures. The anatomical structures that must be identified on ultrasound for 
the safe and effective performance of these blocks will be agreed upon using the Delphi 
method. It will help determine whether these are structures are essential for block 
performance, desirable, expert or not relevant. The experts in the panel will be defined by 
their experience – they will all be recognised nationally or internationally for their 
involvement in regional anaesthesia societies and publications. This information will be 
collated by the Primary and co-investigators to generate a pilot test of UGRA interpretation 
content and questions.   
3.3 Phase 3 – Preliminary testing  
The pilot test will be administered by local investigators via a computer-based application ( 
https://www.classmarker.com/) to 35-50 trainees and consultants, recruited by email and 
flyers sent to schools of anaesthesia across the UK. The cohorts will include an equal number 
of trainees and consultants of varying experience, based on the new RCOA curriculum and 
the extent of their regional experience (stage 1 (CT1-3); stage 2 (ST4-5); stage 3 (ST6-7); stage 
3 + regional fellowship trained; consultant; consultant with special interest in regional; 
consultant + regional fellowship trained). The pilot test will contain short video clips and 
images of ultrasound guided regional anaesthesia with questions in a constructed response 
question (CRQ) format to identify relevant structures, anatomy and clinical knowledge. It is 
important to recognise that due to the change in the RCoA curriculum, some participants will 
be familiar with the CRQ format while others might be more familiar with MCQ-based format, 
or other forms of assessment question. Participants in the pilot and main study will be 
provided with sample questions and answers prior to the test to familiarise themselves with 
the question format. 
Face validity of the test will also be assessed by asking participants to comment on the ease 
of understanding and to identify terms that appear ambiguous. Participants will be able to 
comment in a text box after each item in the questionnaire. They will also be asked to 
comment on how they would improve the test and any relevant anatomy or clinical 

 
 
Research Governance 
 
           and Integrity Team 
 
 
12 
 
knowledge missing from the questions. The format of the test will be amended based on the 
comments. 
A difficulty index of each item will be calculated by dividing the number of correct responses 
by the total number of responses, as detailed in Abubakar et al’s study19, 20, and thus a higher 
value suggests a question is easier to answer. Any item with a difficulty index of < 50% will be 
considered too difficult to answer by the respondents and will be considered for removal. 
This will be compared to the information gathered from the panel of experts on test items, 
for example whether the anatomical structure is essential, desirable, expert or not relevant 
to the specific ultrasound guided nerve block.  
One important step in establishing construct validity is to use the pilot study to identify how 
well the answers for each test item discriminate between the participant’s level of ultrasound 
experience. Differences in ultrasound interpretation skills between study groups will be 
analysed using one-way analysis of variance (ANOVA)/Rasch modelling. Items that 
discriminate well will be retained and those that discriminate poorly will be removed from 
the main test.   
Reliability of the test will be improved by presenting the pilot test items in a random order to 
minimise any effect of cognitive fatigue diminishing performance on subsequent items. The 
average time taken for participants to complete each item in the pilot test will also be 
recorded to guide the optimal final study test length. It is important to balance the maximum 
number of test items whilst minimising the risk of cognitive fatigue and participant drop out.  
While the pilot test will necessarily have more items than the final test, pilot test participants 
will be asked to comment on the point at which they felt fatigued to guide this balance. It will 
also be important to understand whether level of regional anaesthesia experience affects 
fatiguability.   
3.4 Phase 4 – Testing for content validity  
To ensure all items in the assessment tool have the content that tests the domain we intend 
to test, following the pilot the amended test will be presented to the panel of 5 experts in 
regional anaesthesia, who will be asked to rate the suitability of ultrasound images and 
questions to performance of the block. This will be on a 4-point Likert scale (1 = definitely not 
suitable, 2 = probably not suitable, 3 = probably suitable, 4 = definitely suitable). 
The content validity of each item will be calculated by dividing the number of experts who 
rated the item as suitable and very suitable (3 and 4 on the scale) by the total number of 
experts. Items found to be appropriate by a sufficiently high percentage of participants will 
be selected. For example, if 26 out of 40 (65%) participants rate an item as suitable or very 

 
 
Research Governance 
 
           and Integrity Team 
 
 
13 
 
suitable then the lower 95% confidence limit for this proportion is 51%, indicating that more 
than 50% of experts are likely to find it appropriate. 
3.5 Phase 5 – Psychometric main study testing 
The main study assessment tool will be administered to trainees of all levels and consultants, 
aiming to recruit an equal number across different stages of the RCoA curriculum and of 
varying regional anaesthesia experience (stage 1; stage 2; stage 3; stage 3 + regional 
fellowship trained; consultant; consultant with special interest in regional; consultant + 
regional fellowship trained). Participants will be recruited using emails and flyers sent to 
schools of anaesthesia across the UK. Each participant will answer a series of demographic 
questions, including age, gender, handedness, as well as level of training and practical 
experience in UGRA to account for inconsistencies in clinical exposure at different stages of 
training.  
The exam will consist of constructed response questions (CRQs), which will require the 
participant to write free-text responses to questions, in line with RCoA examinations. 
Participants will be provided with sample questions and answers before the test to familiarise 
themselves with the question format. 
The construct validity of the tool will be tested by comparing the scores of anaesthetists with 
their level of experience specifically in regional anaesthesia. The level of experience is a proxy 
outcome measure for their proficiency in ultrasound-guided regional anaesthesia. Differences 
in ultrasound interpretation skills between study groups will be analysed using one-way 
analysis of variance (ANOVA)/ Rasch modelling. At the end of the assessment, there will be a 
short questionnaire to allow for participant feedback regarding ease of understanding and 
suitability of ultrasound images to support the face validity of the test.  
The internal reliability of the item responses will be assessed using Cronbach’s alpha. An alpha 
value of 0.70 will be considered sufficient and acceptable for the questions. A split-test 
method will also be used for each participant to compare the first half of their assessment 
with the second half to ensure internal consistency of their item responses. The difficulty 
index calculated for items in the pilot study will be used to ensure a balanced set of test items, 
in terms of their difficulty, in the first and second halves of the test. All items will be made 
mandatory. The test will be administered by local investigators via a web-based application. 
It will have an automatic system of marking to minimise inter-marker differences. It will also 
be manually checked by investigators who are blinded to the level and identity of the 
participant. We will aim to recruit a minimum of 250 participants so that there is a stable 
reliability value, each of whom will be allocated a participant number to pseudonymise the 
data. The results will be analysed only when recruitment has ended. 

 
 
Research Governance 
 
           and Integrity Team 
 
 
14 
 
4. Study Outcome Measures 
4.1 Primary Outcome 
The primary outcome will be the development of a validated, reliable and defensible 
assessment tool to test UGRA interpretation skills.  
4.1 Secondary Outcomes 
The secondary outcomes are development a validated, reliable and defensible assessment 
tool assessing anatomical, clinical knowledge and the performance of ultrasound guided 
regional anaesthesia: 
• Assessment of anatomical knowledge relevant to performing a local anaesthetic injection 
around peripheral nerves 
• Assessment of clinical regional anaesthesia knowledge relevant to performing a local 
anaesthetic injection around peripheral nerves 
• Assessment of the performance of an ultrasound-guided local anaesthetic injection 
around peripheral nerve 
5. Participant Entry 
5.1 Pre-registration Evaluation 
Participants will be required to complete the following tasks to be involved in this research: 
 
• 
Read a participant information sheet (PIS) about the study. 
• 
Face to face or virtual explanation and consent of the study will be taken by one of 
the study team members (PI's or co-investigators); written or online consent (digital 
signature) 
5.2 Participant Recruitment  
Consultant and trainee anaesthetists across the breadth of the curriculum stages, will be 
invited to participate by email. All potential participants will be given a PIS (approved by a 
Research Ethics Committee) describing the study on recruitment. The participant will have 
time to read the PIS and to discuss their participation with study administrators, as required, 
before providing written consent. 
5..3. 
Inclusion criteria  
Any anaesthetic trainee currently registered with the Royal College of Anaesthetists in a UK-
training programme and practising consultant anaesthetists who are registered with the 
General Medical Council. 

 
 
Research Governance 
 
           and Integrity Team 
 
 
15 
 
5..4. 
Exclusion criteria 
Staff grade, associate specialist and specialty doctors not enrolled in a UK anaesthetic training 
programme will be excluded because they encompass a wide range of different experience in 
regional anaesthesia. Co-investigators, local investigators and pilot test participants will be 
excluded due to prior experience with the assessment tool. 
5.5. 
Participant withdrawal 
Each participant has the right to withdraw at any time. The data will be collected in an 
pseudonymised format. This means that a participant can withdraw from the study.  In order 
to withdraw from the study a participant will need to contact the Chief Investigator. This will 
be explained in the participant information sheet.  
6. Assessment  
6.1 Definition of end of trial 
The end of the trial will be when the required number of anaesthetists have completed the 
web-based test for the validation phase of the study. The data will then be locked for analysis. 
6.2 Likely rate of loss to follow up 
Losses to follow up are expected to be very few, as each participant will be expected to 
complete the online test in one sitting. However, in the case of individuals failing to complete 
the test (e.g. registered but did not take the test, attempted too few questions), their scores 
will be excluded from the final analysis. We will build an expected loss to follow up of 10% 
when powering the study. 
6.3 Expenses 
Participants will not be paid expenses for taking part. 
7. 
Statistics and Data Analysis 
7.1 Sample Size Calculation 
Sample size for the definitive data capture stage, ‘Phase 5 – Psychometric main study 
testing’, will be calculated using a power calculation based on data obtained in the pilot 
work during ‘Phase 3 – Preliminary testing’. For example, internal reliability of the item 
responses is to be assessed using Cronbach’s alpha, an alpha value of 0.70 being considered 
adequate. With 250 participants completing a 50-item questionnaire it would be possible to 
show that Cronbach’s alpha is significantly higher than 0.6 if the true value is 0.7 i.e. there is 

 
 
Research Governance 
 
           and Integrity Team 
 
 
16 
 
a greater than 85% chance (power) that the 95% confidence interval for Cronbach’s alpha 
will exceed 0.6 and include 0.7. 
7.2 Trial Methods 
7.2.1 Blinding 
The investigators will be blinded to the level of training when checking the automatic marking 
of the test answers.  
7.2.2 Measures taken to avoid bias 
All tests will be anonymised and marked automatically by the web-based application to avoid 
bias. Answers will be checked by local investigators, who will be blinded to the identity of the 
participant.  
It is important to recognise that due to the change in the RCoA curriculum, some participants 
will be familiar with the CRQ format while others might be more familiar with MCQ-based 
format, or other forms of assessment question. Participants in the pilot and main study will 
be provided with sample questions and answers prior to the test to familiarise themselves 
with the question format. 
7.3 Plan of Analysis 
We have taken advice from the Imperial College statistician, Mr Roger A’Hern.   
 
The primary outcome will be analysed using a series of calculated indices and statistical tests. 
A difficulty index will be calculated by dividing the number of correct responses by the total 
number of responses, with a difficulty index <50% considered sufficient to exclude an item 
from the test.  Content validity will be calculated by asking an expert panel to rate items in 
the test on a 4-point scale as definitely not suitable, probably not suitable, probably suitable 
and definitely suitable. A validity index of > 0.78 will be considered sufficient to include a 
question. Construct validity will be measured using a one-way analysis of variance (ANOVA)/ 
Rasch modelling, internal reliability using Cronbach’s alpha (with a value of 0.70 considered 
sufficient), and internal consistency using a split-test method.  
7.3.1 Frequency of analyses 
The primary analysis will take place when the pilot study recruitment is complete in phase 3. 
This will be used to guide the generation of the final version used in the next phase. Then the 
analysis of the phase 5 validation study will take place when recruitment for this test is 
complete. No interim analysis is planned. 

 
 
Research Governance 
 
           and Integrity Team 
 
 
17 
 
7.3.2 Criteria for the termination of the trial 
The study may be terminated early if the results of another study make it redundant, or on 
safety grounds. 
7.3.3 Economic issues 
There will be no formal economic analysis. 
7.3.4 Data storage 
All data will be stored in anonymised form and analysed on NHS Trust computers.  
 
Data and all appropriate documentation will be stored for a minimum of 5 years after the 
completion of the study. 
8. 
Regulatory Issues 
 
8.1 
Ethics Approval 
The Study Coordination Centre has obtained approval from the xxx Research Ethics 
Committee (REC) and Health Regulator Authority (HRA). The study must also receive 
confirmation of capacity and capability from each participating NHS Trust before accepting 
participants into the study or any research activity is carried out. The study will be conducted 
in accordance with the recommendations for physicians involved in research on human 
subjects adopted by the 18th World Medical Assembly, Helsinki 1964 and later revisions. 
8.2 
Consent  
Consent to enter the study must be sought from each participant only after a full explanation 
has been given, an information leaflet offered and time allowed for consideration.  Signed 
participant consent should be obtained. The right of the participant to refuse to participate 
without giving reasons will be respected. All participants are free to withdraw at any time 
from the protocol without giving reasons. 
8.3 
Confidentiality 
The Chief Investigator will preserve the confidentiality of participants taking part in the study 
and is registered under the Data Protection Act.  Data will be pseudonymised.   
 
Each participant will be identified by a unique sequential number identifier allocated to them 
and this information and all data collected will be stored in the master file in the Primary 
Investigator’s locked office on the hospital site.  

 
 
Research Governance 
 
           and Integrity Team 
 
 
18 
 
8.4 
Indemnity 
Imperial College Healthcare NHS Trust holds standard NHS Hospital Indemnity and insurance 
cover with NHS Resolution for NHS Trusts in England, which apply to this study. 
8.5 
Sponsor 
Imperial College Healthcare NHS Trust will act as the main Sponsor for this study.   
8.6 
Funding 
This study is unfunded  
8.7 
Audits  
The study may be subject to inspection and audit by Imperial College Healthcare NHS Trust 
under their remit as sponsor and other regulatory bodies to ensure adherence to GCP and 
the UK Policy Frame Work for Health and Social Care Research.  
9. 
Study Management 
 
The trial will be managed by a Trial Management Group (TMG). The TMG will be chaired by a 
Chief Investigator and will include all members of the named research team (see Chief 
Investigators & Research Team Contact Details). 
 
A study principal investigator will be responsible for identifying potential participants, seeking 
informed consent, collecting trial data and ensuring adherence to the trial protocol. 
10. 
Publication Policy 
 
We anticipate that the results will be published in a peer-reviewed journal and may be 
presented at a national and international conferences. 
 
 
 

 
 
Research Governance 
 
           and Integrity Team 
 
 
19 
 
11. References 
 
[1] Turbitt LR, Mariano ER & El-Boghdadly K (2020). Future directions in regional 
anaesthesia: not just for the cognoscenti. Anaesthesia 75, 293-297. 
[2] Royal College of Anaesthetists (2010). Curriculum for a CCT in Anaesthetics. Edition 2, 
Version 1.8. 
[3] Lewis SR, Price A, Walker KJ et al (2015). Ultrasound guidance for upper and lower limb 
blocks. Cochrane Database System Review CD006459. 
[4] Orebaugh SL, Williams BA, Vajello M et al (2009). Adverse outcomes associated with 
stimulator-based peripheral nerve blocks with versus without ultrasound visualisation. 
Region Anaesth Pain Med 34, 251e5. 
[5] Abrahams M, Aziz M, Fu R, Horn J (2009). Ultrasound guidance compared with electrical 
neurostimulation for peripheral nerve block: a systematic review and meta-analysis of 
randomized controlled trials. BJA 102, 408–417. 
[6] Munimara S & McLeod G. A systematic review and meta-analysis of ultrasound versus 
electrical stimulation for peripheral nerve location and blockade. Anaesthesia 70, 1084-
1091. 
[7] Macfarlane AJR, Harrop-Griffiths W & Pawa A. (2020). Regional anaesthesia and COVID-
19: first choice at last? BJA 125, 243-247. 
[8] Hecker K & Violato C. (2009). Validity, reliability and defensibility of assessments in 
veterinary education. JVME 36, 271-275. 
[9] Chuan A, Wan A, Royse C & Forrest K (2018). Competency-based tools for regional 
anaesthesia: a narrative review. BJA 120, 264-273. 
[10] Chin K, Tse C, Chan V et al (2011). Hand-motion analysis using the Imperial College 
surgical assessment device: validation of a novel and objective performance measure in 
ultrasound-guided peripheral nerve blockade. Reg Anaes Pain Med 36, 213-219. 
[11] Hayter M, Friedman Z, Bould M et al. (2009) Validation of the Imperial College surgical 
assessment device (ICSAD) for labour epidural placement. Can J Anaes 56, 419-26. 
[12] Smith H, Kopp S, Johnson R et al (2012). Looking into learning: visuospatial and 
psychomotor predictors of ultrasound-guided procedural performance. Reg Anaes Pain Med 
37, 441-7. 

 
 
Research Governance 
 
           and Integrity Team 
 
 
20 
 
[13] Duce N, Gillet L, Descaller J et al (2016). Visuospatial ability and novice brachial plexus 
sonography performance. Acta Anaesthesiol Scand 60, 1161-1169. 
[14] Shafqat A, Ferguson E, Thanawala V et al (2015). Visuospatial ability as a predictor of 
novice performance in ultrasound-guided regional anaesthesia. Anaesthesiology 123, 1188-
1197. 
[15] Sivaranjan M, Lane P, Miller E et al. (1981). Performance evaluation: continuous lumbar 
epidural anaesthesia skill test. Anaes Analg 60, S43-47. 
[16] Watson M, Wong D, Kluger R et al (2014). Psychometric evaluation of a direct 
observation of procedural skill assessment tool for ultrasound-guided regional anaesthesia. 
Anaesthesia 69, 604-12. 
[17] Woodworth GE, Carney PA, Cohen JM et al (2015). Development and validation of an 
assessment of regional anaesthesia interpretation skills. American Society of Regional 
Anaesthesia and Pain Medicine 40, 306-314. 
[18] Hauer KE, Boscardin C, Brenner JM (2020). Twelve tips for assessing medical knowledge 
with open-ended questions: Designing constructed response examinations in medical 
school. Medical Teacher 42, 880-885. 
[19] Abubakar IJ, Wimmer BC, Bereznicki LR et al. (2020). Development and validation of an 
atrial fibrillation knowledge assessment tool (AFKAT). Int J Env Res and Publ Health 17, 1721. 
[20] University of Washington. Understanding Item Analysis (2019). Available online: 
http://www.washington.edu/assessment/scanning-scoring/scoring/reports/item-analysis/ 
(accessed on 3rd November 2020). 
 
[21] Polit DF & Beck CT (2006). The Content Validity Index: Are you sure you know what’s 
being reported? Critique and Recommendations. Res Nurs Health 29, 489–497. 
 
 
 
 
 
 
 

 
 
Research Governance 
 
           and Integrity Team 
 
 
21 
 
12. Appendix 
The following link allows you to view sample question(s) that would be posed to the participants 
https://1drv.ms/u/s!AoKOa_qdcBwN7xRgYp6889QRs-qh?e=4pd9f2 
All ultrasound images are taken from voluntary study team members or NHS patients.  These images 
are fully anonymised and contain no staff and/or patient information and cannot be identified by 
anyone even study team members.  Verbal consent has been gained from all volunteers and NHS 
patients explaining these images are used for educational purposes as is the nature of this 
assessment tool being developed.