Study of Advice and Decision-making on Prognosis Using the Judge-advisor System Within Multi-disciplinary Teams (ADJUST)
Phase: NA
Conditions: Terminal Illness
Sponsor: University College, London

ADJUST: a study on MDT prognostication 
 
 
 
 
Sponsor code:  [6288/001] 
ADJUST study version 1.0 01-September-2020  
Page 1 
 
 
 
 
 
 
 
 
Full title of study 
 
Study of Advice and Decision-making on prognosis 
using the Judge-advisor System within multi-discipli-
nary Teams (ADJUST) 
Short title 
 
ADJUST: A study on MDT prognostication 
Version and date of protocol 
 
Version 1.0 01-September-2020 
  
Sponsor: 
University College London (UCL) 
Sponsor protocol number: 
6288/001 
 
Funder: 
 
Marie Curie Chair’s grant and programme grant 
 
 
Single site/multi-site: 
Single-site  
Chief investigator: 
Professor Patrick Stone 
Marie Curie Palliative Care Research Depart-
ment 
UCL Division of Psychiatry 
6th Floor Maple House, 
149 Tottenham Court Road, 
London W1T7NF 
 
Sponsor Representative: 
Pushpsen Joshi 
RM&G Manager 
E-mail: pushpsen.joshi1@nhs.net 
Joint Research Office (Research Support Centre), UCL, 
1st Floor Maple House – Suite B, 
149 Tottenham Court Road, 
London W1T 7NF 
 
 
 
 
 

ADJUST: a study on MDT prognostication 
 
 
 
 
Sponsor code:  [6288/001] 
ADJUST study version 1.0 01-September-2020  
Page 2 
 
1 Protocol Version History 
Version Number 
Date 
Protocol Update Final-
ised By (insert name of 
person):  
Reasons for Update 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 

ADJUST: a study on MDT prognostication 
 
 
 
 
Sponsor code:  [6288/001] 
ADJUST study version 1.0 01-September-2020  
Page 3 
 
2 
Signatures 
The Chief investigator agrees to perform the investigations and to abide by this protocol.  
The investigator agrees to conduct the study in compliance with the approved protocol, the UK Data 
Protection Act (1998), the Trust Information Governance Policy (or other local equivalent), the current 
Research Governance Framework, the Sponsor’s SOPs, and other regulatory requirements as 
amended. 
 
Chief investigator 
Professor Patrick Stone 
 
01-September-
2020 
UCL 
Signature 
Date 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 

ADJUST: a study on MDT prognostication 
 
 
 
 
Sponsor code:  [6288/001] 
ADJUST study version 1.0 01-September-2020  
Page 4 
 
3 Contents 
1 
Protocol Version History .......................................................................................... 2 
2 
Signatures ............................................................................................................... 3 
3 
Contents .................................................................................................................. 4 
4 
List of abbreviations ................................................................................................ 6 
5 
Study personnel ....................................................................................................... 7 
6 
Summary ................................................................................................................. 8 
7 
Background and Rationale ....................................................................................... 9 
7.1 
Background ................................................................................................................. 9 
7.2 
Prognostication ........................................................................................................... 9 
7.3 
The Judge-Advisor System (JAS) ................................................................................ 12 
7.4 
JAS Research ............................................................................................................. 12 
7.5 
JAS research in the context of this study .................................................................. 12 
8 
Assessment and Management of Risk .................................................................... 13 
9 
Objectives ............................................................................................................. 13 
9.1 
Primary objective: ..................................................................................................... 14 
9.2 
Secondary objective: ................................................................................................. 14 
10 
Study design .......................................................................................................... 14 
11 
Data collection and study procedures ..................................................................... 14 
11.1 
Arm A (recruited at seminars) ....................................... Error! Bookmark not defined. 
11.2 
Arm B (recruited online) ............................................... Error! Bookmark not defined. 
12 
Selection of Participants ........................................................................................ 15 
12.1 
Inclusion criteria ........................................................................................................ 15 
12.2 
Exclusion criteria ....................................................................................................... 15 
12.3 
Recruitment .............................................................................................................. 15 
12.3.1 
Arm A (recruited at seminars) ................................................................................... 15 
12.3.2 
Arm B (recruited online) ............................................................................................ 16 
12.4 
Informed consent ...................................................................................................... 16 
12.4.1 
Arm A (recruited at seminars) ....................................... Error! Bookmark not defined. 
12.4.2 
Arm B (recruited online) ................................................ Error! Bookmark not defined. 
12.5 
Participant withdrawal .............................................................................................. 17 

ADJUST: a study on MDT prognostication 
 
 
 
 
Sponsor code:  [6288/001] 
ADJUST study version 1.0 01-September-2020  
Page 5 
 
13 
Recording and reporting of adverse events ............................................................. 17 
13.1 
Notification of reportable protocol violations .......................................................... 17 
14 
Data management ................................................................................................. 17 
14.1 
Confidentiality ........................................................................................................... 17 
15 
Data handling ........................................................................................................ 18 
16 
Statistical Considerations ....................................................................................... 19 
16.1 
Sample size calculation ............................................................................................. 19 
16.2 
Statistical analysis ..................................................................................................... 20 
16.2.1 
Summary of baseline data and flow of participants ................................................. 20 
16.2.2 
Primary outcome analysis ......................................................................................... 20 
16.2.3 
Secondary outcome analysis ..................................................................................... 20 
17 
Record keeping and archiving................................................................................. 21 
18 
Study Management Group (SMG) ........................................................................... 21 
19 
Ethics .................................................................................................................... 21 
20 
Patient and public involvement (PPI) ...................................................................... 21 
21 
Monitoring ............................................................................................................ 22 
22 
Finance .................................................................................................................. 22 
23 
Insurance .............................................................................................................. 22 
24 
Publication policy .................................................................................................. 22 
25 
Intellectual property .............................................................................................. 22 
26 
References ............................................................................................................ 24 
27 
Appendix 1 - Schedule of assessments .................................................................... 27 
28 
Appendix 2 – Sample vignette ................................................................................ 27 
 
 
 

ADJUST: a study on MDT prognostication 
 
 
 
 
Sponsor code:  [6288/001] 
ADJUST study version 1.0 01-September-2020  
Page 6 
 
4 List of abbreviations 
AE 
Adverse Event 
APR 
Annual Progress Report 
CI 
Chief Investigator 
CONSORT 
Consolidated Standards of Reporting Trials 
EAPC 
European Association for Palliative Care 
GSF 
Gold Standards Framework 
IPR 
Intellectual Property Rights 
JAS 
Judge-Advisor System 
LCP 
Liverpool Care Pathway 
LACDP 
Leadership Alliance for the Care of Dying People 
MDT 
Multidisciplinary Team 
NHS R&D 
National Health Service Research & Development   
OACC 
Outcome Assessment and Complexity Collaborative 
PaP 
Palliative Prognostic Score 
PI 
Principal Investigator 
PiPS 
Prognosis in Palliative care Study 
PIS 
Participant Information Sheet 
PPI  
Patient and Public Involvement 
PPI  
Palliative Prognostic Index 
PPS 
Palliative Performance Scale 
REC 
Research Ethics Committee 
SD 
Standard Deviation 
SOP 
Standard Operating Procedure  
SMG 
Study Management Group 
WOA 
Weight Of Advice 

ADJUST: a study on MDT prognostication 
 
 
 
 
Sponsor code:  [6288/001] 
ADJUST study version 1.0 01-September-2020  
Page 7 
 
5 Study personnel 
See protocol cover page for Chief Investigator and Sponsor contact details. 
MPhil/PhD Student 
Andrea Bruun 
e-mail: andrea.bruun.19@ucl.ac.uk 
 
tel: 020 767 99713  
 
Associate Professor 
Dr Adam Harris 
 
e-mail: adam.harris@ucl.ac.uk 
 
tel: 020 7679 5332 
 
Senior Research Fellow 
Dr Linda Oostendorp 
 
e-mail: l.oostendorp@ucl.ac.uk  
tel: 020 7679 8079 
 
Research Fellow 
Dr Nicola White 
 
e-mail: n.white@ucl.ac.uk  
 
tel: 020 7679 9057 
 
Website Developer 
Dr Christopher Tomlinson 
 
e-mail: chris.tomlinson@imperial.ac.uk  
 
tel: 020 7594 5072 
 
Statistician/Senior Research Fellow 
Dr Victoria Vickerstaff  
 
e-mail: v.vickerstaff@ucl.ac.uk 
tel:  020 7679 9721 
 
Associate Professor  
Dr Steven Bloch 
e-mail: s.bloch@ucl.ac.uk  
tel: 020 7679 4269  
 
 
 
 
 

ADJUST: a study on MDT prognostication 
 
 
 
 
Sponsor code:  [6288/001] 
ADJUST study version 1.0 01-September-2020  
Page 8 
 
6 Summary 
 
Objective: 
 
To understand in quantitative terms how multidisciplinary teams 
form intuitive judgements about the prognoses of palliative care pa-
tients.  
Type of study: 
Online psychological experiment. 
Study design and meth-
ods: 
In this study, using the Judge-Advisor System, we will recruit clini-
cians working in palliative care for adults. Participants will be asked 
to complete a survey and review five patient summaries (“vignettes”) 
based on real cases derived from a previous study. Clinicians will be 
presented with key prognostic information and will be asked to pro-
vide an estimate of the probability that the patient will survive for 
two weeks (0-100%). After providing this initial estimate, participants 
will receive prognostic advice. While in reality all participants will re-
ceive the same advice, participants will be randomised into two 
groups and these groups will be informed that the advice is either: 
(1) provided by an algorithm; or (2) provided by a team colleague. 
Participants will then be given the opportunity to give a second, pos-
sibly revised estimate in the light of the advice received. 
Study data will be collected using two different recruitment methods 
(Arm A and Arm B). Participants in arm A will be recruited at a series 
of (online) seminars, and participants in arm B will be recruited 
through their hospice employment. All participants will be asked to 
complete an online questionnaire, including five vignettes. Clinicians 
will be asked some demographic questions including age, gender, 
profession and experience.  
Study duration per partic-
ipant: 
Approximately 10-15 minutes 
Estimated total study du-
ration: 
36 months 
Planned study sites: 
For arm A, we will approach clinicians attending a series of (online) 
seminars on palliative care. For Arm B, we will approach clinicians 
working in non-NHS hospices in the UK.  
Total number of partici-
pants planned: 
100-300 
Main inclusion/exclusion 
criteria: 
Participants must be clinicians working in palliative care for adults.  
They must be willing and able to provide informed consent 

ADJUST: a study on MDT prognostication 
 
 
 
 
Sponsor code:  [6288/001] 
ADJUST study version 1.0 01-September-2020  
Page 9 
 
Statistical methodology 
and analysis: 
Analysis will be conducted by the study team. Participant de-
mographics will be described.  
The primary outcome will be the weighting of advice that each judge 
attributes to the advice that they receive. The weighting of advice 
reflects the importance that a judge gives to the advice provided and 
the level to which the judge incorporates this into their decision 
(ranging from ‘0’ if completely discounting the advice to ‘1’ if shifting 
completely to the advice). The weighting of advice will be analysed 
using regression analysis, taking into account the judge’s initial and 
final estimates and the advice given. We will explore whether the 
weighting of advice is influenced by characteristics of (1) the advisor 
(a clinician or algorithm); (2) the judge (e.g. seniority); and (3) the 
advice itself (e.g. distance from judge’s initial estimate).  
 
7 Background and Rationale 
7.1 
Background 
This study is part of the ‘Improving care, assessment, communication and training at end-of-life’ (I-
CAN-CARE) programme of research. The overall aim of this programme is to improve end-of-life care 
for patients and their families by better assessment of dying patients, by improved understanding of 
how clinicians identify dying patients, by improving clinicians’ skills in making prognostic decisions 
and by understanding how prognoses are communicated to relatives of dying patients. 
This project uses an experimental design to study decision-making about prognoses within multidis-
ciplinary teams. 
7.2 
Prognostication 
Studies show that patients, carers and clinicians all value accurate prognostic information (Adams et 
al., 2009, Degner et al., 1997, Kirk et al., 2004, Kutner et al., 1999, Steinhauser et al., 2000, 
Steinhauser et al., 2001). This information can help patients and family members to make decisions 
and feel prepared, can help family members to prioritise commitments, and can help to access fund-
ing and services, and plan treatment and care in the hospital or community (Pontin and Jordan, 
2013). 
Clinicians’ predictions about length of survival are inaccurate and over-optimistic (Glare et al., 2003, 
Gwilliam et al., 2013b, White et al., 2016, Christakis and Lamont, 2000) and no clear guidance exists 
on how clinicians can be taught to perform this task better. Nonetheless, clinical predictions of sur-
vival remain the most common method of arriving at a prognostic estimate and the European Asso-
ciation for Palliative Care (EAPC) recommends that the “clinical prediction of survival (in combination 
with other prognostic factors) is a valid tool to obtain a general prognostic evaluation of patients”.  
In 2013, due to substantial criticism of the Liverpool Care Pathway (LCP), a care plan that had been 
designed to help doctors and nurses provide quality end-of-life care, an independent committee 

ADJUST: a study on MDT prognostication 
 
 
 
 
Sponsor code:  [6288/001] 
ADJUST study version 1.0 01-September-2020  
Page 10 
 
chaired by Baroness Neuberger was asked to review LCP use (Neuberger et al., 2013). A particular 
concern expressed by many relatives and professionals was that clinician predictions of survival were 
inaccurate and that this had the potential to adversely affect patient care. A frequent theme of the 
Neuberger report was that clinicians needed to be better at identifying patients who are dying. The 
Leadership Alliance for the Care of Dying People (LACDP) responded to the Neuberger review and 
identified “the recognition of dying” as one of the five key priority areas for improving end-of-life 
care (Leadership Alliance for the Care of Dying People, 2014, Leadership Alliance for the Care of 
Dying People). Systematic identification of patients approaching the “end-of-life” was also a key rec-
ommendation of the End-of-Life Care Strategy (Department of Health, 2008). The Gold Standards 
Framework (GSF) service improvement programme (widely used in general practice, nursing homes 
and increasingly in acute hospitals) uses a needs-based coding system dependent upon whether pa-
tients are expected to live for “days”, “weeks”, “months” or “years” (Thomas and Free, 2011). Simi-
larly the Outcome Assessment and Complexity Collaborative (OACC) suite of measures  includes a 
“phase of illness” assessment in which clinicians are asked to judge whether a patient is “stable”, 
“unstable”, “deteriorating” or “dying” (Witt et al., 2014). However, clinicians’ predictions about 
length of survival are inaccurate and over-optimistic (Christakis and Lamont, 2000, Glare et al., 2003, 
Gwilliam et al., 2013a, White et al., 2016) and no clear guidance exists on how clinicians can be 
taught to perform this task better.  
Nonetheless, clinical predictions of survival remain the most common method of arriving at a prog-
nostic estimate and the European Association for Palliative Care (EAPC) recommends that the “clini-
cal prediction of survival (in combination with other prognostic factors) is a valid tool to obtain a 
general prognostic evaluation of patients (grade A)1” and that “a second opinion by a more experi-
enced professional could be useful (grade D)2” (Maltoni et al., 2005). This latter recommendation 
was based on experts’ recommendations (Christakis and Lamont, 2000, Higginson and Costantini, 
2002), but more recently a few studies have investigated whether multidisciplinary team discussions 
can improve the accuracy of prognostic estimates (Gwilliam et al., 2011, Kee et al., 2007). One study 
investigated whether 6-month survival predictions for patients with newly diagnosed lung cancer 
made by individual clinicians and the multidisciplinary team (MDT) as a whole became more accu-
rate following MDT discussion (Kee et al., 2007). While clinicians felt more confident following the 
MDT discussion, this did not lead to better accuracy of individual clinicians, and only slightly better 
accuracy was found at the group level. Notably, this study did not ask clinicians to explicitly share 
their prognostic estimates. Our own research has shown that, in patients with advanced cancer, an 
MDT survival estimate was slightly more accurate than either a doctor’s or a nurse’s estimate alone 
(Gwilliam et al., 2013a). More broadly, a systematic review has found indications that MDT discus-
sions can lead to significant changes in assessment and management of cancer patients (Pillay et al., 
2016). It is thought that a multidisciplinary approach can help to reduce variations and improve 
 
 
1 Grade A recommendations relied on research evaluated as level I (impact studies with low risk of bias or ho-
mogeneous meta-analyses) or level  II  (Heterogeneous meta-analyses or confirmatory studies with a low risk 
of bias) 
2 Grade D recommendations relied on research evaluated as level V (expert opinion) or inconsistent or incon-
clusive studies of any level. 

ADJUST: a study on MDT prognostication 
 
 
 
 
Sponsor code:  [6288/001] 
ADJUST study version 1.0 01-September-2020  
Page 11 
 
consistency of care, provide a learning opportunity for junior staff, and improve communication be-
tween different specialties (Ruhstaller et al., 2006).  
Another approach to improving accuracy of predictions is to provide advice in the form of a prognos-
tic score. Several scores are available, including the Palliative Prognostic Score (PaP score) (Maltoni 
et al., 1999, Pirovano et al., 1999), Palliative Prognostic Index (PPI) (Morita et al., 1999), Palliative 
Performance Scale (PPS) (Anderson et al., 1996) and Prognosis in Palliative care Scales 
(PiPS)(Gwilliam et al., 2011). One study found indications that doctors’ estimates became more accu-
rate when they used the PPI to supplement their own intuitive judgments (Morita et al., 2001). An-
other recent study showed that, for a cohort of 38 patients, the PaP and the PPI more accurately 
predicted survival than junior doctors and palliative care experts and the most accurate estimates 
were obtained when palliative care experts used the PaP score (Tavares et al., 2017). 
Qualitative studies have explored how palliative care doctors and nurses formulate a prognostic esti-
mate and how they view this process (Clarkson et al., 2013, Pontin and Jordan, 2013). Clinicians 
mentioned using objective patient data such as Karnofsky Performance Status as well as less tangible 
factors such as intuition. One interviewee commented how prioritizing different patient data to for-
mulate a prognosis could lead to disagreements between professionals from different disciplines 
(Pontin and Jordan, 2013). Furthermore, it was suggested that registered nurses and healthcare as-
sistants working on wards might be best placed to formulate a prognosis, because they spend most 
time with patients and are closely involved in care delivery (Pontin and Jordan, 2013). While, in prac-
tice, many prognostic decisions are taken by a senior doctor, a quantitative study in a multidiscipli-
nary team providing supportive care and palliative radiotherapy for cancer patients has shown that 
prognoses from doctors were equally (in)accurate as those from other professionals, including 
nurses, radiation therapists and allied health professionals (Fairchild et al., 2014). It was suggested 
that nurses, because they spent more time with patients, were therefore better placed to assess cer-
tain symptoms and signs. In sum, involving professionals from various disciplines may help to com-
bine knowledge on various symptoms and signs and reduce error. However, the manner in which 
different estimates are combined to arrive at an MDT estimate is not well understood.   
The research team has recently completed a study investigating the accuracy of doctors, nurses and 
MDT predictions of 14 day survival compared to the accuracy of an algorithm (PiPS2) (Kalpakidou et 
al., 2018). As part of this study we collected observational data about the clinical condition of 1800+ 
palliative care patients. Furthermore, the research team has completed another study investigating 
how individual clinicians decide when a patient is imminently dying by using judgement analysis 
(Cooksey, 1996, White et al., 2018) 
The current study will build on this work by using the Judge-Advisor System (JAS) research methodol-
ogy to investigate the factors which affect MDT decision-making. The ultimate purpose of this re-
search will be to make recommendations about measures that clinicians can take to improve the ac-
curacy of prognostic judgements. It is recognised that real-world MDT decision-making is inevitably 
more complicated than our simple experimental set-up will allow. This is one of the reasons why the 
overall project will also investigate the interactions occurring during MDT meetings using qualitative 
methods (not described in this protocol). By combining the results from these two different ap-
proaches, the project will give a nuanced insight into MDT prognostication.  

ADJUST: a study on MDT prognostication 
 
 
 
 
Sponsor code:  [6288/001] 
ADJUST study version 1.0 01-September-2020  
Page 12 
 
7.3 
The Judge-Advisor System (JAS) 
A key component of the dynamics in a JAS is the differentiation between the roles of the judge and 
the advisor(s). The judge is the decision maker who assesses the information concerning a specific 
decision and makes the final decision. An advisor is the person who offers advice, information, or 
suggestions to the judge (Sniezek and Buckley, 1995). While actual decision-making power resides 
solely with the judge, the advisor may have some stake in the judge’s decision (Sniezek and Buckley, 
1995). 
Using advice is an essential practice in making decisions in real life, whether it is as simple as seeking 
directions in an unfamiliar environment or more complex situations such as those involving legal or 
medical issues (Yaniv, 2004b). Advice is usually sought from someone with more expertise or providing 
a different perspective, because of a need to improve the accuracy of a judgement and the expectation 
that advice will help (Sniezek and Van Swol, 2001, Yaniv, 2004b). In the context of prognostication, a 
doctor may also seek advice for social reasons, including self-affirmation and sharing of responsibility, 
knowing the difficulties of prognostication and having concerns about the consequences of making an 
inaccurate prediction (Kennedy et al., 1997, Yaniv and Milyavsky, 2007). The JAS offers a robust basis 
from which hypotheses on advice-giving in dyads or groups can be tested (Van Swol and Sniezek, 
2005). 
7.4 
JAS Research 
In traditional JAS research, judges are asked to complete an experimental task that usually requires a 
quantitative answer. Judges provide an initial estimate and they may also be asked to express a level 
of confidence regarding the accuracy of this initial decision (Bonaccio and Dalal, 2006). Next, the judge 
will receive advice, which could come from a real advisor or could be fictional, depending on the aims 
of the study. The judge is then invited to reconsider the initial estimate and provide a final, possibly 
revised estimate. Research has focussed on the importance (weighting) that a judge gives to the advice 
provided and the level to which the judge incorporates this into their decision or not (discounting). 
When revising estimates, people tend to use two strategies: choosing between two estimates or av-
eraging them. A surprising result of previous research has been that averaging was the more effective 
strategy across a wide range of commonly encountered environments. Authors have observed that 
despite this finding, choosing was the preferred strategy although greater accuracy would have been 
achieved had they always averaged (Soll and Larrick, 2009). 
One of the main findings reported in the literature is that people tend to discount advice (Yaniv, 2004b, 
Yaniv and Kleinberger, 2000b, Bonaccio and Dalal, 2006). Although the appropriate use of advice leads 
to better judgements (because of the reduction of random error), respondents prefer their own esti-
mates; this is referred to as egocentric discounting (Yaniv and Kleinberger, 2000a). Additionally, the 
distance of the advice from one’s own opinion can affects the weight it receives. High-knowledge re-
spondents discounted the advice given while the low knowledge group averaged less and chose to 
remain with their own initial judgment more (Yaniv, 2004b).  
7.5 
JAS research in the context of this study 
While research is available on advice and information seeking among clinicians (Rappolt, 2002, Weber 
et al., 2007, Weinberg et al., 1981) to the best of our knowledge no experimental studies have used 
the JAS to explore the dynamics of receiving and using advice in the context of medical prognostic 
decision-making.  

ADJUST: a study on MDT prognostication 
 
 
 
 
Sponsor code:  [6288/001] 
ADJUST study version 1.0 01-September-2020  
Page 13 
 
In the context of our study, it is often the responsibility of a (senior) doctor to formulate a prognosis 
to consider hospice enrolment, assess eligibility for a clinical study, plan care and treatment, and pre-
pare patients and families. As described above, doctors could obtain advice in a formal MDT setting 
or ask a more experienced colleague, a colleague from a different background that brings a different 
perspective, or a colleague who has spent more time with the patient. JAS studies have used different 
types of questions, including estimations of the dates of historical events and multiple-choice ques-
tions (Bonaccio and Dalal, 2006). The current study will ask participants to estimate the probability of 
patients surviving two weeks (0-100%). 
A recent series of experiments using a variety of judgement tasks has shown that, contrary to prior 
work, people placed a higher weight on advice from an algorithm as compared to advice from another 
person (Logg et al., 2019). The researchers provided judges in the study with equivalent advice that 
was labelled as coming from an algorithmic versus human source. In the first experiment, judges were 
asked to estimate the weight of a person in a photograph. The actual weight of this person was 164 
pounds. All judges then received the same advice (163 pounds), which was the estimate provided by 
415 participants in another experiment (Moore and Klein, 2008). Half of the judges were told that: ‘An 
algorithm ran calculations based on estimates of participants from a past study. The output that the 
algorithm computed as an estimate was: 163 pounds’. The remaining judges were told that ‘The aver-
age estimate of participants from a past experiment was: 163 pounds’. In this study, we will do an 
experiment similar to this, by randomising clinicians to receive equivalent advice, described as coming 
from a prognostic tool or another professional.  
This research will provide greater understanding of the factors affecting the quality of MDT prognostic 
estimates and will provide an evidence-base to improve clinician training.  
8 Assessment and Management of Risk  
The table below summarises the risks and mitigations of all study procedures: 
Intervention  
Potential risk 
Risk Management 
Completion of an 
online experimental 
task to provide five 
prognostic estimates 
and receive prognostic 
advice from an ‘advi-
sor’  
There are no obvious risks associ-
ated with the completion of this 
task. However, predicting survival is 
a difficult task, therefore partici-
pants may experience some feel-
ings of frustration.  
Participants will be advised that 
they can withdraw from the study 
at any time and without giving a 
reason. 
 
Collection of data, us-
ing a secure online 
platform. 
There is a small risk that data could 
be lost or accessed by unauthorised 
personnel. 
Data will be encrypted, backed up 
and stored securely on UCL com-
puters. 
9 Objectives 
To understand how clinicians form intuitive judgements about the prognoses of palliative care pa-
tients after receiving advice perceived as coming from either a team member or an algorithm.  

ADJUST: a study on MDT prognostication 
 
 
 
 
Sponsor code:  [6288/001] 
ADJUST study version 1.0 01-September-2020  
Page 14 
 
9.1 
Primary objective: 
To assess the level to which clinicians incorporate advice received from other clinicians or an algo-
rithm into their estimates of the prognosis of palliative care patients. 
9.2 
Secondary objective: 
To investigate the extent to which clinicians’ integration of advice is influenced by the characteristics 
of the judge, the advisor or the advice itself. 
10 Study design 
Study data will be collected using two different recruitment methods (arm A and arm B). Both study 
arms involve participants being asked to provide a prognostic estimate of the probability of survival 
for palliative care patients. Participants are asked to provide information about their gender, age, 
role, grade and experience. They will be presented with five case summaries (“vignettes”) (see Ap-
pendix 2 – Sample vignette) based on real cases derived from a previous study (PiPS2) (Kalpakidou et 
al., 2018). In the PiPS2 study, data were collected on palliative care patients in order to calculate a 
prognostic score (the PiPS score). Doctors and nurses were asked to make a prediction about how 
long they expected the patient to survive, and the subsequent length of survival of the patients was 
recorded. We will create vignettes using the anonymised data already collected as part of the PiPS2 
study (see Appendix 2 – Sample vignette). Participants will be presented with key prognostic infor-
mation and will be asked to estimate the probability that the patient will survive the next two weeks 
(0% ‘certain to die’ to 100% ‘certain survival’). 
Participants will be randomised into one of the following two groups: The clinicians’ advice group or 
the algorithm advice group. In the clinicians’ advice group, participants will receive advice labelled as 
coming from another clinician. Doctors will receive advice labelled as coming from nurses and vice 
versa. Seminar attendees with other clinical backgrounds in this group will be further randomised 
into receiving advice labelled as either coming from a doctor or a nurse. In the algorithm group, par-
ticipants will receive advice labelled as coming from the PiPS-B algorithm. In fact, participants in 
both groups will receive the same advice.  After receiving the advice, participants will be given the 
opportunity to give a revised estimate, in the light of the advice received.  
A schedule of study assessments and procedures is set-out in Appendix 1 - Schedule of assessments.  
11 Data collection and study procedures 
We expect that data collection for this study will take approximately ten months starting in August 
2020 and end by April 2021. See the Gantt chart in the appendix.  
Clinicians will be invited to participate in the study as part of a (online) seminar on palliative care by 
the CI or through an invitation email sent to their hospice medical director (see section 12.3). Semi-
nar participants will receive information about the study before the seminar and will be invited to 
complete the online questionnaire in preparation for the seminar. The CI will make it clear that dele-
gates at the seminar are under no obligation to complete the questionnaire.  

ADJUST: a study on MDT prognostication 
 
 
 
 
Sponsor code:  [6288/001] 
ADJUST study version 1.0 01-September-2020  
Page 15 
 
Participants will be provided with a link to the study website and will complete all assessments 
online. The website will provide information regarding the study and a link to a PIS. Participants will 
be asked to tick check boxes to indicate that they are giving informed consent. After filling in the de-
mographics section, eligible participants will be considered to be enrolled and a participant ID will be 
assigned.  Next, participants will be randomly allocated to one of the two study groups. This will be 
performed through the web-based system, using a pre-generated blocked randomisation list. 
Participants will be given tailored instructions, shown a practice vignette and will then be able to 
start the decision tasks and to provide estimates of the probability of patients surviving two weeks. 
Participants will be presented with five vignettes. Vignettes will be presented in random order to 
prevent order effects. 
There will be no time limit for completion, so that each participant will be able to consider their re-
sponses. Participants will be given the opportunity to log out and return to the same place at a more 
convenient time. Participants will not be able to move on to the next page if required information is 
missing or is in an incorrect format (e.g. incorrect email format). The online environment will be pi-
loted by the study team in close collaboration with the study statistician who will review the data 
being collected before, during, and at the start of the recruitment phase to ensure data integrity. 
After completing the series of vignettes, participants will be shown a debrief page to remind the par-
ticipant what the results will be used for and they will be able to download a certificate of participa-
tion. Participants will be asked if they wish to receive a summary of the results at the end of the 
study and whether they would be willing to be approached for subsequent studies.  
Responses collected through the website up until the scheduled seminar will be added to a spread-
sheet to provide the attendees with preliminary study findings during the presentation.  
12 Selection of Participants 
12.1 Inclusion criteria  
1 
Clinicians working in an adult palliative care service. 
2 
Willing and able to provide written informed consent. 
12.2 Exclusion criteria  
Participants who do not meet the inclusion criteria will be excluded 
12.3 Recruitment  
In both study arms, before participants complete the vignettes, they will be asked to complete a 
brief demographic section. Questions in this section will be used to assess the participant’s eligibility 
for the study (i.e. are they a clinician working in palliative care for adults) as well as to describe the 
study population and conduct analyses of associations between judges’ characteristics and their in-
tegration of advice.  
12.3.1 Arm A (recruited at (online) seminars) 
Clinicians will be recruited at (online) seminars where the CI is scheduled to give presentations on 
prognostication. The study will be presented before the seminar and attendees will be invited to fill 

ADJUST: a study on MDT prognostication 
 
 
 
 
Sponsor code:  [6288/001] 
ADJUST study version 1.0 01-September-2020  
Page 16 
 
in the online questionnaire in preparation for the seminar. The CI will stress that participation in the 
study is voluntary. The study can be completed by international clinicians attending the (online) 
seminars.  
12.3.2 Arm B (recruited online) 
We will recruit clinicians to complete the online survey in the time period from August 2020 to April 
2021, and we will identify eligible clinicians in several ways. The Hospice UK website includes a data-
base listing adult and children’s hospice care providers in the UK (https://www.hospiceuk.org/about-
hospice-care/find-a-hospice). The database includes 310 hospices, of which 47 hospices have the 
word ‘children’ in their name. We will start by approaching a small number of the remaining 263 
hospices on the list to explore the ease of recruitment for this study. We will not recruit from NHS 
services. In a first wave of recruitment, we will approach the nine Marie Curie hospices in the UK 
(Belfast, Bradford, Cardiff and the Vale, Edinburgh, Glasgow, Hampstead, Liverpool, Newcastle, and 
West Midlands), as these hospices may be more inclined to participate because of their link with 
Marie Curie. We will approach additional hospices as needed until we reach our lower limit for our 
sample size (see more in section 16.1 Sample size calculation). We will make individual contact with 
each hospice to inform them about the study, which could include approaching the medical director, 
head of nursing or matron, and/or the education and research lead. We will initially approach hos-
pices via email and then follow up with a phone call, we will offer that the research team should 
come and present the study at their unit. If the hospice agrees to participate in the study, we will 
forward the template for the invitation email to be circulated to all doctors and nurses working in 
the hospice.  
If necessary, we will use additional recruitment strategies, including Marie Curie Palliative Care Re-
search Department’s social media platform or newsletters, where interested clinicians will be able to 
contact the research team to participate in the study.  
12.4 Informed consent  
Participants will consent to complete the decision tasks, to have their data collected, stored and pro-
cessed, to receive a results summary, and to be contacted by the research team to be prompted to 
complete the study.  
Participants will receive information about the aims, methods, anticipated benefits and potential 
hazards of the study in the invitation email, on the study website and in the PIS. The PIS will be avail-
able to download from the study website. This information includes an explanation that participants 
are under no obligation to enter the study and that they can withdraw at any time during the study, 
without having to give a reason. It also includes a transparency message to inform participants how 
we will use their information.  
On completion, a debrief page will be provided to remind the participant what the results will be 
used for. Contact details will be provided on the introduction and debrief pages should participants 
have any questions or issues they wish to follow up.  
Informed consent will be obtained via checkboxes before starting the study assessments. The check 
boxes will be to confirm that the participant is willing: 

ADJUST: a study on MDT prognostication 
 
 
 
 
Sponsor code:  [6288/001] 
ADJUST study version 1.0 01-September-2020  
Page 17 
 
1 
To participate. 
2 
To acknowledge the results will be used in future publications and research. 
3 
To provide an email address in order to enable participants to log out and return, and for the 
research team to send reminder emails, send feedback and approach them about future re-
search, if desired. 
12.5 Participant withdrawal 
A participant may be withdrawn from the study whenever continued participation is no longer in the 
participant’s best interests, but the reasons for doing so must be recorded. Reasons for discontinu-
ing the study may include: 
1 
Participants withdrawing consent  
2 
Persistent non-compliance to protocol requirements 
The decision to withdraw a participant from the study will be recorded in the study database. If a 
participant explicitly states they do not wish to contribute further data to the study their decision 
will be respected and recorded in the study database.  
Throughout the study, the research team will monitor and review the responses given by the partici-
pants, and the time taken by each participant to complete the online decision tasks; in order to as-
sess for compliance with the protocol. Participants may be excluded from the analysis if their re-
sponse record strongly suggests that they did not comply with the study protocol (e.g. all vignettes 
answered too speedily or with the same response). 
13 Recording and reporting of adverse events  
This is a very low risk study. Clinicians will be asked to review either one or five vignettes and will re-
ceive advice from an advisor. Clinicians will be familiar with the type of information provided. Clear 
instructions and appropriate debrief will be provided. For these reasons, there are no expected ad-
verse events of our study.   
13.1 Notification of reportable protocol violations 
 A reportable protocol violation is a breach which is likely to effect to a significant degree:  
(a) the safety or physical or mental integrity of the participants of the study; or 
(b) the scientific value of the study. 
The sponsor will be notified immediately of any case where the above definition applies during the 
study conduct phase.    
14  Data management  
14.1 Confidentiality 
All data will be handled in accordance with the General Data Protection Regulation. Participants will 
be asked for their email address as well as being assigned a participant ID. The purpose of this is to: 

ADJUST: a study on MDT prognostication 
 
 
 
 
Sponsor code:  [6288/001] 
ADJUST study version 1.0 01-September-2020  
Page 18 
 
• 
Enable the participant to log out and back in to the study website to continue the series of 
vignettes  
• 
Contact participants with a reminder email if the participant should not complete the series 
of vignettes. One reminder email will be sent, one week after starting the study/the last 
website visit (Arm B only) 
• 
Populate the certificate of participation at the end of the study  
• 
Send feedback at the end of the study  
• 
Contact participants regarding future studies  
The participants will not be asked for any other personal identifiable information.  
15  Data handling  
In the study, demographic data and estimates of probabilities obtained through decision tasks will 
be collected from professionals in accordance with the PIS and participants’ informed consent.  
Data obtained from the study will be kept on a web-based database on UCL servers, which is en-
crypted and password protected. The database will be accessible only to approved members of the 
research team by limiting access to the intranet to their IP addresses. Once online data collection is 
completed, the research team at Marie Curie Palliative Care Research Department will download the 
data from the website for statistical analysis at UCL. UCL will act as the data controller.  
This database will be pseudonymised as only the participant ID will be referenced, but a separate da-
tabase linking email addresses with participant ID numbers (but not with other study data) will be 
kept in a separate location on UCL servers. Email addresses will be encrypted in the database with 
the decryption key stored in a separate location. The only member of the study team who will have 
access to this database is Dr Christopher Tomlinson. These data will only be used if requested by rel-
evant authorities, such as to demonstrate the authenticity of the data. The copy of the final data-
bases including email addresses will be retained until three years after publication of the last paper 
arising from the study, or no longer than five years from the end of data collection.  
A separate database of email addresses will be kept for those participants who have indicated that 
they are willing to be approached for future research.  
A UCL email account will be set up for members of the study team to contact participants. Emails 
held in this account will include participants’ email addresses but not their participant ID. These data 
will be retained until three years after publication of the last paper arising from the study, or no 
longer than five years from the end of data collection.  
Procedures for data handling will be explained clearly in the PIS and informed consent will be 
sought. 
The research team at Marie Curie Palliative Care Research Department will process, store and dis-
pose of all data in accordance with all applicable legal and regulatory requirements, including the 
Data Protection Act 1998, the GDPR, and any amendments thereto. Data will be stored on password 
protected, access-restricted, shared drives (S:Drive) on UCL servers. The S:Drives are maintained by 
UCL and routinely backed up. 

ADJUST: a study on MDT prognostication 
 
 
 
 
Sponsor code:  [6288/001] 
ADJUST study version 1.0 01-September-2020  
Page 19 
 
Data (suitably anonymised) may be shared with other research groups if a reasonable request is sub-
mitted to and agreed by the CI.  
16  Statistical Considerations  
16.1 Sample size calculation 
We aim at recruiting between 100-200 clinicians within the 10 month time period of recruitment 
(August 2020-April 2021). This will be sufficient to address the objectives of the study. We will stop 
recruiting and close down the website when we have accrued 300 respondents with complete data 
within the recruitment period (before April 2021).  
We want to compare the means of the weighting of advice (WOA) of the two groups, (the group re-
ceiving advice labelled as coming from an algorithm and the group receiving advice labelled as coming 
from a clinician), with a confidence interval (CI) of 95%. Based on calculations from a similar study by 
Logg et al. (2019) we expect the standard deviation (SD) to be between 0.3 and 0.4. This will give a 
width of the CI between 0.17 and 0.31, depending on the sample size and the SD (see the table below). 
These numbers are based on and calculated from the guidelines by Machin et al. (2018). This means 
that a sample size between 100-200 respondents should be sufficient to detect a significant difference 
between the mean WOAs of the two groups if the mean difference is greater than 0.085 to 0.22.  
Table 1: The width of the confidence interval varying by sample size and SDs 
Sample 
Size 
Expected SD for mean 
diff in WOA 
σ = 0.3 
𝛔= 0.4 
50 
0.33 
0.44 
100 
0.24 
0.31 
150 
0.19 
0.26 
200 
0.17 
0.22 
250 
0.15 
0.20 
300 
0.14 
0.18 
 
The calculations above are sufficient to address the primary objective of the study which is to assess 
the level to which clinicians incorporate advice received from other clinicians or an algorithm into 
their estimates of the prognosis of palliative care patients.  
The online survey consists of five vignettes. Increasing the number of vignettes would increase gen-
eralisability of the study but would add to the study burden for the participants and increase the risk 
that the study would not be completed by respondents. It was considered that five vignettes was a 
suitable balance between generalisability and study burden. 

ADJUST: a study on MDT prognostication 
 
 
 
 
Sponsor code:  [6288/001] 
ADJUST study version 1.0 01-September-2020  
Page 20 
 
16.2 Statistical analysis 
A fully detailed statistical analysis plan will be drawn up in a separate document prior to data analy-
sis. Participants who do not complete the required number of vignettes or violate the protocol (e.g. 
providing the same estimate for every vignette) will be removed from the analysis. 
16.2.1 Summary of baseline data and flow of participants 
Baseline demographic characteristics (gender, age, work environment, role, seniority, expertise) will 
be summarised by group and overall. As a result of the randomisation process, we expect them to be 
balanced across groups. Categorical data will be described using numbers and percentages. Continu-
ous data will either be described with mean and standard deviation, or median and interquartile 
range, pending the distribution. We will produce a CONSORT flow diagram of all participants 
(http://www.consort-statement.org/).  
16.2.2 Primary outcome analysis  
The primary outcome will be clinicians’ estimates of the probability of a patient surviving for two 
weeks. In order to assess the level to which participants incorporate the advice into their final esti-
mates of the probability of survival, the WOA is calculated for each participant for each vignette(s). 
This is done by comparing their final estimate against the initial estimate and the advice provided. 
The final estimate of probability of surviving two weeks can be represented as a weighted combina-
tion of the participant’s initial estimate and the advice received, with the weights being proportional 
to the extent of the shift towards (away from) the advice. WOA will be defined as = |𝑓−𝑖|/|𝑎−𝑖|, 
where ‘i’, ‘f’, and ‘a’ stand for initial, final, and advice, respectively (Yaniv, 2004a). 
 
If a participant decides to adhere completely to his or her initial estimate (100% discounting of the 
advice), the weight of the advice will be 0. If the participant decides to shift completely to the advice 
then the weight of advice will be 1.0 (0% discounting). Intermediate weights indicate that positive 
weights were assigned to the initial estimate and the advice (partial discounting) (Yaniv and 
Kleinberger, 2000a). 
We will then use regression analyses to compare the means of the WOA scores for the two arms, the 
algorithm arm and the clinicians’ arm.  
16.2.3 Secondary outcome analysis  
The secondary outcome will be clinicians’ estimates of the probability of a patient surviving for two 
weeks. We will perform regression analyses to explore any associations between participants’ 
weighting policies and characteristics of the participants (gender, age, work environment, role, sen-
iority, expertise, confidence).  
We want to explore any associations between participants’ weighting policies and the advice itself 
(e.g. the strength of the advice) as well. Participants will receive five pieces of advice from the PiPS 
data. They will receive advice given with high confidence from 85% and upwards (i.e. the patient has 
90% probability of surviving the next 14 days), with lower confidence (i.e. 75% probability of sur-
vival) and one where the advice is equivocal (i.e. 50% probability of survival). Multilevel regression 
analyses will be used to take into account multiple vignettes are completed by each participant. 

ADJUST: a study on MDT prognostication 
 
 
 
 
Sponsor code:  [6288/001] 
ADJUST study version 1.0 01-September-2020  
Page 21 
 
17 Record keeping and archiving 
At the end of the study, all essential documentation will be archived securely by the CI for a mini-
mum of 20 years from the declaration of end of study.  
Essential documents are those which enable both the conduct of the study and the quality of the 
data produced to be evaluated and show whether the site complied with all applicable regulatory 
requirements.  
The sponsor will notify the research team when study documentation can be archived. All archived 
documents must continue to be available for inspection by appropriate authorities upon request.  
18 Study Management Group (SMG) 
The SMG will be formed of the CI (Patrick Stone) and all of the personnel listed in section 1. The 
group will meet regularly and will review recruitment figures and substantial amendments to the 
protocol prior to submission to the REC.   
19 Ethics 
The sponsor will ensure that the study protocol, PIS, consent form and submitted supporting docu-
ments have been approved by the appropriate Research Ethics Committee, prior to any participant 
recruitment. The protocol, all other supporting documents including and agreed amendments, will 
be documented and submitted for ethical and regulatory approval as required. Amendments will not 
be implemented prior to receipt of the required approval(s).  
An annual progress report (APR) will be submitted to the REC within 30 days of the anniversary date 
on which the favourable opinion was given, and annually until the study is declared ended. The CI or 
delegate will prepare the APR. 
Within 90 days after the end of the study, the CI/Sponsor will ensure that the main REC is notified 
that the study has finished.  If the study is terminated prematurely, those reports will be made 
within 15 days after the end of the study. 
The CI or delegate will supply the Sponsor with a summary report of the study, which will then be 
submitted to the REC within 1 year after the end of the study.  
20 Patient and public involvement (PPI) 
This study does not involve patients or members of the public directly. The participants are all clini-
cians working in palliative care. The CI (Patrick Stone) is a Palliative Care Consultant and a member of 
the Study Management Group. As such, he has been involved in the design of the research and will 
be involved in management of the research, undertaking the research, analysis of results and dis-
semination of findings. In addition, we will ask a small number of other palliative care clinicians to 
review the recruitment materials and pilot the study website.  

ADJUST: a study on MDT prognostication 
 
 
 
 
Sponsor code:  [6288/001] 
ADJUST study version 1.0 01-September-2020  
Page 22 
 
21 Monitoring  
The sponsor will determine the appropriate level and nature of monitoring required for the study.  
Risk will be assessed on an ongoing basis and adjustments made accordingly. 
The degree of monitoring will be proportionate to the risks associated with the study. 
22 Finance 
This study is funded by the Marie Curie Chair’s grant and the Marie Curie I-CAN-CARE Programme 
grant (MCCC-FPO-16-U). There are no financial interests by the CI, PI, or study management mem-
bers.  
23 Insurance 
University College London holds insurance against claims from participants for injury caused by their 
participation in the study. Participants may be able to claim compensation if they can prove that UCL 
has been negligent.  
Participants may also be able to claim compensation for injury caused by participation in this study 
without the need to prove negligence on the part of University College London or another party.  
Participants who sustain injury and wish to make a claim for compensation should do so in writing in 
the first instance to the Chief Investigator, who will pass the claim to the Sponsor’s Insurers, via the 
Sponsor’s office. 
24 Publication policy 
Study results will be published in peer-reviewed, indexed, journals using an open access format, and 
the results will be presented at academic conferences. Furthermore, we will also publish a PhD the-
sis. Authorship eligibility will be in accordance with The International Committee of Medical Journal 
Editors. All proposed publications will adhere to UCL publication policy. We will also publicise our 
findings on the Marie Curie website. 
25 Intellectual property 
All background intellectual property rights (including licences) and know-how used in connection 
with the study shall remain the property of the party introducing the same and the exercise of such 
rights for purposes of the study shall not infringe any third party’s rights. 
All intellectual property rights and know-how in the protocol and in the results arising directly from 
the study, but excluding all improvements thereto or clinical procedures developed or used by each 
participating site, shall belong to UCLH.  Each participating site agrees that by giving approval to con-
duct the study at its respective site, it is also agreeing to effectively assign all such intellectual prop-
erty rights (“IPR”) to UCL and to disclose all such know-how to UCL.  
Each participating site agrees to, at the request and expense of UCL execute all such documents and 
do all acts necessary to fully vest the IPR in UCL.  

ADJUST: a study on MDT prognostication 
 
 
 
 
Sponsor code:  [6288/001] 
ADJUST study version 1.0 01-September-2020  
Page 23 
 
Nothing in this section shall be construed so as to prevent or hinder the participating site from using 
know-how gained during the performance of the study in the furtherance of its normal activities of 
providing or commissioning clinical services, teaching and research to the extent that such use does 
not result in the disclosure or misuse of confidential information or the infringement of an intellec-
tual property right of UCL.  This does not permit the disclosure of any of the results of the study, all 
of which remain confidential. 
 

ADJUST: a study on MDT prognostication 
 
 
 
 
Sponsor code:  [6288/001] 
ADJUST study version 1.0 01-September-2020  
Page 24 
 
26 References 
ADAMS, E., BOULTON, M. & WATSON, E. 2009. The information needs of partners and family 
members of cancer patients: a systematic literature review. Patient Educ Couns, 77, 179-86. 
ANDERSON, F., DOWNING, G. M., HILL, J., CASORSO, L. & LERCH, N. 1996. Palliative performance 
scale (PPS): a new tool. J Palliat Care, 12, 5-11. 
BONACCIO, S. & DALAL, R. S. 2006. Advice taking and decision-making: An integrative literature 
review, and implications for the organizational sciences. Organizational Behavior and Human 
Decision Processes, 101, 127-151. 
CHRISTAKIS, N. A. & LAMONT, E. B. 2000. Extent and determinants of error in doctors' prognoses in 
terminally ill patients: prospective cohort study. BMJ, 320, 469-72. 
CLARKSON, R., SELBY, D. & MYERS, J. 2013. A qualitative analysis of the elements used by palliative 
care clinicians when formulating a survival estimate. BMJ Support Palliat Care, 3, 330-4. 
COOKSEY, R. W. 1996. Judgment Analysis - Theory, Methods, and Applications, San Diego, Academic 
Press. 
DEGNER, L. F., KRISTJANSON, L. J., BOWMAN, D., SLOAN, J. A., CARRIERE, K. C., O'NEIL, J., BILODEAU, 
B., WATSON, P. & MUELLER, B. 1997. Information needs and decisional preferences in 
women with breast cancer. JAMA, 277, 1485-92. 
DEPARTMENT OF HEALTH 2008. End of Life Care Strategy. Promoting high quality care for all adults 
at the end of life. 
FAIRCHILD, A., DEBENHAM, B., DANIELSON, B., HUANG, F. & GHOSH, S. 2014. Comparative 
multidisciplinary prediction of survival in patients with advanced cancer. Support Care 
Cancer, 22, 611-7. 
GLARE, P., VIRIK, K., JONES, M., HUDSON, M., EYCHMULLER, S., SIMES, J. & CHRISTAKIS, N. 2003. A 
systematic review of physicians' survival predictions in terminally ill cancer patients. BMJ, 
327, 195-8. 
GWILLIAM, B., KEELEY, V., TODD, C., GITTINS, M., ROBERTS, C., KELLY, L., BARCLAY, S. & STONE, P. C. 
2011. Development of prognosis in palliative care study (PiPS) predictor models to improve 
prognostication in advanced cancer: prospective cohort study. BMJ, 343, d4920. 
GWILLIAM, B., KEELEY, V., TODD, C., ROBERTS, C., GITTINS, M., KELLY, L., BARCLAY, S. & STONE, P. 
2013a. Prognosticating in patients with advanced cancer--observational study comparing the 
accuracy of clinicians' and patients' estimates of survival. Ann Oncol, 24, 482-8. 
GWILLIAM, B., KEELEY, V., TODD, C., ROBERTS, C., GITTINS, M., KELLY, L., BARCLAY, S. & STONE, P. 
2013b. Prognosticating in patients with advanced cancer-observational study comparing the 
accuracy of clinicians' and patients' estimates of survival. Annals of Oncology, 24, 482-488. 
HIGGINSON, I. J. & COSTANTINI, M. 2002. Accuracy of prognosis estimates by four palliative care 
teams: a prospective cohort study. BMC Palliat Care, 1, 1. 
KALPAKIDOU, A. K., TODD, C., KEELEY, V., GRIFFITHS, J., SPENCER, K., VICKERSTAFF, V., OMAR, R. Z. & 
STONE, P. 2018. The Prognosis in Palliative care Study II (PiPS2): study protocol for a multi-
centre, prospective, observational, cohort study. BMC Palliative Care , 17 , Article 101. 
(2018). 
KEE, F., OWEN, T. & LEATHEM, R. 2007. Offering a prognosis in lung cancer: when is a team of 
experts an expert team? J Epidemiol Community Health, 61, 308-13. 
KENNEDY, J., KLEINMUNTZ, D. N. & PEECHER, M. E. 1997. Determinants of the justifiability of 
performance in ill-structured audit tasks. Journal of Accounting Research, 35, 105-123. 
KIRK, P., KIRK, I. & KRISTJANSON, L. J. 2004. What do patients receiving palliative care for cancer and 
their families want to be told? A Canadian and Australian qualitative study. BMJ, 328, 1343. 
KUTNER, J. S., STEINER, J. F., CORBETT, K. K., JAHNIGEN, D. W. & BARTON, P. L. 1999. Information 
needs in terminal illness. Soc Sci Med, 48, 1341-52. 
LEADERSHIP ALLIANCE FOR THE CARE OF DYING PEOPLE Care for people in the last days and hours of 
life. 
LEADERSHIP ALLIANCE FOR THE CARE OF DYING PEOPLE 2014. One chance to get it right. Improving 
people’s experience of care in the last few days and hours of life. 
LOGG, J. M., MINSON, J. A. & MOORE, D. A. 2019. Algorithm appreciation: People prefer algorithmic 
to human judgment. Organizational Behavior and Human Decision Processes, 151, 90-103. 
MACHIN, D., CAMPBELL, M. J., TAN, S. B. & TAN, S. H. 2018. Sample Sizes for Clinical, Laboratory and 
Epidemiology Studies, Chichester, UK, Chichester, UK: John Wiley & Sons, Ltd. 

ADJUST: a study on MDT prognostication 
 
 
 
 
Sponsor code:  [6288/001] 
ADJUST study version 1.0 01-September-2020  
Page 25 
 
MALTONI, M., CARACENI, A., BRUNELLI, C., BROECKAERT, B., CHRISTAKIS, N., EYCHMUELLER, S., 
GLARE, P., NABAL, M., VIGANO, A., LARKIN, P., DE CONNO, F., HANKS, G., KAASA, S. & 
STEERING COMMITTEE OF THE EUROPEAN ASSOCIATION FOR PALLIATIVE, C. 2005. 
Prognostic factors in advanced cancer patients: evidence-based clinical recommendations--a 
study by the Steering Committee of the European Association for Palliative Care. J Clin Oncol, 
23, 6240-8. 
MALTONI, M., NANNI, O., PIROVANO, M., SCARPI, E., INDELLI, M., MARTINI, C., MONTI, M., 
ARNOLDI, E., PIVA, L., RAVAIOLI, A., CRUCIANI, G., LABIANCA, R. & AMADORI, D. 1999. 
Successful validation of the palliative prognostic score in terminally ill cancer patients. Italian 
Multicenter Study Group on Palliative Care. J Pain Symptom Manage, 17, 240-7. 
MOORE, D. A. & KLEIN, W. M. 2008. Use of absolute and comparative performance feedback in 
absolute and comparative judgments and decisions. Organizational Behavior and Human 
Decision Processes, 107, 60-74. 
MORITA, T., TSUNODA, J., INOUE, S. & CHIHARA, S. 1999. The Palliative Prognostic Index: a scoring 
system for survival prediction of terminally ill cancer patients. Supportive Care in Cancer, 7, 
128-133. 
MORITA, T., TSUNODA, J., INOUE, S. & CHIHARA, S. 2001. Improved accuracy of physicians' survival 
prediction for terminally ill cancer patients using the Palliative Prognostic Index. Palliat Med, 
15, 419-24. 
NEUBERGER, J., GUTHRIE, C. & AARANOVITCH, D. 2013. More care, less pathway: a review of the 
Liverpool care pathway. London: Department of Health. 
PILLAY, B., WOOTTEN, A. C., CROWE, H., CORCORAN, N., TRAN, B., BOWDEN, P., CROWE, J. & 
COSTELLO, A. J. 2016. The impact of multidisciplinary team meetings on patient assessment, 
management and outcomes in oncology settings: A systematic review of the literature. 
Cancer Treat Rev, 42, 56-72. 
PIROVANO, M., MALTONI, M., NANNI, O., MARINARI, M., INDELLI, M., ZANINETTA, G., PETRELLA, V., 
BARNI, S., ZECCA, E., SCARPI, E., LABIANCA, R., AMADORI, D. & LUPORINI, G. 1999. A new 
palliative prognostic score: a first step for the staging of terminally ill cancer patients. Italian 
Multicenter and Study Group on Palliative Care. J Pain Symptom Manage, 17, 231-9. 
PONTIN, D. & JORDAN, N. 2013. Issues in prognostication for hospital specialist palliative care 
doctors and nurses: a qualitative inquiry. Palliat Med, 27, 165-71. 
RAPPOLT, S. 2002. Family physicians' selection of informal peer consultants: implications for 
continuing education. J Contin Educ Health Prof, 22, 113-20. 
RUHSTALLER, T., ROE, H., THÜRLIMANN, B. & NICOLL, J. J. 2006. The multidisciplinary meeting: An 
indispensable aid to communication between different specialities. European Journal of 
Cancer, 42, 2459-2462. 
SNIEZEK, J. A. & BUCKLEY, T. 1995. Cueing and Cognitive Conflict in Judge-Adviser Decision-Making. 
Organizational Behavior and Human Decision Processes, 62, 159-174. 
SNIEZEK, J. A. & VAN SWOL, L. M. 2001. Trust, Confidence, and Expertise in a Judge-Advisor System. 
Organ Behav Hum Decis Process, 84, 288-307. 
SOLL, J. B. & LARRICK, R. P. 2009. Strategies for Revising Judgment: How (and How Well) People Use 
Others' Opinions. Journal of Experimental Psychology-Learning Memory and Cognition, 35, 
780-805. 
STEINHAUSER, K. E., CHRISTAKIS, N. A., CLIPP, E. C., MCNEILLY, M., GRAMBOW, S., PARKER, J. & 
TULSKY, J. A. 2001. Preparing for the end of life: preferences of patients, families, physicians, 
and other care providers. J Pain Symptom Manage, 22, 727-37. 
STEINHAUSER, K. E., CHRISTAKIS, N. A., CLIPP, E. C., MCNEILLY, M., MCINTYRE, L. & TULSKY, J. A. 
2000. Factors considered important at the end of life by patients, family, physicians, and 
other care providers. JAMA, 284, 2476-82. 
TAVARES, T., OLIVEIRA, M., GONCALVES, J., TROCADO, V., PERPETUO, J., AZEVEDO, A., MACHADO, F., 
BARRETO, V. & ROCHA, C. 2017. Predicting prognosis in patients with advanced cancer: A 
prospective study. Palliat Med, 269216317705788. 
THOMAS, K. & FREE, A. 2011. Gold Standards Framework Prognostic Indicator Guidance. Royal 
College of General Practitioners. 
VAN SWOL, L. M. & SNIEZEK, J. A. 2005. Factors affecting the acceptance of expert advice. British 
Journal of Social Psychology, 44, 443-461. 

ADJUST: a study on MDT prognostication 
 
 
 
 
Sponsor code:  [6288/001] 
ADJUST study version 1.0 01-September-2020  
Page 26 
 
WEBER, H., STOCKLI, M., NUBLING, M. & LANGEWITZ, W. A. 2007. Communication during ward 
rounds in internal medicine. An analysis of patient-nurse-physician interactions using RIAS. 
Patient Educ Couns, 67, 343-8. 
WEINBERG, A. D., ULLIAN, L., RICHARDS, W. D. & COOPER, P. 1981. Informal advice- and 
information-seeking between physicians. J Med Educ, 56, 174-80. 
WHITE, N., HARRIES, P., HARRIS, A., REID, F., VICKERSTAFF, V., LODGE, P., MCGOWAN, C., MINTON, 
O., TOMLINSON, C., TOOKMAN, A. & STONE, P. How Is Imminent Death Recognised? A Study 
to Understand the Decision Making Process of Expert Prognosticators.  10th World Research 
Congress of the European Association for Palliative Care (EAPC), 2018 Bern, Switzerland. 
WHITE, N., REID, F., HARRIS, A., HARRIES, P. & STONE, P. 2016. A Systematic Review of Predictions of 
Survival in Palliative Care: How Accurate Are Clinicians and Who Are the Experts? PLoS One, 
11, e0161407. 
WITT, J., MURTAGH, F. E. M., DE WOLF-LINDER, S., HIGGINSON, I. J. & DAVESON, B. A. 2014. 
Introducing the Outcome Assessment and Complexity Collaborative (OACC) Suite of 
Measures. A Brief Introduction. 
YANIV, I. 2004a. Receiving other people's advice: Influence and benefit. Organizational Behavior and 
Human Decision Processes, 93, 1-13. 
YANIV, I. 2004b. Receiving other people’s advice: Influence and benefit. Organizational Behavior and 
Human Decision Processes, 93, 1-13. 
YANIV, I. & KLEINBERGER, E. 2000a. Advice taking in decision making: Egocentric discounting and 
reputation formation. Organizational Behavior and Human Decision Processes, 83, 260-281. 
YANIV, I. & MILYAVSKY, M. 2007. Using advice from multiple sources to revise and improve 
judgments. Organizational Behavior and Human Decision Processes, 103, 104-120. 
YANIV, I. I. & KLEINBERGER, E. 2000b. Advice Taking in Decision Making: Egocentric Discounting and 
Reputation Formation. Organ Behav Hum Decis Process, 83, 260-281. 
 

ADJUST: a study on MDT prognostication 
 
 
 
 
Sponsor code:  [6288/001] 
ADJUST study version 1.0 01-September-2020  
Page 27 
 
27 Appendix 1 - Schedule of assessments    
28 Appendix 2 – Sample vignette  
Please note that this is an example vignette and that the actual vignettes used may vary somewhat 
from this example in style and content. 
Example Vignette 1 
Mr Smith has recently been admitted to the inpatient unit at St Swithin’s hospice. He has two pri-
mary cancers - colorectal (lower GI primary) and oropharyngeal (head and neck primary). He has me-
tastases to his lungs and bones and is currently undergoing palliative radiotherapy.  
He is 63 years old and has full capacity. On assessment, there is no evidence of ascites or peripheral 
oedema. He reports that his eating and drinking are significantly reduced and that he has lost weight, 
but there is no dysphagia. He feels fatigued and is unable to do jobs around the house that he used 
to be able to do. For the most part, he is still independent in self-care tasks. As he talks about his 
symptoms, it is clear that he is experiencing shortness of breath. His pulse rate is 88 (beats/min).  
His blood tests show the following: 
WBC: 
 
 
11x109/L 
(normal range 4.0 to 11.0) 
Lymphocyte: 
 
0x109/L  
(normal range 1.0 to 4.0) 
Neutrophil: 
 
10x109/L 
(normal range 1.7 to 8.0) 
Platelet: 
 
273x109/L 
(normal range 150 to 450) 
Urea: 
 
 
7mmol/L 
(normal range 2.5 to 7.8) 
Albumin: 
 
26g/L  
(normal range 35 to 50) 
Alkaline Phosphatase: 105U/L  
(normal range 30 to 130) 
Alanine Transaminase: 12U/L  
(normal range 0 to 52) 
CRP: 
 
 
288mg/L 
(normal range 0.0 to 10.0) 
LDH: 
 
 
1183U/L 
(normal range 140 to 280)  
 

ADJUST: a study on MDT prognostication 
 
 
 
 
Sponsor code:  [6288/001] 
ADJUST study version 1.0 01-September-2020  
Page 28 
 
 
 
 
What do you think the probability is that this patient will survive the next two weeks? 
 
Please write your answer as a percentage in the box to the right  
(Scale: 0-100. Where 0% means certain to die and 100% means certain to survive) 
 
 
 
                                      
FOR THE CLINICIAN ADVICE: 
A palliative care [nurse/doctor] estimated (based on the provided patient information) that the 
probability this patient will survive the next two weeks was: 
 
FOR THE ALGORITHM ADVICE: 
The PiPS-B14 estimated (based on the provided patient information) that the probability this patient 
will survive the next two weeks was:             
 
 
What is your final estimate of the probability this patient will survive the next two weeks? 
 
…….% 
…….% 
…….% 
…….%