An Online International Comparison of Thresholds for Triggering a Negative Response to the "Surprise Question"
Conditions: Palliative Care, Terminal Illness
Sponsor: University College, London

STUDY PROTOCOL
Open Access
An online international comparison of
thresholds for triggering a negative
response to the “Surprise Question”: a
study protocol
Nicola White1*
, Linda Oostendorp1, Victoria Vickerstaff1, Christina Gerlach2, Yvonne Engels3, Maud Maessen4,
Christopher Tomlinson5, Johan Wens6, Bert Leysen6, Guido Biasco7, Sofia Zambrano4, Steffen Eychmüller4,
Christina Avgerinou1, Rabih Chattat8, Giovanni Ottoboni8, Carel Veldhoven3 and Patrick Stone1
Abstract
Background: The Surprise Question (SQ) “would I be surprised if this patient were to die in the next 12 months?”
has been suggested to help clinicians, and especially General Practitioners (GPs), identify people who might benefit
from palliative care. The prognostic accuracy of this approach is unclear and little is known about how GPs use this
tool in practice. Are GPs consistent, individually and as a group? Are there international differences in the use of the
tool? Does including the alternative Surprise Question (“Would I be surprised if the patient were still alive after 12
months?”) alter the response? What is the impact on the treatment plan in response to the SQ? This study aims to
address these questions.
Methods: An online study will be completed by 600 (100 per country) registered GPs. They will be asked to review
20 hypothetical patient vignettes. For each vignette they will be asked to provide a response to the following four
questions: (1) the SQ [Yes/No]; (2) the alternative SQ [Yes/No]; (3) the percentage probability of dying [0% no
chance – 100% certain death]; and (4) the proposed treatment plan [multiple choice]. A “surprise threshold” for
each participant will be calculated by comparing the responses to the SQ with the probability estimates of death.
We will use linear regression to explore any differences in thresholds between countries and other clinician-related
factors, such as years of experience. We will describe the actions taken by the clinicians and explore the differences
between groups. We will also investigate the relationship between the alternative SQ and the other responses.
Participants will receive a certificate of completion and the option to receive feedback on their performance.
Discussion: This study explores the extent to which the SQ is consistently used at an individual, group, and
national level. The findings of this study will help to understand the clinical value of using the SQ in routine
practice.
Trial registration: Clinicaltrials.gov NCT03697213 (05/10/2018). Prospectively registered.
Keywords: Prognosis, Palliative care, Surprise question, Survival, Death
© The Author(s). 2019 Open Access This article is distributed under the terms of the Creative Commons Attribution 4.0
International License (http://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and
reproduction in any medium, provided you give appropriate credit to the original author(s) and the source, provide a link to
the Creative Commons license, and indicate if changes were made. The Creative Commons Public Domain Dedication waiver
(http://creativecommons.org/publicdomain/zero/1.0/) applies to the data made available in this article, unless otherwise stated.
* Correspondence: n.g.white@ucl.ac.uk
1University College London, London, UK
Full list of author information is available at the end of the article
White et al. BMC Palliative Care           (2019) 18:36 
https://doi.org/10.1186/s12904-019-0413-x

Background
Predicting how long a patient has left to live is an in-
accurate and complicated clinical skill [1–5]. Yet it is a
clinical skill expected of all medical doctors [6]. In the
UK, early identification of people approaching the last
year of life can facilitate referral to specialist services
and funding, which in turn improves the patient’s quality
of life [7–12]. The task of early identification of these
patients often falls on General Practitioners (GPs) in the
community. The timely identification of patients in the
last 12 months of life has been described by GPs as one
of the main challenges they face [13]. It has been re-
ported that GPs often wait until very close to death be-
fore discussing end of life issues [14] which can often
prevent the individual experiencing a “good death” [15,
16]. The “Surprise Question” (SQ) is one such tool avail-
able to identify those who might benefit from palliative
care. The SQ describes the process whereby clinicians
are encouraged to ask themselves whether or not they
would “be surprised” if a patient were to die within a
specific time (e.g. 12 months, 6 months, 1 month, 7 days).
It is suggested that if a clinician “would not be surprised”
if a patient were to die within, for example, the next 12
months then this should act as a trigger to adopting a
more palliative care approach to their care (e.g. placing
the patient on an end-of-life communication register or
referring them to specialist palliative care services). Al-
though the original aim of the SQ was not simply to pre-
dict survival but to identify people with palliative care
needs [17, 18], it is often used as a means to identify
people who might be in the last 12 months of life. The
prognostic capability of the SQ has been reported to be
variable [19–21].
One problem with the SQ is that it is not clear to what
extent a death needs to be expected before a clinician
would be “surprised” if it did or did not occur. It has
previously been highlighted by GPs how subjective and
difficult the SQ can be in clinical practice [22]. If there
are large variations in these “trigger” values, or “thresh-
old levels”, between clinicians then this would have im-
portant implications for how responses to the SQ should
be
interpreted.
The
variation
between
individuals’
threshold levels, and the consistency of the responses to
the SQ, have not previously been evaluated. It is also not
known which other factors may affect trigger values (e.g.
nationality, age, gender, palliative care experience, or
years
of
seniority).
A
better
understanding
about
when clinicians would, or would not, be “surprised” if
a patient were to die would help to standardise and
calibrate the use of the SQ. Being aware of the
threshold probabilities which trigger “surprise” in dif-
ferent groups may also lead to strategies to improve
the accuracy and the consistency with which the SQ
is used in clinical practice.
Research by Carel Veldhoven and colleagues [23, 24]
reported that the use of an additional alternative Sur-
prise Question “would I be surprised if this patient is
still alive after 12 months?”, following a negative re-
sponse (“No”) to the original Surprise Question, im-
proves
the
predictive
value
of
the
question.
Incorporating this question, alongside the original SQ
could help improve the prognostic capability of the tool
in addition to focusing care where it is needed most.
Further research is needed to validate this finding.
In addition to this, there is little evidence about the re-
lationship between the SQ response and the course of
action decided upon. What evidence is available suggests
that the treatment decisions made at the end of life vary
considerable depending on clinician and patient factors
[25]. Do GPs consistently pursue the same course of ac-
tion for patients with similar symptoms, or for those
with the same SQ response?
Objectives
The study has the following objectives:
1. To determine how consistent General Practitioners
are in their response to the SQ, and what their
“threshold” level is in order to provide a negative
response (“No, I would not be surprised”) to the
Surprise Question
2. To determine the relationship between the original
Surprise Question, the alternative Surprise Question
(“Would I be surprised if this person were still alive
after 12 months”), and the estimated probability of
death occurring in the next 12 months.
3. To understand the course of action decided upon in
relation to the prognostic estimates provided.
Methods
Methods: participants, interventions, and outcomes
This protocol follows the SPIRIT reporting guidelines
for research protocols [26]. The checklist can be found
in Additional file 1.
Study design
The Surprise Study is a non-randomised multicentre on-
line study with a single group of GPs who will complete
20 hypothetical patient summaries, or “vignettes”.
Study setting
The online study will be distributed to GPs in the fol-
lowing participating countries: Belgium, Germany, Italy,
The Netherlands, Switzerland, and the United Kingdom.
White et al. BMC Palliative Care           (2019) 18:36 
Page 2 of 9

Eligibility criteria
A convenience sample of GPs with a variety of years of
experience and seniority across six European countries
will be approached to participate.
Inclusion criteria
 Registered GP in one of the six participating
countries
 Able to read and understand the language in which
the study is presented to them
Exclusion criteria
 Decline to participate
Procedure
Participants will receive a Participant Information Sheet
(PIS) if they receive an email to participate and/or they
will be able to download a PIS directly from the study
website. Informed consent will be embedded in the
study website and will be obtained via four checkboxes
before starting the study. The first will be to consent to
participate. The second will be to consent to the results
being used in future publications, research and educa-
tional packages. The third box will be consent to provide
an email address in order to enable participants to log
out and return, and for the research team to send re-
minder emails. The fourth consent box is optional and
will ask participants if they wish to receive feedback on
their performance. The participant will be reminded that
they are free to withdraw at any time. On completion, a
debrief page will be provided to remind the participant
what the results will be used for. The contact details for
the study team will also be displayed should they have
any concerns or issues they wish to raise.
Once consented, participants will be asked to read
through 20 vignettes of patients with advanced diseases
but who have not yet been referred to specialist palliative
care services.
Vignette development
The vignettes were constructed by the collaborators
in the group and were designed to represent typical
patient cases some of whom would be expected to
die within 12 months, some of whom would be ex-
pected to survive for 12 months and some with an
uncertain prognosis. The actual survival of the pa-
tients depicted in the vignettes will not be known
(since the cases will not represent “real” patients).
However the purpose of the exercise is not to evalu-
ate the accuracy of clinicians’ predictions but rather
to investigate the level of probability at which respon-
dents would not be surprised if the patient were to
die within the next 12 months. The vignettes contain
non-specialist information (e.g. they will not contain
specialty specific blood test results or diagnostic test
results)
so
that
generalist
doctors
without
any
specialty-specific training will be able to understand
the information provided. See Fig. 1 for an example
of a patient vignette.
To ensure an equal distribution of symptoms and dis-
ease groups, the vignettes were developed follow a guid-
ance
template,
informed
from
the
Gold
Standards
Framework Proactive Indicator Guidance [27], as well as
disease specific grading measures (Additional file 2).
Each country participating developed a 5 vignettes fol-
lowing this template, from which the final set of vi-
gnettes were selected.
Translation process
We will adhere to the European Organisation for Re-
search and Treatment of Cancer (EORTC) guidelines
[28] for the translation process as much as possible
within the confines of the resources available for this
study. All study documents (the website content, the
PIS, the certificate of completion, and the vignettes)
will firstly be written in English, verified by the UCL
study team. The forward translation into the lan-
guages of all countries participating in this study
(English,
German,
Italian,
and
Dutch)
will
be
Fig. 1 Example hypothetical vignette
White et al. BMC Palliative Care           (2019) 18:36 
Page 3 of 9

completed independently by two translators per lan-
guage. The translators will compare and form one
translated set of documents. The documents will then
be back translated to English by a third translator
who was not involved in the forward translation. The
UCL study team will then review the original version
with the back translated version and any differences
will be discussed between the translators until the dif-
ferences have been resolved.
Outcomes
The primary outcomes will be the continuous estimates
of probability of dying within the next 12 months for the
SQ (0–100%).
The secondary outcomes will be:
1) Dichotomous response to the SQ (Yes/No)
2) Dichotomous response to the alternative SQ (Yes/
No)
3) The options for the course of action selected by the
participants in each vignette
Participant timeline
It is expected that participation should take no longer
than 30 min. However, due to the busy work schedules
of the participants, they will have the option to “log out”
of the study and return at their convenience. This meth-
odology has been used previously and shown to help re-
cruitment as well as minimise attrition [29, 30] Should
they not complete the study on the first assessment, they
will be sent a reminder email 7 days after they started
the study and a final one 3 weeks after starting the study
(see
Table
1
for
the
participant
timeline).
Once
recruitment has started, the website will remain open
until the end of the study period or until the recruitment
target has been met.
Sample size
Twenty vignettes have been developed. This number of
vignettes was estimated in order to keep the task burden
to a minimum, to reduce attrition, and to collect enough
data points to establish the threshold score for each
participant.
We aim to recruit the target sample size of 600
doctors (100 per country) within 8 months of the
study opening. This timeframe should be sufficient to
allow GPs, with busy working schedules, to complete
the study. The sample size is based on calculating the
trigger probability to an acceptable level of precision.
As there is no current evidence about the trigger
probability level for individuals, we assumed a trigger
probability of 50%. Using this conservative estimate,
aiming at a 4% margin of error (equivalently a preci-
sion of 8%), with a level of confidence of 95%, we will
need to recruit 600 participants. From previous re-
search that approached a membership body of doctors
to participate [31], we anticipate a response rate of
approximately 10–15% and therefore will aim to ap-
proach 1000–1500 doctors per country.
Recruitment
The method of recruitment will vary slightly by country
in order to accommodate national variations in the re-
quired approvals. Participants will be contacted via one
of the following methods:
Table 1 Participant Timeline
Study Period
Time window
(only if not completed at time point 1)
Initial access
Reminder email (7 days)a
Reminder email (21 days)a
Study end
Time point
1
2
3
4
ENROLMENT
Informed consent
X
Demographic information
X
Practice vignette
X
MAIN STUDY
20 vignettes
X
X
X
ASSESSMENTS
The Surprise Question
X
X
X
The alternative Surprise Question
X
X
X
Dichotomous prediction
X
X
X
Treatment plan
X
X
X
EXCLUSION (if incomplete)
X
aMain study & assessments only need to be completed if not completed at time point 1
White et al. BMC Palliative Care           (2019) 18:36 
Page 4 of 9

1) An email or receipt of study information within a
newsletter from a membership affiliation email
2) An email or word of mouth from a colleague who
has participated/heard of the study (snow-balling)
In the UK for example, we could use Local Medical
Committees (LMCs), Clinical Commissioning Groups
(CCGs) administrators, and the newsletters of the
British Medical Association (BMA) and the Royal
College of General Practitioners (RCGP) to dissemin-
ate a link to the website to participants. In addition
to this, we will ask any GP who participates to share
the study link with eligible colleagues.
Methods: data collection, management, and analysis
Outcome data collection
Demographic and biographical information about the
participants will be collected (location, job title, spe-
cialty, setting, age, gender, years of experience since
qualifying), and information about their use of the SQ
(frequency of use, confidence).
After each vignette, the participant will be asked the
following four questions in order to collect data for the
primary and secondary outcomes:
1. Would you be surprised if this patient were to die
in the next 12 months? (Y/N) (The Surprise
Question)
2. Would you be surprised if this patient were to
remain alive after 12 months? (Y/N) (The
alternative surprise question)
3. What do you think the probability is of this patient
dying within the next 12 months? 0% (Certain
survival) - 100% (Certain death)
4. What do you think this patient needs? (select more
than one if appropriate)
This final question (Q4) will contain a list of potential
treatment or care options. The precise wording of these
questions will be developed and agreed with the Euro-
pean collaborators to make sure that the questions are
culturally and linguistically meaningful.
Participant retention
At the completion of the study, participants will be
able to download a certificate of participation. Once
the study has closed, they will receive feedback on
their performance (if they requested this on the con-
sent page). It is hoped that these options, as well as
the option to log out of the website and return at a
later date, will promote participant recruitment and
retention.
Data management
The database will be designed by a database specialist
(CT). The database will be rigorously tested by the UCL
study team (NW, LO, VV, & PS) prior to the study
starting.
Data collection tools
The study is accessed by the participants online and as
such,
the
data
will
be
collected
directly
from
participants.
Each participant will be able to complete a practice vi-
gnette prior to completing the main set of vignettes in
order to familiarise themselves with the online environ-
ment. There will be no time limit for completion so that
each participant will be able to take their own time. Par-
ticipants will not be able to move on to the next page if
required information is missing or if responses have
been submitted in an incorrect format (e.g. incorrect
email format). The UCL study team and the statistician
will review the data being collected before, during, and
at the start of the recruitment phase to ensure data
integrity.
Statistical methods
A detailed statistical analysis plan will be drawn up in a
separate document prior to analysis.
Primary outcome analysis
This is an exploratory study to determine how the SQ is
utilised by different GPs in different countries. For this
reason, we will employ a per-protocol analysis where
those participants who do not complete more than 15
vignettes or violate the protocol (e.g. putting the same
answer for every vignette) will be removed from the
analysis.
We will examine each participant’s responses to the 20
vignettes to calculate the ‘trigger probability’ (or ‘thresh-
old’) for each individual. To calculate this threshold we
will examine the participant’s responses to the SQ and
the responses to the probability of dying in the next 12
months. We define the threshold as the probability level
after which all responses to the SQ are ‘no’. We will also
describe the level of uncertainty about this decision. This
uncertainty will be calculated by looking at the differ-
ence between the lowest probability level with which the
participant responded ‘no’ to the SQ and the threshold
level.
Three example GP responses are described below to
explain this further.
GP 1 responded to 20 vignettes. For each vignette the
GP stated the probability of dying and a corresponding
answer to the SQ (yes/no). The probability results
ranged from 0 to 100%. With each probability the GP
also stated ‘yes/no’ to the questions (for ease of display,
White et al. BMC Palliative Care           (2019) 18:36 
Page 5 of 9

the individual probability estimates have not been pro-
vided but the ‘Y’/‘N’ responses have been placed over
the corresponding % to show the point at which the re-
sponses changed from Y to N):
In Fig. 2, the probability threshold would be 60%. This
GP consistently said ‘no’ to the SQ if the probability of
dying was greater than 60%. Consequently the threshold
would be 60% and the uncertainty would be 0%.
GP 2 also responded to 20 vignettes.
In Fig. 3, the lowest probability level with which the
GP responded to ‘no’ on the SQ was 30%. After 60% the
GP consistently said ‘no’ on the SQ. Consequently the
threshold is 60% but their uncertainty is 30% (= 60–
30%).
GP 3 also responded to 20 vignettes.
In Fig. 4, the lowest probability for which the GP an-
swered “no” to the SQ was 10%. After 95% the GP con-
sistently said “no”. Consequently the threshold was 95%
and the level of uncertainty was 85% (= 95–10%).
Once the thresholds are calculated for all participants,
we will use linear regression to explore the differences, if
any, in thresholds between countries and other demo-
graphic detail collected.
Secondary outcome analysis
We will explore the relationship between the alternative
SQ and the other responses. We will also describe the
actions taken by the clinicians and explore the differ-
ences between groups.
Methods: monitoring
Throughout the study, the research team will review re-
cruitment figures and the study will close once each
country has recruited 100 participants. Researchers at
UCL will check the responses given by the participants
to assess for compliance with the protocol. Participants
may be excluded from the analysis if their response rec-
ord strongly suggests that they did not comply with the
study protocol (e.g. all items answered with the same re-
sponse). The UCL Study Management Group (PS, LO,
NW, CT, VV) will be responsible for overseeing the trial
and will meet regularly (at least four times per year) to
review recruitment figures.
This is a very low risk study. There are no expected
side effects of our intervention and this study will not
have a Data Monitoring Committee.
Data management & dissemination
Confidentiality
All data will be handled in accordance with the General
Data Protection Regulation (GDPR) 2018 [32].
The study will ask the participants for their name and
email address as a personal identifier as well as being
assigned a participant ID. The purpose of this is to:
1. Enable the participant to log out and back in to the
series of vignettes at the same place
2. To contact participants who do not complete the
study with a reminder email at the one week and 3
week time points after the last website visit.
The participants will not be asked for any other per-
sonal identifiable information. During the study, all data
will be kept securely on a web-based database, which is
encrypted and password protected. The database will be
accessible to approved members of the UCL research
team only. Once all data have been reviewed, the names
and email addresses will be retained for 12 months after
publication, at which point all personal information will
be destroyed and only the participant ID will be refer-
enced. This will be explained clearly in the PIS and con-
sent will be sought.
Access to the data
In the study, demographic data and outcome data will
be collected from participants in accordance with the
PIS and this protocol.
The final study database will be downloaded from the
website by the research team (Marie Curie Palliative
Care Research Department, Division of Psychiatry, UCL)
for statistical analysis and UCL will act as the data con-
troller of such data for the study.
The research team (Marie Curie Palliative Care Re-
search Department, Division of Psychiatry, and UCL)
Fig. 2 GP 1’s responses
White et al. BMC Palliative Care           (2019) 18:36 
Page 6 of 9

will process, store and dispose of the final study database
in accordance with all applicable legal and regulatory re-
quirements,
including
the
General
Data
Protection
Regulation (GDPR) and any amendments thereto. Data
will be stored electronically, on an encrypted hard drive,
with restricted access. The hard drive is maintained by
UCL and routinely backed up.
The anonymous dataset will be shared with the other
research groups participating in the study.
Record keeping and archiving
At the end of the study, all essential documentation will
be archived securely by the CI for a minimum of 20
years from the declaration of end of study.
Essential documents are those which enable both the
conduct of the study and the quality of the data pro-
duced to be evaluated and show whether the site com-
plied with all applicable regulatory requirements.
The sponsor will notify the site when study documen-
tation can be archived. All archived documents will be
continued to be available for inspection by appropriate
authorities upon request.
Dissemination policy
Study
results
will
be
published
in
peer-reviewed,
indexed, journals using an open access format, and the
results will be presented at national and international
conferences. Authorship eligibility will be in accordance
with The International Committee of Medical Journal
Editors. All proposed publications will adhere to UCL
publication policy.
Marie Curie supports this study and we will work with
them to disseminate findings through blogs, newsletters
and social media.
Discussion
Four members of the Marie Curie Expert Research
Voices Group have reviewed a summary of this study to
ensure its aims, objectives, study design and outputs are
in line with patient and family needs. All respondents
felt it was important to identify patients who may re-
quire palliative care early, and they felt the Surprise
Question could be a useful way to get doctors to think
about the sorts of patients who may or may not require
palliative care. Other comments included respondents’
own experiences when a loved one died. These com-
ments were taken in to consideration when developing
the study documents.
It is expected that the recruitment of GPs will be chal-
lenging due to the nature of their work. We further an-
ticipate that attrition may be an issue due to time
constraints on the participants. We have added a func-
tion for the doctor to “log out” of the study and return
at a more convenient time. We have also endeavoured
to minimise the number of vignettes. In addition to this,
we will keep the study open as long as possible to try
and maximise recruitment. We have adopted recruit-
ment strategies that have proven to be successful in an-
other study which also recruited GPs from the UK [33].
Fig. 3 GP 2’s responses
Fig. 4 GP 3’s responses
White et al. BMC Palliative Care           (2019) 18:36 
Page 7 of 9

Additional files
Additional file 1: SPIRIT checklist. (PDF 172 kb)
Additional file 2: Template for vignettes. (PDF 604 kb)
Abbreviations
BMA: British Medical Association; CCG: Clinical Commissioning Groups;
GDPR: General Data Protection Regulation; GP: General Practitioner;
LMC: Local Medical Committees; PIS: Participant Information Sheet;
RCGP: Royal College of General Practitioners; SQ: Surprise Question;
UCL: University College London; UK: United Kingdom
Acknowledgements
We would like to thank the members of the PPI group at Marie Curie Expert
Voices for their invaluable feedback on the study documents.
Funding
This study is partially funded by the Marie Curie I-CAN-CARE Program grant
(MCCC-FPO-18-U). Nicola White, Linda Oostendorp, Victoria Vickerstaff, and
Patrick Stone are partly supported by the UCLH NIHR Biomedical Research
Centre. There are no financial interests by the CI, PI, or study management
members. University College London (UCL) is the sponsor. The sponsor will
have no role in the design, collection, analysis, interpretation, or writing of
this data.
Availability of data and materials
Not applicable.
Protocol version
3.0 2019.01.21
Authors’ contributions
The study will be led by the group based in the UK. There will be a group of
lead researchers in each country who will manage the project within each
country. United Kingdom: NW will create the vignettes and the online study,
recruitment, analyse the data, and write up the results. LO will assist in the
creation of the online study, recruitment, analyse the data, and write up the
results. PS will oversee all actions of the study team. PS will oversee the
analysis and will contribute to the final report. VV will provide support
regarding the statistical analysis of the data and will review the statistical
content within the final report. CT will build the online website and
database which will hold the data gathered from the participants. CA will
assist in the study design and will assist in the write up of the results. The
researchers in all participating countries (CG, YE, MM, JW, BL, GB, SZ, SE, RC,
GO, CV) helped to create the vignettes and the online study, the translation
of study materials into each of the applicable languages, support with
recruitment, and interpretation, write up and dissemination of the results. All
authors have read and approved the manuscript.
Ethics approval and consent to participate
This study gained ethics approval from University College London REC on
the 9th August 2018 (ref 8675/003), University of Antwerp REC on the 7th
January 2019 (ref 18/50/589), University of Mainz REC on the 26th November
2018, Radboud University Medical Center REC on the 20th December 2018
(ref 2018–4949), Bern Ethics Committee on the 29th August 2018 (ref 2018–
00710) and University of Bologna REC on the 25th January 2019 (ref 12,590).
In case any protocol amendments are required, these will be reviewed by
the study team before submission to the relevant committees, and the trial
registry will be updated where necessary. Consent is obtained electronically
through a series of check boxes, see section “Methods, Methods: Participants,
interventions, and outcomes” for further detail.
Consent for publication
Not applicable.
Competing interests
The authors declare that they have no competing interests.
Publisher’s Note
Springer Nature remains neutral with regard to jurisdictional claims in
published maps and institutional affiliations.
Author details
1University College London, London, UK. 2University of Mainz, Mainz,
Germany. 3Radboud University Medical Centre, Nijmegen, The Netherlands.
4University Hospital of Bern, Bern, Switzerland. 5Imperial College London,
London, UK. 6University of Antwerp, Antwerp, Belgium. 7University of
Bologna & Academy of the Sciences of Palliative Medicine, Bologna, Italy.
8University of Bologna, Bologna, Italy.
Received: 4 February 2019 Accepted: 6 March 2019
References
1.
Glare P, Virik K, Jones M, Hudson M, Eychmuller S, Simes J, Christakis N. A
systematic review ofphysicians' survival predictions in terminally ill cancer
patients. BMJ. 2003;327:195.
2.
Glare PA, Sinclair CT. Palliative medicine review: prognostication. J Palliat
Med. 2008;11(1):84–103.
3.
Maltoni M, Caraceni A, Brunelli C, Broeckaert B, Christakis N, Eychmueller S,
Glare P, Nabal M, Viganò A, Larkin P, et al. Prognostic factors in advanced
Cancer patients: evidence-based clinical recommendations—a study by the
steering Committee of the European Association for palliative care. J Clin
Oncol. 2005;23(25):6240–8.
4.
White N, Reid F, Harris A, Harries P, Stone P. A systematic review of
predictions of survival in palliative care: how accurate are clinicians and
who are the experts? PLoS One. 2016;11(8):e0161407.
5.
Zhou M, Holden L, Lao N, Lam H, Zeng L, Chow E. Accuracy of clinicians’
prediction of survival and prognostic factors indicative of survival: a
systematic review. Hong Kong Journal of Radiology. 2013;16(3):168–82.
6.
Gibbins J, McCoubrie R, Maher J, Wee B, Forbes K. Recognizing that it is part
and parcel of what they do: teaching palliative care to medical students in
the UK. Palliat Med. 2010;24(3):299–305.
7.
Henson L, Gao W, Higginson I, Smith M, Davies J, Ellis-Smith C, Daveson B.
Emergency department attendance by patients with cancer in the last
month of life: a systematic review and meta-analysis. Lancet. 2015;385:S41.
8.
Seow H, Brazil K, Sussman J, Pereira J, Marshall D, Austin PC, Husain A,
Rangrej J, Barbera L. Impact of community based, specialist palliative care
teams on hospitalisations and emergency department visits late in life and
hospital deaths: a pooled analysis. BMJ. 2014;348:3496.
9.
El-Jawahri A, Greer JA, Temel JS. Does palliative care improve outcomes for
patients with incurable illness? A review of the evidence. J Support Oncol.
2011;9(3):87–94.
10.
Higginson IJ, Evans CJ. What is the evidence that palliative care teams
improve outcomes for Cancer patients and their families? The Cancer
Journal. 2010;16(5):423–35.
11.
Higginson IJ, Finlay IG, Goodwin DM, Hood K, Edwards AGK, Cook A,
Douglas H-R, Normand CE. Is there evidence that palliative care teams Alter
end-of-life experiences of patients and their caregivers? J Pain Symptom
Manag. 2003;25(2):150–68.
12.
Hately J, Laurence V, Scott A, Baker R, Thomas P. Breathlessness clinics
within specialist palliative care settings can improve the quality of life
and functional capacity of patients with lung cancer. Palliat Med. 2003;
17(5):410–7.
13.
Mitchell S, Loew J, Millington-Sanders C, Dale J: Providing end-of-life care in
general practice: findings of a national GP questionnaire survey. Br J Gen
Pract 2016:bjgpsep-2016-2066-2650-mitchell-fl-p.
14.
Abarshi E, Echteld M, Donker G, Van den Block L, Onwuteaka-Philipsen B,
Deliens L. Discussing end-of-life issues in the last months of life: a
nationwide study among general practitioners. J Palliat Med. 2011;14(3):
323–30.
15.
The End of Life Care Strategy. In. Edited by Department of Health; 2008.
16.
Meier EA, Gallegos JV, Thomas LPM, Depp CA, Irwin SA, Jeste DV. Defining a
good death (successful dying): literature review and a call for research and
public dialogue. Am J Geriatr Psychiatry. 2016;24(4):261–71.
17.
Lynn J. Living long in fragile health: the new demographics shape end of
life care. Hastings Cent Rep. 2005;35(7):s14–8.
18.
Lynn J. The utility and value of the “surprise question” for patients with
serious illness. Can Med Assoc J. 2017;189(33):E1072–3.
White et al. BMC Palliative Care           (2019) 18:36 
Page 8 of 9

19.
Downar J, Goldman R, Pinto R, Englesakis M, Adhikari NK. The “surprise
question” for predicting death in seriously ill patients: a systematic review
and meta-analysis. Can Med Assoc J. 2017;189(13):E484–93.
20.
White N, Kupeli N, Vickerstaff V, Stone P. How accurate is theSurprise
question’ at identifying patients at the end of life? A systematic review and
meta-analysis. BMC Med. 2017;15(1):139.
21.
Mitchell GK, Senior HE, Rhee JJ, Ware RS, Young S, Teo PC, Murray S, Boyd K,
Clayton JM. Using intuition or a formal palliative care needs assessment
screening process in general practice to predict death within 12 months: a
randomised controlled trial. Palliat Med. 2018;32(2):384–94.
22.
Elliott M, Nicholson C. A qualitative study exploring use of the surprise
question in the care of older people: perceptions of general practitioners
and challenges for practice. BMJ Support Palliat Care. 2014:bmjspcare–2014-
000679.
23.
Veldhoven C, Weijers F, Verhagen S, Vissers KYE. Combining the original and
alternative surprise question for the early identification of palliative patients:
a Dutch pilot study. In: European Association of Palliative Care. Madrid.
2017:189. http://www.eapc-2017.org/files/EAPC17/dl/EJPC-Abstract-Book-
2017.pdf.
24.
Weijers F, Veldhoven C, Verhagen C, Vissers K, Engels Y. Adding a second
surprise question triggers general practitioners to increase the
thoroughness of palliative care planning: results of a pilot RCT with cage
vignettes. BMC palliative care. 2018;17(1):64.
25.
Hinkka H, Kosunen E, Lammi U-K, Metsänoja R, Puustelli A, Kellokumpu-
Lehtinen P. Decision making in terminal care: a survey of Finnish doctors'
treatment decisions in end-of-life scenarios involving a terminal cancer and
a terminal dementia patient. Palliat Med. 2002;16(3):195–204.
26.
Standard Protocol Items: Recommendations for Interventional Trials (SPIRIT).
http://www.spirit-statement.org/. Accessed 1 Apr 2019.
27.
The Gold Standards Framework Proactive Identification Guidance (PIG).
http://www.goldstandardsframework.org.uk. Accessed 1 Apr 2019.
28.
European Organisation for Research and Treatment of Cancer (EORTC):
EORTC quality of life group translation procedure. In., 4th edn; 2017.
29.
Oostendorp L, White N, Harries P, Yardley S, Tomlinson C, Ricciardi F, Gokalp
H, Stone P: 35 online randomised controlled trial to improve clinical
estimates of survival (ORACLES): study design. BMJ Support Palliat Care
2018, 8(3):373–373.
30.
White N, Harries P, Harris AJL, Vickerstaff V, Lodge P, McGowan C, Minton O,
Tomlinson C, Tookman A, Reid F, et al. How do palliative care doctors
recognise imminently dying patients? A judgement analysis. BMJ Open. (in
press).
31.
White N, Harries P, Harris AJ, Vickerstaff V, Lodge P, McGowan C, Minton O,
Tomlinson C, Tookman A, Reid F et al: 44 an evidenced-based heuristics
model (or rule of thumb) to improve doctors’ intuition about when patients
are imminently dying. BMJ Support Palliat Care 2018, 8:376–376.
32.
General Data Protection Regulation (GDPR). https://ico.org.uk/for-
organisations/guide-to-the-general-data-protection-regulation-gdpr/.
Accessed 1 Apr 2019.
33.
Curtis P, Taylor G, Harris M. How preferred learning approaches change with
time: a survey of GPs and GP specialist trainees. Education for Primary Care.
2018:1–6.
White et al. BMC Palliative Care           (2019) 18:36 
Page 9 of 9