Consensus Survey - Round 2

I-CoCO
Cardiff University led contribution to the IPDAS Collaboration
A Cardiff University led contribution to the IPDASCollaboration

Voting Document
Thank you for taking part in the 1st round of voting. The purpose of the 2nd round of voting
is to reach consensus on the core criteria that decision support interventions should meet. It
is very important that voters from the 1st round of voting also participate in the 2nd roundof voting to ensure that the results are valid, and we would be very grateful of yourparticipation in this round.
Results from the 1st round have been summarised for you to review. Please consider theseresults when you vote in the 2nd round.

Pre-voting instructions
1.

Please plan to take approximately 30 minutes to complete the survey.

2.

The 2nd round survey follows a similar format to the 1st round survey. In addition, you
you will be asked to consider the 1st round voting results.

3.

If you are unfamiliar with any of the terms, please refer to the Glossary, which can befound at the end of this survey.

If you have any questions or are having difficulty opening the documents, please contact the
research team at ipdas@cardiff.ac.uk
Documents can also be downloaded from the Consensus Process Website:www.decisionlaboratory.com/i-coco-participant.php

Privacy Statement
The information collected during this survey will remain confidential, and will only be made available to the research team.
Your name will not appear on any publications or be associated with the data you provide. Contact information provided
will only be used for the purpose of contacting you during the survey process. Any personal information entered will only beviewed by the research team, and will not be passed on to any third parties.

2

Voting instructions

1. For each dimension, read the 1st round voting results summary.
2. For each item, consider the mean (average) score obtained in the 1st round.
3. Considering the mean score, please assess again whether the subject of the item should be
included as a ‘must have’ component of patient decision support interventions (DESIs).
Score on the basis that if the patient decision support intervention did not meet therequirement of the item, or it was of low quality, then there would be a risk to the user ofharmful bias.
The rating scales range from 1-5.
1 = minimal impact on risk of harmful bias
5 = definite risk of harmful bias
A score of 1 would indicate that the item is not a ‘must have’ item. If the patient decision
support intervention did not meet the requirement of the item, or it was of low quality, there
would be minimal or no impact on the risk of harmful bias to the user.
A score of 5 would indicate that the item is a ‘must have’ item. If the patient decision support
intervention did not meet the requirement of the item, or it was of low quality, there would bea definite risk of harmful bias to the user.
NOTE: Please rate on the basis of risk of harmful bias, and not how feasible it is to implement.

3Dimension 1 - Information
Providing information about options in sufficient detail for making a specificdecision

What is this dimension?
This dimension has 8 items. The decision to be made and the options available to test
or treat the health problem are expected to be described. This may include:
description of the condition; what the procedure involves; possible benefits andadvantages; and possible harms, side effects, or disadvantages. The harms andbenefits of each option should be presented with equal detail and a comparison
between options should be made possible. It may include the benefits and harms of
not being tested or treated.

How might this affect the quality of the decision?
In theory, patient decision support interventions may lead to poor decisions if thefacts given are incomplete or superficial. For ethical and legal reasons, patients havethe right to get full and accurate information before giving their consent to a medical
test or treatment. For patients actively involved in choosing options, more detailed
information about options may be needed.
If you are unfamiliar with any of the terms used, please refer to the Glossary at theend of this survey.

4

Dimension 1 – Information
Voting page
Please assess whether the subject of the item should be included as a ‘must have’ component of patientdecision support interventions (DESIs).
Score on the basis that if the patient decision support intervention did not meet the requirement of theitem, or it was of low quality, then there would be a risk to the user of harmful bias.
The rating scales range from 1-5.
A score of 1 would indicate that the item is not a ‘must have’ item. If the patient decision support
intervention did not meet the requirement of the item, or it was of low quality, there would be minimal orno impact on the risk of harmful bias to the user.
A score of 5 would indicate that the item is a ‘must have’ item. If the patient decision support
intervention did not meet the requirement of the item, or it was of low quality, there would be a definite
risk of harmful bias to the user.
NEW - 1st Round Voting Results
The voters' responses to each item are displayed in 2 ways:•

Percentages (%) are shown in red above the 1-5 rating scale. These are the percentage of the 101
Voters who assigned this score to the item in round 1•

The mean (average) scores are shown in the yellow box to the left of the rating scale
The lowest mean item score in round 1 was 3.09 and the highest mean item score was 4.85

1.

The decision support intervention describes thehealth condition or problem (treatment,
procedure or investigation) for which the index
decision is required

Mean

4.43
Mean

2.

The decision support intervention explicitly
states the decision that needs to be considered(index decision)

4.37
Mean

3.

The decision support intervention describes theoptions available for the index decision

4.

The decision support intervention describes therationale for the decision and the natural courseof the health condition or problem, if no action
is taken

4.85
Mean

4.39

If the DESI did not meet therequirement of the item, or it wasof low quality, there would be…
(Circle one response)
2
4
11
16
67
1
2
3
4
5
Minimal impact onrisk of harmful bias

5
1

1
2

Definite risk ofharmful bias

6
3

Minimal impact onrisk of harmful bias

0
1

1
2

1
2

Minimal impact onrisk of harmful bias

59
5

Definite risk ofharmful bias

1
3

Minimal impact onrisk of harmful bias

1
1

29
4

10
4

88
5

Definite risk ofharmful bias

12
3

31
4

55
5

Definite risk ofharmful bias

Mean
5.

The decision support intervention describes the
positive features (benefits or advantages) ofeach option

6.

The decision support intervention describes thenegative features (harms, side effects, ordisadvantages) of each option

7.

The decision support intervention makes it
possible to compare the positive and negative
features of the available options

8.

The decision support intervention shows thenegative and positive features of options withequal detail (for example, using similar fonts,
sequence, display or statistical information)

4.63
Mean

4.74
Mean

4.05
Mean

4.38

0
1

1
2

5
3

Minimal impact onrisk of harmful bias

0
1

0
2

4
2

3
3

2
2

Minimal impact onrisk of harmful bias

20
4

77
5

Definite risk ofharmful bias

24
3

Minimal impact onrisk of harmful bias

1
1

70
5

Definite risk ofharmful bias

Minimal impact onrisk of harmful bias

2
1

24
4

28
4

43
5

Definite risk ofharmful bias

13
3

27
4

57
5

Definite risk ofharmful bias

6

Dimension 2 - Probabilities
Presenting outcome probabilities

What is this dimension?
This dimension has 8 items. It assesses the way that probabilistic information ispresented. A 'probability' is the chance or likelihood that something will happen. It
could mean the chance of a disease, benefit, harm, or side effect. It is often
estimated by seeing what happens to large groups of patients in the natural course of
the disease or after they have tests or treatments. The groups are usually defined by adisease or condition. Sometimes probabilities are described according to factors thatchange one's chances (e.g. age, sex, severity of disease, presence of other healthproblems).
The patient decision support intervention should provide precise and balanced
information about the outcome probabilities associated with the options. Information
should be presented in a way that facilitates understanding and accounts for individual
differences in processing complex information:- Use of different formats to present probabilities: words, numbers, and pictures;- Comparison of outcome probabilities across options;- Levels of uncertainty associated;- Multiple ways of presenting the probabilities should be used to limit framing biases.

How might this affect the quality of the decision?
In theory, patient decision support interventions may lead to poor decisions if theprobabilities are miscalculated or presented in ways that are misleading. Probabilities
are usually based on research studies and calculated by experts in statistics andmodelling. Decision aids on the same topic sometimes use different numbers so it may
be helpful to identify the source of the probabilities the steps used to calculate them.
If you are unfamiliar with any of the terms used, please refer to the Glossary at theend of this survey.

7

Dimension 2 - Probabilities
Voting page
Please assess whether the subject of the item should be included as a ‘must have’ component of patientdecision support interventions (DESIs).
Score on the basis that if the patient decision support intervention did not meet the requirement of theitem, or it was of low quality, then there would be a risk to the user of harmful bias.
The rating scales range from 1-5.
A score of 1 would indicate that the item is not a ‘must have’ item. If the patient decision support
intervention did not meet the requirement of the item, or it was of low quality, there would be minimal orno impact on the risk of harmful bias to the user.
A score of 5 would indicate that the item is a ‘must have’ item. If the patient decision support
intervention did not meet the requirement of the item, or it was of low quality, there would be a definite
risk of harmful bias to the user.
NEW - 1st Round Voting Results
The voters' responses to each item are displayed in 2 ways:•

Percentages (%) are shown in red above the 1-5 rating scale. These are the percentage of the 101
Voters who assigned this score to the item in round 1

The mean (average) scores are shown in the yellow box to the left of the rating scale
The lowest mean item score in round 1 was 3.09 and the highest mean item score was 4.85

1.

The decision support intervention provides
information about outcome probabilities
associated with the options (i.e. the likely
consequences of decisions)2.

The decision support intervention specifies thedefined group (reference class) of patients forwhich the outcome probabilities apply

3.

The decision support intervention specifies theevent rates for the outcome probabilities (innatural frequencies)

4.

The decision support intervention specifies thetime period over which the outcome
probabilities apply

Mean

4.32
Mean

4.17
Mean

4.10
Mean

3.87

If the DESI did not meet therequirement of the item, or it wasof low quality, there would be…
(Circle one response)
2
1
13
32
53
1
2
3
4
5
Minimal impact onrisk of harmful bias

0
1

6
2

Definite risk ofharmful bias

19
3

Minimal impact onrisk of harmful bias

5
1

2
2

28
2

Minimal impact onrisk of harmful bias

48
5

Definite risk ofharmful bias

11
3

Minimal impact onrisk of harmful bias

36
1

28
4

43
4

40
5

Definite risk ofharmful bias

28
3

6
4

3
5

Definite risk ofharmful bias

8

Mean
5.

The decision support intervention allows the user
to compare outcome probabilities across options
using the same denominator and time period

4.19
Mean

6.

The decision support intervention provides
information about the levels of uncertainty
around event or outcome probabilities (e.g. by
giving a range, ‘our best estimate is’)

3.56
Mean

7.

The decision support intervention provides morethan one way of viewing the probabilities (e.g.
words, numbers, and diagrams)

8.

The decision support intervention provides
balanced information about event or outcome
probabilities to limit framing biases

3.50
Mean

4.18

1
1

4
2

16
3

Minimal impact onrisk of harmful bias

3
1

16
2

14
2

24
3

5
2

Minimal impact onrisk of harmful bias

37
4

21
5

Definite risk ofharmful bias

30
3

Minimal impact onrisk of harmful bias

3
1

46
5

Definite risk ofharmful bias

Minimal impact onrisk of harmful bias

5
1

34
4

30
4

22
5

Definite risk ofharmful bias

12
3

32
4

49
5

Definite risk ofharmful bias

9

Dimension 3 - Values
Clarifying and expressing values

What is this dimension?
This dimension has 4 items. It assesses whether the intervention facilitates theexpression and clarification of patients/users’ values and attitudes regarding theoptions available. It involves a 2-step process of thinking about and thencommunicating the personal importance of different positive and negative features of
options. Ways of helping patients to clarify and express their values include:- describing features of options in ways that helps patients imagine what it is like toundergo procedures and to live with the physical, psychological, and social
consequences;- providing examples of how different values may lead to different choices;- helping patients rate or trade-off different features of the options;- recording and sharing values with others involved in the decision

How might this affect the quality of the decision?
In theory, there may not be a good fit between which features matter most to thepatient and the option that is chosen if patient decision support interventions only
help patients to consider the facts and not personal values.
If you are unfamiliar with any of the terms used, please refer to the Glossary at theend of this survey.

10Dimension 3 - Values
Voting page
Please assess whether the subject of the item should be included as a ‘must have’ component of patientdecision support interventions (DESIs).
Score on the basis that if the patient decision support intervention did not meet the requirement of theitem, or it was of low quality, then there would be a risk to the user of harmful bias.
The rating scales range from 1-5.
A score of 1 would indicate that the item is not a ‘must have’ item. If the patient decision support
intervention did not meet the requirement of the item, or it was of low quality, there would be minimal orno impact on the risk of harmful bias to the user.
A score of 5 would indicate that the item is a ‘must have’ item. If the patient decision support
intervention did not meet the requirement of the item, or it was of low quality, there would be a definite
risk of harmful bias to the user.
NEW - 1st Round Voting Results
The voters' responses to each item are displayed in 2 ways:•

Percentages (%) are shown in red above the 1-5 rating scale. These are the percentage of the 101
Voters who assigned this score to the item in round 1

The mean (average) scores are shown in the yellow box to the left of the rating scale
The lowest mean item score in round 1 was 3.09 and the highest mean item score was 4.85

Mean
1.

The decision support intervention describes thefeatures of options to help patients imagine what
It is like to experience the physical effects

2.

The decision support intervention describes thefeatures of options to help patients imagine what
It is like to experience the psychological effects

3.

The decision support intervention describes thefeatures of options to help patients imagine what
It is like to experience the social effects

4.

The decision support intervention asks patientsto think about which positive and negative
features of the options matter most to them

3.65
Mean

3.43
Mean

3.24
Mean

4.05

If the DESI did not meet therequirement of the item, or it wasof low quality, there would be…
(Circle one response)
3
9
28
41
20
1
2
3
4
5
Minimal impact onrisk of harmful bias

5
1

14
2

Definite risk ofharmful bias

30
3

Minimal impact onrisk of harmful bias

5
1

20
2

5
2

Minimal impact onrisk of harmful bias

15
5

Definite risk ofharmful bias

36
3

Minimal impact onrisk of harmful bias

6
1

35
4

26
4

14
5

Definite risk ofharmful bias

17
3

23
4

50
5

Definite risk ofharmful bias

11

Dimension 4 – Decision Guidance
Structured guidance in deliberation and communication

What is this dimension?
This dimension has 2 items. It assesses whether the patient decision support
intervention provides structured guidance towards making a decision. Guidance
provides a step-by-step was of thinking about the options and discussing them with
their practitioner. Guidance is provided within a patient decision support intervention.
It may be implicit in the way the intervention is organised or explicit by providing a list
or worksheet outlining the steps.

How might this affect the quality of the decision?
Patients are not able to participate in decision making about health care options if
they lack skills in the process of thinking about a decision and discussing it with others.
Those without these skills may benefit from guidance in a step-by-step process.
If you are unfamiliar with any of the terms used, please refer to the Glossary at theend of this survey.

12

Dimension 4 – Decision Guidance
Voting page
Please assess whether the subject of the item should be included as a ‘must have’ component of patientdecision support interventions (DESIs).
Score on the basis that if the patient decision support intervention did not meet the requirement of theitem, or it was of low quality, then there would be a risk to the user of harmful bias.
The rating scales range from 1-5.
A score of 1 would indicate that the item is not a ‘must have’ item. If the patient decision support
intervention did not meet the requirement of the item, or it was of low quality, there would be minimal orno impact on the risk of harmful bias to the user.
A score of 5 would indicate that the item is a ‘must have’ item. If the patient decision support
intervention did not meet the requirement of the item, or it was of low quality, there would be a definite
risk of harmful bias to the user.
NEW - 1st Round Voting Results
The voters' responses to each item are displayed in 2 ways:•

Percentages (%) are shown in red above the 1-5 rating scale. These are the percentage of the 101
Voters who assigned this score to the item in round 1

The mean (average) scores are shown in the yellow box to the left of the rating scale
The lowest mean item score in round 1 was 3.09 and the highest mean item score was 4.85

Mean
1.

The decision support intervention provides a
step-by-step way to make a decision2.

The decision support intervention includes tools
like worksheets or lists of questions to use when
discussing options with a practitioner

3.13
Mean

3.32

If the DESI did not meet therequirement of the item, or it wasof low quality, there would be…
(Circle one response)
12
17
30
30
12
1
2
3
4
5
Minimal impact onrisk of harmful bias

10
1

12
2

Minimal impact onrisk of harmful bias

Important &definitely essential

31
3

32
4

16
5

Important &definitely essential

13Dimension 5 – Development
Using a systematic development process

What is this dimension?
This dimension has 6 items. It assesses whether the patient decision support
intervention was designed using a systematic development process. This should
include:- Needs assessment with patients and health professionals;- Expert review by patients and health professionals not involved in developing theintervention;- Field-testing with patients and health professionals.

How might this affect the quality of the decision?
In theory, patient decision support interventions may lead to poor decisions if they aredeveloped by people who do not have the knowledge and skills to understand thedecision situation and to help patients make decisions. Even qualified people may notdesign a good decision support intervention, if they do not take time to develop it tomeet the needs of the patients who face the specific decision and the practitioners
who counsel them about the options. Outside experts may also help to identify things
that were missed during development.
If you are unfamiliar with any of the terms used, please refer to the Glossary at theend of this survey.

14

Dimension 5 - Development
Voting page
Please assess whether the subject of the item should be included as a ‘must have’ component of patientdecision support interventions (DESIs).
Score on the basis that if the patient decision support intervention did not meet the requirement of theitem, or it was of low quality, then there would be a risk to the user of harmful bias.
The rating scales range from 1-5.
A score of 1 would indicate that the item is not a ‘must have’ item. If the patient decision support
intervention did not meet the requirement of the item, or it was of low quality, there would be minimal orno impact on the risk of harmful bias to the user.
A score of 5 would indicate that the item is a ‘must have’ item. If the patient decision support
intervention did not meet the requirement of the item, or it was of low quality, there would be a definite
risk of harmful bias to the user.
NEW - 1st Round Voting Results
The voters' responses to each item are displayed in 2 ways:•

Percentages (%) are shown in red above the 1-5 rating scale. These are the percentage of the 101
Voters who assigned this score to the item in round 1

The mean (average) scores are shown in the yellow box to the left of the rating scale
The lowest mean item score in round 1 was 3.09 and the highest mean item score was 4.85

Mean
1.

The development process included finding out
what clients or patients need to prepare them todiscuss a specific decision2.

The development process included finding out
what health professionals need to prepare themto discuss a specific decision with patients3.

The development process included review byclients/patients not involved in producing thedecision support intervention

4.

The development process included review byprofessionals not involved in producing thedecision support intervention

4.31
Mean

3.96
Mean

3.70
Mean

3.63

If the DESI did not meet therequirement of the item, or it wasof low quality, there would be…
(Circle one response)
2
2
15
26
55
1
2
3
4
5
Minimal impact onrisk of harmful bias

4
1

6
2

Definite risk ofharmful bias

22
3

Minimal impact onrisk of harmful bias

4
1

15
2

13
2

Minimal impact onrisk of harmful bias

42
5

Definite risk ofharmful bias

19
3

Minimal impact onrisk of harmful bias

4
1

27
4

32
4

31
5

Definite risk ofharmful bias

24
3

35
4

25
5

Definite risk ofharmful bias

15

Mean
5.

The decision support intervention was field
tested with patients who are facing the decision

6.

The decision support intervention was field
tested with practitioners who counsel patients
who face the decision

4.35
Mean

3.83

1
1

4
2

8
3

Minimal impact onrisk of harmful bias

4
1

9
2

Minimal impact onrisk of harmful bias

34
4

54
5

Definite risk ofharmful bias

19
3

37
4

32
5

Definite risk ofharmful bias

16

Dimension 6 – Evidence
Using evidence

What is this dimension?
This dimension has 5 items. It assesses the quality of the scientific evidence used indeveloping the patient decision support intervention. The patient decision support
intervention is expected to provide citations to the studies selected. It should describe
how research evidence was synthesized and information about the quality of theresearch evidence used. A production or publication date should be displayed andinformation about the proposed update policy should be provided. The patient decision
support intervention should use up-to-date research from the best available scientificstudies.

How might this affect the quality of the decision?
In theory, decision support interventions may lead to poor decision if they contain
information from studies that are inaccurate, biased, incomplete, out-dated or are
based on patients who are different from those most likely to use the decision support
intervention.
If you are unfamiliar with any of the terms used, please refer to the Glossary at theend of this survey.

17

Dimension 6 - Evidence
Voting page
Please assess whether the subject of the item should be included as a ‘must have’ component of patientdecision support interventions (DESIs).
Score on the basis that if the patient decision support intervention did not meet the requirement of theitem, or it was of low quality, then there would be a risk to the user of harmful bias.
The rating scales range from 1-5.
A score of 1 would indicate that the item is not a ‘must have’ item. If the patient decision support
intervention did not meet the requirement of the item, or it was of low quality, there would be minimal orno impact on the risk of harmful bias to the user.
A score of 5 would indicate that the item is a ‘must have’ item. If the patient decision support
intervention did not meet the requirement of the item, or it was of low quality, there would be a definite
risk of harmful bias to the user.
NEW - 1st Round Voting Results
The voters' responses to each item are displayed in 2 ways:•

Percentages (%) are shown in red above the 1-5 rating scale. These are the percentage of the 101
Voters who assigned this score to the item in round 1

The mean (average) scores are shown in the yellow box to the left of the rating scale
The lowest mean item score in round 1 was 3.09 and the highest mean item score was 4.85

Mean
1.

The decision support intervention (or associateddocumentation) provides citations to the studies
selected

2.

The decision support intervention (or associateddocumentation) describes how research evidence
was selected or synthesised

3.

The decision support intervention (or associateddocumentation) provides a production or
publication date4.

The decision support intervention (or associateddocumentation) provides information about the
update policy

3.56
Mean

3.35
Mean

3.83
Mean

3.09

If the DESI did not meet therequirement of the item, or it wasof low quality, there would be…
(Circle one response)
10
13
20
26
32
1
2
3
4
5
Minimal impact onrisk of harmful bias

9
1

21
2

Definite risk ofharmful bias

22
3

Minimal impact onrisk of harmful bias

5
1

12
2

17
2

Minimal impact onrisk of harmful bias

25
5

Definite risk ofharmful bias

17
3

Minimal impact onrisk of harmful bias

16
1

24
4

28
4

39
5

Definite risk ofharmful bias

27
3

24
4

17
5

Definite risk ofharmful bias

18

Mean
5.

The decision support intervention (or associateddocumentation) describes the quality of theresearch evidence used

3.68

5
1

11
2

Minimal impact onrisk of harmful bias

22
3

36
4

27
5

Definite risk ofharmful bias

19

Dimension 7 – Disclosure
Disclosure

What is this dimension?
This dimension has 2 items. It assesses the patient decision support intervention’stransparency regarding the funding used for development, developers credentials or
qualifications, and affiliations of the developers

How might this affect the quality of the decision?
Often, decisions about medical tests or treatments have financial implications forpractitioners, their institutions, and for commercial companies that make an sellrelated products. In theory, these financial interests may influence the content of apatient decision support intervention, particularly when individuals or groups who
stand to gain (or lose) are involved with developing the patient decision aid. By clearly
identifying individuals or groups involved who have a potential financial interest in thecontent of an intervention, patients can decide for themselves whether or not thecontent is biased.
If you are unfamiliar with any of the terms used, please refer to the Glossary at theend of this survey.

20

Dimension 7 – Disclosure
Voting page
Please assess whether the subject of the item should be included as a ‘must have’ component of patientdecision support interventions (DESIs).
Score on the basis that if the patient decision support intervention did not meet the requirement of theitem, or it was of low quality, then there would be a risk to the user of harmful bias.
The rating scales range from 1-5.
A score of 1 would indicate that the item is not a ‘must have’ item. If the patient decision support
intervention did not meet the requirement of the item, or it was of low quality, there would be minimal orno impact on the risk of harmful bias to the user.
A score of 5 would indicate that the item is a ‘must have’ item. If the patient decision support
intervention did not meet the requirement of the item, or it was of low quality, there would be a definite
risk of harmful bias to the user.
NEW - 1st Round Voting Results
The voters' responses to each item are displayed in 2 ways:•

Percentages (%) are shown in red above the 1-5 rating scale. These are the percentage of the 101
Voters who assigned this score to the item in round 1

The mean (average) scores are shown in the yellow box to the left of the rating scale
The lowest mean item score in round 1 was 3.09 and the highest mean item score was 4.85

Mean
1.

The decision support intervention (or associateddocumentation) provides information about thefunding used for development

4.08
Mean

2.

The decision support intervention includes
author/developers credentials or qualifications

3.50

If the DESI did not meet therequirement of the item, or it wasof low quality, there would be…
(Circle one response)
6
8
10
25
52
1
2
3
4
5
Minimal impact onrisk of harmful bias

8
1

10
2

Minimal impact onrisk of harmful bias

Definite risk ofharmful bias

28
3

34
4

21
5

Definite risk ofharmful bias

21

Dimension 8 – Plain Language
Using plain language

What is this dimension?
This dimension has 1 item. It assesses the use of plain language in the patient decision
support intervention. The intervention should incorporate ways to share healthinformation that promote understanding and improve readability for all audiences,including those with limited health literacy. This includes the use of everyday language
and information that is structured, flows logically, and is focused. Readability is one
measure of the reading ease or difficulty of text, expressed as a 'grade level' score.
Some valid measures of readability include SMOG and FRY.

How might this affect the quality of the decision?
In theory, patient decision support interventions may lead to poor decisions if mostpatients cannot read, understand, and use the information. Patient decision support
interventions that use plain language make it easier for patients to learn about theiroptions.
If you are unfamiliar with any of the terms used, please refer to the Glossary at theend of this survey.

22

Dimension 8 – Plain Language
Voting page
Please assess whether the subject of the item should be included as a ‘must have’ component of patientdecision support interventions (DESIs).
Score on the basis that if the patient decision support intervention did not meet the requirement of theitem, or it was of low quality, then there would be a risk to the user of harmful bias.
The rating scales range from 1-5.
A score of 1 would indicate that the item is not a ‘must have’ item. If the patient decision support
intervention did not meet the requirement of the item, or it was of low quality, there would be minimal orno impact on the risk of harmful bias to the user.
A score of 5 would indicate that the item is a ‘must have’ item. If the patient decision support
intervention did not meet the requirement of the item, or it was of low quality, there would be a definite
risk of harmful bias to the user.
NEW - 1st Round Voting Results
The voters' responses to each item are displayed in 2 ways:•

Percentages (%) are shown in red above the 1-5 rating scale. These are the percentage of the 101
Voters who assigned this score to the item in round 1

The mean (average) scores are shown in the yellow box to the left of the rating scale
The lowest mean item score in round 1 was 3.09 and the highest mean item score was 4.85

Mean
1.

The decision support intervention (or associateddocumentation) reports readability levels (using
one or more of the available scales)

3.35

If the DESI did not meet therequirement of the item, or it wasof low quality, there would be…
(Circle one response)
15
12
23
25
26
1
2
3
4
5
Minimal impact onrisk of harmful bias

Definite risk ofharmful bias

23

Dimension 9 – Evaluation
Evaluating the decision support intervention

What is this dimension?
This dimension has 2 items. It assesses whether the patient decision support
intervention (after evaluation) helped patients/users make an informed choice:improved knowledge about the features of the options; improved match between
patient preferences and option chosen.

How might this affect the quality of the decision?
Most people agree that patient decision support interventions should help patients andtheir practitioners improve decision making. It is important to know what to look for inorder to conclude that an intervention does more good than harm.
If you are unfamiliar with any of the terms used, please refer to the Glossary at theend of this survey.

24

Dimension 9 – Evaluation
Voting page
Please assess whether the subject of the item should be included as a ‘must have’ component of patientdecision support interventions (DESIs).
Score on the basis that if the patient decision support intervention did not meet the requirement of theitem, or it was of low quality, then there would be a risk to the user of harmful bias.
The rating scales range from 1-5.
A score of 1 would indicate that the item is not a ‘must have’ item. If the patient decision support
intervention did not meet the requirement of the item, or it was of low quality, there would be minimal orno impact on the risk of harmful bias to the user.
A score of 5 would indicate that the item is a ‘must have’ item. If the patient decision support
intervention did not meet the requirement of the item, or it was of low quality, there would be a definite
risk of harmful bias to the user.
NEW - 1st Round Voting Results
The voters' responses to each item are displayed in 2 ways:•

Percentages (%) are shown in red above the 1-5 rating scale. These are the percentage of the 101
Voters who assigned this score to the item in round 1

The mean (average) scores are shown in the yellow box to the left of the rating scale
The lowest mean item score in round 1 was 3.09 and the highest mean item score was 4.85

1.

There is evidence that the decision support
intervention improves the match between thepreferences of the informed patient and theoption that is chosen

2.

There is evidence that the decision support
intervention helps patients improve theirknowledge about options’ features

Mean

3.37
Mean

3.63

If the DESI did not meet therequirement of the item, or it wasof low quality, there would be…
(Circle one response)
18
6
21
33
23
1
2
3
4
5
Minimal impact onrisk of harmful bias

9
1

8
2

Minimal impact onrisk of harmful bias

Definite risk ofharmful bias

23
3

32
4

29
5

Definite risk ofharmful bias

25

Dimension 10 – Test
For decision support interventions directed at investigations or screening tests

What is this dimension?
This dimension has 9 items and is only applicable to patient decision support
interventions that address screening or diagnostic tests. It assesses whether the
specific features of a screening or diagnostic test are described in sufficient details.
Accurate information about false positive/negative and true positive/negative test
results should be provided. This information should be framed in multiple ways tofacilitate understanding. The intervention is expected to include information about
the implications and consequences of accepting or declining the investigation or
screening procedure, including possible further tests or treatments. This also includes
information about 'over-detection'; this happens when a disease is found that would
not have been diagnosed or caused symptoms within a person's lifetime had the person
not been screened.

How might this affect the quality of the decision?
In theory, patient decision support interventions may lead to poor decisions if thefacts given are incomplete or superficial. This includes information related to the testor screening procedure. For ethical and legal reasons, patients have the right to getfull and accurate information before giving their consent to a medical test or
treatment. For patients actively involved in choosing options, more detailed
information about test outcomes may be needed.
If you are unfamiliar with any of the terms used, please refer to the Glossary at theend of this survey.

26

Dimension 10 – Test
Voting page
Please assess whether the subject of the item should be included as a ‘must have’ component of patientdecision support interventions (DESIs).
Score on the basis that if the patient decision support intervention did not meet the requirement of theitem, or it was of low quality, then there would be a risk to the user of harmful bias.
The rating scales range from 1-5.
A score of 1 would indicate that the item is not a ‘must have’ item. If the patient decision support
intervention did not meet the requirement of the item, or it was of low quality, there would be minimal orno impact on the risk of harmful bias to the user.
A score of 5 would indicate that the item is a ‘must have’ item. If the patient decision support
intervention did not meet the requirement of the item, or it was of low quality, there would be a definite
risk of harmful bias to the user.
NEW - 1st Round Voting Results
The voters' responses to each item are displayed in 2 ways:•

Percentages (%) are shown in red above the 1-5 rating scale. These are the percentage of the 101
Voters who assigned this score to the item in round 1

The mean (average) scores are shown in the yellow box to the left of the rating scale
The lowest mean item score in round 1 was 3.09 and the highest mean item score was 4.85

Mean
1.

The decision support intervention describes what
the test is designed to measure2.

The decision support intervention includes
information about the chances of having a true
positive test result

3.

The decision support intervention includes
information about the chances of having a true
negative test result

4.

The decision support intervention includes
information about the chances of having a false
positive test result

4.58
Mean

4.49
Mean

4.41
Mean

4.50

If the DESI did not meet therequirement of the item, or it wasof low quality, there would be…
(Circle one response)
0
4
4
22
70
1
2
3
4
5
Minimal impact onrisk of harmful bias

1
1

1
2

Definite risk ofharmful bias

6
3

Minimal impact onrisk of harmful bias

2
1

2
2

1
2

Minimal impact onrisk of harmful bias

59
5

Definite risk ofharmful bias

6
3

Minimal impact onrisk of harmful bias

1
1

33
4

34
4

56
5

Definite risk ofharmful bias

6
3

32
4

60
5

Definite risk ofharmful bias

27

Mean
5.

The decision support intervention includes
information about the chances of having a false
negative test result

6.

If the tests detects the condition or problem, thedecision support intervention describes the next
steps typically taken

7.

The decision support intervention describes thenext steps if the condition or problem is notdetected

8.

The decision support intervention describes the
chances the disease is detected with and without
the use of the test

9.

The decision support intervention has
information about the consequences of detecting
the condition or disease that would never havecaused problems if screening had not been done
(lead time bias)

4.50
Mean

4.38
Mean

4.04
Mean

4.15
Mean

4.34

1
1

1
2

8
3

Minimal impact onrisk of harmful bias

2
1

2
2

5
2

5
3

5
2

39
4

53
5

Definite risk ofharmful bias

16
3

Minimal impact onrisk of harmful bias

2
1

62
5

Definite risk ofharmful bias

Minimal impact onrisk of harmful bias

3
1

28
4

38
4

39
5

Definite risk ofharmful bias

13
3

Minimal impact onrisk of harmful bias

37
4

44
5

Definite risk ofharmful bias

0

4

10

35

52

1

2

3

4

5

Minimal impact onrisk of harmful bias

Definite risk ofharmful bias

28

Thank you for completing the survey
We are very grateful for your contribution to the consensus on core criteria for IPDASi
If you have any further questions about the consensus process, please do not hesitate
to contact the research team at ipdas@cardiff.ac.uk

29

Glossary
Authors. The people with primary responsibility for the patient decision support intervention’s contents arelisted, including their credentials and affiliations. For example: Natasha Fatale, M.D., Chief of Urology,
Pottsylvania Medical Center.
Balanced presentation of options and their features. The use of the same level of detail and degree ofprominence when displaying information in favour of and against the options/consequences (FDA Fair
Balance Prescription Drug Advertising Act of 2001). Whether the information is balanced depends on threefactors: format, sequencing, and framing.
Benefits. Intended positive features or consequences of an option. Benefits can be temporary or
permanent. Patients may also get a benefit from having no treatment. Benefits should be described interms of how strong the positive effect will be, how long it might last, and how often someone can expect
to enjoy the positive outcome.
Cochrane Review. Conducted as part of the Cochrane Collaboration, a Cochrane review is a systematic
review of research studies to learn about the effect of health interventions such as patient decision support
interventions. Systematic reviews are defined as a rigorous process, specified at the beginning of thestudy, to a) conduct a comprehensive search of the literature for all relevant studies on a specific topic; b)
appraise the quality of the identified studies; and c) synthesis the findings into a single report. The
Cochrane Collaboration is an international not-for-profit organisation that produces and circulates
systematic reviews of health care interventions and promotes the search for evidence to help support
health care decisions (www.cochrane.org).
Cochrane Review of Patient Decision Aids. A systematic review of randomized controlled trials
evaluating the effect of patient decision support interventions for patients facing actual screening or
treatment decisions.
Conditional probability. A measure of how likely something is to happen, based on existing information.
For example, a measure of how likely someone is to suffer from a heart attack, if that person is a smoker.
The likelihood would be different if the person were not a smoker.
Conflict of interest. “A set of conditions in which professional judgment concerning a primary interest(such as patient’’s welfare or the validity of research) tends to be unduly influenced by secondary interest(such as financial gain)”. (Thompson DF. Understanding financial conflicts of interest. New Engl J Med
1993; 329:573-576)
Decision support. Helping another person make a decision. It may be provided before a visit to apractitioner (in preparation for making a decision) or during a visit with a practitioner (while making thedecision).
Deliberation. The process of thoughtfully considering and discussing all sides of a decision that involves
choosing among alternative actions.
Denominator. A mathematical term that appears on the bottom of a fraction. For example, in the fraction
1/100, the denominator is 100. When probabilities are stated in the decision support intervention, theyshould use the same denominator in order to facilitate comparison across options. For example, imagine adecision support intervention that presented two different drug treatment options. If the intervention stated
10/100 people will experience nausea from treatment option A, they should presented the risk of nausea
for treatment option B using the same denominator e.g. 2/100 and not 1/50.
Decision support interventions (decision aids): Decision support interventions help people think about
choices they face; they describe where and why choice exists, in short, conditions of dual equipoise; they
provide information about options, including, where reasonable, the option of taking no action. These
interventions help people to deliberate, independently or in collaboration with others, about options by30

considering relevant attributes; they support people to forecast how they might feel about short,intermediate, and long-term outcomes that have relevant consequences, in ways that help the process ofconstructing preferences and eventual decision making appropriate to their individual situation
Development process. The steps taken by the authors/creators of the patient decision support
intervention to understand the needs of potential users (e.g. patients and health care practitioners), and to
consult experts and users to design, revise, and guide the development of patient decision support
interventions.
Dimension (IPDASi). This refers to the broad area within which quality is assessed. The broad
dimensions are made up of more specific items that are scored. The dimension score is the mean of theitem scores within the dimension (see Item for further detail).
Disclosure of conflicts of interest. Providing information on factors that have the potential to unduly
influence the content of patient decision support interventions, such as the source of funding (for theirdevelopment and production; people primarily responsible for the content) and affiliations of patientdecision support developers.
Event rate. The number of events (events might be benefits, harms, or side effects) that occur in a definedgroup of people (population) over a defined period of time. For example: if 1000 patients are treated ourbest guess is 2 patients may die. If 1000 patients are not treated our best guess is 4 patients may die.
(Users Guides to the Medical Literature 2002).
Evidence. Knowledge gained through scientific research.
False negative. This refers to a test result that tells you a disease or condition is not present, when in fact
it is present. A false negative test result is an error, which means the result is not giving you the correct
information. For example, if a person had a blood test designed to detect colon cancer, and the results
show that the person does not have colon cancer, when in fact they do have the disease, this is a false
negative.
False positive. This is the opposite of a false negative and refers to a test result that tells you a disease orcondition is present, when in fact it is not present. A false positive is an error, which means the result is not
giving you the correct information.
Field-testing. Formal testing in “real life” settings where the patient decision support intervention isintended to be used, with patients and practitioners who are actually involved in making a decision.
Results of field-testing are used to improve the decision support intervention.
Financial interests. The potential financial interests of any company related to the patient decision
support intervention's clinical content are clearly explained in plain language. For example: The NewName
Pharmaceutical Company, Inc., a for-profit company, makes and sells Borisol, one of the treatment
choices described in this program. The Richards Foundation is a charitable, not-for- profit groupcompletely funded by the NewName Pharmaceutical Company, Inc.
Framing. The way health statistics and information is presented with:- Similar use of positive and negative words to describe the intensity of the benefits or harms
without being too sensational or alarmist;- Similar display of numbers of the chances of experiencing benefits and harms;- Similar level of detail for both benefits and harms; and- Balanced examples of patients’ experiences so that equal space is given to different viewpoints
and choices.
Framing can make patients feel differently about the options. For example, it can make patients feeldifferently to be told the ‘‘death rate’’ of a procedure rather than the ‘‘survival rate’’. It is alarming to hearthat 20% of patients might die from a surgery but reassuring to remember that 80% survive. Hence,framing is the expression of the same information in different ways.

31Format. This refers to how the information about benefits and harms looks or sounds:••

For written resources, the size of the type, the spacing, and the page layout of information shouldbe the same throughout the patient decision support.
For audio or video resources, consistent volume and length of airtime is important for all options.

Funding sources. The agencies or organisations that provide the money to pay for the patient decision
support intervention. It must be is clearly and prominently acknowledged. For example: This program wasfunded by grants from the Government of Pottsylvania, the Richards Foundation, and the Badenov
Pharmaceutical Company, Inc. It is important that the information about sources of funding is presented ina way that ensures that it will be seen and understood by the user. Ideally, this information should appear
in clear large print at the beginning of the patient decision support.
Guidance in communication. To outline possible questions and concerns, discuss barriers tocommunicating with practitioners (such as shyness, defensiveness and confusion,) and then provide asystematic approach to helping overcome these barriers to make sure the patients’’ questions andconcerns are raised and addressed. Guidance can be provided in a variety of ways including by printed
materials, an internet site, audio or video tapes.
Guidance in deliberation. To outline possible actions, and then provide a systematic approach tocomparing and contrasting those actions. Guidance can be provided in a variety of ways including printed
materials, an internet site, audio or video tapes.
Guidance methods. Instruction or direction in the steps of decision making. Some examples may include
a list of the steps for making a high quality decision, filling in a worksheet that helps patients to clarify theirvalues about treatment options, or a list of questions to ask the doctor or decision coach.
Harmful bias. The aim of the consensus process is to identify those criteria, which if not present, or of low
quality, would lead to a risk of harmful bias and negatively affect patients’ decision-making. In this context,
we take harmful bias to mean the possibility that decision making that is influenced by factors or
representations that have not been addressed by: attention to the evidence base; a systematicdevelopment process; accurate information representations; decision guidance/support.
Harms and side effects. Unintended negative features or consequences of an option. These can be
temporary or permanent and major or minor. Descriptions of harms and side effects should include how
severe the negative effect will be, how long it might last, and how often someone can expect to experience
the negative effect. An example of harm is the development of breast cancer from taking estrogen andprogesterone. An example of a side effect is upset stomach from taking an antibiotic pill.
Health care consumer. A person who faces a health-related issue (for example, a decision to be tested
for a disease) or who has been diagnosed with a disease or condition.
Health literacy. An individual’s ability to perform basic reading, listening, computing and observing toobtain, understand, and use health information.
Index decision. This refers to the specific decision that the decision support intervention is designed toaddress; the decision that needs to be considered.
Informed consent. A conscious action taken by a person to give permission for something to occur which
involves their person or privacy. For example, giving permission for a doctor to perform a medical test or
for personal information to be published.
Item (IPDASi). This refers to the specific area within which quality is assessed. Each broad dimension ismade up of the specific items (see Dimension for further detail).
Lead time bias. The intention of screening is to diagnose a disease earlier than it would be without
screening. Without screening the disease may be discovered later, when symptoms appear. Even if inboth cases a person will die at the same time, because we diagnosed the disease earlier with screening
the survival time since diagnosis is longer with screening; but life span has not been prolonged, and there
will be added anxiety as the patient must live with knowledge of the disease for longer.

32

Lead time bias is the bias that occurs when screening diagnoses the disease earlier, but there is no effecton the outcome of the disease - it may appear that the test prolonged survival, when in fact it only resulted
in earlier diagnosis when compared to traditional methods.
Measurable improvements in the quality of decision-making. The most convincing evidence that apatient decision support intervention would really improve decision quality is likely to be generated bycontrolled study designs that compare two groups of patients who both face the decision: one group thatexperiences use of the patient decision support in practice and another group that experiences usual
practice (without a patient decision support). An improvement in the quality of decision making with thepatient decision support would be shown if the group who used the patient decision support in practice had
a greater match between their values individual values and the health care options they selected.
Medical tests (screening and diagnostic). Tests performed on healthy people to see if they have a
disease (diagnostic) or to see if they are likely to have a disease or condition (screening).
Medical treatments. Effective treatments that medical professionals judge to be ethical and which areavailable to patients in their community.
Natural frequency. This term means the same as “Event rate”. (Gigerenzer 2003).
Needs. Gaps between what is known and what is needed to make a decision.
Needs Assessment. There are two-types of needs assessment used in the development of patientdecision support interventions: patient and health professional. A needs assessment can tell us what
information patients need to prepare them to discuss a specific decision, and what health professionals
need to prepare them to discuss the decision with patients. It can help determine what information patientsneed to make decisions that match the personal importance they feel toward the positive and negative
features of their options. For example, before coming to a decision about treatment, patients first need toknow basic information about their condition. They also need to know about possible treatment options(including doing nothing) and the potential benefits and harms involved.
Over-detection (or over-diagnosis). Detection by screening tests of disease which, without screening,would not have been diagnosed or caused symptoms within the person’s lifetime. A common example isdetection of low grade prostate cancer by PSA testing which would not have harmed the person in their
lifetime. Also described as detection of pseudo-disease.
Patient. A health care consumer who faces a health-related issue (e.g. a screening decision) or who has
been diagnosed with a disease (e.g. diagnostic test or treatment decisions). Other names include public,
people, person, health consumer, or patient decision support users.
Patient Decision Support Intervention (DESI). Resources created to help patients make specific healthdecision when there is more than one option (including the status quo). They providing (at the minimum)
information on the options and the features of these options (benefits, harms, pros, cons, side effects,
inconveniences) (O’’Connor et al., BMJ, 1999).
Patient stories. Accounts from patients with a health condition describing their experiences with thecondition, options, decision-making, and consequences or outcomes. These may be collected as audio orvideo recorded interviews or as written stories. The characteristics of stories in patient decision support
usually vary depending on whether they involve a health issue faced by otherwise healthy people or atreatment decision for a life-threatening illness. Many examples of “others’ experiences” are a combinationof details of several real cases (O’’Connor, Drake et al, 1999; O’’Connor & Jacobsen, 2003).
Plain language. The use of ways to clearly communicate health information that creates optimalunderstanding.
Practitioner. A health care professional that provides direct care to patients or public. This includes
physicians, nurses, physiotherapists, pharmacists, and social workers.

33

Probability. The chance that something will occur. Probabilities are typically estimated by observing what
happens to large groups of patients. For example, watching 10,000 men aged 60 years for 10 years to see
how many die. The probability of dying in this group over 10 years might be 1,500 out of 10,000. If apatient is similar to the patients in the group, it would say the best guess at a patient’s chance of dying
over the next 10 years is 1,500 out of 10,000.
Quality of the Evidence. How accurate the descriptions and the estimated effects of different options are.
This evidence often comes from one or more studies or resources. The quality of evidence depends on
several factors, including the strength of the study design, the quality of the methods, whether there isagreement among studies of the same health option, and how directly the studies address the healthoption.
Readability. A measure of the reading ease or difficulty of printed materials. It is usually measured using
proven formulas that gauge the length of words in syllables and length of sentences (e.g. SMOG, Fry).
Reference class. This is the group of patients who might have a side effect or suffer from a disease in aset period of time. It is the denominator of an event rate, which is the group of patients who, over a
specified period of time, are at risk of an event (Gigerenzer 2003).
Relative risk reduction. Relative risk measures how much the likelihood of something happening is
reduced in a group of patients who take a treatment, compared to a group of patients who don’t. For
example, if 60% of a non-treated group of patients died and only 30% of the treated group died, thetreatment would have a relative risk reduction of 0.5 or 50%. Therefore, the death rate in the treated groupis half of that in the control group. The relative risk reduction is the ratio of the event rate in patients
exposed (intervention group) to the event rate in patients not exposed (control group) to a specified
intervention (Users Guides to the Medical Literature 2002).
Risk. In this context, risk can mean the same as probability – that is, the chance that something will occur.
For example, risk can be the chance that a positive or negative feature of an option will occur.
Sequence. The order in which the information on options and consequences (benefits and harms) is
provided. It should be the same for all the options.
True negative. The extent to which a test result is accurate and free from false positives. The fewer thenumber of false positives, the greater the specificity of the test. The proportion of individuals without adisease or condition that are correctly identified by a test.
True positive. The extent to which a test result is accurate and free from false negatives. The fewer thenumber of false negatives, the greater the specificity of the test. The proportion of individuals with adisease or condition that are correctly identified by a test.
Uncertainty. In the context of outcome measurements, uncertainty refers to the estimated amount orpercentage by which an observed or calculated value may differ from the true value. In decision support
interventions, this can be presented as confidence intervals, ranges, or by using phrases such as 'our best
estimate is'.
Users. Patients who face the decision and health practitioners who help patients learn about their options
to make the decision.
Values. How a person feels about or rates the importance of options and their positive and negative
features. These preferences are based on how their health might be affected by the decision, theirattitudes about the chances each option holds for bringing benefits or harms, their willingness to maketrade-offs over time, how they feel about certain medical procedures or anything else that might be usefulin making the decision.
Values clarification. Ways to help patients form and share how important various options and theirfeatures are. Examples include: a) describing features of options so patients can imagine and value what it
is like to undergo procedures and live with the consequences; b) providing examples of how other
patients’’ values led them to make different choices; c) bringing forth values by guiding patients to rate ortrade-off different features of options; and d) recording, guiding, or coaching patients to help them share
their values with others involved in the decision.

34