IPDAS Voting Document – 2nd round

Page 1 of 27

International Patient Decision Aid Standards(IPDAS) Collaboration

IPDAS Voting Document – 2nd Round
April 11, 2005
Voting Document Editors: A O’Connor, G Elwyn, D Stacey
Voting Document Contributors
Steering Committee: A O’Connor (CA) & Glyn Elwyn (UK) (co-leaders) with A Barratt (AU),M Barry (US), A Coulter (UK), M Holmes-Rovner (US), N Moumjid (FR), H LlewellynThomas (US), M O’Kane (US), R Thomson (UK), D Stacey (CA), T Whelan (CA)Leaders of Quality Criteria Panels. A Barratt (AU), M Barry (US), P Butow (AU), A Clarke(UK), A Edwards (UK), V Entwistle (UK), D Feldman-Stewart (CA), M Holmes-Rovner (US),
Hilary Llewellyn-Thomas (US), N Moumjid (FR), A Mulley (US), A O’Connor (CA), C Ruland(NO), K Sepucha (US), T Whelan (CA),Methods Group: G Elwyn (leader, UK) with S Bernstein (US), P Shekelle (US), R Thomson(UK), R Volk (US)Background Document Contributors:
Quality Criteria Panels: A O’Connor (CA) & Hilary Llewellyn-Thomas (US) (editors) with J
Austoker (UK), A Barratt (AU), M Barry (US), H Bekker (UK), J Belkora (US), C Braddock(US), P Butow (AU), E Chan (US), A Charvet (Switz), A Clarke (UK), J Davison (CA), J Dolan(US), A Edwards (UK), V Entwistle (UK), A Fagerlin (US), D Feldman-Stewart (CA), J Fowler(US), D Frosch (US), P Hewitson (UK), M Holmes-Rovner (US), T Hope (UK), MJ Jacobsen(CA), A Kennedy (Switz), S Knight (US), M Kupperman(US), B Ling (US), T Marteau (UK), K
McCaffery (AU), N Moumjid (FR), A Mulley (US), M O’Connor (US), E Ozanne (US), M
Pignone (US), A Raffle (UK), C Ruland (NO), L Schwartz (US), K Sepucha (US), S Sheridan(US), S Stableford (US), D Stacey (CA), D Stilwell (US), V Tait (CA), D Timmermans (NL), L
Trevena (AU), T Whelan (CA), C Wills (US), S Woloshin (US), S Ziebland (UK)IPDAS Voting Document – 2nd round

Page 2 of 27

Introduction
The International Patient Decision Aid Standards (IPDAS) Collaboration is made up of people fromaround the world who design, test, compile, provide, or use patient decision aids. Their common interest
is to reach agreement about how to judge the quality of a patient decision aid. The leaders of the IPDASCollaboration are Professor Annette O’Connor in Canada and Dr. Glyn Elwyn in the United Kingdom.
Why are standards needed?
The Cochrane Review Team compiled over 500 patient decision aids in use or being tested. However
without standards (or a set of criteria), it is difficult to know whether or not a particular patient decision
aid is a good one.
What are patient decision aids and why are they needed?
Patient decision aids are tools to help patients participate in their health decisions in ways that they
prefer. Patient decision aids are used when there is more than one medically reasonable option todiagnose or treat a health problem. Each of the options has good and bad features that patients value
differently. Even when two patients are in the same situation, what is important for one person may bedifferent for another person. Therefore, there is no clear choice that applies to everyone. The best choice
involves matching which features matter most to a person with the option that has these features. To
make a good decision, you need an expert on the facts (e.g. a health practitioner), an expert to whom thefeatures matter most (e.g. the patient), and a process that helps these two experts talk to each other.
Patient decision aids can help with that process.
Patient decision aids aim to do three things to prepare a person for decision making.
• They provide facts about a person’s condition, the options, and their features.
• They help patients to clarify their values (the features that matter most to them).
• They help patients to share their values with their health care practitioner and others, so a courseof action can be planned that matches their values.
Patient decision aids do not advise patients to choose one option over another. They do not replacecounseling from a health care practitioner. Instead, they prepare patients to discuss the options with theirhealth care practitioner.
An international group of researchers, known as the ‘Cochrane Review Team of Patient Decision Aids’
is compiling decision aids and summarizing the results of research trials. The latest review of 34 studies
shows that patients and practitioners who use patient decision aids make better decisions. Patients
participate more, know more, and have more realistic expectations of what might happen. They are more
likely to receive an option with features they most value (O’Connor et al., Cochrane Library, 2003).
What is the goal of the IPDAS Collaboration?
The goal is to reach agreement on a set of criteria that will help patients judge the quality of patientdecision aids. These criteria will be helpful to a wide variety of patients and organizations from aroundthe world that use or develop patient decision aids. For example, the set of criteria will help patients whoface the decision, practitioners who counsel them about the decision, patients who design and test
decision aids, and patients who deliver or make decisions about purchasing decision aids for the public.

IPDAS Voting Document – 2nd round

Page 3 of 27

Instructions for 2nd Round of Voting on Paper Version
Thank you for participating in the 1st round of voting! The purpose of the 2nd round of voting is to reachconsensus on the importance of these criteria when judging the quality of patient decision aids. It is
critical that voters from the 1st round of voting also participate in the 2nd round of voting to ensure thatthe results are valid.
We have summarized the 1st round results for you to review. Some items were changed or added basedon voter feedback. These changes are shown in red print and square brackets. All voters’ commentsfrom the 1st round have been saved for a background technical report.
Please plan to take about 20-40 minutes to complete the second round of voting.
1. In each section, please read the summary of ‘1st round voting results’.
2. For each item, consider the median (middle) score and then rate again how important eachcriterion is to you when judging the quality of a patient decision aid:
From 1 = not importantto

9 = very important

NOTE: please rate how ‘important’ the criterion is and not how ‘feasible’ it is to implement.
3. Then, continue with the next set of criteria.
4. When you have voted on all of the criteria, please fax the completed document to the IPDASCollaboration (613) 761-5402.
If you have any questions, please do not hesitate to contact the IPDAS Collaboration at IPDAS@ohri.ca
Privacy statement
The information collected during this voting process will be kept confidential. A code number will be
used to identify the information so your name will not appear on any publications. All relevant
information obtained while you are voting will only be made available to the research team and, if
requested, to the Research Ethics Committee at Baylor College of Medicine, Houston, Texas. If theresults of the study are published, your name will not be associated with the data.

IPDAS Voting Document – 2nd round

Page 4 of 27

I. Using a systematic development process
What is this criterion? The logical steps taken to build a patient decision aid. Steps may include:• To form groups to develop decision aids (decision experts, patient users, practitioner users);• To identify the needs of potential users;• To draft, review, field test, and revise the decision aid;
• To have the decision aid reviewed by outside experts who were not involved in its development and field testing.
How might this affect the quality of decision making? In theory, decision aids may lead to poor decisions if they are developed bypeople who do not have the knowledge and skills to understand the decision situation and to help patients make decisions. Even
qualified people may not design a good decision aid, if they do not take the time to develop it to meet the needs of the patients
who face the specific decision and the practitioners who counsel them about the options. Outside experts may also help to identify
things that were missed during development.
What is the evidence to support including or excluding this criterion? The Cochrane Collaboration review team examined the way
19 decision aids were developed. Of these, 17 reported the credentials of the developers (e.g. MD, RN, PhD), and 11 reported on
the steps taken to develop the decision aid. There were no studies comparing different ways of developing patient decision aids.

NEW: 1st Round Voting Results
The voters’ responses to each item are displayed in two ways.
••

Percentages (%) show in red above the numbers 1 to 9 are the proportion of 122 voters who assigned this score to theitem.
The median (middle) score shown in the [green] box was calculated to give an equal voice from all 4 groups of voters:a) patients/consumers; b) researchers/developers; c) practitioners; and d) policy makers/health plan executives. (If you
want to know the raw median, add the percentages from left to right until you reach 50% or more)

How consistent were voters in this section? All items range between ‘7 and 8’.
Did the 4 voter groups differ in scores? NO.
Were any voting items changed? YES, for item #8 voters requested clarification of ‘expert’. Some votersthought patients with experience needed to be included as an expert. Two items replace this item, one
focused on the expertise of health professionals and one focused on the expertise of patients, who had madethe decision.
How important is this criterion in judging the quality of a decision aid?
(circle your response)

1. The patient decision aid has information about thecredentials of the people who developed it.
2. Patients were asked what they need to prepare them todiscuss a specific decision.
3. Practitioners were asked what they need to discuss a
specific decision with patients.
4. Patients who were facing the decision field tested thedecision aid.
5. Practitioners who counsel patient on the options field
tested the decision aid.

1

2

6 2

9 12 18 12 39

1 2 3 4 5 6 7 8 9notimportant
2 0 1 05

6

very
important
11 16 60

1 2 3 4 5 6 7 8 9notimportant
1 0 1 238

very
important
21 20 45

1 2 3 4 5 6 7 8 9notimportant
1 0 0 03

3

very
important
9 21 64

1 2 3 4 5 6 7 8 9notimportant0 0 0 13

9very
important
19 26 41

1 2 3 4 5 6 7 8 9notimportant0 0 0 032

very
important
13 21 62

median7
median8
median7
median8
median7
median□Unable toevaluate□
Unable toevaluate□
Unable toevaluate□
Unable toevaluate□
Unable toevaluate

IPDAS Voting Document – 2nd round

6. Field testing showed that the decision aid was acceptableto patients.
7. Field testing showed that the decision aid was acceptableto practitioners.
8. [a] The decision aid was reviewed by outside experts[health professionals] who were not involved in its
development or field testing.
8. [b] The decision aid was reviewed by outside experts[patients who previously faced the decision] who were not
involved in its development or field testing.
Comments:

Page 5 of 27
0

0

0

0

3

2

13 21 62

1 2 3 4 5 6 7 8 9notimportant
0 0 1 33

very
important
12 21 30 31

1 2 3 4 5 6 7 8 9notimportant
0 0 3 57very
important
17 16 15 38

1 2 3 4 5 6 7 8 9notimportant
0 0 3 57very
important
17 16 15 38

1 2 3 4 5 6 7 8 9notimportantvery
important

NOTE: Criteria about the effects of decision aids on the quality of decision making are presented later8
median7
median7
median7□
Unable toevaluate□
Unable toevaluate□
Unable toevaluate□
Unable toevaluate

IPDAS Voting Document – 2nd round

Page 6 of 27

II. Providing information about options
What is this criterion? Giving the facts about the positive and negative features of the different choices to test or treat a healthproblem. This may include: what the procedure involves; possible benefits and advantages; and possible harms, side effects, ordisadvantages. It may include the benefits and harms of not being tested or treated. In the case of medical tests (to screen or
diagnose), facts may be given about what the positive or negative results might mean and what further tests or treatments might berequired. It may also include facts about ‘over-detection; this happens when a disease is found that would not have been diagnosed
or caused symptoms within a person’s lifetime had the person not been screened.
How might this affect the quality of the decision? In theory, decision aids may lead to poor decisions if the facts given are
incomplete or superficial. For ethical and legal reasons, patients have the right to get full and accurate information before giving
their consent to a medical test or treatment. For patients actively involved in choosing options, more detailed information about
options may be needed.
What is the evidence to support including or excluding this criterion? In the Cochrane Collaboration review, all 19 patient decision
aids provided information about the condition, the options, the benefits, and the harms. The amount of information given varied. In
studies that tested patients’ knowledge, patients who used decision aids had knowledge scores of 48 to 95% correct, compared toscores of 31 to 67% correct among patients who received standard care without decision aids. Patients who used patient decision
aids also felt more informed about their options.

NEW: 1st Round Voting Results
The voters’ responses to each item are displayed in two ways••

Percentages (%) show in red above the numbers 1 to 9 are the proportion of 122 voters who assigned this score to theitem.
The median (middle) score shown in the [green] box was calculated to give an equal voice from all 4 groups of voters:a) patients/consumers; b) researchers/developers; c) practitioners; and d) policy makers/health plan executives. (If you
want to know the raw median, add the percentages from left to right until you reach 50% or more)

How consistent were voters in this section? All items range between ‘7 and 8’.
Did the 4 voter groups differ in scores? NO
Were any voting items changed? YES, for items #3 and 8 voters requested clarification of terms. See
additions [in square brackets in red].
How important is this criterion in judging the quality of a decision aid?
(circle your response)

1. The patient decision aid describes the health condition
related to the decision.
2. The patient decision aid lists the health care options.
3. The option of choosing none of the health care options[e.g. doing nothing] is included
4. The patient decision aid describes what happens in thenatural course of a health condition if none, of the healthcare options, is chosen.
5. The patient decision aid has information about theprocedures involved (e.g. what is done before, during, andafter the health care option)

3

0

0

1

3

5 12 11 66

1 2 3 4 5 6 7 8 9notimportant
1 0 1 00

very
important
2 4 12 81

1 2 3 4 5 6 7 8 9notimportant
1 1 0 02 4

veryimportant
9 18 65

1 2 3 4 5 6 7 8 9notimportant
2 1 1 01

very
important
4 12 16 63

1 2 3 4 5 6 7 8 9notimportant
2 0 0 12

very
important
7 13 17 59

1 2 3 4 5 6 7 8 9notimportant
1 0 0 010very
important
4 17 77

median8
median8
median8
median8
median8
median□Unable toevaluate□
Unable toevaluate□
Unable toevaluate□
Unable toevaluate□
Unable toevaluate

IPDAS Voting Document – 2nd round

Page 7 of 27
1

0

0

0

1

0

4 17 77

6. The patient decision aid has information about the positive 1 2 3 4 5 6 7 8 9
notvery
features of the options (e.g. benefits, advantages)
7. The patient decision aid has information about negative
features of the options (e.g. harms, side effects,
disadvantages)

important
1 0 0 0

1 1

important
4 15 79

1 2 3 4 5 6 7 8 9notimportant
1 0 0 2

3

3

very
important
8 20 64

8. The information about [outcomes] of options (positive and 1 2 3 4 5 6 7 8 9
notvery
negative) includes the chances they [may] happen.
9. The patient decision aid has information about what thetest is supposed to measure.

important
1 0 0 04

important
4 14 23 54

1 2 3 4 5 6 7 8 9notimportant
1 0 0 2

very
important
8 15 25 47

3
10. The patient decision aid has information about the chances
12
3
4
5 6 7 8 9of receiving a true positive, true negative, false positive,notvery
and false negative test result.
importantimportant

11. The patient decision aid describes possible next steps
based on the test results.

1

0

0

0

2

2

9

25 60

1 2 3 4 5 6 7 8 9notimportant
1 0 1 3very
important
3 11 18 18 48

12. The patient decision aid has information about the chances 1 2 3 4 5 6 7 8 9notveryof disease being found with and without screening.
13. The patient decision aid has information about detection
and treatment of disease that would never have caused
problems if screening had not been done.
Comments:

important
3 0 1 1

important
5 8 16 17 50

1 2 3 4 5 6 7 8 9notimportantvery
important8
median8Median8
median7
median7
median8
median7
median8□
Unable toevaluate□
Unable toevaluate□
Unable toevaluate□
Unable toevaluate□
Unable toevaluate□
Unable toevaluate□
Unable toevaluate□
Unable toevaluate

IPDAS Voting Document – 2nd round

Page 8 of 27

III. Presenting probabilities
What is this criterion? A ‘probability’ is the chance or likelihood that something will happen. It could mean the chance of adisease, benefit, harm, or side effect. It is often estimated by seeing what happens to large groups of patients in the natural courseof the disease or after they have tests or treatments. The groups are usually defined by a disease or condition. Sometimes
probabilities are described according to factors that change one’s chances (e.g. age, sex, severity of disease, presence of other
health problems). Probabilities can be presented as words, numbers, or pictures. Some of the criteria are there to make it easier tocompare probabilities and some to make it easier to imagine the probability.
How might this affect the quality of the decision? In theory, decision aids may lead to poor decisions if the probabilities are
miscalculated or presented in ways that are misleading. Probabilities are usually based on research studies and calculated byexperts in statistics and modeling. Decision aids on the same topic sometimes use different numbers so it may be helpful toidentify the source of the probabilities and the steps used to calculate them. The different ways to present probabilities are
discussed below.
Describing probabilities: It is easier to understand the number of times something happens in a group (known as ‘event rates’)
than using other ways of presenting probabilities. To illustrate, identical information about probabilities is presented below using
different methods:•

event rates: “If 1000 patients are treated our best guess is 2 patients may die. If 1000 patients are not treated our best
guess is 4 patients may die”
• relative risk reduction “Our best guess is that treating patients cuts the chances of dying in half (by 50%) compared to notreatment”;• absolute risk reduction “Our best guess is that treating 1000 patients prevents 2 patients from dying compared to notreatment”• number need to treat: “Our best guess is that you have to treat 500 patients to prevent one death”.
Consistent denominators. It is easier to understand options when the same denominator (the bottom number in a fraction) is used.
Most patients understand that ‘4 out of 1000’ is a higher chance of something happening than ‘2 out of 1000’. Many patients do
not realize that ‘1 out of 250’ is a higher chance of something happening than ‘2 out of 1000’.
Same time frames. It is easier to compare options when probabilities are presented using the same time frames. For example, it isdifficult to decide about an option if you are told “If 1000 patients are treated, 2 patients may die within 10 years; if 1000 patientsare not treated 4 patients may die within 5 years”.
Balanced framing of probabilities. Patients feel differently about options when the probability of dying is presented (4 patientsout of 1000 may die) compared to the probability of living (996 patients out of 1000 may live). By using both frames (4 may die
and 996 may live), patients may be less likely to focus only on the positive or negative information.
Using visual aids. Some patients find it easier to understand probabilities when they are displayed using faces, stick figures, or bar
charts. Visual aids comparing options should also use consistent denominators (e.g. what happens to 1000 figures without
treatment compared to 1000 figures taking treatment). Similarly, two bar charts comparing options should use the same scale forcomparison, starting from zero and increasing by the same amount (e.g. both charts show increases by 50 patients up to 1000
patients).
Comparisons to the rates of other events. Very small chances of unfamiliar events (e.g. 1 in 10 million chance of dying from acomplication) are difficult to comprehend. It may help to compare the chances of a rare unfamiliar event to those of more familiar
situations (dying from crossing the street, being struck by lightening, childbirth, any cause).
What is the evidence to support including or excluding this criterion? In the Cochrane Collaboration review, 17of 19 patientdecision aids presented probabilities and 3 described the uncertainty associated with the evidence. The methods of displaying
probabilities varied. In all 3 studies comparing decision aids with and without probabilities, the number of patients out of 100
reporting probabilities that were realistic, was higher if they used a patient decision aid with probabilities (63-72% correct) than apatient decision aid without probabilities (43-46% correct).

IPDAS Voting Document – 2nd round

Page 9 of 27

NEW: 1st Round Voting Results
The voters’ responses to each item are displayed in two ways••

Percentages (%) show in red above the numbers 1 to 9 are the proportion of 122 voters who assigned this score to theitem.
The median (middle) score shown in the [green] box was calculated to give an equal voice from all 4 groups of voters:a) patients/consumers; b) researchers/developers; c) practitioners; and d) policy makers/health plan executives. (If you
want to know the raw median, add the percentages from left to right until you reach 50% or more)

How consistent were voters in this section? Items vary widely between ‘4 and 8’.
Did the 4 voter groups differ in scores? NO
Were any voting items changed? YES, for items #11 and 12 that explain how probabilities were calculated.
Some voters thought that the calculations or use of tools to estimate probabilities were too technical for apatient decision aid. Others suggested that this information could be placed in a reference section oraccessible technical document.
How important is this criterion in judging the quality of a decision aid?
(circle your response)

1. The patient decision aid presents probabilities using
event rates in a defined group of patients for a
specified time.
2. The patient decision aid compares probabilities of
options using the same denominator.
3. The patient decision aid compares probabilities of
options over the same period of time.
4. The patient decision aid describes the uncertainty
around the probabilities (e.g. by giving a range or byusing phrases such as ‘our best guess is’).
5. The patient decision aid uses visual diagrams to show
the probabilities (e.g. faces, stick figures, or bar
charts).
6. The patient decision aid uses the same scales in the
diagrams comparing options.

1

0

1

1

3

6

15 20 54

1 2 3 4 5 6 7 8 9notimportant
1 0 1 0

1 2

very
important
15 20 61

1 2 3 4 5 6 7 8 9notimportant
1 0 2 13

very
important
3 13 25 53

1 2 3 4 5 6 7 8 9notimportant
2 2 3 0very
important
2 7 24 23 39

1 2 3 4 5 6 7 8 9notimportant
1 2 2 4very
important
3 13 16 20 40

1 2 3 4 5 6 7 8 9notimportant
1 1 3 12

very
important
2 10 14 67

1 2 3 4 5 6 7 8 9notimportant
2 1 4 3very
important
9 18 23 37

3
7. The patient decision aid provides more than one way
1
2
3
4
5
6 7 8 9of explaining the probabilities (e.g. words, numbers,
notvery
diagrams).
importantimportant
8. The patient decision aid allows patients to select a way 4 6 9 3 8 16 18 12 24
1 2 3 4 5 6 7 8 9of viewing the probabilities (e.g. words, numbers,
notvery
diagrams).
importantimportant
1 3 3 1 6 9 22 22 33
9. The patient decision aid allows patients to see theprobabilities of what might happen based on their own 1 2 3 4 5 6 7 8 9
notvery
individual situation (e.g. specific to their age or severity ofimportantimportantmedian8
median8
median8
median7
median7
median8
median7
median6
median7

their disease).

3

5

6

8

10 10 28 8 22

median□Unable toevaluate□
Unable toevaluate□
Unable toevaluate□
Unable toevaluate□
Unable toevaluate□
Unable toevaluate□
Unable toevaluate□
Unable toevaluate□
Unable toevaluate

IPDAS Voting Document – 2nd round

10. The patient decision aid places the chances of whatmight happen in the context of other situations (e.g.
chances of developing other diseases, dying of other diseases, ordying from any cause).

11. The way the probabilities were calculated is described[in a reference section or accessible technicaldocument].
12. If the chance of disease is provided by sub-groups[e.g., younger, middle-age, or older people], the toolthat was used to estimate these risks is described [in areference section or accessible technical document].
13. The patient decision aid presents probabilities using
both positive and negative frames (e.g. showing bothsurvival rates and death rates).
Comments:

Page 10 of 27
3

5

6

8

10 10 28 8 22

1 2 3 4 5 6 7 8 9notimportant
9

6

very
important

7 10 13 20 8 20 68

1 2 3 4 5 6 7 8 9

median4

notvery
importantimportant
9 7 10 16 16 9 18 9 7

median1 2 3 4 5 6 7 8 95notimportant
2

0

1

very
important6

11 10 16 20 35

1 2 3 4 5 6 7 8 9notimportantvery
importantmedian7□
Unable toevaluate□
Unable toevaluate□
Unable toevaluate□
Unable toevaluate

IPDAS Voting Document – 2nd round

Page 11 of 27

IV. Clarifying and expressing values
What is this criterion?: A 2-step process of first thinking about and then communicating the personal importance of different
positive and negative features of options. Ways of helping patients to clarify and express their values may include:• describing features of options in ways that helps patients imagine what it is like to undergo procedures and to live with thephysical, emotional, and social consequences;• providing examples of how different values may lead to different choices;• helping patients rate or trade-off different features of the options;• recording and sharing values with others involved in the decision.
How might this affect the quality of the decision?: In theory, there may not be a good fit between which features matter most to thepatient and the option that is chosen if patient decision aids only help patients to consider the facts and not personal values.
What is the evidence to support including or excluding this criterion?: Of 19 patient decision aids included in the Cochrane
Review, all 19 described the features of the options but in varying levels of detail; 14 provided examples of different patients’values; 8 guided patients to rate or trade-off different features of options; and 9 suggested ways to share their values with others.
Of the 3 studies that measured the fit between a patient’s values and their choice, all 3 showed an improved fit after using apatient decision aid. Ten studies measured how clear patients felt about their personal values. Six of these studies showed thatdecision aids were better in helping patients feel clear about their values (2 to 17 points out of 100) and the other 4 studies foundno differences.
It is not clear how many ways of clarifying and expressing values should be in a decision aid or which way is best. When decision
aids describe the features of options only briefly, having more than one way helps. When decision aids describe features in detail(e.g. options’ effects on a person’s physical, emotional, social situation), more ways may not be more helpful.

NEW: 1st Round Voting Results
The voters’ responses to each item are displayed in two ways••

Percentages (%) show in red above the numbers 1 to 9 are the proportion of 122 voters who assigned this score to theitem.
The median (middle) score shown in the [green] box was calculated to give an equal voice from all 4 groups of voters:a) patients/consumers; b) researchers/developers; c) practitioners; and d) policy makers/health plan executives. (If you
want to know the raw median, add the percentages from left to right until you reach 50% or more)

How consistent were voters in this section? All items range between ‘6 and 8’.
Did the 4 voter groups differ in scores? NO
Were any voting items changed? YES, for item #3 voters requested clarification of the item. See revisions[in square brackets in red].
How important is this criterion in judging the quality of a decision aid?
(circle your response)

1. The patient decision aid describes the features of options
to help patients imagine what it is like to experience their
physical, emotional, and social effects.
2. The patient decision aid asks patients to think about which
positive and negative features of the options matter mostto them.
3. The patient decision aid suggests ways for patients toshare [what matters most to them when] others are
involved in the decision.
Comments:

1

2

0

1

3 12 17 25 39

1 2 3 4 5 6 7 8 9notimportant
0 3 2 06

very
important
8 12 20 50

1 2 3 4 5 6 7 8 9

median7
Medi8an

8

not
very
importantimportant
2 2 3 3 14 10 24 16 27

median1 2 3 4 5 6 7 8 96notimportantvery
important□
Unable toevaluate□
Unable toevaluate□
Unable toevaluate

IPDAS Voting Document – 2nd round

Page 12 of 27

V. Using patient stories
What is this criterion? Describes the experiences of other patients who have faced the decision. Stories may include living with thecondition, going through the steps of decision making, or living with the consequences of their choice.
How might this affect the quality of the decision? Some patients find stories more meaningful than factual information. However,
the way patient stories are selected may sway (or bias) a patient’s decision, divert patients from the facts, or not represent thedifferent views of patients.
What is the evidence to support including or excluding this criterion? Of the 19 patient decision aids in the Cochrane Review, 14
provided patient stories. No studies compared decision aids with patient stories to those without stories.

NEW: 1st Round Voting Results
The voters’ responses to each item are displayed in two ways••

Percentages (%) show in red above the numbers 1 to 9 are the proportion of 122 voters who assigned this score to theitem.
The median (middle) score shown in the [green] box was calculated to give an equal voice from all 4 groups of voters:a) patients/consumers; b) researchers/developers; c) practitioners; and d) policy makers/health plan executives. (If you
want to know the raw median, add the percentages from left to right until you reach 50% or more)

How consistent were voters in this section? Items vary widely between ‘4 and 7’.
Did the 4 voter groups differ in scores? NO
Were any voting items changed? YES, for items #2 to 5. For some voters, because they did not endorse thefirst item, they also scored the other items lower. For other voters, although they did not endorse the first
item, they said that if stories were included, they should meet some standards. Therefore, these items wererevised. As well, for items #3 to 5, some voters thought that there would be too much detail in the patientdecision aid and suggested that this information could be placed in a reference section or accessible technicaldocument.
How important is this criterion in judging the quality of a decision aid?
(circle your response)

1. The patient decision aid provides stories of other
patients’ experiences.
2. [If stories are used] in a patient decision aid, thestories represent a range of experiences (positive andnegative).
3. [If stories are used] in a patient decision aid, the steps
used to select these stories are described [in areference section or accessible technical document].
4. [If stories are used] in a patient decision aid, the steps
that experts used to review the information containedin these stories is included [in a reference section oraccessible technical document].
5. [If stories are used] in a patient decision aid, astatement that the patients gave informed consent toinclude their stories is included [in a reference sectionor accessible technical document].
Comments:
Note: balance in stories is discussed in a later section.

12 4

6

3

12 19 21 15 8

1 2 3 4 5 6 7 8 9notimportant
5 2 6 5very
important
6 15 9 23 29

1 2 3 4 5 6 7 8 9

median5
median7

notvery
importantimportant
8 10 11 11 18 7 11 9 14

median1 2 3 4 5 6 7 8 9

4

not
very
importantimportant
9 10 11 10 15 15 14 5 10

median1 2 3 4 5 6 7 8 95notimportant
12 12 8very
important
6 17 46 12 23

1 2 3 4 5 6 7 8 9notimportantvery
importantmedian4□
Unable toevaluate□
Unable toevaluate□
Unable toevaluate□
Unable toevaluate□
Unable toevaluate

IPDAS Voting Document – 2nd round

Page 13 of 27

VI. Guiding/Coaching in deliberation and communication
What is this criterion? Both guidance and coaching provide a step-by-step way of thinking about the options and discussing themwith their practitioner. Guidance is provided within a patient decision aid. It may be implicit in the way the patient decision aid isorganized or explicit by providing a list or worksheet outlining the steps. ‘Coaching’ is provided in-person, one-on-one, by atrained person who is supportive but neutral in the decision. It may be given before or after using a patient decision aid, or as partof the delivery of one.
How might this affect the quality of the decision? Patients are not able to participate in decision making about health care options
if they lack skills in the process of thinking about a decision and discussing it with others. Those without these skills may benefit
from guidance or coaching in a step by step process.
What is the evidence to support including or excluding this criterion? Of 19 patient decision aids in the Cochrane Review, 17
studies provide guidance and/or coaching. One large study found that coaching along with a patient decision aid was more costeffective and improved patient satisfaction compared to using the decision aid alone or using neither coaching or the decision aid.
In other studies, it was not possible to identify the effects of guidance and coaching separately from other parts of a decision aid.

NEW: 1st Round Voting Results
The voters’ responses to each item are displayed in two ways••

Percentages (%) show in red above the numbers 1 to 9 are the proportion of 122 voters who assigned this score to theitem.
The median (middle) score shown in the [green] box was calculated to give an equal voice from all 4 groups of voters:a) patients/consumers; b) researchers/developers; c) practitioners; and d) policy makers/health plan executives. (If you
want to know the raw median, add the percentages from left to right until you reach 50% or more)

How consistent were voters in this section? All items range between ‘5 and 7’.
Did the 4 voter groups differ in scores? YES, for items #4 and 5. Practitioners and researchers rated these
items lower than patients and policy makers. Some voters were concerned that the coach role was not
feasible or too costly. Other voters thought coaching would be helpful.
Were any voting items changed? NO
How important is this criterion in judging the quality of a decision aid?
(circle your response)

1. The patient decision aid provides a step-by-step way
to make a decision.
2. The patient decision aid suggests ways to talk about
the decision with a health practitioner.
3. The patient decision aid includes tools like worksheetsor lists of questions to use when discussing options
with a practitioner.
4. The patient decision aid offers the option of working
with a trained ‘coach’ to help patients consider the
options.
5. The patient decision aid offers the option of working
with a trained ‘coach’ to help patients prepare to talkabout the decision with a practitioner.
Comments:

3

3

3

5 10 11 15 18 33

1 2 3 4 5 6 7 8 9notimportant
0 1 2 5very
important
7 10 15 27 33

1 2 3 4 5 6 7 8 9notimportant
1 1 1 6

very
important
7 8 27 19 31

1 2 3 4 5 6 7 8 9

median7
median7
median7

notvery
importantimportant
13 5 7 8 16 15 13 12 11

median1 2 3 4 5 6 7 8 9

5

not
very
importantimportant
14 5 8 9 15 14 14 14 9

median1 2 3 4 5 6 7 8 95notimportantvery
important□
Unable toevaluate□
Unable toevaluate□
Unable toevaluate□
Unable toevaluate□
Unable toevaluate

IPDAS Voting Document – 2nd round

Page 14 of 27

VII. Disclosing conflicts of interest
What is this criterion: To be open and honest in stating: the funding source for creating and producing patient decision aids,
 the financial support for practitioners who are responsible for creating the patient decision aid, and the affiliations of patient decision aid developers that might influence the content of patient decision aids.
How might this affect the quality of the decision? Often, decisions about medical tests or treatments have financial implications
for practitioners, their institutions, and for commercial companies that make and sell related products. In theory, these financial
interests may influence the content of a patient decision aid, particularly when individuals or groups who stand to gain (or lose),are involved with developing the patient decision aid. By clearly identifying individuals or groups involved who have a potential
financial interest in the content of the patient decision aid, patients can decide for themselves whether or not the content is biased.
What is the evidence to support including or excluding this criterion? Of the 131 patient decision aids in the Cochrane review, 126
provided information on the funding sources. There have been no studies showing whether funding sources or affiliations arelikely to influence the content of patient decision aids. However, in medical publishing and professional organisations there isstrong evidence to suggest that financial interests and specialty advice can bias the information provided.

NEW: 1st Round Voting Results
The voters’ responses to each item are displayed in two ways••

Percentages (%) show in red above the numbers 1 to 9 are the proportion of 122 voters who assigned this score to theitem.
The median (middle) score shown in the [green] box was calculated to give an equal voice from all 4 groups of voters:a) patients/consumers; b) researchers/developers; c) practitioners; and d) policy makers/health plan executives. (If you
want to know the raw median, add the percentages from left to right until you reach 50% or more)

How consistent were voters in this section? All items range between ‘7 and 8’.
Did the 4 voter groups differ in scores? NO
Were any voting items changed? NO
How important is this criterion in judging the quality of a decision aid?
(circle your response)

1. The patient decision aid reports where the money
came from to develop the decision aid
2. The patient decision aid reports where the money
came from to copy and distribute the decision aid.
3. The patient decision aid reports whether the authors ofthe decision aid stand to gain or lose by the choices
patients make after using a decision aid.
4. The patient decision aid reports whether theaffiliations of the authors stand to gain or lose by the
choices patients make after using a decision aid.
5. If the patient decision aid includes stories of other
patients’ experiences, it reports if there was somefinancial or other reason why patients decided to sharethem.
Comments:

2

0

1

2

4

7 12 12 61

1 2 3 4 5 6 7 8 9notimportant
3 2 5 44

very
important
8 19 10 44

1 2 3 4 5 6 7 8 9notimportant
1 1 1 28

very
important
4 11 15 57

1 2 3 4 5 6 7 8 9notimportant
1 2 3 38

very
important
8 18 13 47

1 2 3 4 5 6 7 8 9notimportant
9 5 7 3very
important
4 11 12 9 40

1 2 3 4 5 6 7 8 9notimportantvery
importantmedian8
median7
median8
median8
median7□
Unable toevaluate□
Unable toevaluate□
Unable toevaluate□
Unable toevaluate□
Unable toevaluate

IPDAS Voting Document – 2nd round

Page 15 of 27

VIII. Delivering patient decision aids on the Internet
What is this criterion? One of several formats that can be used for a patient decision aid. The Internet is the use of a global
network of millions of computers. Putting patient decision aids on the Internet makes them available to patients around the world.
How might this affect the quality of the decision? Increasingly, patients use the Internet to find health information for making
decisions. Most decision aid developers are making aids for the Internet. The advantages are: widespread access by practitioners
and patients; ability to tailor to each patient; cost savings in distribution; ease in updating; and stopping the circulation of out ofdate material. However, not all patients have access to or use the Internet. Internet-based patient decision aids may be more usable
if they are simple to use, organised in a logical way, easy to search, interactive, and offer links to other helpful information.
What is the evidence to support including or excluding this criterion? Over the last 5 years, there has been a steady increase in theuse of the Internet as the medium of delivery for patient decision aids. Currently over 98 of the 131 available patient decision aidsin the Cochrane Review Inventory are on the Internet. Only one study compared the use of the Internet to another format (videocassette). This study found that both formats were acceptable, and that patients who reviewed either decision aid in their entirety,
improved their knowledge. However, the study also found that patients were more likely to view the patient decision aid whenassigned to watch the video at the doctor’s office just before their appointment, compared to using the Internet patient decision aid
at home.

NEW: 1st Round Voting Results
The voters’ responses to each item are displayed in two ways••

Percentages (%) show in red above the numbers 1 to 9 are the proportion of 122 voters who assigned this score to theitem.
The median (middle) score shown in the [green] box was calculated to give an equal voice from all 4 groups of voters:a) patients/consumers; b) researchers/developers; c) practitioners; and d) policy makers/health plan executives. (If you
want to know the raw median, add the percentages from left to right until you reach 50% or more)

How consistent were voters in this section? All items range between ‘6 and 8’.
Did the 4 voter groups differ in scores? YES, for item #2. Researchers rated this item the lowest (least
important) compared to the other groups; voters did not elaborate on reasons behind the rating for this item.
Were any voting items changed? YES, all items. Voters were concerned that these criteria required that all
patient decision aids be avialble on the internet. This was not the initial intent of these items. Therefore, alead statement was added to all items. For item #3, voters requested clarification of the term ‘feedback’; see
addition [in square brackets in red]. Item #6 was suggested by some voters.
How important is this criterion in judging the quality of a decision aid?
(circle your response)

1. [If the patient decision aid is used on the internet], it
provides a step-by-step way to move through the web
pages (screens) on the Internet.
2. [If the patient decision aid is used on the internet], it
allows patients to search for key words in the decisionaid.
3. [If the patient decision aid is used on the internet], it
provides feedback on personal health information thatis entered into the decision aid [e.g. the chances you
may get a complication ]4. [If the patient decision aid is used on the internet], thewebsite provides security for personal healthinformation entered into the decision aid.

3

1

3

3

8

3 16 21 44

1 2 3 4 5 6 7 8 9

median8

notvery
importantimportant
4 7 5 7 10 10 24 17 18

median1 2 3 4 5 6 7 8 9

6

not
very
importantimportant
4 2 3 11 15 12 14 16 23

median1 2 3 4 5 6 7 8 96notimportant
2

0

1

very
important
0

2

2 2

13 79

1 2 3 4 5 6 7 8 9notimportantvery
importantmedian8□
Unable toevaluate□
Unable toevaluate□
Unable toevaluate□
Unable toevaluate

IPDAS Voting Document – 2nd round

5. [If the patient decision aid is used on the internet], it iseasy for patients to find their way back to the point
they were at in the decision aid when they clicked on
links to other web pages.

Page 16 of 27
3

0

1

1

3 5

14 22 52

1 2 3 4 5 6 7 8 9notimportantvery
important

6. [If the patient decision aid is on the internet, it can also 1 2 3 4 5 6 7 8 9very
be printed as a single document (e.g., pdf document)]. notimportant

Comments:

importantmedian

8

NEW
ITEM

□
Unable toevaluate□
Unable toevaluate

IPDAS Voting Document – 2nd round

Page 17 of 27

IX. Balancing the presentation of options
What is this criterion? Using equal importance and level of detail to display information about the positive and negative features of
options. Some ways to look at the balance of positive compared to negative features is to look at:
 The format for options: the font, spacing, layout, volume of sound, and amount of airtime; The order each is discussed; and The framing of the options in terms of adjectives used and the display of numbers and statistics
How might this affect the quality of the decision? The way information is presented affects how patients see their health and how
they make decisions. In theory, unbalanced information may influence patients’ understanding, expectations, and value judgments
about the options. As a result, patients may select options that would not be acceptable if they had been presented in a balanced
way. It is dishonest and unethical to create false expectations or make it difficult for patients to understand the probabilities (orchances) of positive or negative features of options.
What is the evidence to support including or excluding this criterion? Of 19 patient decision aids in the Cochrane Review, all ofthem present both the harms as well as the benefits. However, only 4 of 19 patient decision aids were evaluated to find out if
patients thought they provided balanced information. In these 4 studies, 60 to 79% of patients rated the patient decision aids ascompletely balanced, while the others rated the aids as slanted in favour of one option. However, patients who found the patientdecision aids slanted were more likely to be leaning toward a particular option before using the decision aid. The patient decision
aids have not been evaluated for balance in format, order, or framing of information.

NEW: 1st Round Voting Results
The voters’ responses to each item are displayed in two ways••

Percentages (%) show in red above the numbers 1 to 9 are the proportion of 122 voters who assigned this score to theitem.
The median (middle) score shown in the [green] box was calculated to give an equal voice from all 4 groups of voters:a) patients/consumers; b) researchers/developers; c) practitioners; and d) policy makers/health plan executives. (If you
want to know the raw median, add the percentages from left to right until you reach 50% or more)

How consistent were voters in this section? All items were ‘8’.
Did the 4 voter groups differ in scores? NO
Were any voting items changed? NO
How important is this criterion in judging the quality of a decision aid?
(circle your response)

1. The patient decision aid makes it possible to compare
the positive and negative features of the available
options.
2. The patient decision aid shows the negative andpositive features of options with equal detail (forexample using similar fonts, order, display ofstatistical information).
3. Field testing showed that undecided patients felt the
information was presented in a balanced way.
Comments:

1

0

0

1

1

1 10 12 74

1 2 3 4 5 6 7 8 9notimportant
1 1 1 23

2

very
important
8 13 69

1 2 3 4 5 6 7 8 9notimportant
1

1

3

very
important
27

6 17 15 50

1 2 3 4 5 6 7 8 9notimportantvery
importantmedian8
median8
median8□
Unable toevaluate□
Unable toevaluate□
Unable toevaluate

IPDAS Voting Document – 2nd round

Page 18 of 27

X. Using plain language
What is this criterion? Ways to share health information that promote understanding and improve readability for all audiences,including those with limited health literacy. This includes the use of everyday language and information that is structured, flows
logically, and is focused; as well as, follows guidelines for document design. Readability is one measure of the reading ease ordifficulty of text, expressed as a ‘grade level’ score. Some valid measures of readability include SMOG and FRY.
How might this affect the quality of the decision? In theory, decision aids may lead to poor decisions if most patients cannot readunderstand and use the information. Patient decision aids that use plain language make it easier for patients to learn about theiroptions.
Common factors that limit understanding include: being unfamiliar with health terms and language, having limited reading skills,
and aging. For example, the average US adult reads at the 7th or 8th grade level. Even patients with above average reading skills
may be limited by emotional distress or medications that affect thinking abilities.
In general, people are able to understand spoken language at about two grade levels higher than written language. Patient decision
aids with spoken words (audio, video, in person discussion) may be easier to understand.
What is the evidence to support including or excluding this criterion? The Cochrane review of patient decision aids found nostudies that tested the effect of readability of the text on the quality of the decisions made. Health literacy studies found that
improved "reading ease" of written materials increases understanding.

NEW: 1st Round Voting Results
The voters’ responses to each item are displayed in two ways••

Percentages (%) show in red above the numbers 1 to 9 are the proportion of 122 voters who assigned this score to theitem.
The median (middle) score shown in the [green] box was calculated to give an equal voice from all 4 groups of voters:a) patients/consumers; b) researchers/developers; c) practitioners; and d) policy makers/health plan executives. (If you
want to know the raw median, add the percentages from left to right until you reach 50% or more)

How consistent were voters in this section? All items vary widely between ‘5 and 8’.
Did the 4 voter groups differ in scores? YES, for item #1. Practitioners and researchers rated this item lower(less important) than patients and policy makers; voters did not elaborate on reasons behind the rating forthis item.
Were any voting items changed? YES. For item #2 voters requested clarification of a term and for item #4
voters requested broader application to other education systems. See revisions [in square brackets in red].
How important is this criterion in judging the quality of a decision aid?
(circle your response)

1. The patient decision aid describes the ‘professional
standards for plain language materials’ that guided its
development (e.g. Plain Language Association International)
2. The patient decision aid identifies the reading level atwhich it is written and the formula [method] used todetermine the level.
3. The patient decision aid is written at a level that can beunderstood by at least half of the patients for whom it
is intended.
4. The patient decision aid is written at a level no higher
than grade 8 [or equivalent] according to a readability
formula (e.g., SMOG, FRY).

10 11 10 9 12 13 11 7 16

median1 2 3 4 5 6 7 8 9

5

not
very
importantimportant
14 8 9 11 10 13 14 9 11

median1 2 3 4 5 6 7 8 95notimportant
3 1 2 12

4

very
important
9 16 62

1 2 3 4 5 6 7 8 9notimportant
9 1 3 4very
important
8 12 17 14 34

1 2 3 4 5 6 7 8 9notimportant
3 2 2 28

very
important
12 12 20 40

median8
median7
median□Unable toevaluate□
Unable toevaluate□
Unable toevaluate□
Unable toevaluate

IPDAS Voting Document – 2nd round

Page 19 of 27

5. The patient decision aid provides ways to help patients 3 2 2 2 8 12 12 20 40
understand information other than reading (e.g. audio, 1 2 3 4 5 6 7 8 9
notvery
video, or in-person discussion).
importantimportant
6. Field testing showed that the patient decision aid wasunderstood by patients with limited reading skills.
Comments:

3

2

3

2

3 10 15 25 39

1 2 3 4 5 6 7 8 9notimportantvery
important7
median7□
Unable toevaluate□
Unable toevaluate

IPDAS Voting Document – 2nd round

Page 20 of 27

XI. Basing information on up-to-date scientific evidence
What is this criterion? The use of up-to-date research from the best available scientific studies. This may include:• using standard systematic steps in selecting the evidence• judging the quality of the studies• describing how consistent the results are among studies• reporting how similar the patients in the studies were to those who would use the patient decision aid.
• recording the sources of evidence• updating the evidence regularly
How might this affect the quality of the decision? In theory, decision aids may lead to poor decisions if they contain informationfrom studies that are inaccurate, biased, incomplete, out-dated or are based on patients who are different from those most likely to
use the decision aid.
What is the evidence to support including or excluding this criterion? Of 19 patient decision aids in the Cochrane Review, 13
studies provided references to the scientific evidence and 5 studies described the quality of the evidence.

NEW: 1st Round Voting Results
The voters’ responses to each item are displayed in two ways••

Percentages (%) show in red above the numbers 1 to 9 are the proportion of 122 voters who assigned this score to theitem.
The median (middle) score shown in the [green] box was calculated to give an equal voice from all 4 groups of voters:a) patients/consumers; b) researchers/developers; c) practitioners; and d) policy makers/health plan executives. (If you
want to know the raw median, add the percentages from left to right until you reach 50% or more)

How consistent were voters in this section? All items range between ‘6 and 8’.
Did the 4 voter groups differ in scores? NO
Were any voting items changed? YES. For item #2, some voters thought that describing the steps forselecting scientific evidence was too technical for a patient decision aid and others suggested that thisinformation could be placed in a reference section or accessible technical document. For item #5, somevoters suggested that the patient decision aid should also clearly state when no evidence is available; hence,an extra item was added to ‘include lack of evidence’.
How important is this criterion in judging the quality of a decision aid?
(circle your response)

1. The patient decision aid provides references toscientific evidence used.
2. The steps used to select the scientific evidence (e.g.
finding, appraising, summarizing) is included [in areference section or accessible technical document].
3. The patient decision aid reports the date when it waslast updated.
4. The patient decision aid reports how often theinformation in the decision aid is updated.

0

1

3

1

5

3 17 18 54

1 2 3 4 5 6 7 8 9

median8

notvery
importantimportant
4 7 7 8 17 12 12 10 23

median1 2 3 4 5 6 7 8 96notimportant
0 0 1 02

very
important
2 4 23 69

1 2 3 4 5 6 7 8 9notimportant
3 3 8 29very
important
8 22 11 35

1 2 3 4 5 6 7 8 9notvery
importantimportant
0 2 3 3 10 9 15 19 39

5. [a] The patient decision aid describes the quality of the 1 2 3 4 5 6 7 8 9
notvery
scientific evidence (e.g. quality of research studies).
importantimportantmedian8
median7
median7□
Unable toevaluate□
Unable toevaluate□
Unable toevaluate□
Unable toevaluate□
Unable toevaluate

IPDAS Voting Document – 2nd round

5. [b] The patient decision aid describes the quality ofthe scientific evidence (e.g. quality of research studies, 1 2 3 4 5 6 7 8 9
notvery[including lack of evidence]).
importantimportant
5. The patient decision aid uses evidence taken from studies 2 0 2 3 3 9 22 17 43
1 2 3 4 5 6 7 8 9
on patients that are similar to the patients who would use
notvery
the decision aid (e.g. age, gender).
importantimportant
Comments:
Note: the level of uncertainty around the evidence is described in the probabilities section

Page 21 of 27

New
Item
median8□
Unable toevaluate□
Unable toevaluate

IPDAS Voting Document – 2nd round

Page 22 of 27

XII. Establishing the effectiveness
What is this criterion? To identify the things that you would need to observe in order to say that after using a patient decision aid,
the way the decision was made was good, and that the choice that was made was good. The health care decision you need to keep
in mind is one which has no clear answers for everyone. There is more than one medically reasonable option. Each option has
different features (procedures, benefits, harms, side effects). Not everyone agrees on the features that matter most to them.
Why is this criterion important? Most people agree that decision aids should help patients and their practitioners improve decision
making. It is important to know what to look for in order to conclude that a decision aid does more good than harm.
What is the evidence on definitions of good decision making and decisions? People are still discussing a definition of a ‘good’decision. The most common agreement between researchers is that decisions should be based on adequate information andpersonal values. A national survey found that the public thought it was important to be informed, have clear values, make choicesthat matched with their values, and be satisfied with the decision. A survey of cancer doctors stressed the importance of patients
being clear about values and being informed. In the Cochrane review of 34 studies, 18 measured knowledge, 10 measured ‘feelingclear about values’, and 3 measured ‘agreement between values and choices’.

NEW: 1st Round Voting Results
The voters’ responses to each item are displayed in two ways••

Percentages (%) show in red above the numbers 1 to 9 are the proportion of 122 voters who assigned this score to theitem.
The median (middle) score shown in the [green] box was calculated to give an equal voice from all 4 groups of voters:a) patients/consumers; b) researchers/developers; c) practitioners; and d) policy makers/health plan executives. (If you
want to know the raw median, add the percentages from left to right until you reach 50% or more)

How consistent were voters in this section? All items range between ‘7 and 8’.
Did the 4 voter groups differ in scores? NO
Were any voting items changed? NO
How important is this criterion in judging the quality of a decision aid?
(circle your response)

1. There is evidence that the patient decision aid helps
patients recognize that a decision needs to be made.
2. There is evidence that the patient decision aid helps
patients know about the available options.
3. There is evidence that the patient decision aid helps
patients know about different features of the options.
4. There is evidence that the patient decision aid helps
patients understand that values affect the decision.
5. There is evidence that the patient decision aid helps
patients be clear about which features of options
matter most to them.
6. There is evidence that the patient decision aid helps
patients discuss values with their health practitioners.

2

0

3

3

8

8 14 20 42

1 2 3 4 5 6 7 8 9notimportant
1 0 0 13 3

very
important
8 22 63

1 2 3 4 5 6 7 8 9notimportant
1 0 0 2

4 3

very
important
12 25 53

1 2 3 4 5 6 7 8 9notimportant
2 1 1 3very
important
3 3 14 22 52

1 2 3 4 5 6 7 8 9notimportant
1 0 2 12

very
important
9 15 22 50

1 2 3 4 5 6 7 8 9notimportant
3 0 1 4very
important
10 10 23 20 29

1 2 3 4 5 6 7 8 9notimportant
2 0 0 33

very
important
3 12 23 54

median7
median8
median7
median8
median8
median7
median□Unable toevaluate□
Unable toevaluate□
Unable toevaluate□
Unable toevaluate□
Unable toevaluate□
Unable toevaluate

IPDAS Voting Document – 2nd round

7. There is evidence that the patient decision aid helps
patients become involved in decision making in ways
they prefer.
8. There is evidence that the patient decision aid
improves the match between the features that mattermost to the informed patient and the option that ischosen.
Comments:

Page 23 of 27
2

0

0

3

3

3 12 23 54

1 2 3 4 5 6 7 8 9notimportant
1 0 0 3

2 5

very
important
12 25 52

1 2 3 4 5 6 7 8 9notimportantvery
important8
median8□
Unable toevaluate□
Unable toevaluate

IPDAS Voting Document – 2nd round

Page 24 of 27
Glossary

Absolute risk reduction. This type of measurement tells us if a treatment is likely to help patients who take it, compared to patients who don’t take it. In
other words, it is the difference in event rates between patients exposed (intervention group) and not exposed (control group) to a specified
intervention. For example, improvement of 1% from 4% to 3%. (Users Guides to the Medical Literature 2002).
Applicability of evidence. Whether the patients and practitioners involved in the study, the screening or treatment that was tested; and the outcomesmeasured, are similar to the patients who would be using the patient decision aid and their situation.
Authors. The people with primary responsibility for the patient decision aid’s contents are listed along with their specialty and their place of work.. For
example: Clinical Advisor: Natasha Fatale, M.D., Chief of Urology, Pottsylvania Medical Center
Balanced presentation of options and their features. The use of the same level of detail and degree of prominence when displaying information in favour ofand against the options/consequences (FDA Fair Balance Prescription Drug Advertising Act of 2001). Whether the information is balanced
depends on 3 factors: format, sequencing, and framing.
Benefits. Intended positive features or consequences of an option. Benefits can be temporary or permanent. Patients may also get a benefit from having notreatment. Benefits should describe how strong the positive effect will be, how long it might last, and how often someone can expect to enjoy the
positive outcome.
Coaching in communication. To provide balanced instruction, suggestions, feedback, and support (for example, through role-playing) to patients who aregoing to be meeting with their practitioner to prepare them to openly ask questions and make their preferences known. Coaching can be provided
in-person or over the phone, in a one-to-one session, or in a group situation with a leader.
Coaching in deliberation. To provide balanced instruction, suggestions, feedback, and support to patients who are deliberating on a health decision.
Coaching can be provided in-person or over the phone, in a one-to-one session, or in a group situation with a leader.
Coaching methods. Personal attention from a trained professional to help patients improve the quality of decision making. Examples include a nurse, who
might tailor information to a patient’s specific situation, provide an exercise to help patients clarify their values, help patients prepare a list ofquestions for their doctor, and helps patients think through and compare options. It could also be provided by a trained person who helps a groupof newly diagnosed patients to sort through the process of making the decision.
Cochrane Review. Conducted as part of the Cochrane Collaboration, a Cochrane review is a systematic review of research studies to learn about the effect ofhealth interventions such as patient decision aids. Systematic reviews are defined as a rigorous process, specified at the beginning of the study, to:a) conduct a comprehensive search of the literature for all relevant studies on a specific topic; b) appraise the quality of the identified studies; andc) synthesize the findings into a single report. The Cochrane Collaboration is an international not-for-profit organisation that produces andcirculates systematic reviews of healthcare interventions and promotes the search for evidence to aid in making healthcare decisions(http://www.cochrane.org).
Cochrane Review of Patient Decision Aids. A systematic review of randomized controlled trials evaluating the effect of patient decision aids for patientsfacing actual screening or treatment decisions.
Communication about decision making. A discussion of facts, possible outcomes and attitudes, and different ways to make tradeoffs between choices.
Conditional probability. A measure of how likely something is to happen, based on existing information. For example, a measure of how likely someone isto suffer from a heart attack, if that person is a smoker. The likelihood would be different if the person were not a smoker.
Conflict of interest. “A set of conditions in which professional judgment concerning a primary interest (such as patient’s welfare or the validity of research)tends to be unduly influenced by secondary interest (such as financial gain)”. (Thompson DF. Understanding financial conflicts of interest. New
Engl J Med 1993; 329:573-576)
Consistency of evidence. Whether there is agreement among many studies on the effect of the same treatment option.
Decision quality. The extent to which patients choose and receive health care options that match their informed and considered values.
Decision Support. Helping another person make a decision. It may be provided before a visit to a personal practitioner (in preparation for decision making)or during the visit with the personal practitioner (while making the decision).
Deliberation. The process of thoughtfully considering and discussing all sides of a decision that involves choosing among alternative actions.
Development process. The steps taken by the creators of the patient decision aid to understand the needs of potential users (e.g. patients and health carepractitioners), and to consult experts and users to design, revise, and guide the development of the patient decision aids.
Disclosure of conflicts of interest. Providing information on factors that have the potential to unduly influence the content of patient decision aids such as
the source of funding (for their development and production; people primarily responsible for the content) and affiliations of patient decision aid
developers.
Event rate. The number of events (events might be benefits, harms, or side effects) that occur in a defined group of people (population) over a defined periodof time. For example: if 1000 patients are treated our best guess is 2 patients may die. If 1000 patients are not treated our best guess is 4 patients
may die. (Users Guides to the Medical Literature 2002).
Evidence. Knowledge gained through scientific research.
Field testing. Formal testing in “real life” settings where the patient decision aid is intended to be used, with patients and practitioners who are actually
involved in making a decision. Results of field testing are used to improve the patient decision aid and make it more useful.
Financial interests. The potential financial interests of any company related to the patient decision aid’s clinical content are clearly explained in plain
language. For example: The Badenov Pharmaceutical Company, Inc., a for-profit company, makes and sells Borisol, one of the treatment choices
described in this program. The Richards Foundation is a charitable, not-for-profit group completely funded by the Badenov Pharmaceutical
Company, Inc.
Framing. The way health statistics and information is presented with:

IPDAS Voting Document – 2nd round

Page 25 of 27

Similar use of positive and negative words to describe the intensity of the benefits or harms without being too sensational or alarmist;
Similar display of numbers of the chances of experiencing benefits and harms;
Similar level of detail for both benefits and harms; andBalanced examples of patients’ experiences so that equal space is given to different viewpoints and choices.
Framing can make patients feel differently about the options. For example, it can make patients feel differently to be told the ‘death rate’ of a procedure
rather than the ‘survival rate’. It is alarming to hear that 20% of patients might die from a surgery but reassuring to remember that 80% survive.
Hence, framing is the expression of the same information in different ways
Format. This refers to how the information about benefits and harms looks or sounds:
For written resources, the size of the type, the spacing, and the page layout of information should be the same throughout the patient decision aid.
For audio or video resources, consistent volume and length of airtime is important for all options.
Funding sources. The agencies or organisations that provide the money to pay for the patient decision aid. It must be is clearly and prominently
acknowledged. For example: This program was funded by grants from the Government of Pottsylvania, the Richards Foundation, and theBadenov Pharmaceutical Company, Inc. It is important that the information about sources of funding is presented in a way that ensures that it will
be seen and understood by the user. Ideally, this information should appear in clear large print at the beginning of the patient decision aid.
Guidance in communication. To outline possible questions and concerns, discuss barriers to communicating with practitioners (such as shyness,
defensiveness and confusion,) and then provide a systematic approach to helping overcome these barriers to make sure the patients’ questions andconcerns are raised and addressed. Guidance can be provided in a variety of ways including by printed materials, an internet site, audio or video
tapes.
Guidance in deliberation. To outline possible actions, and then provide a systematic approach to comparing and contrasting those actions. Guidance can beprovided in a variety of ways including printed materials, an internet site, audio or video tapes.
Guidance methods. Instruction or direction in the steps of decision making. Some examples may include a list of the steps for making a high quality
decision, filling in a worksheet that helps patients to clarify their values about treatment options, or a list of questions to ask the doctor or decisioncoach.
Harms and side effects. Unintended negative features or consequences of an option. These can be temporary or permanent and major or minor. Descriptionsof harms and side effects should include how severe the negative effect will be, how long it might last, and how often someone can expect toexperience the negative effect. An example of harm is the development of breast cancer from taking estrogen and progesterone. An example of aside effect is upset stomach from taking an antibiotic pill.
Health care consumer. A person who faces a health-related issue (for example, a decision to be tested for a disease) or who has been diagnosed with adisease or condition.
Health literacy. An individual’s ability to perform basic reading, listening, computing and observing to obtain, understand, and use health information.
Indirect commercial funding. Any company that pays money to practitioners who have primary responsibility for the patient decision aid’s information
should be clearly acknowledged. For example: Dr. Fatale receives salary support from the Richards Foundation and research funding from the
Badenov Pharmaceutical Company, Inc.
Informed consent. A conscious action taken by a person to give permission for something to occur which involves their person or privacy. For example,
giving permission for a doctor to perform a medical test or for personal information to be published.
Interactivity on the Internet. A website is interactive if there is the ability to enter information and receive feedback based on the information that wasentered. The ability for patients to enter and receive information from each other is also a way for a website to be interactive. When a website
provides an interactive service, such as tailoring information to the patient based on information that the person provides, it should also explain
the reasoning behind the feedback it provides.
Internal search capability. A function to locate content within a patient decision aid. The search engine should be capable of searching for natural language
words or phrases and retrieving only relevant materials. It should also have a way for a person to enter words that is easy to understand and use.
Internet. A network connecting millions of computers around the world for communications purposes.
Internet Design. Describes the accessibility, organization, and internal search capacity as well as graphical and multimedia design. For example: Is the site
available each time patients want to use it? Is the information on it organized and easy to manoeuvre? Has attention been paid to presenting theinformation as error-free (e.g., spelling, punctuation) as possible? Is there a readily identifiable link back to the institutional or organizational
home page?
Links on the Internet. Connections to other internal pages or to external websites that form the web-like structure of information.
Link Architecture. The way a patient decision aid organises its links and is important for patients to find their way around a website. The links should beapparent, organized and logical to the reader. If the patient decision aid uses small pictures instead of words to indicate a link, they should bemeaningful and consistent throughout the website.
Link Content. Links should be accurate, current, credible, and relevant. It helps to be alerted when leaving the patient decision aid website and going to one
that is separate from the patient decision aid.
Logical organization (navigability). Simple, consistent, and easy to use with references to other sections provided to help a person understand the overall
structure of the decision aid on the website. More logically organized patient decision aids are designed to ensure that a person does not skip but
views all relevant information that is necessary for informed decision making.
Measurable improvements in the quality of decision-making. The most convincing evidence that a patient decision aid would really improve decisionquality is likely to be generated by controlled study designs that compare two groups of patients who both face the decision: one group thatexperiences use of the patient decision aid in practice and another group that experiences usual practice (without a patient decision aid). An
improvement in the quality of decision making with the patient decision aid would be shown if the group who used the patient decision aid inpractice had a greater match between their values individual values and the health care options they selected.

IPDAS Voting Document – 2nd round

Page 26 of 27

Medical tests (screening and diagnostic). Tests performed on healthy people to see if they have a disease (diagnostic) or to see if they are likely to have a
disease or condition (screening).
Medical treatments. Effective treatments that medical professionals judge to be ethical and which are available to patients in their community.
Natural frequency. This term means the same as “Event rate”. (Gigerenzer 2003).
Needs. Gaps between what is known and what is needed to make a decision. A patient decision aid needs assessment can tell us what information patients need tomake decisions that match the personal importance they feel toward the positive and negative features of their options. For example, before
coming to a decision about treatment, patients first need to know basic information about their condition. They also need to know about possible
treatment options (including doing nothing) and the potential benefits and harms involved.
Number needed to treat (screen). Not all medical tests or treatments work for every patient. This is a way of telling how many patients would need to havethe treatment or test to prevent one person from suffering from a disease or a harmful treatment side effect. Difference in occurrence rates ofadverse outcomes between patients exposed to a treatment (intervention group) and not exposed (control group). It is the inverse of absolute risk
reduction (Users Guides to the Medical Literature 2002).
Over-detection (or over-diagnosis): Detection by screening tests of disease which, without screening, would not have been diagnosed or caused symptoms
within the person’s lifetime. A common example is detection of low grade prostate cancer by PSA testing which would not have harmed theperson in their lifetime. Also described as detection of pseudo-disease
Patient. A health care consumer who faces a health-related issue (e.g. a screening decision) or who has been diagnosed with a disease (e.g. diagnostic test or
treatment decisions). Other names include public, people, person, health consumer, or patient decision aid users.
Patient Decision Aid. Resources created to help patients make specific health decision when there is more than one option (including the status quo). They
providing (at the minimum) information on the options and the features of these options (benefits, harms, pros, cons, side effects, inconveniences)(O’Connor et al., BMJ, 1999)
Patient stories. Accounts from patients with a health condition describing their experiences with the condition, options, decision-making, and consequences
or outcomes. These may be collected as audio or video recorded interviews or as written stories. The characteristics of stories in patient decision
aids usually vary depending on whether they involve a health issue faced by otherwise healthy people or a treatment decision for a lifethreatening illness. Many examples of “others’ experiences” are a combination of details of several real cases (O’Connor, Drake et al, 1999;
O’Connor & Jacobsen, 2003).
Plain language. The use of ways to clearly communicate health information that create optimal understanding.
Practitioner. A health care professional that provides direct care to patients or public. This includes physicians, nurses, physiotherapists, pharmacists, andsocial workers.
Probability. The chance that something will occur. Probabilities are typically estimated by observing what happens to large groups of patients. For example,
watching 10,000 men aged 60 years for 10 years to see how many die. The probability of dying in this group over 10 years might be 1,500 out of
10,000. If a patient is similar to the patients in the group, it would say the best guess at a patient’s chance of dying over the next 10 years is 1,500
out of 10,000.
Quality of the Evidence. How accurate the descriptions and the estimated effects of different options are. This evidence often comes from one or morestudies or resources. The quality of evidence depends on several factors, including the strength of the study design, the quality of the methods,
whether there is agreement among studies of the same health option, and how directly the studies address the health option.
Readability. A measure of the reading ease or difficulty of printed materials. It is usually measured using proven formulas that gauge the length of words insyllables and length of sentences (e.g. SMOG, Fry).
Reference class. This is the group of patients who might have a side effect or suffer from a disease in a set period of time. It is the denominator of an event
rate, which is the group of patients who, over a specified period of time, are at risk of an event (Gigerenzer 2003).
Relative risk reduction. Relative risk measures how much the likelihood of something happening is reduced in a group of patients who take a treatment,
compared to a group of patients who don’t. For example, if 60% of a non-treated group of patients died and only 30% of the treated group died,
the treatment would have a relative risk reduction of 0.5 or 50%. Therefore, the death rate in the treated group is half of that in the control group.
The relative risk reduction is the ratio of the event rate in patients exposed (intervention group) to the event rate in patients not exposed (control
group) to a specified intervention (Users Guides to the Medical Literature 2002).
Rigorous social process. A method used to gather information from many sources on the key factors influencing the decision. For example, it may include
interviews, focus groups, surveys, or decision analysis with individuals who have faced or are facing the decision (e.g. patients, family) andhealth practitioners (e.g. doctors, nurses, social workers).
Risk. In this context, risk can mean the same as probability – that is, the chance that something will occur. For example, risk can be the chance that a positiveor negative feature of an option will occur.
Sequence. The order in which the information on options and consequences (benefits and harms) is provided. It should be the same for all the options.
Severity. A description of how intense the harm or side effect might be. It’s best if described in a practical way that means something to patients’ daily life.
For example, if a side effect was “fatigue”, it would be helpful to know how severe the fatigue might be in practical terms such as “fatigued sothat you can’t walk a block” or “fatigued so that you can’t get out of bed”.
Study design. The basic method or approach used in a research project. Some common designs may be a systematic review of several studies (with orwithout combining the results across studies; meta-analysis), a randomized controlled trial, an observational study (for example a cohort or casecontrol study), or a descriptive study (for example, a case series).
Study quality. Details of the study source’s, design, and execution that ensure the study findings are fair and unbiased. For example, the quality of arandomized trial depends on making sure that the same number and type of patients received the treatment, that the health professionals were notaware of which groups of patients were receiving the treatment, and that the patients were followed for long enough to tell find the longer-term
effects of the treatment.

IPDAS Voting Document – 2nd round

Page 27 of 27

Suitability. The ability of health information resources to match the language, logic, and experience of the target audience. Suitability, as used by Doak,Doak, and Root (1996), measures how well health resource content, approach, and design match the culture, experiences, abilities, values andbeliefs of the intended audience. With the development of multi-media documents, suitability should also include assessing usability-audiovisualfactors and technological accessibility (e.g. ease of navigation).
Usability. A measure of the quality of a person's experience when interacting with a web-site, and generally tells how easy a patient decision aid is easy touse. Usability is a combination of factors that affect a person's experience, including: how efficient it is, whether it is easy to remember how to
use it the next time, whether there are many errors and if they are serious or not (http://usability.gov/basics/index.html). An option to remember
what has been viewed is important, in case the entire patient decision aid cannot be viewed in one sitting. A person may want to return at a latertime without having to start from the beginning.
Users. Patients who face the decision and health practitioners who help patients learn about their options to make the decision.
Values. How a person feels about or rates the importance of options and their positive and negative features. These preferences are based on how their health might
be affected by the decision, their attitudes about the chances each option holds for bringing benefits or harms, their willingness to make trade-offs overtime, how they feel about certain medical procedures or anything else that might be useful in making the decision.
Values clarification. Ways to help patients form and share how important various options and their features are. Examples include: a) describing features of
options so patients can imagine and value what it is like to undergo procedures and live with the consequences; b) providing examples of how
other patients’ values led them to make different choices; c) bringing forth values by guiding patients to rate or trade-off different features of
options; and d) recording, guiding, or coaching patients to help them share their values with others involved in the decision.