Establishing the Effectiveness2012 UPDATED CHAPTER L:ESTABLISHING THE EFFECTIVENESSSECTION 1:
AUTHORS/AFFILIATIONSHarvard Medical School, Boston

USA

Newcastle University, Newcastle Upon Tyne

UK

Women’s College Hospital, Toronto

CanadaNewcastle upon Tyne Hospitals NHS Trust

UK
USA

Daniel D. Matlock

Informed Medical Decisions Foundation,
BostonUniversity of Colorado, Denver

Chirk-Jenn Ng

University of Malaya, Kuala Lumpur

Malaysia

Mary Ropka

University of Virginia

USA

Dawn Stacey

University of Ottawa

CanadaNatalie Joseph Williams

Cardiff University

UK

Celia E. Wills

Ohio State University, Columbus

USA

Karen Sepucha(co-lead)
Richard Thomson(co-lead)Cornelia M. (Cory)
Borkhoff
Joanne “Jo” Lally
Carrie A. LevinUSA

SECTION 2:
CHAPTER SUMMARY
What is this dimension?
In order to establish the effectiveness of a patient decision aid (PtDA), it is critical toprovide evidence that the PtDA improves i) the quality of the decision-making process
and ii) the quality of the choice that is made (i.e., “decision quality”). The quality of thedecision-making process refers to the extent to which a PtDA helps patients to: recognizethat a decision needs to be made; feel informed about the options and theircharacteristics; be clear about what matters most to them in this decision; discuss goals,
concerns, and preferences with their health care providers; and be involved in decision
making. The quality of the choice that is made (i.e., “decision quality”) is defined as the

Suggested Citation:
Sepucha K, Thomson R, Borkhoff CM, Lally J, Levin CA, Matlock DD, Ng CJ, Ropka M, Stacey D, Joseph-Williams N,
Wills CE. (2012). Establishing the effectiveness. In Volk R & Llewellyn-Thomas H (editors). 2012 Update of the International
Patient Decision Aids Standards (IPDAS) Collaboration's Background Document. Chapter L. http://ipdas.ohri.ca/resources.html.

Establishing the Effectivenessextent to which patients’ eventual choices are consistent with their informed values andare actually implemented.
What is the theoretical rationale for including this dimension?
There are a number of complex reasons for assessing the effectiveness of PtDAs. In thischapter, we have organized this rationale into scientific, ethical, conceptual, and healthpolicy sub-sections. We have also clearly indicated the different rationales for measuring
the quality of the decision-making process and the quality of the choice that is made (i.e.,“decision quality”).
What is the evidence to support including or excluding this dimension?
This updated overview of the evidence incorporates: a) the results of the 2011 Cochrane
Collaboration’s review of the 86 randomized controlled trials of PtDAs; b) an
examination of the reported performance of instruments used to measure primary process
and outcome variables in studies included in the 2011 Cochrane Collaboration’s review;and c) good practice examples of how investigators should report on the performance of
instruments used to measure decision making process and outcomes. This overview
raises a number of issues to be addressed in future conceptual and empiricalinvestigations. For example, when and how should we measure the effects of PtDAs on
the decision-making process and on decision quality? Are we overlooking importantmeasurement constructs? Are decision-making process variables predictive of decisionquality? Can we develop a minimum “tool-kit” of key theory-based measures of PtDA
effectiveness? In what ways do options’ behavioral-change characteristics (e.g., diet andexercise versus surgery for obesity/weight management) change our strategies (if at all)for the evaluation of PtDAs?

SECTION 3:
DEFINITION (CONCEPTUAL/OPERATIONAL) OF THIS QUALITYDIMENSIONa) Updated Definition
In order to establish the effectiveness of a patient decision aid (PtDA), it is critical toprovide evidence that the PtDA improves i) the quality of the decision-making process
and ii) decision quality – that is, the quality of the choice that is made.
The Quality of the Decision-Making Process
The core areas/domains about the decision-making process that should be captured, andsome details about how each area/domain might be measured, include the extent to which
PtDAs help patients to:
Recognize that a decision needs to be made (e.g., one approach is to assess whether
the patient understands that there is more than one reasonable approach to thesituation).
2

Establishing the EffectivenessFeel informed about the options and the risks, benefits, and consequences of theoptions (e.g., the uninformed subscale of the Decisional Conflict Scale (O’Connor,
1995)).
Be clear about what matters most to them for this decision (e.g., the unclear values
subscale of the Decisional Conflict Scale).
Discuss goals, concerns, and preferences with their health care providers.
Be involved in decision making (e.g., adaptation of Degner’s Control Preferences
Scale that assesses who made the decision (Degner, 1992)).

Decision Quality -- The Quality of the Choice That is Made
The quality of the choice that is made is referred to as “decision quality” and, for the
purposes of the prior round of IPDAS voting, was defined as the extent to which patients’decisions are consistent with their informed values (Elwyn 2006). Other definitions ofdecision quality emphasize the additional need for the choice to be implemented(O’Connor 1997, Sepucha 2004). It follows from the definition that the impact of PtDAs
on the following outcomes should be measured whenever possible:


Informed patient: This is most commonly measured by assessing patients’knowledge of the options and outcomes. Note that this is not assessed withpatients’ perceptions; instead, factual items are used to assess their understandingof the information. This may include an assessment of accurate risk perceptions,when applicable.
and

Concordance between what matters most to the patient and the chosen option:
Despite considerable variation in assessment of this construct, most approaches
require the following: (1) elicitation of patients’ goals and/or treatment
preferences; (2) identification of the patient’s chosen or implemented option; and(3) calculation of the extent to which the chosen option best meets the statedgoals.

Many other process and outcome variables that are related to decision quality have beenused to evaluate PtDAs; these include decision self efficacy, percentage of patients who
are able to state a clear preference (as opposed to being unsure), decision regret, andpatient satisfaction with decision making and choice of option. Further, there are many
surveys and scales that cover some, but not all, of the process and outcome domains
presented above. The definition presented here focuses on the core set of domains that are
critical and some examples of scales that have been used; it does not imply that these are
the only aspects that should or could be measured.
b) Changes from the Original Definition
The core of the original definition has been retained; however, we have edited the details
3

Establishing the Effectivenessto make it consistent with the definition used in the Cochrane Systematic Review of
PtDAs, and to more clearly differentiate between decision quality as an outcome variable
and the key domains for the decision-making process variables (Stacey et al 2011).
c) Emerging Issues/Research Areas in Definition
See Section 6.

SECTION 4:
THEORETICAL RATIONALE FOR INCLUSION OF THIS QUALITYDIMENSIONa) Updated Theoretical Rationale
Scientific Rationale
Establishing the effectiveness of any health care intervention, including decision aids, iscritical. There is a strong consensus and scientific evidence that PtDAs: a) improve thequality of the decision-making process; and b) increase decision quality or the likelihoodthat individuals choose and/or receive health care interventions that are most consistent
with their informed and considered values (Briss et al., 2004; O’Connor et al., 1997;
O’Connor, Stacey et al., 2003; Ratliff et al., 1999; Sepucha et al., 2004, Sepucha 2008,
Stacey et al., 2011; Elwyn et al., 2006). However, the field needs to continue to generate
high-quality evidence regarding the benefits and harms of these tools to improve ourunderstanding of their impact across various populations, health conditions, andhealthcare systems, including across countries and cultures.
The best study design, when applicable, for determining effectiveness of health careinterventions is the double blind randomized controlled trial that attempts to minimize
risk of bias between groups to allow for head-to-head comparisons. However, these typesof studies are limited as they tend to include a fairly homogeneous sample of welleducated, white, middle income patients, and less is known about the impact PtDAs haveon disadvantaged patients (e.g., individuals with low literacy, or low incomes). In thecase of decision aid trials, it is also challenging (or impossible) to blind participants(patients or providers) to the intervention. Furthermore, there is the challenge of taking
the results observed within such trial conditions and applying them to real world settings,
especially when the issues of implementation, such as the timing of the use of thedecision aid, are difficult to standardize.
Ethical Rationale
Whilst this chapter focuses on the effectiveness of PtDAs, these are commonly seen as ameans of supporting a shift from paternalism to informed and shared decision making(SDM). SDM is a process by which a decision is made between the patient, with or4

Establishing the Effectivenesswithout his/her family, and one or more healthcare professionals. It offers a model toapproach patient engagement, particularly in preference-sensitive decisions – decisions inwhich there are reasonable options and in which the choice among those options is likely
to be influenced by patient preferences and values.
Furthermore, not only is there a very clear ethical imperative to engage patients indecisions about their own care, but also PtDAs can support better informed consent. Kingand colleagues (2006) have argued that traditional informed consent methods are
inadequate to engage and inform patients about treatment options in preference-sensitive
decisions, and that SDM, because it extends beyond information-giving to supporting theformulation and communication of informed preferences, may offer an ethically andlegally supported means for fostering informed choice. Research shows that, when usual
care is compared to use of decision aids, current approaches without decision aids are
inadequate for ensuring that patients are informed and have realistic expectations (Stacey
et al., 2011).
As in any area of the evaluation of interventions, we should be concerned with addressing
not only the benefits but also any adverse effects. In the context of PtDAs, the literature isrelatively light on the exploration of adverse effects, although such effects have been
posited. Adverse effects might include, for example, an increase in inequalities (throughbeing more accessible to, or used by, well-educated patients) or selection of “lesseffective” interventions with impact on population health, particularly in preventive
health choices (Thomson, 2005). For example, well informed patients may select aspirinrather than (the more effective) warfarin for prevention of stroke in atrial fibrillation as aresult of balancing how they value the benefits and risks of these alternatives (Thomson,2007). Other adverse effects might include increased patient anxiety when patients are
faced with clinical uncertainty (Politi et al., 2011)), or are offered an unexpected role indecision making and are initially wary of engaging in decision making (Say et al., 2006),
or feel unsupported or ‘abandoned’ if decision making is not actually shared but is unduly
delegated to patients (Quill & Cassel, 1995). Adverse effects may also occur if PtDAs
are neither well developed nor kept up to date, and therefore might bias decisions – hence
the importance of standards such as these.
Conceptual Rationale
The measures of “the quality of the decision-making process” and “decision quality”
highlighted in this chapter have underpinnings in theories of decision making. Normative
theories of decision making, such as subjective expected utility theory, are based on theideal that all patients can approach decisions rationally and are able to weigh the risks
and benefits of various interventions (von Neumann and Morgenstern, 1947).
Descriptive theories of decision making, such as prospect theory, demonstrate thathumans are subject to cognitive biases that cause decision making to deviate from thenormative/rational (Kahneman and Tversky, 1979). For example, a well-known cognitive
bias has to do with the effects of framing, where people tend to be risk-averse when
statistics are presented as gains and risk-seeking when they are presented as losses
5Establishing the Effectiveness(Tversky and Kahneman, 1974). These kinds of biases can threaten a person’s ability toacquire accurate knowledge or to make a decision concordant with their values, thus
threatening the quality of their decision. The Dual-Process Theory of decision making(Kahneman, 2003) argues that people make decisions either ‘intuitively’ (i.e., quickly
drawing on past experiences), or ‘reasonably’ (i.e., using a thoughtful, analytic
approach), with the latter being less subject to many of the cognitive biases. PtDAs aredesigned to encourage a more thoughtful or reasoned process in decision making, thus
minimizing the influence of cognitive bias. If a more reasoned, normative approach ispursued, the actual choice will more likely to be informed and value concordant – the key
domains of “decision quality”.
Although conceptually many of these theories share similar underpinnings (e.g., anemphasis on information, and the use of deliberative processes to align choices withgoals), there is considerable debate about how that gets translated into specific measures(see section 5). The debate involves not only how to measure the construct (e.g.,measuring patients’ preferences using the standard gamble in formal decision analysis
versus using attitude scales), but also the timing of the measurement. For example, somehave argued that it is only legitimate to measure the quality of the decision making
process before and immediately after exposure to the decision aid, because any latermeasurement will be prone to potential bias of the practitioner and longer-term
measurement, especially after treatments have been experienced, may be influenced byhindsight bias based on health outcomes. (Elywn and Miron-Shatz, 2010)
A key goal of health care is to improve health outcomes, and many outside the field ask
about what is the impact of decision aids on health outcomes. There are several
challenges to the use of clinical health outcomes (such as quality of life, pain, or
mortality) to assess the effectiveness of decision aids. First, the nature of the situation
addressed by decision aids requires that there are multiple reasonable options; thus, bydefinition, there is usually not one clearly superior treatment or intervention. Second,
many of these decisions are made under uncertainty, and are essentially making a bet;
evaluation of the bet depends on the odds not the outcome. For example, a patient may
chose to have surgery, feeling that the benefits outweigh the harms, and yet may suffer asevere, unanticipated complication during the procedure. Using health outcomes toevaluate that decision would suggest it was poor, whereas using decision-making process
or decision quality measures would suggest it was strong. A third challenge with using
health outcomes as a measure is that studies have shown that patients vary in their
willingness to trade off quality of life and quantity of life. For example, patients may
elect to forego chemotherapy if their desire to avoid short-term severe side effects
outweighs their desire for increasing short-term survival. To the extent that patients feeldifferently about the potential health outcomes, it is necessary to measure theeffectiveness of PtDAs by the extent to which they enable patients to achieve theoutcomes they most desire (and avoid those they most dislike).

6

Establishing the Effectiveness
Policy Rationale
There are several recent, widespread health policy drivers that align with theimplementation of PtDAs and SDM, hence emphasizing the need for a robust evidence
base. Achieving decisions that are based on informed patients’ values and preferences is
consistent with the aims of patient-centered care (Institute of Medicine, 2001). Patientand family engagement in care has been identified as one of six national priorities forhealth care quality by the National Priorities Partnership in U.S. (National Priorities
Partnership). The US National Quality Forum has identified SDM as one of top fivepriority areas for measurement gaps (Table 2 in NQF report to HHS June 11, 2011
Measure Prioritization Advisory Committee Report).
In the UK, SDM (“Nothing about me, without me”) is included within the latest UK
Government health policy (Department of Health, 2010), and is embedded in legislationpassing through parliament (Health and Social Care Bill 2011). The Department of
Health has commissioned an extensive programme of development of PtDAs (NHS,2012).
Further description on policy developments internationally can be found in the special
issue of The German Journal for Quality in Healthcare, 2011.
From the policy perspective, however, comes a different pressure for measurement—
costs. If health systems are to fund access to PtDAs, then they want to know theintervention is not only effective but also cost-effective. The impact of decision aids on
utilization has been demonstrated in a small number of decisions, but, as of yet, there isnot strong evidence that the tools are cost-effective or reduce costs. The impact on cost
seems to depend on baseline utilisation (NICE, 2011). As implementation efforts expand,examining the impact on costs/cost-effectiveness will be an important outcome to include
in those efforts.
b)

Changes from the Original Theoretical Rationale

The original chapter did not provide an extensive rationale for including the concept ofeffectiveness; instead, it focused on proposing methods to operationalize the definition of
the outcome concept of decision quality.
We have modified the discussion of the rationale in a number of ways:i)
we have organized the rationale into scientific, ethical, conceptual, and healthpolicy sub-sections;ii)
we have more clearly indicated the different rationales for measuring the “qualityof the decision-making process” and the “quality of the choice” (i.e., “decisionquality”); andiii)
we have removed most of the section that dealt with operationalizing themeasurement of decision quality.
7

Establishing the Effectivenessc) Emerging Issues/Research Areas in Theoretical Rationale
See Section 6.

SECTION 5:
EVIDENCE BASE UNDERLYING THIS QUALITY DIMENSIONa) Updated Evidence
Cochrane Collaboration Review
The latest update of the Cochrane Collaboration’s Systematic Review includes 86 RCTs
comparing individual PtDAs for treatment or screening decisions to usual care and/or
alternate interventions. The trials were identified by searching through December 2009
on MEDLINE, EMBASE, PsycINFO, and Cochrane databases, and through September
2008 for CINAHL. Fifty-one trials measured knowledge (59%), 16 measured accurate
risk perceptions (19%), 13 measured agreement between values and choice (15%), 23
measured feeling clear about values, and 26 measured feeling informed (Stacey et al.,2011).
Evidence About The Quality of the Decision-Making Process
PtDAs resulted in:a) reduction in feeling uninformed (MD -6.4 of 100; 95% CI -9.2 to -3.7);b) reduction in feeling unclear about personal values (MD -4.8; 95% CI -7.2 to -2.4); andc) fewer people passive in decision making (RR 0.6; 95% CI 0.5 to 0.8). There were no
studies evaluating the decision-making process attributes relating to helping patients torecognize that a decision needs to be made or understand that their values affect the
choice (Stacey et al. 2011).
Decision aids also appear to have a positive effect on patient-practitioner communicationin the four studies that measured this outcome. For satisfaction with the decision (numberof studies (n) = 12) and/or with the decision making process (n = 12), those exposed to adecision aid were either more satisfied or there was no difference between the decision
aid versus comparison interventions. Decision aids do not appear to be different fromcomparisons in terms of anxiety (n = 20), and general health outcomes (n = 7) orcondition-specific health outcomes (n = 9). The effects of decision aids on otheroutcomes (adherence to the decision, costs/resource use) were inconclusive (Stacey et al.,2011).
Evidence About Decision Quality - The Quality of the Choice That is Made
The latest results indicate that: PtDAs improve knowledge by 14% (mean difference 13.8
out of 100, 95% CI 11.4-16.2), with greater knowledge gains with more complex PtDAs;
and improve accurate risk perception by 74% (relative risk 1.7; 95% CI 1.5-2.1) and8

Establishing the Effectivenessmore so when the probabilities are expressed in numbers than words. PtDAs also resultin fewer people being undecided (more have clear treatment preference) (RR 0.6; 95% CI
0.4 to 0.7); and, in the presence of explicit values clarification, improve agreement
between values and choice by 25% (RR 1.3; 95% CI 1.1-1.5) (Stacey et al., 2011). Noneof the trials measured decision quality defined as decisions based on informed values.
Review of the Reported Performance of Instruments/Measures
There are many different survey instruments and measures used to assess decisionmaking process and decision quality. It is important to clearly document and appraise theperformance of the instruments used to evaluate the effectiveness of PtDAs. A few
recent studies exist that examine the properties of instruments such as:Kryworuchko et al.’s 2008 appraisal of selected “primary outcome” measuresused in decision aid trials, which included some decision-making process anddecision quality measures. The study concluded that most publications provided
inadequate detail for appraising the included instruments.
 Sepucha and Ozanne’s 2009 systematic review of methods to calculate value
concordance, which found considerable variability in definitions andmethodology and minimal reports of validation of instruments or approaches.
 Scholl et al.’s 2011 review of shared decision making instruments (an update ofSimon 2007), which showed that, whilst reliability of most scales is good, theydiffer in their extent of validation.
To extend this work, we identified all reported outcomes from the 86 studies in the latest
Cochrane Collaboration’s review, and determined whether they measured one or more of
the elements of the “quality of the decision-making process” or “decision quality” asdefined earlier. We extracted data on the reported development and performance of each
instrument, focusing on the reporting of key criteria for high-performing patient-reported
measures including reliability, validity, responsiveness, precision, interpretability,
feasibility, and acceptability. The initial review limited the abstraction to the details
provided within the published trial papers – based on how a reader might evaluate themeasures as described by the trial authors.
We found that the majority measured at least one aspect of decision quality or decisionmaking process (80/86 trials). Overall, we abstracted 220 measures that mapped onto one
or more of the constructs, with knowledge being the most commonly measured construct.
There was fairly limited reporting on the performance of the measures used. The studies
included information on the psychometrics of only 64/220 or 29% of these measures. The
most commonly reported criteria were reliability and validity; however these werereported less than 20% of the time. All the other criteria that we abstracted were reported
less than 5% of the time. A little more than half of these measures were previously
published or adapted from previously published instruments (129/220 or 58%), and thisinformation may be available in prior publication. However, a significant proportion of
the instruments or measures (72/220 or 33%) was new and reported nothing about the9Establishing the Effectivenessproperties of the measure. More detailed analysis of the properties of the measurement isongoing and will be published separately.
Despite the general lack of reporting on the performance of instruments, there are a few
examples of high quality reporting. One instrument that had wide and very strong
reporting on its performance was the Decision Conflict Scale (DCS). It is also one of themost widely-used measures (in 39 of the 86 trials). The DCS measures two of thedecision-making process domains—whether patients feel that they understandtesting/treatment options and outcomes, and whether they feel clear about what mattersmost to them.
Tables 1 and 2 illustrate notable examples of how investigators included fairly
comprehensive but parsimonious reporting on the performance of instruments.
Table 1 reports on the Decisional Conflict Scale.

TABLE 1: Extract from Vodermaier A et al., 2009.
“MeasuresPatients were assessed with the following instrumentation:
Primary outcome variable
Decisional conflict. The Decisional Conflict Scale (DCS; O'Connor, 1995) measurespatients' uncertainty about which treatment to choose, factors contributing to uncertainty(believing to be uninformed, unclear values, and unsupported in decision making), andperceived effectiveness of decision making. Questions have to be answered on a 5-point
Likert scale [from strongly agree to strongly disagree]. Higher scores on the scale orsubscales reflect higher decisional conflict, uncertainty, and a less effective choice. The
German version of the scale demonstrated subscale and total score internal consistencies
in the present sample between 0.73 and 0.94. The scale discriminates between patients
who make and those who delay decisions (O'Connor, 1995; Bunn and O'Connor, 1996)
and is sensitive to change (O'Connor et al., 1998).”10

Establishing the Effectiveness
Table 2 reports on a knowledge measure that was developed for the featured study.

TABLE 2. Extracts from Barry et al., 1997.
“The quality of the treatment decisions was examined using a multifaceted approach.
First, we determined whether subjects were better informed through a twenty question
test of BPH Knowledge … developed by a panel including a general internist, a urologist,a survey researcher, and a lawyer with a special interest in informed consent. Correct
responses were scored +1, incorrect responses -1, and “not sure” responses were scored 0(total range -20 to +20). The test of knowledge was administered two weeks after
exposure to either the [patient decision aid]) or the control brochure.”
and“Validation of New Outcome Measures
Cronbach’s alpha statistic for the items testing BPH knowledge was 0.68. The criterion
validity of this test was assessed by comparing scores for a convenience sample of 12
urologic nurses with the scores of the 167 BPH patients enrolled in the baseline period.
The nurses had a mean score of 14.8 [out of 20], compared to 5.6 for the patients (p <0.001). Nurses answered an average of 85% of the questions correctly, compared to 48%
for the patients (p < 0.001). Furthermore, a modest correlation between these patients’knowledge scores and their educational levels was seen, r = 0.23 (p < 0.001).”

Other work that has advanced the measurement of decision quality is not yet reflected in
the Cochrane Collaboration’s Review. Sepucha and colleagues have publishedpsychometric analyses of three decision quality instruments (for osteoarthritis of the kneeor hip, herniated disc, and breast cancer surgery) that assess the extent to which patientsare informed and receive treatments that match their goals (Sepucha, Stacey et al., 2011;
Sepucha, Feibelmann et al., 2012; Sepucha, Belkora, et al., 2012). In general, these
instruments met several criteria for patient-reported outcomes, including test-retest
reliability, content and discriminant validity, acceptability, and feasibility. It will be
important to further evaluate these as outcome measures in trials of decision aids in orderto achieve the outcome of decision quality as defined in this chapter and in previous
IPDAS publications (Elwyn et al., 2006).
b) Changes to Original Evidence
The above summary of evidence has been updated, incorporating: a) the results of the2011 Cochrane Collaboration’s review of the 86 randomized controlled trials of PtDAs;b) an examination of the reported performance of instruments used to measure primaryprocess and outcome variables in studies included in the 2011 Cochrane Collaboration’s
11Establishing the Effectiveness
review; and c) good practice examples of how investigators should report on theperformance of instruments used to measure decision making process and outcomes.
c) Emerging Issues/Research Areas in Evidence
See Section 6.

SECTION 6:
EMERGING ISSUES AND RESEARCH AREAS IN THIS QUALITYDIMENSION
Inevitably, the revision of this chapter has also identified a number of questions thatremain to be answered. A large number emerged in our discussions, which we havereviewed and summarized as high level questions that we suggest are most worthy of
pursuit in future research.
How should we measure impact on decision-making process and on decision quality?








Current measures are almost exclusively patient-reported measures; what
advantages/disadvantages would be involved in using provider-reported measures ormeasures of the patient-provider interaction?
What are the best measures of decision quality?
How much, and what type of patient knowledge, is needed to evaluate/support highquality decisions?
What is the value of decision quality measurement at an individual and aggregate
level?
What is the difference between risk perception and risk understanding?
What methods are most appropriate for determining concordance? Is it best to focus
on the chosen option or the implemented option?
How do we measure impact on patient-provider interaction?

What else should we measure? Examples include:







What is the role, if any, for including measurement of health outcomes? Measurementof costs/cost effectiveness?
Should harms/adverse effects of decision aids be explicitly measured and, if so, how?
(e.g., bias, cognitive burden, decision regret)
What is the impact of PtDAs on health inequalities and literacy?
What is the role of brief PtDAs compared to complex PtDAs?
What are the (most) active ingredients of PtDAs? Which components are core toeffectiveness?
What is the impact of decision aids on treatment rates, adherence, service utilization,and service costs?
12

Establishing the Effectiveness
When should we measure impact? Examples include:Measuring knowledge or patients’ preferences too long after the decision has been
made may lead to hindsight bias in the appraisal of that decision.
Measuring immediately after a decision aid but before the decision has been made isalso problematic as provider consults will likely influence both knowledge andpatients’ preferences.

Theoretical Issues
Multiple theories are relevant to the development of decision aids and improving
decision quality (decision making theories, information processing theories,
communication theories, etc.) – and each may suggest different outcomes. How toreconcile and/or link to each? Can we develop a minimum dataset of measures ofdecision quality/PtDA effectiveness?

Increasingly, one or more options in situations covered in decision aids involve alarge behavior change component (e.g., surgery versus diet and exercise forobesity/weight management). In what ways does this behavior change component
change our strategies (if at all) for the evaluation of PtDAs (e.g., do we need to assess
levels of self-efficacy and motivation in addition to knowledge and concordance)?

Do we have evidence that the decision-making process variables are predictive ofdecision quality?

REFERENCES
Barry MJ, Cherkin DC, et al. A randomized trial of a multimedia shared decisionmaking program for men facing a treatment decision for benign prostatic hyperplasia.
Disease Management and Clinical Outcomes. 1997; 1(1): 5-14.
Bekker HL, Legare F, Stacey D, O'Connor A, Lemyre L. Is anxiety an appropriate
measure of decision aid effectiveness: a systematic review? Patient Education andCounselling. 2003; 50: 255-262.
Briss P, Rimer B, Reilley B, Coates R, Lee N, Mullen P, et al. Promoting informed
decisions about cancer screening in communities and healthcare systems. Am J Prev
Med. 2004; 26:67-80.
Collins ED, Moore C, Clay K, Kearing S, O’Connor AM, Llewellyn-Thomas H, Barth R,
Sepucha K. Can women make an informed decision for mastectomy? Journal of ClinicalOncology. 2009; 27(4):519-25.

13

Establishing the Effectiveness
Degner LF, Sloan JA. Decision making during serious illness: What role do patients
really want to play? J Clin Epidemiol. 1992; 45: 941-950.
Department of Health. Equity and excellence: liberating the NHS . London: 2010.
http://www.dh.gov.uk/en/Publicationsandstatistics/Publications/PublicationsPolicyAndG
uidance/DH_117353 (accessed 6th April 2012)
Elwyn G, O’Connor A, Stacey D, Volk R, Edwards A, Coulter A, Thomson R, Barratt A,
Barry M, Bernstein S, Clarke A, Entwistle V, Feldman-Stewart D, Holmes-Rovner M,
Llewellyn-Thomas H, Moumjid N, Mulley A, Ruland C, Sepucha K, Whelan T.
Developing a quality criteria framework for patient decision aids: an online international
Delphi consensus process. BMJ, Aug 2006; 333: 417 ; doi:10.1136/bmj.38926.629329.
Elwyn G, and Miran-Shatz T. Deliberation before determination: the definition andevaluation of good decision making. Health Expectations, 2010 Jun;13(2):139-47. Epub
2009 Sep 9.
German Journal for Quality in Health Care - Zeitschrift für ärztliche Fortbildung und
Qualität im Gesundheitswesen. Policy and practice developments in the implementationof shared decision making: An international perspective. Edited by Martin Harter, Trudyvan der Weijden and Glyn Elwyn. 2011 Volume 105, Issue 4, Pages 227-326.
Health and Social Care Bill 2010-11, 132-EN, House of Commons, (2011).
Institute of Medicine. Crossing the quality chasm: A New Health System for the 21stCentury. IOM, 2001.
Kahneman D. A perspective on judgement and choice. American Psychologist. 2003; 58,
697-720.
Kahneman D, Tversky A. Prospect theory: an analysis of decision under risk,
Econometrica. 1979; XLVII, 263-291.
King JS, Moulton BW. Rethinking informed consent: the case for shared medical
decision-making. American Journal of Law & Medicine. 2206; 32(4):429-501.
King JS, Eckman MH, et al. (2011). The potential of shared decision making to reduce
health disparities. The Journal of Law, Medicine & Ethics. 2011; 39: 30-33.
Kryworuchko, J., D. Stacey, et al. Appraisal of primary outcome measures used in trialsof patient decision support. Patient Education and Counseling. 2008; 73(3): 497-503.
Lee C, Barry M, Cosenza C, Dominik R, Mulley A, O’Connor A, Sepucha K.
Development of instruments to measure the quality of breast cancer treatment decisions.
Health Expectations. 2010; Sep;13(3):258-72.
14

Establishing the Effectiveness

Moulton B, King JS. Aligning ethics with medical decision-making: The quest for
informed patient choice. Journal of Law, Medicine and Ethics. 2010: 85-97.
NICE (National Institute for Health and Clinical Excellence) (2011). Patient experiencein adult NHS services quality standard.
http://www.nice.org.uk/guidance/qualitystandards/patientexperience/home.jsp (accessed
12th July 2012)
National Priorities Partnership.
http://www.qualityforum.org/Setting_Priorities/NPP/National_Priorities_Partnership.asp
x (accessed 6th April 2012)
National Quality Forum. Measure Prioritization Advisory Committee Report: Measuredevelopment and endorsement agenda. 2011
http://www.qualityforum.org/News_And_Resources/Press_Releases/2011/National_Qual
ity_Forum_Releases_Measure_Development_and_Endorsement_Agenda__Prioritized_Li
st_of_Measure_Gaps.aspx (Accessed 6th April 2012)
NHS Right Care Programme. Shared Decision Making
http://www.rightcare.nhs.uk/index.php/shared-decision-making/ (Accessed 6th April
2012)
O’Connor AM. Validation of a decision conflict scale. Med Decis Making. 1995; 15: 25–
30.
O'Connor AM, Drake ER, Wells GA, Tugwell P, Laupacis A, Elmslie T. A survey of thedecision-making needs of Canadians faced with complex health decisions. Health
Expectations. 2003;6:97-109.
O'Connor A, Llewellyn-Thomas H, Sawka C, Pinfold S, To T, Harrison D. Physicians'
opinions about decision aids for patients considering systemic adjuvant therapy foraxillary-node negative breast cancer. Patient Education & Counseling. 1997;30:143-53.
O’Connor AM, Stacey D, Entwistle V, Llewellyn-Thomas H, Rovner D, Holmes-Rovner
M, Tait V, Tetroe J, Fiset V, Barry M, Jones J. Decision aids for people facing healthtreatment or screening decisions (Cochrane Review). In: The Cochrane Library, Issue 2
2003. Oxford: Update Software.
http://www.cochrane.org/cochrane/revabstr/ab001431.htm.
Politi MC et al. Communicating uncertainty can lead to less decision satisfaction: anecessary cost of involving patients in shared decision making? Health Expectations.
2001; 14(1), 84-91.

15

Establishing the Effectiveness
Quill T, Cassel C. Non-abandonment: a central obligation for physicians. Annals ofInternal Medicine. 1995; 122, 368-74.
Ratliff A, Angell M, Dow R, Kupperman M, Nease R, Fisher R, et al. What is a gooddecision? Effective Clinical Practice. 1999;2:185-97.
Say R, Murtagh M, Thomson, R. Patients’ preference for involvement in medical
decision making: a narrative review. Patient Education and Counseling. 2006; 60(2), p.
102.
Scholl, I., M. K. Loon, et al. Measurement of shared decision-making-a review of
instruments. Zeitschrift für Evidenz, Fortbildung und Qualität im Gesundheitswesen.
2011;105: 313-34.
Sepucha KR, Belkora JK, Chang Y, Cosenza C, Levin CA, Moy B, Partridge A, Lee CN.
Measuring decision quality: A psychometric evaluation of new instrument for breast
cancer surgery. BMC Med Inform Decis Mak. 2012 Jun 8;12(1):51.[Epub ahead of print]
Sepucha KR, Feibelmann S, Abdu WA, Clay CF, Cosenza C, Kearing S, Levin CA, Atlas
SJ. Psychometric evaluation of a decision quality instrument for treatment of lumbar
herniated disc. Spine. 2012; Mar 15. [Epub ahead of print].
Sepucha, KR, Fowler, FJ, Mulley AG. Policy support for patient-centered care: The needfor measurable improvements in decision quality. Health Affairs 2004.
http://content.healthaffairs.org/content/early/2004/10/07/hlthaff.var.54.full.pdf
Sepucha K, Ozanne E, Silvia K, Partridge A, Mulley A. An approach to measuring thequality of breast cancer decisions. Patient Education and Counseling. 2007; 65(2):261-9.
Sepucha K, Ozanne E. How to define and measure concordance between patients’preferences and medical treatments: A systematic review of approaches andrecommendations for standardization. Patient Education and Counseling. 2010;
Jan;78(1):12-23.
Sepucha K, Levin C, Uzogara E, Barry M, O’Connor A, Mulley A. Developing
instruments to measure the quality of decisions: Early results for a set of symptom-driven
decisions. Patient Education and Counseling. 2008; 73:504-510.
Sepucha K, Stacey D, Clay C, Chang Y, Cosenza C, Dervin G, Dorrwachter J,
Feibelmann S, Katz JN, Kearing S, Malchau H, Taljaard M, Tomek I, Tugwell P, Levin
C. Decision quality instrument for treatment of hip and knee osteoarthritis: a
psychometric evaluation. BMC Musculoskelet Disord. 2011; Jul 5;12(1):149.
Simon D, Loh A, Härter M. Measuring (shared) decision making -- a review ofpsychometric instruments. Zärtzl. Fortbild. Qual. Gesundh.wes . 2207; 101; 259-267.
16

Establishing the Effectiveness

Stacey D, Bennett CL, Barry MJ, Col NF, Eden KB, Holmes-Rovner M, LlewellynThomas H, Lyddiatt A, Légaré F, Thomson R. Decision aids for people facing healthtreatment or screening decisions. Cochrane Database of Systematic Reviews 2011, Issue
10. Art. No.: CD001431. DOI: 10.1002/14651858.CD001431.pub3.
http://decisionaid.ohri.ca/cochsystem.html
Thomson RG, Murtagh MJ, Khaw FM. Tensions in public health policy: patientengagement, evidence-based public health and health inequalities QSHC. 2005;14:398400
Thomson R, Eccles M, Steen N, Greenaway J, Stobbart L, Murtagh M, May C. A patientdecision aid to support shared decision making on antithrombotic treatment of patientswith atrial fibrillation: randomised controlled trial. QSHC. 2007; 16: 216-223
Tversky A, Kahneman D. Judgement under uncertainty: heuristics and biases. Science.
1974; 185: 1124-1131.
Vodermaier A, Caspari C, Koehm J , Kahlert S , Ditsch N, Untch M. Contextual factors
in shared decision making: a randomised controlled trial in women with a strong
suspicion of breast cancer. Br J Cancer. 2009; 100: 590-597
Von Neumann, J. and O. Morgenstern (1947). Theory of games and economic behavior.
New York, Wiley.

17

Establishing the Effectiveness
APPENDIX:
ORIGINAL CHAPTER L
Original Authors
Vikki Entwistle(co-lead)
Al Mulley(co-lead)Annette O’Connor

University of Aberdeen

UK

Harvard Medical School, Boston

USA

University of Ottawa

CanadaTheresa Marteau

King's College London Research Group

UK

Andrew Kennedy

Council on Health Research forDevelopment (COHRED)
Harvard Medical School, Boston

Switzerland

Karen Sepucha

USA

Original Rationale / Theory
There is a reasonable consensus that: a) patient decision aids aim to improve the qualityof decision making; and b) quality decisions are those that result in individuals choosing
and/or receiving the health care interventions that are most consistent with their informed
and considered values (Briss et al., 2004; O’Connor et al., 1997; O’Connor, Stacey et al.,
2003; Ratliff et al., 1999; Sepucha et al., 2004).
An assessment of the effectiveness of a patient decision aid should, therefore, comprises
evaluation of the extent to which it improves the proportion of patients who choose
and/or receive health care interventions that are consistent with their individual values.
There is, as yet, no approach or metric for measuring the congruence between an
individual’s values and the health care options they choose and/or receive.
Given the current state of methodological knowledge and experience, two strategies
might now be pursued to further the evaluation of the effectiveness of patient decision
aids:1) The development of methods and measures for assessing the primary endpoint
criterion (e.g., decisions that are consistent with the individual’s informed and considered
attitudes towards health states that might be affected by the decision; attitudes towards
the risks associated with the relevant options; willingness to make trade-offs over time;and position in relation to other value–relevant issues involved in the decision);
2) The assessment of process criteria that are likely to support the achievement of the
primary endpoint criterion.
18

Establishing the EffectivenessFirst we outline here one promising approach for measuring the primary endpoint
criterion. Then we identify some process criteria that are likely to support theachievement of the primary endpoint criterion.
An Approach For Measuring The Primary Endpoint Criterion
For most, if not all, of the decisions that patient decision aid developers seek to support, it
will be possible to identify a number of option attributes, risk considerations, time trade
off considerations, and other value-relevant issues that are most usually salient forpatients facing the decision of interest.
(This identification step would require a rigorous social process involving a grouprepresenting all relevant perspectives and sources of expertise relating to the particulardecision that the patient decision aid is intended to support).
Then the quality of decision-making could be reasonably estimated for a population ofpatients facing the decision. This could be done by gauging the (aggregate) proportion ofpatients for whom the health care option selected and/or received is consistent with theirattitudes towards the most usually salient attributes and considerations.
We recognize that there will probably always be some individuals for whom the mostusually salient option attributes and value considerations are not the ones on which theirdecision turns, because other attributes and value considerations are more important tothem. However, assuming all other things are equal, when populations of patients that did
and did not use a patient decision aid when facing the decision are compared, the odds
ratio of the aggregate proportion of patients for whom the option chosen and/or received
was consistent with their attitudes towards the most usually salient attributes andconsiderations should serve as a reasonable measure of the decision quality achieved withand without the patient decision aid.
Methodological development will be needed to refine both (a) the basic approach to thiskind of measurement, as well as (b) the decision-specific measures of the options chosen,
individuals’ knowledge about the most usually salient option attributes (in order to ensurethat patients’ attitudes are well informed), and individuals’ attitudes towards the mostusually salient option attributes and value considerations.
Process Criteria Likely to Support the Achievement of the Primary Endpoint Criterion
Process criteria that are likely to support the achievement of the primary endpoint
criterion include the knowledge assessment mentioned above and also the extent to which
patients: recognise there is a decision to be made; understand that no single option is bestfor everyone because patients have different values and preferences; appreciate that theirown goals, values, and preferences matter in the decision; and have reflected on anddiscussed their attitudes towards the most usually salient option attributes and value
19

Establishing the Effectivenessconsiderations with the practitioner(s) with whom they are making the decision.
Other Outcome Criteria of Interest
The use of patient decision aids in practice may affect a number of health care processes
and outcomes in addition to the ones mentioned above. The focus in this document on
decision quality as the primary endpoint criterion is not meant to imply that otheroutcomes are unimportant. Patients’ perceptions of the decision making process andconfidence in the decisions made are, for example, important aspects of the quality ofhealth care that may be affected by the use of patient decision aids. However, they are
not the best indicators of whether a patient decision aid has fulfilled its aim of improving
decision quality; for example, patients who are not well informed about their health care
options may express confidence in a particular choice even if it is not consistent with
their reported values.
Original Evidence
There is a reasonable consensus that (a) patient decision aids aim essentially to improve
the quality of decision making, and (b) good decisions are those that result in individuals
choosing and/or receiving the health care interventions that are most consistent with theirinformed and considered values.
For example, a national survey of the public endorsed the criteria of being informed
about options, outcomes and probabilities, having clear values, making a choice that iscongruent with values, and being satisfied with the choice (O’Connor, Drake et al.,2003). A survey of oncologists place the highest endorsement (>95%) on the patient
criteria of being clear about the values tradeoffs in the decision and being informed oftreatment alternatives, harms, and benefits (O’Connor et al., 1997).
Trials of patient decision aids also frequently assess these criteria. In 34 trials of
individual patient decision aids, 18 measured knowledge, 10 measured feeling clear about
values, and 3 measured agreement between values and choices (O’Connor, Stacey et al.,2003).
Original References
Bekker HL, Legare F, Stacey D, O'Connor A, Lemyre L. Is anxiety an appropriate measure ofdecision aid effectiveness: a systematic review? Patient Education and Counseling. 2003; 50:
255-262.
Briss P, Rimer B, Reilley B, Coates R, Lee N, Mullen P et al. Promoting informed decisions
about cancer screening in communities and healthcare systems. Am J Prev Med 2004;26:67-80.
O'Connor AM, Drake ER, Wells GA, Tugwell P, Laupacis A, Elmslie T. A survey of thedecision-making needs of Canadians faced with complex health decisions. Health Expectations
2003;6:97-109.
20

Establishing the EffectivenessO' Connor A, Llewellyn-Thomas H, Sawka C, Pinfold S, To T, Harrison D. Physicians' opinions
about decision aids for patients considering systemic adjuvant therapy for axillary-node negative
breast cancer. Patient Education & Counseling 1997;30:143-53.
O’Connor AM, Stacey D, Entwistle V, Llewellyn-Thomas H, Rovner D, Holmes-Rovner M, Tait
V, Tetroe J, Fiset V, Barry M, Jones J. DAs for people facing health treatment or screening
decisions (Cochrane Review). In: The Cochrane Library, Issue 2 2003. Oxford: Update Software.
http://www.cochrane.org/cochrane/revabstr/ab001431.htm.
Ratliff A, Angell M, Dow R, Kupperman M, Nease R, Fisher R et al. What is a good decision?
Effective Clinical Practice 1999;2:185-97.
Sepucha, KR, Fowler, FJ, Mulley AG. Policy support for patient-centered care: The need formeasurable improvements in decision quality. Health Affairs 2004.

21