Using a Systematic Development Process
2012 UPDATED CHAPTER A:
USING A SYSTEMATIC DEVELOPMENT PROCESSSECTION 1:
AUTHORS/AFFILIATIONSAngela Coulter (lead) Informed Medical Decisions Foundation andUniversity of Oxford
Jennifer
University of Saskatchewan
Kryworuchko
Patricia Dolan Mullen University of Texas Health Science Center at Houston

UK

USA

Chirk-Jenn Ng

University of Malaysia

Malaysia

Diana Stilwell

Informed Medical Decisions Foundation

USA

Trudy van der
Weijden

University of Maastricht

The Netherlands

CanadaSECTION 2:
CHAPTER SUMMARY
What is this dimension?
While authors differ in the emphasis they place on particular aspects, key features common to all
patient decision aid (PtDA) development processes include scoping and design, development of a
prototype, ‘alpha’ testing with patients and clinicians in an iterative process, ‘beta’ testing in ‘real
life’ conditions (field tests), and production of a final version for use and/or further evaluation.
What is the theoretical rationale for including this dimension?
It is important that PtDAs are carefully developed, user-tested, and open to scrutiny, with a welldocumented and systematically applied development process. Users require assurance that thedevelopment process has been carried out to acceptable standards. Because poor quality decision
aids have the potential to cause harm to patients and cause doubt to be cast on the concept of shareddecision making more broadly, it is essential that those not involved in developing them areprovided with sufficient documentation to check the validity and reliability of their development
processes.
What is the evidence to support including or excluding this dimension?
To date, only about half of patients’ decision aids appear to have been field tested with patients, andeven fewer had been reviewed or tested by clinicians not involved in the development process. Veryfew described a distribution strategy, and surprisingly few (17%) described a method for reviewing
and synthesizing the clinical evidence. In many cases it was difficult to gauge from the trial reports
whether a development process along the lines recommended in IPDAS had been followed or not.
However, it is important to bear in mind that most of the PtDAs described in the RCTs included in
the Cochrane Collaboration’s review were developed before the publication of the IPDAS criteria.
Suggested Citation:
Coulter A, Kryworuchko J, Mullen P, Ng CJ, Stilwell D, van der Weijden T. (2012). Using a systematic development process. In
Volk R & Llewellyn-Thomas H (editors). 2012 Update of the International Patient Decision Aids Standards (IPDAS)
Collaboration's Background Document. Chapter A. http://ipdas.ohri.ca/resources.html.

Using a Systematic Development Process

SECTION 3:
DEFINITION (CONCEPTUAL/OPERATIONAL) OF THIS QUALITY DIMENSIONa)

Updated Definition

Patient decision aids (PtDAs) aim to provide evidence-based information to help people make healthdecisions. They are usually developed when there is more than one reasonable treatment, prevention,screening, or care management option. They take a variety of forms, including leaflets or booklets,
computer programs, DVDs, or interactive tools for use online or in the clinic. Some include
extensive information and filmed interviews to illustrate options and outcome probabilities, whileothers use decision analytic tools to elicit reactions to specific features and trade-offs. Some use
face-to-face discussions or educational methods in combination with written material. All aim topresent outcomes, risks, and uncertainties in a clear, comprehensible, scientifically valid andunbiased manner to help people make personally relevant value-based health decisions.
The PtDA development process has been described in a number of trial reports and associatedarticles (see below). In addition, several organizations developing suites of decision aids forcommercial use have described their development processes. While different authors tend to place
greater or lesser emphasis on particular aspects, key features common to all include scoping anddesign, development of a prototype, ‘alpha’ testing with patients and clinicians in an iterative
process, ‘beta’ testing in ‘real life’ conditions (field tests), and production of a final version for use
and/or further evaluation (see model development process below, in Figure 1).
The process is often overseen by a multidisciplinary steering group that includes patient andclinician representatives and other relevant stakeholders. Further detail on each of the elements is
included below in Table 1.

2

Using a Systematic Development Process
Figure 1: Model Development Process for Decision Aids
Scoping: Define scope andpurpose of decision aid andtarget audience

Steering 1: Assemble
steering group, including
clinical experts and patients

Design 1: Assesspatients’ views on
decisional needs

Design 2: Assess
clinicians’ views on
patients’ needs

Design 3: Determine
format anddistribution plan

Design 4: Review andsynthesize evidence

Prototype: Draft decisionaid, incl. storyboard, script,
design, etc.

Alpha testing 1: Withpatients to checkcomprehensibility andacceptability

Steering 2: Review bysteering group; redraft andredesign if necessary

Beta testing 1: Field
test with patients toassess usability

Alpha testing 2: Withclinicians to checkacceptability andfeasibility

Beta testing 2: Field
test with clinicians toassess usability

Steering 3: Steering group
reviews field test results
and finalizes decision aid

3

Using a Systematic Development Process
Table 1: Key Elements of Decision Aid Development Process1
Element
Scoping

Definition

Methods

Comments

Describe health condition or
problem; state the decisionthat needs to be considered;specify target audience

Developer advised by multidisciplinary steering group, ideallyinvolving topic experts, clinicians,and patients

Explicit statement should be included in
PtDA.

Likely to be informed by specifictheoretical approach which may be
explicit or implicit.

All decision aid RCTs in Cochrane review
included a description of the purpose andscope of the aid they were evaluating.
Theoretical framework often unstated.

Steering group

A team of stakeholders who
advise on the development,
evaluation, andimplementation of the PtDA

Steering group members will haverelevant expertise in decision making
for the specific topic: patient
representatives, clinicians, patient
educators, shared decision making
expertise, policy makers.

Members should be familiar with and/or
sympathetic to the concept of SDM.

Design 1 and 2 –Assess decisional
needs [see Chapter B
for further details]

Elicit patients’ andclinicians’ views on patients’
information and decision
support needs

Focus groups

Few recommendations regarding optimal
approaches to assessing decisional needs.

Stakeholder interviews
Surveys
Systematic literature review(including qualitative andquantitative studies)

RCTs included in Cochrane review
reported patients’ perspective on
decisional needs more frequently thanclinicians’ perspective (43% vs 15%).

Direct observation

1

Collated from a review of papers reporting trials that were included in the latest Cochrane Collaboration’s review of decision aids (Stacey et al., 2011) plus associated papers describing 84 decision aids
evaluated in randomized controlled trials (RCTs).

4

Using a Systematic Development Process

Design 3 – Determine
format and distribution
plan [see Chapter H
for further details]

Includes choice of media andformat of decision aid,
setting, timing of
introduction into patient
pathway, how and when
decision aid will be
distributed to patients and/or
clinicians

Formats may include print media,
audio recordings, DVDs, videos,
websites, computer programs,
decision boards, face-to-face
discussions, group education, andany combination of these.

Should be considered early in thedevelopment process.

Distribution methods include
handing out in clinic, mailing,
telephone coaching, or direct-topatient via websites or other means.

Less than a third (31%) of RCTs in
Cochrane review included a descriptionof how the decision aid would be
distributed and used in routine clinicalpractice.

Settings include primary care,secondary care, health coaches,
community
Design 4 – Review
Summary of clinicaland synthesize
evidence relevant to theevidence [see Chapters decision and options
C, I, and K for further
details]

Some RCTs report complex methods
that may not be suitable for widespreaduse.

Comprehensive literature search
Frameworks provide little guidance onwith emphasis on systematic reviews selection of relevant outcomes, how to(when available).
minimize bias, address potential
financial conflicts of interest, reach
The evidence may include empirical consensus, or deal with poor-quality or
studies of patients’ experience
inadequate evidence.
and/or preferences.
How clinical evidence was appraisedUse quality criteria to assess clinicaland selected for inclusion in the decision
practice guidelines when these are
aid was reported in only a small minority
used as evidence source.
of RCTs included in Cochrane review(17%).
It may be more efficient to develop
PtDAs alongside clinical practiceguidelines, since they draw on thesame evidence base.

5

Using a Systematic Development Process

Prototypedevelopment

Draft decision aid, including
storyboard, script, graphics,
web design, video, etc.

Ranges from basic to highlysophisticated.

This aspect of the development process
is rarely reported in any detail.

Alpha testing

Direct feedback from‘typical’ users sought during
the development process.
This may include membersof the steering group andothers involved in thedevelopment process.

Review by key stakeholders(patients, clinicians) via focus
groups, cognitive interviews, direct
observation, usability, andacceptability testing.

Specific methods, observations, andresults often not reported.

Testing with patients (andoccasionally providers) in‘real-world’ settings toassess feasibility.

Small-scale observational pilot
studies often precede larger
randomized controlled trials.

Beta (field) testing

Feedback may be sought at various
stages in an iterative process.

Review and field testing should be
carried out with patients andclinicians who have not been
involved in the development
process.
Offering clinicians the opportunity
to review and comment on thematerials may be essential if they are
to be persuaded to recommend the
PtDAs to their patients.

6

RCTs included in Cochrane review
reported patient testing more frequently
than clinician testing (37% vs 21%).

Field testing often focuses on use of tool
in settings that may not reflect ‘realworld’ use; provider reactions notroutinely assessed.
RCTs included in Cochrane review
reported results of field tests withpatients more often than clinicians’
reactions to decision aid (51% vs 19%).

Using a Systematic Development Processb)

Changes from Original Definition

The new definition includes all the elements of the original, but they have been renamed andreordered to clarify the different phases of the development process. The description of PtDA design
has been expanded to include consideration of format and distribution plans, and we have also added
a section on prototype development. Our aim was to provide a clearer overview of the entire
development process. Where appropriate, we have included pointers to other sections in the updated
IPDAS document where more detailed descriptions will be found.
c) Emerging Issues/Research Areas in Definition








Optimal methods for determining decisional needs require further development and testing.
There may be much to be learnt from development processes and quality standards in relatedareas; examples include general information materials, user-centred web design, or guidelines
for evaluating complex interventions (McGee, 2010; Cato, 2001; Craig et al, 2008).
Comparison of PtDAs and clinical practice guidelines and the scope for developing these inparallel should be evaluated.
There is no consensus on how to select material for inclusion in decision aids. Studies should
evaluate how much information patients want/need and how much detail is required.
Studies should compare the effectiveness of different PtDA formats and delivery mechanisms.
More guidance is needed to inform PtDA alpha- and beta- tests, including user-centred design
methods, acceptability, usability, and feasibility testing. The process of designing the PtDA
remains rather subjective. When does the iteration stop? What is saturation?
There is little evidence on the relative importance of each of the above-mentioned features of thedevelopment process. Most have emerged from practical experience supported by consensus, but
we do not claim they are evidence-based.

SECTION 4:
THEORETICAL RATIONALE FOR INCLUDING THIS QUALITY DIMENSIONa) Updated Theoretical Rationale
It is important that PtDAs are carefully developed, user-tested, and open to scrutiny, with awell-documented and systematically applied development process. Some decision aids havebeen designed for one-off use in studies to advance knowledge, while others are intended for
wider use in a range of real-life clinical settings. Some have been developed by academics,
some by clinicians, some by voluntary organizations, and some by commercial companies.
Whatever their provenance or purpose, users require assurance that the development process
has been carried out to acceptable standards. Poor quality decision aids have the potential tocause harm to patients and they could also cast doubt on the concept of shared decision
making more broadly, so it is essential that those not involved in developing them areprovided with sufficient documentation to check the validity and reliability of theirdevelopment processes.

7

Using a Systematic Development Processb) Changes to Original Rationale
Wording changes only.
c) Emerging Issues/Research Areas in RationaleThere is a need to assess the strengths and limitations of different theoretical frameworks andtheir usefulness or otherwise in designing and implementing PtDAs.



Better understanding of the barriers and facilitators to adoption of shared decision making
and the needs of the various stakeholders will be essential to ensure successful development
and implementation of high quality, useful, and relevant decision aids.



There are efforts under way to develop systems for certifying patient decision aids in boththe US and the UK, for example, as part of regulations supporting widespread use of certified
PtDAs in routine practice. Any accreditation scheme will require a set of agreed-uponstandards and careful documentation of the processes by which the PtDA was developed.
These schemes should be monitored and evaluated to see if they help to raise standards indecision aid development.

SECTION 5:
EVIDENCE BASE UNDERLYING THIS QUALITY DIMENSIONa)

Updated Evidence Base

Despite the proliferation of patient decision aids, detailed information on the processes by which
they are developed is limited. Some groups that have developed multiple PtDAs have proposed
guidelines for their development (Elwyn et al., 2011; O’Connor et al., 1998; Raats et al., 2008), or
described insights generated by a particular approach (Montori et al., 2007). The following is a brief
overview of selected approaches to decision aid development.
Ottawa Decision Support Framework: O’Connor (1998) is among the earliest authors to describe
the development of a PtDA, and the Ottawa Decision Support Framework (ODSF) guided thedevelopment of at least 22 of the PtDAs included in the Cochrane Collaboration review. Based on
expectancy value, decisional conflict, and social support theories, the framework includes three key
elements: 1) assessment of determinants of decisions (both patients’ and providers’); 2) provision ofdecision support interventions to prepare the patient and provider to make and implement a decision;and 3) evaluation of the success of the interventions at improving the quality and outcomes of thedecision process. Additional detail is provided to define determinants of decisions, such associodemographic and clinical characteristics; patients’ and providers’ perceptions of the decisionand of what important others think about the decision; and resources (both personal and external)
available to make the decision. The authors note that the goals of decision support are to address
modifiable and suboptimal decision determinants, such as inadequate knowledge, unrealistic
expectations, unwanted pressure, and inadequate support, and they encourage the use of tailoredoutcome probabilities, detailed descriptions of benefits and risks, and information on the opinions
and perspectives of others (both clinicians and patients) on the decision.
8

Using a Systematic Development Process
Using the example of a decision aid aimed at helping women decide about use of postmenopausal
hormone therapy, O’Connor (1998) outlines an iterative development process involving the researchteam and panels of patients and experts, with the PtDA content based on clinical guidelines, andstructured guidance in clarifying values and implementing a decision is provided by a personal
worksheet.
Commentary: The Ottawa Framework is particularly relevant to ‘preference-sensitive’ decisions,which involve careful deliberation and consideration of tradeoffs among options. Perhaps because
the PtDA developed by O’Connor was based on an existing high-quality clinical guideline, theframework provides little advice for how developers should review and synthesize the relevant
clinical evidence. The framework also does not address how developers might deal with conflicts ofinterest, achieving consensus on the evidence, maintaining the PtDA content over time, or
implementation outside research settings.
Cardiff University: Based on their experience developing three web-based PtDAs over a sevenyear period, Elwyn and colleagues (2010) proposed a development process for web-based decision
support interventions. This systematic “process map” includes 3 main steps: 1) content specification,with an emphasis on ensuring that patients’ perspectives on the proposed options are sought andincluded in addition to synthesis of the scientific evidence; 2) design, including storyboarding, an
iterative phase of trial and experimentation called “sandpit” testing, and usability testing; and 3)
field testing with patients facing the decision and clinicians who are interacting with them. The
process calls for key documentation, including: a protocol document that explains the decision andhighlights the rationale for developing a PtDA; evidence synthesis based on systematic reviews orcomprehensive literature searches; storyboard; and technical specification document to guide thewebsite development. Their projects were overseen by a project management group, which retained
editorial control, and included involvement at all steps by key stakeholders including clinicians,patients, and policymakers. Unique challenges faced by developers of web-based tools arehighlighted, including decisions regarding navigation (free vs. mandated) and use of interactivity(audio, video, gaming, avatars, etc.) in ways that add value and enhance ease of use yet avoid overengineering. The authors find little evidence to inform best practices in these areas.
Commentary: The process outlined by Elwyn and colleagues is widely applicable across a range ofsituations for which decision support interventions may be developed (i.e., screening, treatment, etc.)
and a variety of media (although some of the concepts included, such as storyboarding, are adaptedfrom film production). However, as the authors acknowledge, the process is time-consuming andcostly: three PtDAs developed using this process each took two to three years to develop and test.
Insights gained from early efforts could be generalized to create templates to allow more efficient,less costly future development. The process outlined does not offer recommendations regarding
conflict of interest, processes for achieving consensus on the evidence, maintenance of PtDA content
over time, or implementation outside research settings.
Dutch Institute for Healthcare Improvement: Researchers at the Dutch Institute for Healthcare
Improvement reported on the development over a 12-month period of 6 decision aids based on
existing evidence-based clinical guidelines (Raats, 2008). Citing the Ottawa Decision Support
Framework (O’Connor, 1998) and the IPDAS standards, the authors followed four key steps: 1)
establishment of criteria and selection of topics; 2) assessment of patients’ information needs via
literature review and focus groups; 3) drafting of the aid, including iterative review by a9

Using a Systematic Development Process
multidisciplinary working group of health professionals, guideline developers, decision-making
experts, and patients, and with reference to existing aids on the topic; and 4) endorsement of the aid
and establishment of ownership and responsibility for the maintenance and updating of both thesupporting guideline and the decision aid itself. With regard to implementation, the authors call forconcomitant development and coordinated release of clinical practice guidelines and accompanying
PtDAs that support their application, and for identifying and acknowledging early in the guideline
development process any so-called ‘grey zones’ of uncertainty regarding patient preferences.
Commentary: The outlined process appears efficient and scalable when high-quality evidence-based
practice guidelines are available. However, the authors acknowledge that additional research isneeded to evaluate the effect of the aids in practice within the Dutch health care system. The
evidence synthesis step used by other developers is addressed by use of national evidence-based
guidelines; the authors note that these should also meet internationally accepted quality criteria (e.g.,
AGREE, 2003). The resulting tools include a values-clarification method, but the process by which
the method was selected and ‘populated’ with non-directive, standardized questions is not defined.
Mayo Clinic: Montori and colleagues describe insights gleaned from the pragmatic process they
followed to develop the Statin Choice decision aid for patients with diabetes (Montori, 2007), which
was evaluated in an RCT included in the Cochrane Collaboration review. In particular, the authors
highlight how deliberate observations of how patients and clinicians make decisions during officevisits, and of early prototypes in use during patient-provider encounters can inform the ultimate
format, design, and content of the final PtDA. Similar to other developers, their experience
reinforces the importance of flexibility, iteration, and involvement of patients and clinicians
throughout the process.
Commentary: This article does not recommend a particular development process, but rather offers
insights unique to the approach their research group chose. The direct observation methodology that
Montori and colleagues describe may complement more traditional needs assessment approaches for
informing developers about what patients and physicians need from a PtDA. Observing early
prototypes in the setting they are being designed for can also be an important step in ensuring thatthe intervention will work as intended and have the desired effect on the decision making process.
These insights also highlight the importance of flexibility during the early stages of design anddevelopment.
Informed Medical Decisions Foundation: Developers that produce PtDAs for both research
purposes and for public distribution, such as the Informed Medical Decisions Foundation (IMDF),
provide details of their development process on their organization’s website. Ten of the RCTs
included in the most recent Cochrane Collaboration review were conducted using PtDAs developedthat used this process. IMDF lists the following elements in their PtDA development process: 1)
involvement of healthcare providers representing key clinical specialties, overseen by a clinician
who divests him or herself of any potential financial conflicts of interest; 2) involvement of patientsat several stages, including needs assessment via focus groups and literature reviews; and 3) review
and evaluation of PtDA drafts by providers and patients prior to their release for general use. The
approach outlines processes for evidence review and synthesis, disclosure of funding source andconflicts of interest, and periodic review and updates.
Healthwise: Healthwise has almost certainly developed more PtDAs than any other organization.
They provide details of their tools and development process on request, but their PtDAs have not
10

Using a Systematic Development Process
been evaluated in published trials. Their materials are developed by multi-disciplinary project teams
using a four-step process: planning, research, writing, and review, following guidelines that ensure
the content is accurate, easy-to-use, and easy to understand (K. Baker, personal communication).
Ongoing surveillance and updating is a key step in the process, and hundreds of patients provided
input during a recent redesign of the organization’s decision aid portfolio. However, few details areavailable regarding how the developers select and synthesize the relevant clinical evidence or deal
with conflicts of interest.
Review Methods for this Update
We reviewed papers describing 84 decision aids that had been evaluated in RCTs and included in thelatest update of the Cochrane Collaboration review of decision aids (Stacey et al., 2011). Our review
focused on descriptions of the development process. These were assessed against a speciallydesigned checklist that included the original IPDAS description of a systematic development process
as well as additional components drawn from the various processes described above and illustrated
in our model. Since many trial reports included only cursory descriptions of decision aid
development, we also reviewed linked papers cited in the trial reports, of which the following werethe most useful (Kasper et al., 1992; Barry et al., 1995; Spunt et al., 1996; Sawka et al., 1998; Green& Fost, 1997; O’Connor et al., 1999; Kasper et al., 2006; Breslin et al., 2008; Nassar et al., 2007;
Vandemheen et al., 2010; Whelan et al., 1995; Whelan et al., 1999).
As indicated above, only about half of the PtDAs appear to have been field tested with patients, andeven fewer had been reviewed or tested by clinicians not involved in the development process. Veryfew described a distribution strategy, and surprisingly few (17%) described a method for reviewing
and synthesizing the clinical evidence. In many cases it was difficult to gauge from the trial reports
whether a development process along the lines recommended in IPDAS had been followed or not.
However, it is important to bear in mind that most of the PtDAs described in the RCTs included in
the Cochrane Collaboration’s review were developed before the publication of the IPDAS criteria.
An analysis of data from the iCoCo study carried out for the purposes of this review showed that
PtDAs that scored highly on having a systematic development process also scored highly against
other IPDAS criteria. However, there was no evidence from the small sub-sample of studies in
iCoCo to support the hypothesis that a systematic development process results in a demonstrably
better gain when using PtDAs in randomized trial conditions (Joseph-Williams, 2011).
b)

Changes from Original Evidence BaseThis section has been completely rewritten. The original described evidence on patients’ informationneeds which is now covered in Chapter B.
c)Emerging Issues/Research Areas In Evidence BaseSee above.

11

Using a Systematic Development Process

REFERENCES
AGREE 2003. Development and validation of an international appraisal instrument for assessing
the quality of clinical practice guidelines: the AGREE project. Quality and Safety in HealthCare, 12, 18-23.
BARRY, M. J., FOWLER, F. J., MULLEY, A. G., HENDERSON, J. V. & WENNBERG, J. E.
1995. Patient reactions to a program designed to facilitate patient participation in treatment
decisions for benign prostatic hyperplasia. Medical Care, 33, 771-782.
BRESLIN, M., MULLAN, R. J. & MONTORI, V. M. 2008. The design of a decision aid about
diabetes medications for use during the consultation with patients with type 2 diabetes.
Patient Educ Couns, 73, 465-72.
CATO, J. 2001. User-Centered Web Design, London, Pearson Education.
CRAIG, P., DIEPPE, P., MACINTYRE, S., MICHIE, S., NAZARETH, I. & PETTICREW, M.
2008. Developing and evaluating complex interventions: the new Medical Research Council
guidance. BMJ, 337.
ELWYN, G., KREUWEL, I., DURAND, M. A., SIVELL, S., JOSEPH-WILLIAMS, N., EVANS,
R. & EDWARDS, A. 2011. How to develop web-based decision support interventions forpatients: a process map. Patient Educ Couns, 82, 260-5.
GREEN, M. J. & FOST, N. 1997. An interactive computer program for educating and counseling
patients about genetic susceptibility to breast cancer. J Cancer Educ, 12, 204-8.
JOSEPH-WILLIAMS, N., NEWCOMBE, R., POLITI, M., DURAND, M-A., SIVELL, S.,
STACEY, D., O'CONNOR, A., VOLK, R. J., EDWARDS, A., BENNETT, C., PIGNONE,
M., THOMSON, R., ELWYN, G. 2011. Toward minimum standards for certifying patientdecision aids: a correlation analysis and modified Delphi consensus process. J Clin Epid, 2nd
submission 16-12-11.
KASPER, J., KOPKE, S., MUHLHAUSER, I. & HEESEN, C. 2006. Evidence-based patient
information about treatment of multiple sclerosis--a phase one study on comprehension andemotional responses. Patient Educ Couns, 62, 56-63.
KASPER, J. F., MULLEY, A. G., JR. & WENNBERG, J. E. 1992. Developing shared decisionmaking programs to improve the quality of health care. QRB Qual Rev Bull, 18, 183-90.
MCGEE, J. 2010. Toolkit for making written material clear and effective. US Department of Healthand Human Services, Center for Medicare and Medicaid Services.
MONTORI, V. M., BRESLIN, M., MALESKA, M. & WEYMILLER, A. J. 2007. Creating a
conversation: insights from the development of a decision aid. PLoS Med, 4, e233.

12

Using a Systematic Development Process
NASSAR, N., ROBERTS, C. L., RAYNES-GREENOW, C. H. & BARRATT, A. 2007.
Development and pilot-testing of a decision aid for women with a breech-presenting baby.
Midwifery, 23, 38-47.
O'CONNOR, A. M., DRAKE, E. R., FISET, V., GRAHAM, I. D., LAUPACIS, A. & TUGWELL,
P. 1999. The Ottawa patient decision aids. Eff Clin Pract, 2, 163-70.
O'CONNOR, A. M., TUGWELL, P., WELLS, G. A., ELMSLIE, T., JOLLY, E.,
HOLLINGWORTH, G., MCPHERSON, R., BUNN, H., GRAHAM, I. & DRAKE, E. 1998.
A decision aid for women considering hormone therapy after menopause: decision support
framework and evaluation. Patient Education and Counseling, 33, 267-279.
RAATS, C. J., VAN VEENENDAAL, H., VERSLUIJS, M. M. & BURGERS, J. S. 2008. A generic
tool for development of decision aids based on clinical practice guidelines. Patient EducCouns, 73, 413-7.
SAWKA, C. A., GOEL, V., MAHUT, C. A., TAYLOR, G. A., THIEL, E. C., O'CONNOR, A. M.,
ACKERMAN, I., BURT, J. H. & GORT, E. H. 1998. Development of a patient decision aid
for choice of surgical treatment for breast cancer. Health Expect., 1, 23-36.
SPUNT, B. S., DEYO, R. A., TAYLOR, V. M., LEEK, K. M., GOLDBERG, H. I. & MULLEY, A.
G. 1996. An interactive videodisc program for low back pain patients. Health Educ Res, 11,535-41.
STACEY, D., BENNETT, C. L., BARRY, M. J., COL, N. F., EDEN, K. B., HOLMES-ROVNER,
M., LLEWELLYN-THOMAS, H., LYDDIATT, A., LEGARE, F. & THOMSON, R. 2011.
Decision aids for people facing health treatment or screening decisions. Cochrane Database
Syst Rev, 10, CD001431.
VANDEMHEEN, K. L., AARON, S. D., POIRIER, C., TULLIS, E. & O'CONNOR, A. 2010.
Development of a decision aid for adult cystic fibrosis patients considering referral for lungtransplantation. Prog Transplant, 20, 81-7.
WHELAN, T., LEVINE, M., GAFNI, A., SANDERS, K., WILLAN, A., MIRSKY, D.,SCHNIDER, D., MCCREADY, D., REID, S., KOBYLECKY, A. & REED, K. 1999.
Mastectomy or lumpectomy? Helping women make informed choices. J Clin Oncol, 17,1727-35.
WHELAN, T. J., LEVINE, M. N., GAFNI, A., LUKKA, H., MOHIDE, E. A., PATEL, M. &STREINER, D. L. 1995. Breast irradiation postlumpectomy: development and evaluation ofa decision instrument. Journal of Clinical Oncology, 13, 847-853.

13

Using a Systematic Development Process

APPENDIX:
ORIGINAL CHAPTER A
Original Authors
Aileen Clarke (lead)

Queen Mary, University of LondonUK

M J Jacobsen

University of Ottawa

CanadaMary Ann O’Connor

Dartmouth Medical School, Dartmouth College

USA

Diana Stilwell

Foundation for Informed Medical Decision Making

USA

Deb Feldman-Stewart

Queen’s University, Kingston

CanadaOriginal Rationale/Theory
Patient decision aids are meant to support informed values-based decision making. They areusually developed when there is more than one reasonable option and there is considerable
variation in how patients value the features of different options. Practitioners and patients may
find it challenging to arrive at a good decision without advance preparation using a patientdecision aid that helps patients understand the options and clarify the personal value of theirdifferent features.
Patient decision aids take considerable effort to develop, and can have an important effect ondecision quality and the use of health services. Therefore, it is important that they are developed
using a systematic and replicable process.
Specific developmental steps common to many patient decision aids (O’Connor et al., 2003;
Bekker et al., 1999) include:
Assessing Decisional Needs
Groups with relevant perspectives and expertise are assembled and engaged in a rigorous social
process to analyze:
 The characteristics of the decision such as: all potentially relevant health care options; protocols
involved in each option; evidence regarding outcomes, probabilities, and variation in patients.’
values for different features of options; sensitivity of the decision to variation in values andprobabilities; and other characteristics such as the burden of condition and costs.
 Patients’ information needs. Although information needs vary widely from one
patient to the next, in general all patients require information that includes the natural course of
the condition, the procedures involved in the treatments or tests, the potential consequences, theirseverity, and their likelihoods of occurring (e.g. Feldman-Stewart, Brundage & Van Manen,2004).
14

Using a Systematic Development Process

Patients’ decisional needs such as: current perceptions of options; salience of outcomes,
probabilities, and values in decision making; the degree of difficulty making the decision andfactors contributing to that difficulty; usual and preferred decision making roles; decisional
barriers and facilitators; feasibility and local attitudes regarding the use of patient decision aids.

Formation of Groups to Develop And Review Patient Decision Aids
Patient decision aid developers usually include experts in clinical care, evidence-based decision
making, patient education, and patient experience. Patient decision aid reviewers usually include
potential users such as patients who are experienced with the decision and the practitioners whocounsel them about the decision.
Drafting, Reviewing, And Revising
The elements included in patient decision aids (e.g. information about the condition, options, andoutcomes; values clarification; examples of others’ experiences with decision making; and guidance
in decision making and communication) are described elsewhere. Through an iterative process, apatient decision aid is drafted, reviewed, and revised until it is ready for field testing. At this stage,
part of the review may include acceptability questionnaires eliciting, for example: reviewers’
perceptions of the appropriateness and amount of information; ability to help patients decide what ismost important to them; appropriate length; balanced presentation of options and outcomes; ability
to hold their interest; ability to help them understand the various patient roles in decision making;and usefulness for decision making.
Field Testing
Field testing is conducted with patients at the point of decision making. The objectives
focus on feasibility, acceptability to users, potential to improve knowledge, and potential
to clarify personal values regarding the features of options.
External Peer Review
The patient decision aid undergoes critical appraisal by those who were not involved in its
development and evaluation.
Original Evidence
Patients’ Information Needs
Of the 14 screening patient decision aids verified by the Cochrane Review (O’Connor et al., 2003),
10 developers described how they arrived at the content of their aids. Of these 10, 5 (50%) consulted
their respective patient populations about their information needs using interviews (with individual
patients and/or focus groups) or through surveys. Of these 5 populations, 20% wanted information
about the health condition, 83% on the “no test” option, 83% on the test procedure(s), 80% on therisks of the procedures, 80% on the rates of true/false positives and of true/false negatives, 67% on
the potential consequences of a positive test result, and 40% on the potential consequences of anegative test result.

15

Using a Systematic Development Process
Of the 45 treatment patient decision aids verified by the Cochrane Review (O’Connor etal., 2003), 32 developers described how they arrived at the content of their aid. Of these32, 21 (66%) developers consulted their respective patient populations about theirinformation needs, using interviews (either with individuals and/or focus groups) or
surveys. Of these 21 populations, 96% wanted information about the health condition,
81% on the multiple options, 50% on the ‘no treatment’ option, 100% on the treatment
procedures, 100% on the potential benefits, and 100% on the potential risks of the various
treatment options.
RCTs Involving Patients Facing Actual Choices
Of the 29 individual patient decision aids, evaluated in the 34 RCTs included in the Cochrane
Review, 19 were available for review (O’Connor et al., 2003). Of these: 89% (17 of 19) listed the credentials of developers; and 58% (11 of 19) reported a published or easily accessible description of the development
process.
Overall, there is limited evidence about how these development processes affect decisionquality.
Original References
Bekker H. Thornton JG. Airey CM. Connelly JB. Hewison J. Robinson MB. Lilleyman J. MacIntosh M.
Maule AJ. Informed Decision Making an Annotated Bibliography and Systematic Review. Health Tech.
Assess. 1999;3(1):1-156.
Feldman-Stewart D, Brundage MD, Van Manen L & Svenson O. Patient-focussed decision-making in earlystage prostate cancer: Insights from a cognitively based decision aid. Health Expectations 2004; 7(2):126141.
O'Connor AM, Stacey D, Entwistle V, Llewellyn-Thomas H, Rovner D, Holmes-Rovner M, Tait V,
Tetroe J, Fiset V, Barry M, Jones J. Decision aids for people facing health treatment or screening decisions[Cochrane Review]. In: The Cochrane Library, 2003: Issue 2. Oxford: Update Software.

16