Basing Information on
Comprehensive, Critically Appraised, and Up-to-Date Scientific Evidence

2012 UPDATED CHAPTER K:
BASING INFORMATION ON COMPREHENSIVE, CRITICALLY APPRAISED, AND
UP-TO-DATE SYNTHESES OF THE SCIENTIFIC EVIDENCE(Original Title: Basing Information on Up-to-Date Scientific Evidence)

SECTION 1:
AUTHORS/AFFILIATIONS
Victor Montori (lead)

Mayo Clinic, Rochester, Minnesota

USA

Annie LeBlanc (colead)Angela Buchholz

Mayo Clinic, Rochester, Minnesota

USA

University Medical Center Hamburg-Eppendorf, Hamburg

Germany

Diana Stilwell

Informed Medical Decisions Foundation, Boston

USA

Apostolos Tsapas

Aristotle University, Thessaloniki

GreeceNote
The original title for this dimension (“Basing Information on Up-to-Date Scientific Evidence”) is
now considered inadequate. The original title implied that the single criterion subsumed within
this quality dimension would be that, when a decision aid is presenting scientific evidence, only
recent publications needed to be cited. However, given the theoretical justification forpresenting scientific information (see Section 3, below), it is no longer sufficient to cite someevidence. Rather, the onus is on the decision aid developer to ensure the aid is based on thepertinent body of evidence, systematically identified and rigorously summarized.
Therefore, this updated dimension chapter has been re-titled, and presents entirely new sections
regarding definition, theoretical rationale, evidence base, and references.
The original dimension chapter is included here as an Appendix.

Suggested Citation:
Montori V, Leblanc A, Buchholz A, Stilwell D, Tsapas A. (2012). Basing information on comprehensive, critically appraised,and up-to-date syntheses of the scientific evidence. In Volk R & Llewellyn-Thomas H (editors). 2012 Update of the International
Patient Decision Aids Standards (IPDAS) Collaboration's Background Document. Chapter K. http://ipdas.ohri.ca/resources.html.

Basing Information on
Comprehensive, Critically Appraised, and Up-to-Date Scientific Evidence

SECTION 2:
CHAPTER SUMMARY
What is this quality dimension?
Decision aids should be based on comprehensive, critically appraised, and up-to-date scientificevidence. “Comprehensive” refers to the extent to which decision aid developers havethoroughly considered all the pertinent scientific evidence addressing each aspect they chose topresent in their decision aid. The best approach to do so is to use systematic reviews that haveavoided selection bias, carefully and reproducibly assessed the quality of the incorporated
reviewed studies (i.e., the studies’ protection from error and bias), summarized the estimatedpertinent effects (ideally quantitatively in a meta-analysis), indicated the extent to which theseestimates are trustworthy, and assessed the extent to which selective reporting and publication
bias may have corrupted this body of evidence. Decision aid developers should indicate—using
symbols, numbers, or phrases—the degree of confidence attributed to that information, given thequality of the scientific evidence upon which it is based. The dates when the relevant systematic
reviews were used by decision aid developers should be reported and should be sufficiently
recent given the pace of progress in the particular field. Developers should report a version date,and, when pertinent, a “better by” or “expiration date” to communicate to decision aid users
about the speed of evidence accrual in the field and the optimal timing of future updates.
What is the theoretical rationale for including this quality dimension?
Ethical (informed patient choice), quality (patient-centered care), and scientific (evidence-based
medicine) justifications make it imperative that decision aids be based on comprehensive,critically appraised, and up-to-date scientific evidence. To be completely and honestly informed
so that autonomy can be exercised requires patients to access unbiased information based on ahigh-quality synthesis of the available evidence. Patient-centered care requires patients toactively participate in decision making and to be provided with the information and support theyneed to make informed choices. Evidence-based medicine calls for recognizing both that
confident decision making requires confidence in the estimates of effect and that these estimatesalone cannot drive decisions. Therefore, the information presented in patient decision aids should
include not only evidence-based estimates of the effects of the various relevant options, but also
an indication of the extent to which these estimates can be trusted. This will help patients
consider this information in light of their context and values and preferences, in order to make adecision that fits their goals and circumstances.
What is the evidence to support including or excluding this quality dimension?
Half of current decision aids appear to use systematic reviews and another quarter use practiceguidelines, both of which are of variable quality. Very few used the GRADE system to describe
the quality of the evidence. About 5% offered an expiration date and an update policy.

2Basing Information on
Comprehensive, Critically Appraised, and Up-to-Date Scientific Evidence

SECTION 3:
DEFINITION (CONCEPTUAL/OPERATIONAL) OF THIS QUALITY DIMENSIONa)

Original Definition

No explicit definition of this quality dimension was provided in the original chapter.

b)

Updated Definition“Information”
As noted in Chapter B (“Providing Information about Options”), patient decision aids present
the individual patient with a summary of the patient’s current health situation, descriptions of the
protocols involved in the various relevant options, and descriptions – and the likelihoods - ofthose options’ effects on the outcomes of most importance to patients.
“Scientific Evidence”
In the context of patients’ decision aids, this term refers to a body of empirical observations
about the options and their consequences. These observations should be conducted using thescientific method, and will have, to different extents, protections against systematic and random
error.
“Comprehensive”
By comprehensive, we do not mean the scope and range of the information presented in thedecision aid. For this criterion, “comprehensive” refers to the extent to which decision aid
developers have thoroughly considered all the pertinent scientific evidence addressing each
aspect they chose to present in their decision aid.
“Critically Appraised”
This means that decision aid developers will use evidence derived from systematic reviews thata) avoid selection bias; b) carefully and reproducibly assess the quality of the incorporated
reviewed studies (i.e., the studies’ protection from error and bias); c) summarize the estimatedpertinent effects (ideally quantitatively in a meta-analysis); d) indicate the extent to which theseestimates are trustworthy; and e) assess the extent to which selective reporting and publication
bias may corrupt the body of evidence.
Then developers present this synthesized evidence in the decision aid itself. [For definitions,
theoretical rationales, evidence, and emerging issues about particular presentation formats, see
Chapters B (“Providing Information about Options”), C (“Presenting Probabilities”), E(“Using Personal Stories”), I (Balancing the Presentation of Options”), and J (“Addressing
Health Literacy”).]3Basing Information on
Comprehensive, Critically Appraised, and Up-to-Date Scientific Evidence

Whatever the presentation format, note that decision aid developers clearly state that the aid isoffering the “best available” synthesized information, and indicate—using symbols, numbers, orphrases—the degree of confidence attributed to that information, given the quality of thescientific evidence upon which it is based.
“Up-to-Date”
The dates when the relevant systematic reviews were searched for, compiled, criticallyappraised, and synthesized by decision aid developers are reported and are sufficiently recent
given the pace of progress in the particular field. This implies that decision aid developers should
develop a sense of the speed with which evidence that matters accrues in their area of work, andimplement an update policy. Developers should report a version date, and, when pertinent, a“better by” or “expiration date” to communicate to decision aid users about the speed ofevidence accrual in the field and the optimal timing of future updates.
c)

Emerging Issues/Research Areas in Definition

1. What are the duties of decision aid developers in relation to evidence accrual?
As stated, the new standard goes much beyond citation of published literature, the prior standard.
The new standard requires high-quality synthesis of the available evidence about the issues thedecision aid developers seek to present in their aid.
Decision aid developers could make use of existing syntheses they find to be of highmethodological quality and reasonably recent. This suggests that developers should be able tojudge the quality of these reviews. Some tools exist to assist with this work1. If high quality
syntheses were not available, then developers need to conduct their own reviews or commission
their conduct to credible parties. In these cases, decision aid users would need to critically review
the synthesis, which implies that the review must be fully reported and placed in the publicdomain and subject to peer review.
This process is similar to that followed by rigorous guideline developers and would require
methodological research to further understand how it affects the process and costs of decision aid
development and how it affects the aids’ credibility and value as a tool to translate this evidence
into practice. In particular, can and should decision aid developers prevent the dissemination ofcorrupt evidence through their decision aids?
2. What triggers an update?
Besides specifying a regular update policy for a decision aid, its developers also need to specify
a literature-monitoring policy, since, for example, clinical trials may occasionally generateactionable data, which, in turn, should trigger an update of the decision aid. Of note, ouremphasis on systematic syntheses of evidence requires that when a new data point emerges it besubject to synthesis along with the previous body of evidence. Thus new evidence should not
directly update the aid, but rather an updated synthesis should update the aid. How might
decision aid developers build these monitoring policies? What should trigger an update for new

4Basing Information on
Comprehensive, Critically Appraised, and Up-to-Date Scientific Evidence

evidence of benefit? Of harm? When is the evidence stable enough that updates can be further
spaced? Is this likely to change by area of work, by intervention, by outcome?

SECTION 4:
THEORETICAL RATIONALE FOR INCLUSION OF THIS QUALITY DIMENSIONa)

Original Theoretical Rationale

The original rationale for this quality dimension referred to patients’ and health professionals’
expectations that the information presented in a decision aid will be accurate, derived from highquality sources, directly applicable to the patients and practitioners using the decision aid,
tailored to individual patients’ characteristics, and regularly revised as new evidence
accumulates.
b)

Updated Theoretical Rationale

The updated rationale for basing information on comprehensive, critically appraised, and up-todate scientific evidence is basically consistent with the original rationale.
However, in this revised chapter we push the rationale further, by:i.)
Proposing several underlying justifications for the existence of these patients’ /professionals’ expectations in the first place; andii.)
Expanding on the approaches to rating the quality of evidence about the effects of atherapeutic option.
i. Underlying Justifications
Informed Patient Choice
One key principle driving the development of patient decision aids is the ethical argument for
informing patients about their health care choices. Respect for patient autonomy is a governing
principle of medical ethics2 and is generally understood to refer to an individual’s ability to makeand carry out informed health care decisions based on unbiased and thoughtful deliberation. The
American Board of Internal Medicine, the American College of Physicians, and the European
Federation of Internal Medicine state in their charter on medical professionalism that“[p]hysicians must be honest with their patients…and ensure that patients are completely andhonestly informed before the patient has consented to treatment and after treatment has
occurred.”3 To be ‘completely and honestly informed’ so that autonomy can be exercised
requires access to unbiased information that is based on a high-quality synthesis of the available
evidence that is relevant to the patient’s clinical situation and acknowledges where uncertainty
exists because of the quality or quantity of that evidence. When used as part of a shared decision
making process, high-quality patient decision aids that are based on the best available clinicalevidence support clinicians in fulfilling their ethical obligation to promote autonomy by ensuringthat patients are fully informed about their health care choices.

5Basing Information on
Comprehensive, Critically Appraised, and Up-to-Date Scientific Evidence

Patient-Centered Care
The growing emphasis on patient-centered care is another driving force behind the developmentof patient decision aids. Principles of patient-centered care require that patients actively
participate in decision making and be provided with the information and support they need tomake informed choices. Work by the Picker Institute and others has identified respect for thepatient’s values and preferences and access to clear, high-quality information and education to beamong the important characteristics of patient-centered care.4 Patient decision aids support thepractice of patient-centered care by ensuring that patients’ preferences are informed and based on
accurate expectations; this requires in turn that the tools be based in high-quality evidence that,when possible, is relevant to patients’ individual risk profiles.
Evidence-Based Medicine
Over the last 20 years, evidence-based medicine (EBM) has strongly influenced the practice of
medicine5 as it had required use of the best available evidence alongside clinical expertise toformulate recommendations to patients that were pertinent to their context and sensitive to theirvalues and preferences. To this extent, EBM follows two principles.
The first principle recognizes that not all observations and experiments are similarly protectedfrom random and systematic error. The degree of protection from error offers confidence in theestimates of effect. Therefore, in the interests of fostering EBM, the information presented inpatients’ decision aids should include not only evidence-based estimates of the effects of thevarious relevant options, but also an indication of the extent to which this evidence is protectedfrom error (i.e., is trustworthy).
The second principle holds that the evidence alone is never sufficient to fully inform a clinicaldecision. The evidence will be considered during the transactions between the treating clinician
with a particular level of expertise and the patient with unique goals, values, and preferences;
furthermore, the evidence will be applied in a particular biological, psychological, andsociocultural context. Therefore, in the interests of fostering EBM, the information presented inpatients’ decision aids needs to be directly applicable to the patients and practitioners using thedecision aid, and tailored to individual patients’ characteristics.
Ethical (informed patient choice), quality (patient-centered care), and scientific (evidence-based
medicine) justifications make it imperative that decision aids be based on comprehensive,critically appraised, and up-to-date scientific evidence.
ii. Some Approaches to Rating Quality of Evidence
It is important to note that decision aid developers may want to include information based on
different forms of evidence. Natural history and prognostic information, for instance, often
requires the developer to use large and long observational studies. Adverse effects, particularly
those that are rare, may be better characterized in case reports, and their linkage to exposuresascertained through case-control studies. The original chapter echoed the notion of a hierarchy ofevidence about each of these questions and considered rating approaches6. Since then, an
important international consensus emerged in the guidelines movement about how to assess the6Basing Information on
Comprehensive, Critically Appraised, and Up-to-Date Scientific Evidence

quality of evidence in support of a recommended course of action: the GRADE approach. While
we will now focus on this approach, we recognize that other approaches to grading the evidence
exist, but they have substantial shortcomings that this method obviates. Furthermore, consistency
in rating evidence from guideline to decision aid may facilitate the development of decision aids
in conjunction with clinical practice guidelines efforts.
The GRADE Approach
GRADE (Grading of Recommendations Assessment, Development and Evaluation) is the mostcomprehensive approach developed for the purposes of formulating clinical practice guidelines7.
It could be particularly helpful for decision aid developers, as an approach to explaining topatients the extent to which one can have confidence in the pertinent estimates of an option’s
effects.
We will summarize the GRADE approach to grading evidence of effectiveness here, butdevelopers should review the extensive published guidance (www.gradeworkinggroup.org).
A key feature of this approach is that the assessment of quality applies to the body of evidence(not just to the individual study). This includes: a) the likelihood of bias (from the absence ofprotective features such as concealed randomization, blinding of pertinent groups, and analysis
of participants as randomized)8; b) the likelihood of reporting or publication bias9, leading toinconsistency in results across studies10; c) any imprecision in the estimates of effect (e.g., wide
confidence intervals)11; and d) the degree of indirectness (in which the results do not directly
apply to the pertinent patients, the comparisons are inadequate, or the studies measure asurrogate of limited validity)12.
Randomized trials start as high quality evidence and “lose points” as they accumulate problems.
Observational studies start as low quality evidence, and “lose points” in a similar manner, but
could rise in scored quality when they show strong evidence of a dose-response relationship or
when the magnitude of the association of interest is very large13. Importantly, the quality ofevidence supporting the association of treatment with outcomes often differs by outcome. Higher
quality evidence may exist for benefits than for harms, for example.
c.

Emerging Issues/Research Areas in Theoretical Rationale

1. Relevant Information
In the original chapter for this dimension, the authors paid attention to ensuring that the evidence
used to inform a decision was pertinent to the patients who are the intended audience of the tool.
The extent to which evidence is pertinent is subject to much judgment. Most subgroup analysis
yield findings with limited trustworthiness (e.g., subgroup effects that cannot be confirmed or arevery imprecise), such that it is sometimes advisable to use estimates from the general
population14. The challenge of applying evidence from somewhat different patients,
interventions, or outcomes to the situation of interest falls under the general rubric ofindirectness12. The degree with which these differences are likely to make a difference, theextent of indirectness, reduces the confidence that the estimates of effect are correct and this
could be reflected in the decision aid. Efforts to improve the volume and quality of comparative7

Basing Information on
Comprehensive, Critically Appraised, and Up-to-Date Scientific Evidence

effectiveness research may enhance the evidence base for decision aids, because this researchrequires direct comparisons that matter to clinical stakeholders – that is, measuring the effect ofinterventions on patient important outcomes.
2. Tailored Estimates
Decision aids often need to present absolute risk estimates that relate to the individual
characteristics of the patient of interest (see Chapter C, “Presenting Probabilities”). Prognostic
calculators offer the opportunity to tailor such risk estimates. Given the importance of theseestimates, decision aid developers must report which calculator they are using and provide anassessment of its validity, a matter that often requires independent evaluation from the
population from which the formula was derived, as well as comparisons with competing risk
estimators.15
3. Will Linking Information to Evidence-Based Practice Guidelines Foster Decision Aid
Uptake?
A key issue with decision aids is their underuse in practice, despite increasing evidence of theireffectiveness and surging policy support in some regions. Thus, efforts to explicitly link theirdesign and content to clinical policy and workflow may facilitate their adoption. This supports inpart our approach of linking the development and content of decision aids to state-of-the-art
approaches to the development and content of evidence-based practice guidelines.
The “ideal” situation indicating the potential usefulness of a patient decision aid is one in which
there is high-quality evidence linking options to outcomes, but the options are closely matched,
and the choice of the “best” course of action will depend mostly on the patient’s values andpreferences. Guideline developers following GRADE usually will offer a weak or conditionalsuggestion in those circumstances. A suggestion based on high-quality evidence would require
the incorporation of patient values and preferences for implementation, that is, would benefit
from a decision aid. This linkage between guidelines and decision aids may affect development
and uptake of decision aids. An obvious advantage from using the same approach for grading thequality of evidence in guidelines and in decision aids is that decision aid developers can use theevidence supporting current high-quality guidelines as their source for up-to-date evidence.
Furthermore, implementation of the guidelines, i.e., quality improvement efforts, could then belinked to the implementation of decision aids for weak or conditional suggestions.

SECTION 5:
EVIDENCE BASE UNDERLYING THIS QUALITY DIMENSIONa) Original Evidence Base
The original evidence base about providing up-to-date scientific evidence in patients’ decision
aids cited the 2003 Cochrane Collaboration’s review of 29 patient decision aids tested in 34
RCTs in which patients were facing actual choices16 .

8Basing Information on
Comprehensive, Critically Appraised, and Up-to-Date Scientific Evidenceb) Updated Evidence BaseTo evaluate the current practice of including evidence in decision aids, we examined the Ottawa
Decision Aid Inventory. Out of 257 decision aids included in the inventory, 134 provided
references to scientific evidence used, when they were last updated (N=135), and whether they
were available on the Internet (N=134). In a random sample of those decision aids (N = 20),
stratified to correct for differences between providers, ten aids used a high quality systematic
review/meta-analysis (AMSTAR score range 2-11 on a scale of 1-11), and five aids were basedon practice guidelines (AGREE II score range 1-6 on a scale 1-7). Two decision aids used datafrom multiple sources of original research, whereas two used only a narrative review or an
expert’s opinion or a single piece of original research as source for evidence. Four decision aids
did not explicitly cite, hence we could not locate and evaluate, the evidence they used. One of the
20 aids explicitly stated en expiration date and an update policy, whereas eight refer to a policy
statement of the complete contents of the provider-website (as part of a site notice on the website
but not as part of the aid itself). Three used the GRADE system in their presentation to clarify
the quality of the evidence to the user.
c. Emerging Issues/Research Areas in the Evidence Base
1. How Often Should Systematic Reviews be Up-dated?
The frequency with which systematic reviews should be updated – and by extension products
derived from these – is still subject to research question. A comprehensive technical review
commissioned by the Agency for Healthcare Research and Quality published in 200717 foundthat the median time for the emergence of a signal that a review should be updated (e.g.,
substantial new evidence of effectiveness or harm, new alternatives, revelations about the natureof the old evidence) was 5.5 years; yet about 25% of the reviews could benefit from updating
within 2 years of publication. The authors were not able to identify predictors of more urgent
review and suggested yearly surveillance of systematic reviews.
2. Effects of Quality & Up-Dating on Use in PracticeGiven that the uptake in practice of decision aids remains limited, there is no strong evidence
that decision aids supported by high quality and updated summaries of the body of evidence
would be more likely to be taken up in practice. There is no a priori reason to believe that thequality of the process of evidence synthesis and their adaptation into evidence-based decision
aids will lead to better decisional processes and outcomes.
3. When There are Discrepancies
A key aspect worthy of surveillance is the fate of decision aids developed based on the best
available evidence, but which disagree in their presentation with extant guidelines and with thequality-of-care parameters derived from these guidelines. These discrepancies can appear
because decision aids may use more recently updated summaries of evidence, or because therecommendations apply to a different context or resulted from panels with distorted values (e.g.,

9Basing Information on
Comprehensive, Critically Appraised, and Up-to-Date Scientific Evidence

pharmaceutical lobbying). In one example, this apparent divergence led to difficulties in the useof the decision aid and to nonuse18.
4. Communicating Quality of Evidence
Communicating the confidence decision aids developers have around the estimates of thebenefits and harms associated with the selected options would help decision aids users makesense of the magnitude and trustworthiness of the estimates. The communication of thisconfidence in the evidence should be done in a way that is simple and understandable, yetremains precise, without adding cognitive burden on decision aid users.
There is limited empirical work on how to communicate quality of evidence to stakeholders ingeneral, let alone decision aids users. A systematic review of information in decision aids
reported that very few decision aids developers had addressed or incorporated quality ofevidence in their tool.19 The ways by which these developers were representing quality of theevidence varied greatly, from icons (i.e., bronze-gold medals), to verbal labels (i.e., high,moderate, low), and numeric intervals, and have yielded mixed results.19
The GRADE approach for guidelines offers a simple way to report on quality of evidence, using
categories20 21. An obvious advantage gained from using the same approach for grading thequality of evidence in guidelines and in decision aids is that decision aid developers could use
the evidence supporting current high-quality guidelines as their source for up-to-date evidence.
REFERENCES
1. Shea BJ, Hamel C, Wells GA, Bouter LM, Kristjansson E, Grimshaw J, et al. AMSTAR is areliable and valid measurement tool to assess the methodological quality of systematic
reviews. Journal of Clinical Epidemiology 2009;62(10):1013-20.
2. Beauchamp TL, Childress J. Principles of Biomedical Ethics. Oxford: Oxford University
Press, 2009.
3. Brennan TA. Editorial: renewing professionalism in medicine: the physician charter. Spine
2002;27(19):2087.
4. Gerteis M, Edgman-Levitan S, Daley J, Delbanco T. Through the patient's eyes. San
Francisco, CA: Jossey-Bass, 1993.
5. Montori VM, Guyatt GH. Progress in evidence-based medicine. Jama 2008;300(15):1814-6.
6. Atkins D, Best D, Briss PA, Eccles M, Falck-Ytter Y, Flottorp S, et al. Grading quality ofevidence and strength of recommendations. BMJ 2004;328(7454):1490.
7. Guyatt GH, Oxman AD, Vist GE, Kunz R, Falck-Ytter Y, Alonso-Coello P, et al. GRADE: anemerging consensus on rating quality of evidence and strength of recommendations. BMJ
2008;336(7650):924-6.

10

Basing Information on
Comprehensive, Critically Appraised, and Up-to-Date Scientific Evidence

8. Guyatt GH, Oxman AD, Vist G, Kunz R, Brozek J, Alonso-Coello P, et al. GRADE
guidelines: 4. Rating the quality of evidence--study limitations (risk of bias). Journal of
Clinical Epidemiology 2011;64(4):407-15.
9. Guyatt GH, Oxman AD, Montori V, Vist G, Kunz R, Brozek J, et al. GRADE guidelines: 5.
Rating the quality of evidence--publication bias. Journal of Clinical Epidemiology
2011;64(12):1277-82.
10. Guyatt GH, Oxman AD, Kunz R, Woodcock J, Brozek J, Helfand M, et al. GRADE
guidelines: 7. Rating the quality of evidence--inconsistency. Journal of Clinical
Epidemiology 2011;64(12):1294-302.
11. Guyatt GH, Oxman AD, Kunz R, Brozek J, Alonso-Coello P, Rind D, et al. GRADE
guidelines 6. Rating the quality of evidence--imprecision. Journal of Clinical
Epidemiology 2011;64(12):1283-93.
12. Guyatt GH, Oxman AD, Kunz R, Woodcock J, Brozek J, Helfand M, et al. GRADE
guidelines: 8. Rating the quality of evidence--indirectness. Journal of Clinical
Epidemiology 2011;64(12):1303-10.
13. Guyatt GH, Oxman AD, Sultan S, Glasziou P, Akl EA, Alonso-Coello P, et al. GRADE
guidelines: 9. Rating up the quality of evidence. Journal of Clinical Epidemiology
2011;64(12):1311-6.
14. Sun X, Briel M, Walter SD, Guyatt GH. Is a subgroup effect believable? Updating criteria toevaluate the credibility of subgroup analyses. BMJ 2010;340:c117.
15. Chamnan P, Simmons RK, Sharp SJ, Griffin SJ, Wareham NJ. Cardiovascular risk
assessment scores for people with diabetes: a systematic review. Diabetologia
2009;52(10):2001-14.
16. O'Connor AM, Stacey D, Entwistle V, Llewellyn-Thomas H, Rovner D, Holmes-Rovner M,
et al. Decision aids for people facing health treatment or screening decisions. Cochrane
Database of Systematic Reviews 2003(2):CD001431.
17. Shojania KG, Sampson M, Ansari MT, Ji J, Garritty C, Rader T, et al. Updating Systematic
Reviews. Rockville (MD), 2007.
18. Abadie R, Weymiller AJ, Tilburt J, Shah ND, Charles C, Gafni A, et al. Clinician's use of theStatin Choice decision aid in patients with diabetes: a videographic study nested in arandomized trial. Journal of Evaluation in Clinical Practice 2009;15(3):492-7.
19. Feldman-Stewart D, Brennenstuhl S, McIssac K, Austoker J, Charvet A, Hewitson P, et al. A
systematic review of information in decision aids. Health Expectations : An International
Journal Of Public Participation In Health Care And Health Policy 2007;10(1):46-61.

11

Basing Information on
Comprehensive, Critically Appraised, and Up-to-Date Scientific Evidence

20. Akl EA, Maroun N, Guyatt G, Oxman AD, Alonso-Coello P, Vist GE, et al. Symbols weresuperior to numbers for presenting strength of recommendations to health careconsumers: a randomized trial. Journal of Clinical Epidemiology 2007;60(12):1298-305.
21. Guyatt G, Oxman AD, Sultan S, Brozek J, Glasziou P, Alonso-Coello P, et al. GRADE
guidelines 11-making an overall rating of confidence in effect estimates for a single
outcome and for all outcomes. Journal of Clinical Epidemiology 2012.

12

Basing Information on
Comprehensive, Critically Appraised, and Up-to-Date Scientific Evidence

APPENDIX:
ORIGINAL CHAPTER K
Original Authors
Tim Whelan (lead)

McMaster University

CA

Michael Pignone

University of North Carolina

US

Original Rationale
Use of Accurate Information
There is an expectation by patients and health professionals that patient decision aids are basedon the most accurate information available. For a particular clinical decision, the patient decision
aid should use the best available evidence to describe all the relevant health care options andtheir associated outcomes (benefits and risks). Ideally, the patient decision aid’s evidence should
be based on data from a systematic review(s), in which the published and unpublished literature
is systematically searched for the highest quality studies, then summarized, if applicable, using
meta-analyses. Normally, practice guidelines will carefully outline the quality of evidence of thebenefits of different health care options. Ideally, the harms associated with the health care
options described in the patient decision aid should also be supported by the highest quality ofevidence available. On occasion, patient decision aid developers will have to perform their own
systematic reviews to adequately identify the best available option/outcome evidence. In thissituation, details of these reviews should be described in the patient decision aid’s backgroundmaterial.
Source for Accurate Information
The quality of the information used in the patient decision aid has implications for the level ofcertainty placed on that information; patients have a right to know the limitations of the evidence
supporting the effectiveness of different options. The best available option/outcome evidence
should be characterized in terms of its quality, so that users (patients and practitioners) canappreciate the level of uncertainty regarding the likelihood that a particular screening/treatment
option causes a particular therapeutic/harmful outcome. There are many rating systems for
quality of evidence, ranging from high grade (usually a metaanalysis or randomized trial) torelatively low grade (often a cohort study or a case series) (Oxman et al., 2005). For example,
data from a non randomized comparison cohort study regarding the efficacy of treatment is likely
to be less certain than data from a large randomized trial.
For studies of treatments, the highest quality of evidence regarding effectiveness comes fromrigorous meta-analyses or large randomized trials.

13Basing Information on
Comprehensive, Critically Appraised, and Up-to-Date Scientific Evidence

Observational studies (e.g. cohort studies or case control studies) are less likely to provide
high quality evidence about the relationship between a particular screening/treatment option
and a therapeutic effect.

Descriptive studies, including case series, provide lower quality evidence about therelationship between an option and an outcome.

With respect to the incidence of adverse effects, observational and descriptive studies may
provide as good or better evidence than data from randomized trials.

Relevance of Evidence
Information describing the therapeutic effectiveness of different treatment options in the patientdecision aid should be directly applicable to the patients and practitioners who use it. This
pertains not only to the population who will use the instrument, but also, in particular, to theintervention being described. For example, a patient decision aid describing the benefits andharms of adjuvant chemotherapy in women with breast cancer should be based on data fromrandomized trials or systematic reviews involving patients of similar age and stage of disease as
the women who will be using the patient decision aid.
Tailored Information to Individual Characteristics
When outcome probabilities are tailored to the clinical risk for different patient populations,evidence for the different risk groups or the risk assessment tool employed in the patient decision
aid should be provided. Such evidence might include data from those studies and any secondary
validation studies.
Regular Updating of Evidence
Information supporting different health care options can quickly become out-dated. Ideally,
evidence supporting the information contained in the patient decision aid should be regularly
updated at least every two to three years and preferably on a yearly basis. The process used forthe update (i.e. whether a systemic review was performed) should be described. By providing astatement regarding the update policy, patients and practitioners can form their own opinions
about the degree of confidence to be placed in the patient decision aid’s information.
Original Evidence
RCTs Involving Patients Facing Actual Choices
Of 29 individual patient decision aids, evaluated in 34 RCTs included in the Cochrane review, 19
were available for review of their content (O’Connor et al., 2003). Of these, 13 of 19 (68%) of DAs provided specific citation either within the patient decision aid (3 of13) or in a separate resource (10 of 13).
 5 of 19 (26%) described the quality of the evidence with 3 patient decision aids that were14

Basing Information on
Comprehensive, Critically Appraised, and Up-to-Date Scientific Evidence

based on clinical practice guidelines
3 of 19 (16%) described the uncertainty in the evidence presented within the patient decisionaid.

Two studies described how different risk groups used in the patient decision aid were identified.
Original References
O’Connor AM, Stacey D, Entwistle V, Llewellyn-Thomas H, Rovner D, Holmes-Rovner M, Tait V,
Tetroe J, Fiset V, Barry M, Jones J. DAs for people facing health treatment or screening decisions(Cochrane Review). In: The Cochrane Library, Issue 2 2003. Oxford: Update Software.
http://www.cochrane.org/cochrane/revabstr/ab001431.htm.
Oxman A, et al. and the GRADE Working Group. Grading evidence and formulating recommendations.
Submitted for publication.

15