APPRAISAL OF GUIDELINESFOR RESEARCH & EVALUATION II

INSTRUMENT___________________________________________________________________________________The AGREE Next Steps Consortium
May 2009

UPDATE: December 2017
	
  
	
  
	
  
	
  
	
  
	
  

	
  	
  	
  	
  
	
  

	
  

COPYRIGHT AND REPRODUCTION
This document is the product of an international collaboration.
It may be reproduced and used for educational purposes,
quality assurance programmes and critical appraisal of
guidelines. It may not be used for commercial
purposes or product marketing. Only approved non-English
language versions of the AGREE II Instrument should be used,
where available. Offers of assistance in translation into otherlanguages are welcome, provided they conform to theprotocol set out by The AGREE Research Trust.
DISCLAIMER
The AGREE II Instrument is a generic tool designed primarily tohelp guideline developers and users assess themethodological quality of guidelines.
The authors do not take responsibility for the improper use of
the AGREE II Instrument.
© The AGREE Research Trust, May 2009.
© The AGREE Research Trust, September 2013.
© AGREE Enterprise December 2017.
SUGGESTED CITATION FOR AGREE II PUBLICATION
Brouwers MC, Kho ME, Browman GP, Burgers JS, Cluzeau F, Feder G, Fervers B, Graham ID, Grimshaw J, Hanna
SE, Littlejohns P, Makarski J, Zitzelsberger L, for the AGREE Next Steps Consortium. AGREE II: Advancing guideline
development, reporting and evaluation in healthcare. CMAJ 2010;182:E839-842.
SUGGESTED CITATION FOR AGREE II PDF VERSIONAGREE Next Steps Consortium (2017). The AGREE II Instrument [Electronic version]. Retrieved <Month, Day,
Year>, from http://www.agreetrust.org.
FUNDING
The development of the AGREE II Instrument was funded by agrant from the Canadian Institutes of Health Research (FRN77822)

FOR FURTHER INFORMATION ABOUT THE AGREE II, PLEASE CONTACT
AGREE Project Office, agree@mcmaster.ca
AGREE Website, www.agreetrust.org 	
  

	
  
	
  

	
  
	
  

ii	
  

	
  

	
  
	
  

AGREE NEXT STEPS CONSORTIUM MEMBERSHIP
Dr. Melissa C. Brouwers
Principal Investigator, AGREE Next Steps Consortium
McMaster University, Hamilton, Ontario, Canada
Consortium Members:
Dr. G.P. Browman, Victoria, British Columbia, Canada
Dr. J.S. Burgers, Dutch College of General Practitioners, Utrecht, The Netherlands
Dr. F. Cluzeau, Global Health and Development Group, Imperial College London, UK
Dr. D. Davis, Association of American Medical Colleges, Washington DC, USA
Dr. G. Feder, University of Bristol, UK
Dr. B. Fervers, Cancer et Environement, Centre Léon Bérard, France
Dr. I. Graham, Ottawa Hospital Research Institute, Ottawa, Ontario, Canada
Dr. J. Grimshaw, Ottawa Hospital Research Institute, Ottawa, Ontario, Canada
Dr. S.E. Hanna, McMaster University, Hamilton, Ontario, Canada
Ms. M.E. Kho, McMaster University, Hamilton, Ontario Canada
Dr. P. Littlejohns, Kings College London, UK
Ms. J. Makarski, Hamilton, Ontario, Canada
Dr. L. Zitzelsberger, Quebec, Canada

	
  
	
  
	
  

AGREE	  II	  VERSIONS	
  &	
  UPDATES	  AGREE	  II	  Original	
  Public	
  Release	
  and	
  Publication	
  Date:	
  2009/2010	
  AGREE	  II	  Update:	
  September	
  2013	
  AGREE	  II	  Update:	
  December	
  2017	  	
  What’s	  new	  in	
  the	  December	
  2017	
  update?	
  	
  
The	
  August	
  2017	
  update	
  includes	
  revisions	
  to	
  the	
  following	
  sections	
  of	
  the	  Introduction:	
  ‘AGREE	  Website:	  Resources	
  and	
  References’,	  ‘10	
  Years	
  of	
  AGREE’,	
  and	
  ‘Scoring	
  the	
  AGREE	
  II’.	
  Guidance	
  has	  been	  added	  regarding	
  the	
  use	  of	
  AGREE	
  II	
  score	  thresholds	
  to	
  distinguish	
  between	  higher	
  and	
  lower	
  quality	
  guidelines.	
  In	
  addition,	
  minor	
  editorial	
  changes	
  have	  been	  made	  throughout	
  the	  Introduction	  and	
  User’s	
  Manual.	
  The	  content	
  of	
  the	
  AGREE	
  II	
  Instrument	
  itself	
  has	  not	  been	  modified	
  since	  2009	  and	  all	  versions	
  of	
  the	
  AGREE	
  II	
  remain	
  valid	
  for	
  use.	
  	
  	
  
	
  

	
  
	
  

iii	
  

	
  
	
  

	
  

TABLE OF CONTENTS
I. INTRODUCTION
I.

Overview........................................................................................................................ 1

II.

Applying the AGREE II

III.

AGREE Website: Resources and References .............................................................. 3IV.

10 Years of AGREE....................................................................................................... 4........................................................................................... 3

II. USER’S MANUAL: INSTRUCTIONS FOR USING THE AGREE II ......................................... 6
I.

Preparing to Use the AGREE II ..................................................................................... 7

II.

Structure and Content of the AGREE II ......................................................................... 7

III.

Rating Scale and User’s Manual Sections .................................................................... 8

IV.

Scoring the AGREE II .................................................................................................... 9V.

Overall Assessment..................................................................................................... 10

VI.

Guidance for Rating Each Item
a. Domain 1. Scope and Purpose .............................................................................. 11
b. Domain 2. Stakeholder Involvement ...................................................................... 15
c. Domain 3. Rigour of Development ........................................................................ 19
d. Domain 4. Clarity of Presentation .......................................................................... 28
e. Domain 5. Applicability .......................................................................................... 32f.

Domain 6. Editorial Independence ........................................................................ 37

g. Overall Guideline Assessment .............................................................................. 40
III. AGREE II INSTRUMENT ...................................................................................................... 42

	
  
	
  

I.

Domain 1. Scope and Purpose ................................................................................... 43

II.

Domain 2. Stakeholder Involvement ........................................................................... 44

III.

Domain 3. Rigour of Development .............................................................................. 45

IV.

Domain 4. Clarity of Presentation ................................................................................ 48

V.

Domain 5. Applicability ................................................................................................ 49

VI.

Domain 6. Editorial Independence .............................................................................. 51

VII.

Overall Guideline Assessment .................................................................................... 52

iv	
  

	
  
	
  

I. INTRODUCTION
I. OVERVIEWi) Purpose of the AGREE II Instrument
Clinical practice guidelines (‘guidelines’) are systematically developed statements to assist
practitioner and patient decisions about appropriate health care for specific clinicalcircumstances (1). In addition, guidelines can play an important role in health policy formation(2,3) and have evolved to cover topics across the health care continuum (e.g., health promotion,screening, diagnosis).
The potential benefits of guidelines are only as good as the quality of the guidelines themselves.
Appropriate methodologies and rigorous strategies in the guideline development process areimportant for the successful implementation of the resulting recommendations (4-6). The qualityof guidelines can be extremely variable and some often fall short of basic standards (7-9).
The Appraisal of Guidelines for REsearch & Evaluation (AGREE) Instrument (10) was
developed to address the issue of variability in guideline quality. To that end, the AGREE
instrument is a tool that assesses the methodological rigour and transparency in which a
guideline is developed. The original AGREE instrument was refined, which resulted in the
AGREE II, and a User’s Manual was developed (11-13).
The purpose of the AGREE II, is to provide a framework to:
1. Assess the quality of guidelines;
2. Provide a methodological strategy for the development of guidelines; and3. Inform what information and how information ought to be reported in guidelines.
The AGREE II replaces the original instrument as the preferred tool and can be used as part ofan overall quality mandate aimed to improve health care.
ii) History of the AGREE Project
The original AGREE Instrument was published in 2003 by a group of international guideline
developers and researchers, the AGREE Collaboration (10). The objective of the Collaboration
was to develop a tool to assess the quality of guidelines. The AGREE Collaboration definedquality of guidelines as the confidence that the potential biases of guideline development havebeen addressed adequately and that the recommendations are both internally and externallyvalid, and are feasible for practice (10). The assessment includes judgments about the methods
used for developing the guidelines, the components of the final recommendations, and thefactors that are linked to their uptake. The result of the Collaboration’s effort was the original
AGREE Instrument, a 23-item tool comprising 6 quality domains. The AGREE Instrument has
been translated into many languages, has been cited in over 600 publications, and is endorsed
by several health care organizations. More details about the original instrument and related
publications are available on the AGREE Website (http://www.agreetrust.org/).
As with any new assessment tool, it was recognized that ongoing development was required tostrengthen the measurement properties of the instrument and to ensure its usability andfeasibility among intended users. This led several members of the original team to form the

	
  
	
  

0	
  

	
  
	
  

AGREE Next Steps Consortium (Consortium). The objectives of the Consortium were to further
improve the measurement properties of the instrument, including its reliability and validity; torefine the instrument’s items to better meet the needs of the intended users; and to improve thesupporting documentation (i.e., original training manual and user’s guide) to facilitate the abilityof users to implement the instrument with confidence.
The result of these efforts is the AGREE II, which is comprised of the new User’s Manual and 23
item tool organized into the same six domains, described here. The User’s Manual is asignificant modification of the original training manual and user’s guide and provides explicit
information for each of the 23 items. Table 1 compares the items of the original AGREE to theitems in the AGREE II. 	
  Table 1. Comparison of the original AGREE and AGREE II items.
Original AGREE Item

AGREE II Item

Domain 1. Scope and Purpose
1.

The overall objective(s) of the guideline is (are)specifically described.

No change

2.

The clinical question(s) covered by the guideline is (are)specifically described.

The health question(s) covered by the guideline is (are)specifically described.

3.

The patients to whom the guideline is meant to apply
are specifically described.

The population (patients, public, etc.) to whom the guideline ismeant to apply is specifically described.

Domain 2. Stakeholder Involvement
4.

The guideline development group includes individuals
from all the relevant professional groups.

No change

5.

The patients’ views and preferences have been sought.

The views and preferences of the target population (patients,
public, etc.) have been sought.

6.

The target users of the guideline are clearly defined.

No change

7.

The guideline has been piloted among end users.

Delete item. Incorporated into user guide description of item
19.

Domain 3. Rigour of Development
8.

Systematic methods were used to search for evidence.

No change in item. Renumbered to 7.

9.

The criteria for selecting the evidence are clearly
described.

No change in item. Renumbered to 8.
NEW Item 9. The strengths and limitations of the body ofevidence are clearly described.

10. The methods for formulating the recommendations areclearly described.

No change

11. The health benefits, side effects, and risks have been
considered in formulating the recommendations.

No change

12. There is an explicit link between the recommendations
and the supporting evidence.

No change

13. The guideline has been externally reviewed by experts

No change

	
  
	
  

1	
  

	
  
	
  

Original AGREE Item

AGREE II Item

prior to its publication.
14. A procedure for updating the guideline is provided.

No change

Domain 4. Clarity of Presentation
15. The recommendations are specific and unambiguous.

No change

16. The different options for management of the condition
are clearly presented.

The different options for management of the condition or healthissue are clearly presented.

17. Key recommendations are easily identifiable.

No change

Domain 5. Applicability
18. The guideline is supported with tools for application.

The guideline provides advice and/or tools on how therecommendations can be put into practice.
AND Change in domain (from Clarity of Presentation) AND
renumbered to 19

19. The potential organizational barriers in applying therecommendations have been discussed.

The guideline describes facilitators and barriers to its
application.
AND change in order – renumbered to 18

20. The potential cost implications of applying therecommendations have been considered.

The potential resource implications of applying therecommendations have been considered.

21. The guideline presents key review criteria for monitoring
and/ or audit purposes.

The guideline presents monitoring and/ or auditing criteria.

Domain 6. Editorial Independence
22. The guideline is editorially independent from the fundingbody.

The views of the funding body have not influenced the content
of the guideline.

23. Conflicts of interest of guideline development members
have been recorded.

Competing interests of guideline development group members
have been recorded and addressed.

	
  

	
  
	
  

2	
  

	
  
	
  

II. APPLYING THE AGREE IIi) Which guidelines can be appraised with the AGREE II?
As with the original instrument, the AGREE II is designed to assess guidelines developed by
local, regional, national or international groups or affiliated governmental organizations. These
include original versions of and updates of existing guidelines.
The AGREE II is generic and can be applied to guidelines in any health or disease areatargeting any step in the health care continuum, including those for health promotion, publichealth, screening, diagnosis, treatment or interventions. It is suitable for guidelines presented in
paper or electronic format. The AGREE II has not been designed to assess the quality of
guidance documents that address health care organizational issues. Its role in the assessmentof health technology assessments has not been formally evaluated.
ii) Who can use the AGREE II?
The AGREE II is intended to be used by the following stakeholder groups:
•
•

•
•

by health care providers who wish to undertake their own assessment of a guideline
before adopting its recommendations into their practice;by guideline developers to follow a structured and rigorous development methodology,
to conduct an internal assessment to ensure that their guidelines are sound, or toevaluate guidelines from other groups for potential adaptation to their own context;
by policy makers to help them decide which guidelines could be recommended for use
in practice or to inform policy decisions; andby educators to help enhance critical appraisal skills amongst health professionals andto teach core competencies in guideline development and reporting.

III. AGREE WEBSITE: RESOURCES AND REFERENCES
The AGREE Enterprise website, www.agreetrust.org, contains a variety of tools to assist users
in applying the AGREE II.
i) Publications of AGREE Research• Access publications related to the AGREE II and other AGREE tools.
• The Key publications: AGREE II page provides access to publications related to thedevelopment and testing of the AGREE II.
ii) AGREE II Training Tools• Two online tools are available to train new users of the AGREE II:o AGREE II Overview Tutorial,
o AGREE II Practice Exercise.
iii) AGREE II Language Translations• The AGREE II has been translated into various languages, thanks to members of theinternational practice guideline community.
• Copies of these translations are available to the public on this webpage.
• If you would like to undertake a new translation, please contact the AGREE Project
Office by emailing agree@mcmaster.ca.

	
  
	
  

3	
  

	
  
	
  

iv) My AGREE PLUS
• An online platform called My AGREE PLUS is freely available to the public to complete
and track AGREE II appraisals.
• The platform can be used to:o Complete individual AGREE II appraisals,o Contribute to a group AGREE II appraisal, ando Coordinate a group AGREE II appraisal.
• Click the My AGREE PLUS tab at www.agreetrust.org to register and use the platform.
v) Other AGREE tools• Access other AGREE tools to support the development, reporting and appraisal ofclinical practice guidelines and health systems guidance:o AGREE Reporting Checklist: A checklist based on the AGREE II to guide thereporting of clinical practice guidelines (14).
o AGREE GRS: A 4-item tool to assess the quality of clinical practice guidelines
when scarce time or resources make it unfeasible to use the morecomprehensive AGREE II.
o AGREE Recommendations Excellence (AGREE-REX): A tool to assess thequality and direct the development and reporting of clinical practice guideline
recommendations.
o AGREE Health Systems (AGREE-HS): A tool to assess the quality and direct
the development and reporting of health systems guidance documents.
o CheckUp: A checklist to guide the reporting of updated clinical practiceguidelines (15).

IV. 10 Years of AGREE
In 2013, the AGREE Enterprise marked its 10th anniversary since the original AGREE
Instrument was first published and made available for use. To mark this anniversary, an article
was published summarizing the academic journey the AGREE instrument has taken, noting themany accomplishments along the way (16).
REFERENCES
1. Woolf SH, Grol R, Hutchinson A, Eccles M, Grimshaw J. Clinical guidelines: potential
benefits, limitations, and harms of clinical guidelines. BMJ. 1999;318(7182):527-530.
2. Committee to Advise the Public Health Service on Clinical Practice Guidelines IoM. Clinicalpractice guidelines: directions for a new program. Washington: National Academy Press;
1990.
3. Browman GP, Snider A, Ellis P. Negotiating for change. The healthcare manager as catalyst
for evidence-based practice: changing the healthcare environment and sharing experience.
Healthc Pap. 2003;3(3):10-22.
4. Grol R. Success and failures in the implementation of evidence-based guidelines for clinicalpractice. Med Care. 2001;39(8 Suppl 2):1146-54.
5. Davis DA, Taylor-Vaisey A. Translating guidelines into practice: a systematic review oftheoretic concepts, practice experience and research evidence in the adoption of clinicalpractice guidelines. CMAJ. 1997;157(4):408-16.
6. Grimshaw J,.Russell I. Effect of clinical guidelines on medical practice: a systematic review
of rigorous evaluations. Lancet. 1993;342:1317-22.

	
  
	
  

4	
  

	
  
	
  

7. Shaneyfelt TM, Mayo-Smith MF Rothwangl J. Are guidelines following guidelines? The
methodological quality of clinical practice guidelines in the peer-reviewed medical literature.
JAMA 1999:281(20):1900-5.
8. Grilli R, Magrini N, Penna A, Mura G, Liberati A. Practice guidelines developed by specialty
societies: the need for critical appraisal. Lancet. 2000;355:103-6.
9. Burgers JS, Fervers B, Haugh M, Brouwers M, Browman G, Phillip T, Cluzeau FA.
International assessment of the quality of clinical practice guidelines in oncology using theAppraisal of Guidelines and Research and Evaluation Instrument.
J Clin Oncol.
2004;22:2000-7.
10. AGREE Collaboration. Development and validation of an international appraisal instrument
for assessing the quality of clinical practice guidelines: the AGREE project. Qual Saf HealthCare. 2003;12(1):18-23.
11. Brouwers MC, Kho ME, Browman GP, Burgers JS, Cluzeau F, Feder G, Fervers B, Graham
ID, Grimshaw J, Hanna S, Littlejohns P, Makarski J, Zitzelsberger L, for the AGREE Next
Steps Consortium. AGREE II: advancing guideline development, reporting and evaluation in
health care. CMAJ. 2010;182(18):E839-42.
12. Brouwers MC, Kho ME, Browman GP, Burgers JS, Cluzeau F, Feder G, Fervers B, Graham
ID, Hanna SE, Makarski J, for the AGREE Next Steps Consortium. Development of the
AGREE II, part 1: performance, usefulness and areas for improvement. CMAJ
2010;182(10):1045-52.
13. Brouwers MC, Kho ME, Browman GP, Burgers J, Cluzeau F, Feder G, Fervers B, Graham
ID, Hanna SE, Makarski J, for the AGREE Next Steps Consortium. Development of the
AGREE II, part 2: assessment of validity of items and tools to support application. CMAJ
2010;182(10):E472-8.
14. Brouwers MC, Kerkvliet K, Spithoff K, on behalf of the AGREE Next Steps Consortium. The
AGREE Reporting Checklist: A tool to improve reporting of clinical practice guidelines. BMJ
2016;352:i1152.
15. Vernooij RWM, Alonso-Coello P, Brouwers M, Martinez Garcia L, CheckUp Panel. Reporting
items for updated clinical guidelines: Checklist for the Reporting of Updated Guidelines(CheckUp). PLOS Medicine 2017;14(1):e1002207.
16. Makarski J, Brouwers MC. The AGREE Enterprise: a decade of advancing clinical practiceguidelines. Implement Sci. 2014;9:103.

	
  
	
  

5	
  

	
  
	
  

AGREE II:
USER’S MANUAL

	
  
	
  

6	
  

	
  
	
  

II. USER’S MANUAL: INSTRUCTIONS FOR USING THE AGREE II
This User’s Manual has been designed specifically to guide appraisers in the use of the
instrument. We suggest reading the following instructions before using the instrument.

I. Preparing to Use the AGREE IIi) Accompanying Guideline Documents
Before applying the AGREE II, users should first carefully read the guideline document in full. In
addition to the guideline document, users should attempt to identify all information about theguideline development process prior to the appraisal. This information may be contained in thesame document as the guideline recommendations or it may be summarized in a separate
technical report, methodological manual or guideline developer policy statement. These
supporting documents may be published or may be available publicly on web sites. While it is
the responsibility of the guideline authors to advise readers on the existence and location of
relevant additional technical and supporting documents, every effort should be made by the
AGREE II users to locate and include them as part of the materials appropriate for assessment.
ii) Number of Appraisers
We recommend that each guideline be assessed by at least 2 appraisers, and preferably 4, as
this will increase the reliability of the assessment.

II. Structure and Content of the AGREE II
The AGREE II consists of 23 key items organized within 6 domains followed by 2 global rating
items (“Overall Assessment”). Each domain captures a unique dimension of guideline quality.
Domain 1. Scope and Purpose is concerned with the overall aim of the guideline, the specifichealth questions, and the target population (items 1-3).
Domain 2. Stakeholder Involvement focuses on the extent to which the guideline was developedby the appropriate stakeholders and represents the views of its intended users (items 4-6).
Domain 3. Rigour of Development relates to the process used to gather and synthesize theevidence, the methods to formulate the recommendations, and to update them (items 7-14).
Domain 4. Clarity of Presentation deals with the language, structure, and format of the guideline(items 15-17).
Domain 5. Applicability pertains to the likely barriers and facilitators to implementation,
strategies to improve uptake, and resource implications of applying the guideline (items 18-21).
Domain 6. Editorial Independence is concerned with the formulation of recommendations not
being unduly biased with competing interests (items 22-23).
Overall assessment includes the rating of the overall quality of the guideline and whether theguideline would be recommended for use in practice.

	
  
	
  

7	
  

	
  
	
  

III. Rating Scale and User’s Manual Sections
Each of the AGREE II items and the two global rating items are rated on a 7-point scale (1–strongly disagree to 7–strongly agree). The User’s Manual provides guidance on how to rateeach item using the rating scale and also includes 3 additional sections to further facilitate theuser’s assessment. The sections include User’s Manual Description, Where to Look, and How
to Rate.
i) Rating Scale
All AGREE II items are rated on the following 7-point scale:
1
Strongly Disagree

2

3

4

5

6

7
Strongly Agree

Score of 1 (Strongly Disagree). A score of 1 should be given when there is no information that isrelevant to the AGREE II item, if the concept is very poorly reported, or if the authors state
explicitly that criteria were not met.
Score of 7 (Strongly Agree). A score of 7 should be given if the quality of reporting is
exceptional and where the full criteria and considerations articulated in the User’s Manual havebeen met.
Scores between 2 and 6. A score between 2 and 6 is assigned when the reporting of the
AGREE II item does not meet the full criteria or considerations. A score is assigned dependingon the completeness and quality of reporting. Scores increase as more criteria are met andconsiderations addressed. The “How to Rate” section for each item includes details about
assessment criteria and considerations specific to the item.
ii) User’s Manual Description
This section defines the concept underlying the item in broad terms and provides examples.
iii) Where to Look
This section directs the appraiser to where the information in the guideline can usually be found.
Included in this section are common terms used to label guideline sections or chapters. These
are suggestions only. It is the responsibility of the appraiser to review the entire guideline andaccompanying material(s) to ensure a fair evaluation.
iv) How to Rate
This section includes details about assessment criteria and considerations specific to each item.
• The criteria identify explicit elements that reflect the operational definitionof the item. The more criteria that are met, the higher the score theguideline should receive on that item.
• The considerations are aimed to help inform the assessment. As in any
evaluation, judgments by the appraisers are required. The more theconsiderations have been taken into account in the guideline, the higher
the score the guideline should receive on that item.
It is important to note that guideline ratings require a level of judgment. The criteria andconsiderations are there to guide, not to replace, these judgments. Thus, none of the AGREE II
items provide explicit expectations for each of the 7 points on the scale.

	
  
	
  

8	
  

	
  
	
  

v) Other Considerations when Applying the AGREE II
On occasion, some AGREE II items may not be applicable to the particular guideline underreview. For example, guidelines narrow in scope may not provide the full range of options forthe management of the condition (see item 16). The AGREE II does not include a “NotApplicable” response item in its scale. There are different strategies to manage this situation
including having appraisers skip that item in the assessment process or rating the item as 1
(absence of information) and providing context about the score. Regardless of strategy chosen,decisions should be made in advance, described in an explicit manner, and if items are skipped,
appropriate modifications to calculating the domain scores should be implemented. As aprinciple, excluding items in the appraisal process is discouraged.

IV. Scoring the AGREE II
A quality score is calculated for each of the six AGREE II domains. The six domain scores are
independent and should not be aggregated into a single quality score.
i) Calculating Domain Scores
Domain scores are calculated by summing up all the scores of the individual items in a domain
and by scaling the total as a percentage of the maximum possible score for that domain.
Example:If 4 appraisers give the following scores for Domain 1 (Scope & Purpose):
Item 1

Item 2Item 3

Total

Appraiser
1

5

6

6

17

Appraiser
2

6

6

7

19

Appraiser
3

2

4

3

9

Appraiser
4

3

3

2

8

Total

16

19

18

53

Appraiser
Maximum possible score = 7 (strongly agree) x 3 (items) x 4 (appraisers) = 84
Minimum possible score = 1 (strongly disagree) x 3 (items) x 4 (appraisers) = 12
The scaled domain score will be:
Obtained score – Minimum possible score
Maximum possible score – Minimum possible score
53 – 1284 – 12	  	
  X 100 =

41
72

X 100 =

9	
  0.5694 x 100 =

	
  
	
  

57 %

If items are not included, appropriate modifications to the calculations of maximum andminimum possible scores are required.
ii) Interpreting Domain Scores	
  Domain scores can be used to identify strengths and limitations of guidelines, to compare
methodological quality between guidelines, or to select high quality guidelines for adaptation,
endorsement, or implementation. At present, there are no empirical data to link specific quality
scores with specific implementation outcomes (e.g., speed of adoption, spread of adoption) or
specific clinical outcomes; this makes selection of quality thresholds to differentiate between
high, moderate, and low quality guidelines a challenge. In the absence of these data, we
provide examples of approaches that can be used to set quality thresholds:
•

•

•

•

Prioritizing one domain: Through consensus or based on decisions by leadership, one
quality domain may be prioritized over the others. Thus, thresholds can be createdbased on scores for the prioritized domain (e.g., high quality guidelines are those with a
Domain 3 score >70%).
Staged AGREE II appraisal: If users value one domain over the others, they can first
appraise the guidelines using that domain only. Only those guidelines that meet a quality
threshold for that domain (e.g., >70%) are then appraised using the other five AGREE II
domains.
Considering all domain scores: Users can create a threshold across all six domainscores based on consensus or decisions by leadership (e.g., high quality guidelines arethose with domain scores that are all >70%). Alternatively, users might create different
thresholds for each of the domains.
Thresholds for improvement over time: If evaluating changes in scores for guidelines
over time, users can create thresholds for improvement (e.g., at least 10% improvementin each domain score for guidelines by a particular developer over a period of five
years).

Any decisions about how to define quality thresholds should be made by a panel of all relevant
stakeholders before beginning the AGREE II appraisals. Decisions should be guided by thecontext in which the guideline is to be used and by evaluating the importance of the different
domains and items in that context.

V. Overall Assessment
Upon completing the 23 items, AGREE II users will provide 2 overall assessments of theguideline. The overall assessment requires the user to make a judgment as to the quality of theguideline, taking into account the criteria considered in the assessment process. The user isalso asked whether he/she would recommend use of the guideline.
	
  
	
  
________________________________________________________________________________________The next pages include, by domain, guidance for rating each of the 23 items of the AGREE II
when appraising a guideline. Each item includes a description, suggestions for where to find theitem information, and guidance for how to rate.
	
  
	
  
	
  
	
  

	
  
	
  

10	
  

	
  
	
  

DOMAIN 1. SCOPE AND PURPOSE

1. The overall objective(s) of the guideline is (are) specifically described.
2. The health question(s) covered by the guideline is (are) specificallydescribed.
3. The population (patients, public, etc.) to whom the guideline is meant
to apply is specifically described.

	
  
	
  

11	
  

	
  
	
  

SCOPE AND PURPOSE
1. The overall objective(s) of the guideline is (are) specifically described.
1
Strongly Disagree

2

3

4

5

6

7
Strongly Agree	
  Comments	
  
	
  
	
  
	
  
	
  

User’s Manual Description:
This deals with the potential health impact of a guideline on society and populations of patients or individuals.
The overall objective(s) of the guideline should be described in detail and the expected health benefits from theguideline should be specific to the clinical problem or health topic. For example, specific statements would be:• Preventing (long term) complications of patients with diabetes mellitus• Lowering the risk of subsequent vascular events in patients with previous myocardial infarction• Most effective population-based colorectal screening strategies• Providing guidance on the most effective therapeutic treatment and management of patients with diabetes
mellitus.

Where to Look:
Examine the opening paragraphs/chapters for a description of the scope and purpose of the guideline. In somecases, the rationale or need for the guideline is described in a document separate from the guideline, forinstance, in the guideline proposal. Examples of commonly labeled sections or chapters in a guideline where
this information can be found include: introduction, scope, purpose, rationale, background, and objectives.

How to Rate:
Item content includes the following CRITERIA:• health intent(s) (i.e., prevention, screening, diagnosis, treatment, etc.)• expected benefit or outcome• target(s) (e.g., patient population, society)

Additional CONSIDERATIONS:• Is the item well written? Are the descriptions clear and concise?
• Is the item content easy to find in the guideline?

	
  
	
  

12	
  

	
  
	
  

SCOPE AND PURPOSE
2. The health question(s) covered by the guideline is (are) specifically described.
1
Strongly Disagree

2

3

4

5

6

7
Strongly Agree

Comments

User’s Manual Description:
A detailed description of the health questions covered by the guideline should be provided, particularly for the
key recommendations (see Item 17), although they need not be phrased as questions. Following the examples
provided in question 1:• How many times a year should the HbA1c be measured in patients with diabetes mellitus?
• What should the daily aspirin dosage for patients with proven acute myocardial infarction be?
• Does population-based colorectal screening using the fecal occult blood test reduce mortality of colorectal
cancer?
• Is self-monitoring effective for blood glucose control in patients with Type 2 diabetes?

Where to Look:
Examine the opening paragraphs/chapters for a description of the scope and purpose of the guideline. In somecases, the questions are described in a document separate from the guideline, for instance in a search
specification. Examples of commonly labeled sections or chapters in a guideline where this information can befound include: questions, scope, purpose, rationale, and background.

How to Rate:
Item content includes the following CRITERIA:• target population• intervention(s) or exposure(s)• comparisons (if appropriate)• outcome(s)• health care setting or context	  Additional CONSIDERATIONS:• Is the item well written? Are the descriptions clear and concise?
• Is the item content easy to find in the guideline?
• Is there enough information provided in the question(s) for anyone to initiate the development of a guidelineon this topic or to understand the patients/populations and contexts profiled in the guideline?

	
  
	
  

13	
  

	
  
	
  

SCOPE AND PURPOSE
3. The population (patients, public, etc.) to whom the guideline is meant to applyis specifically described.
1
Strongly Disagree

2

3

4

5

6

7
Strongly Agree

Comments

User’s Manual Description:
A clear description of the population (i.e., patients, public, etc.) covered by a guideline should be provided. The
age range, sex, clinical description, and comorbidity may be provided. For example:• A guideline on the management of diabetes mellitus only includes patients with non-insulin dependent
diabetes mellitus and excludes patients with cardiovascular comorbidity.
• A guideline on the management of depression only includes patients with major depression according to the
DSM-IV criteria, and excludes patients with psychotic symptoms and children.
• A guideline on screening of breast cancer only includes women, aged between 50 and 70 years, with nohistory of cancer and with no family history of breast cancer.

Where to Look:
Examine the opening paragraphs/chapters for a description of the target population of the guideline. The
explicit exclusion of some populations (for instance children) is also covered by this item. Examples ofcommonly labeled sections or chapters in a guideline where this information can be found include: patient
population, target population, relevant patients, scope, and purpose.

How to Rate:
Item content includes the following CRITERIA:• target population, gender and age• clinical condition (if relevant)• severity/stage of disease (if relevant)• comorbidities (if relevant)• excluded populations (if relevant)Additional CONSIDERATIONS:• Is the item well written? Are the descriptions clear and concise?
• Is the item content easy to find in the guideline?
• Is the population information specific enough so that the correct and eligible individuals would receive

the action recommended in the guideline?

	
  
	
  

14	
  

	
  
	
  

DOMAIN 2. STAKEHOLDER INVOLVEMENT

4. The guideline development group includes individuals from all
relevant professional groups.
5. The views and preferences of the target population (patients, public, etc.)
have been sought.
6. The target users of the guideline are clearly defined.

	
  
	
  

15	
  

	
  
	
  

STAKEHOLDER INVOLVEMENT
4. The guideline development group includes individuals from all relevant
professional groups.
1
Strongly Disagree

2

3

4

5

6

7
Strongly Agree	
  Comments

User’s Manual Description:
This item refers to the professionals who were involved at some stage of the development process. This may
include members of the steering group, the research team involved in selecting and reviewing/rating theevidence and individuals involved in formulating the final recommendations. This item excludes individuals who
have externally reviewed the guideline (see Item 13). This item excludes target population representation (seeItem 5). Information about the composition, discipline, and relevant expertise of the guideline developmentgroup should be provided.

Where to Look:
Examine the opening paragraphs/chapters, acknowledgement section or appendices for the composition of theguideline development group. Examples of commonly labeled sections or chapters in a guideline where thisinformation can be found include: methods, guideline panel member list, acknowledgements, and appendices.

How to Rate:
Item content includes the following CRITERIA:• For each member of the guideline development group, the following information is included:Ø nameØ discipline/content expertise (e.g., neurosurgeon, methodologist)Ø institution (e.g., St. Peter’s hospital)Ø geographical location (e.g., Seattle, WA)Ø a description of the member’s role in the guideline development group	
  Additional CONSIDERATIONS:• Is the item well written? Are the descriptions clear and concise?
• Is the item content easy to find in the guideline?
• Are the members an appropriate match for the topic and scope? Potential candidates include relevant
clinicians, content experts, researchers, policy makers, clinical administrators, and funders.
• Is there at least one methodology expert included in the development group (e.g., systematic review expert,
epidemiologist, statistician, library scientist, etc.)?

	
  
	
  

16	
  

	
  
	
  

STAKEHOLDER INVOLVEMENT
5. The views and preferences of the target population (patients, public, etc.) havebeen sought.
preferences have been sought.
1
Strongly Disagree

2

3

4

5

6

7
Strongly Agree

Comments
	
  
	
  
	
  

User’s Manual Description:
Information about target population experiences and expectations of health care should inform thedevelopment of guidelines. There are various methods for ensuring that these perspectives inform the different
stages of guideline development by stakeholders. For example, formal consultations with patients/public todetermine priority topics, participation of these stakeholders on the guideline development group, or external
review by these stakeholders on draft documents. Alternatively, information could be obtained from interviewsof these stakeholders or from literature reviews of patient/public values, preferences or experiences. There
should be evidence that some process has taken place and that stakeholders’ views have been considered.

Where to Look:
Examine the paragraphs on the guideline development process. Examples of commonly labeled sections or
chapters in a guideline where this information can be found include: scope, methods, guideline panel member
list, external review, and target population perspectives.

How to Rate:
Item content includes the following CRITERIA:• statement of type of strategy used to capture patients’/public’s’ views and preferences (e.g., participation in
the guideline development group, literature review of values and preferences)• methods by which preferences and views were sought (e.g., evidence from literature, surveys, focus groups)• outcomes/information gathered on patient/public information• description of how the information gathered was used to inform the guideline development process and/or
formation of the recommendations	  Additional CONSIDERATIONS:• Is the item well written? Are the descriptions clear and concise?
• Is the item content easy to find in the guideline?

	
  
	
  

17	
  

	
  
	
  

STAKEHOLDER INVOLVEMENT
6. The target users of the guideline are clearly defined.
1
Strongly Disagree

2

3

4

5

6

7
Strongly Agree	
  Comments
	
  
	
  
	
  
	
  

User’s Manual Description:
The target users should be clearly defined in the guideline, so the reader can immediately determine if theguideline is relevant to them. For example, the target users for a guideline on low back pain may include
general practitioners, neurologists, orthopaedic surgeons, rheumatologists, and physiotherapists.

Where to Look:
Examine the opening paragraphs/chapters for a description of the target users of the guideline. Examples ofcommonly labeled sections or chapters in a guideline where this information can be found include: target user
and intended user.

How to Rate:
Item content includes the following CRITERIA:• clear description of intended guideline audience (e.g. specialists, family physicians, patients, clinical or
institutional leaders/administrators)• description of how the guideline may be used by its target audience (e.g., to inform clinical decisions, toinform policy, to inform standards of care)	
  Additional CONSIDERATIONS:• Is the item well written? Are the descriptions clear and concise?
• Is the item content easy to find in the guideline?
• Are the target users appropriate for the scope of the guideline?

	
  
	
  

18	
  

	
  
	
  

DOMAIN 3. RIGOUR OF DEVELOPMENT

7. Systematic methods were used to search for evidence.
8. The criteria for selecting the evidence are clearly described.
9. The strengths and limitations of the body of evidence are clearly
described.
10. The methods for formulating the recommendations are clearly
described.
11. The health benefits, side effects, and risks have been considered informulating the recommendations.
12. There is an explicit link between the recommendations and the
supporting evidence.
13. The guideline has been externally reviewed by experts prior to its
publication.
14. A procedure for updating the guideline is provided.

	
  
	
  

19	
  

	
  
	
  

RIGOUR OF DEVELOPMENT
7. Systematic methods were used to search for evidence.
1
Strongly Disagree

2

3

4

5

6

7
Strongly Agree

Comments	
  
	
  
	
  
	
  
	
  

User’s Manual Description:
Details of the strategy used to search for evidence should be provided including search terms used, sources
consulted, and dates of the literature covered. Sources may include electronic databases (e.g. MEDLINE,EMBASE, CINAHL), databases of systematic reviews (e.g. the Cochrane Library, DARE), handsearching
journals, reviewing conference proceedings, and other guidelines (e.g. the US National Guideline
Clearinghouse, the German Guidelines Clearinghouse). The search strategy should be as comprehensive as
possible and executed in a manner free from potential biases and sufficiently detailed to be replicated.

Where to Look:
Examine the paragraphs/chapters describing the guideline development process. In some cases the searchstrategies are described in separate documents or in an appendix to the guideline. Examples of commonlylabelled sections or chapters in a guideline where this information can be found include: methods, literaturesearch strategy, and appendices.

How to Rate:
Item content includes the following CRITERIA:• named electronic database(s) or evidence source(s) where the search was performed (e.g., MEDLINE,EMBASE, PsychINFO, CINAHL)• time periods searched (e.g., January 1, 2004 to March 31, 2008)• search terms used (e.g., text words, indexing terms, subheadings)• full search strategy included (e.g., possibly located in appendix)	  Additional CONSIDERATIONS:• Is the item well written? Are the descriptions clear and concise?
• Is the item content easy to find in the guideline?
• Is the search relevant and appropriate to answer the health question? (e.g., all relevant databases and,appropriate search terms used)• Is there enough information provided for anyone to replicate the search?

	
  
	
  

20	
  

	
  
	
  

RIGOUR OF DEVELOPMENT
8. The criteria for selecting the evidence are clearly described.
1
Strongly Disagree

2

3

4

5

6

7
Strongly Agree	
  Comments
	
  
	
  
	
  
	
  

User’s Manual Description:
Criteria for including/excluding evidence identified by the search should be provided. These criteria should be
explicitly described and reasons for including and excluding evidence should be clearly stated. For example,
guideline authors may decide to only include evidence from randomized clinical trials and to exclude articles
not written in English.

Where to Look:
Examine the paragraphs/chapters describing the guideline development process. In some cases, the inclusion
or exclusion criteria for selecting the evidence are described in separate documents or in an Appendix to theguideline. Examples of commonly labeled sections or chapters in a guideline where this information can befound include: methods, literature search, inclusion/exclusion criteria, and appendices.

How to Rate:
Item content includes the following CRITERIA:• description of the inclusion criteria, includingØ target population (patient, public, etc.) characteristics
Ø study designØ comparisons (if relevant)Ø outcomesØ language (if relevant)Ø context (if relevant)• description of the exclusion criteria (if relevant; e.g., French only listed in the inclusion criteria statement could
logically preclude non-French listed in the exclusion criteria statement)	  Additional CONSIDERATIONS:• Is the item well written? Are the descriptions clear and concise?
• Is the item content easy to find in the guideline?
• Is there a rationale given for the chosen inclusion/exclusion criteria?
• Do inclusion/exclusion criteria align with the health question(s)?
• Are there reasons to believe that relevant literature may not have been considered?

	
  
	
  

21	
  

	
  
	
  

RIGOUR OF DEVELOPMENT
9. The strengths and limitations of the body of evidence are clearly described.
1
Strongly Disagree

2

3

4

5

6

7
Strongly Agree	
  Comments
	
  
	
  
	
  
	
  

User’s Manual Description:
Statements highlighting the strengths and limitations of the evidence should be provided. This ought to include
explicit descriptions - using informal or formal tools/methods - to assess and describe the risk of bias forindividual studies and/or for specific outcomes and/or explicit commentary of the body of evidence aggregatedacross all studies. This may be presented in different ways, for example: using tables commenting on different
quality domains; the application of a formal instrument or strategy (e.g., Jadad scale, GRADE method); or
descriptions in the text.

Where to Look:
Examine the paragraphs/chapters describing the guideline development process for information on how themethodological quality of the studies (e.g., risk of bias) were described. Evidence tables are often used tosummarize quality features. Some guidelines make a clear distinction between description and interpretation ofevidence, for instance, in a results section and a discussion section, respectively.

How to Rate:
Item content includes the following CRITERIA:• descriptions of how the body of evidence was evaluated for bias and how it was interpreted by members of
the guideline development group• aspects upon which to frame descriptions include:Ø study design(s) included in body of evidenceØ study methodology limitations (sampling, blinding, allocation concealment, analytical methods)Ø appropriateness/relevance of primary and secondary outcomes consideredØ consistency of results across studiesØ direction of results across studiesØ magnitude of benefit versus magnitude of harmØ applicability to practice context	  Additional CONSIDERATIONS:• Is the item well written? Are the descriptions clear and concise?
• Is the item content easy to find in the guideline?
• Are the descriptions appropriate, neutral, and unbiased? Are the descriptions complete?

	
  
	
  

22	
  

	
  
	
  

RIGOUR OF DEVELOPMENT
10. The methods for formulating the recommendations are clearly described.
1
Strongly Disagree

2

3

4

5

6

7
Strongly Agree	
  Comments
	
  
	
  
	
  
	
  

User’s Manual Description:
A description of the methods used to formulate the recommendations and how final decisions were arrived atshould be provided. For example, methods may include a voting system, informal consensus, and formal
consensus techniques (e.g., Delphi, Glaser techniques). Areas of disagreement and methods of resolving themshould be specified.

Where to Look:
Examine the paragraphs/chapters describing the guideline development process. In some cases, the methods
used to formulate the recommendations are described in separate documents or in an appendix to theguideline. Examples of commonly labeled sections or chapters in a guideline where this information can befound include methods and guideline development process.

	  How to Rate:
Item content includes the following CRITERIA:• description of the recommendation development process (e.g., steps used in modified Delphi technique,voting procedures that were considered)• outcomes of the recommendation development process (e.g., extent to which consensus was reached using
modified Delphi technique, outcome of voting procedures)• description of how the process influenced the recommendations (e.g., results of Delphi technique influence
final recommendation, alignment with recommendations and the final vote)

Additional CONSIDERATIONS:• Is the item well written? Are the descriptions clear and concise?
• Is the item content easy to find in the guideline?
• Was a formal process used to arrive at the recommendations?
• Were the methods appropriate?

	
  
	
  

23	
  

	
  
	
  

RIGOUR OF DEVELOPMENT
11. The health benefits, side effects, and risks have been considered informulating the recommendations.
1
Strongly Disagree

2

3

4

5

6

7
Strongly Agree	
  Comments
	
  
	
  
	
  
	
  

User’s Manual Description:
The guideline should consider health benefits, side effects, and risks when formulating the recommendations.
For example, a guideline on the management of breast cancer may include a discussion on the overall effects
on various final outcomes. These may include: survival, quality of life, adverse effects, and symptom
management or a discussion comparing one treatment option to another. There should be evidence that theseissues have been addressed.

Where to Look:
Examine the paragraphs/chapters describing the guideline development process for a description of the bodyof evidence, its interpretation, and the translation to practice recommendations. Examples of commonly labeledsections or chapters in a guideline where this information can be found include: methods, interpretation,
discussion, and recommendations.

How to Rate:
Item content includes the following CRITERIA:• supporting data and report of benefits• supporting data and report of harms/side effects/risks• reporting of the balance/trade-off between benefits and harms/side effects/risks• recommendations reflect considerations of both benefits and harms/side effects/risks	
  Additional CONSIDERATIONS:• Is the item well written? Are the descriptions clear and concise?
• Is the item content easy to find in the guideline?
• Is the discussion an integral part of the guideline development process? (i.e., taking place duringrecommendation formulation rather than post-formulation as an afterthought)• Has the guideline development group considered the benefits and harms equally?

	
  
	
  

24	
  

	
  
	
  

RIGOUR OF DEVELOPMENT
12. There is an explicit link between the recommendations and the supporting
evidence.
1
Strongly Disagree

2

3

4

5

6

7
Strongly Agree	
  Comments
	
  
	
  
	
  
	
  

User’s Manual Description:
An explicit link between the recommendations and the evidence on which they are based should be included in
the guideline. The guideline user should be able to identify the components of the body of evidence relevant toeach recommendation.

Where to Look:
Define and examine the recommendations in the guideline and the text describing the body of evidence that
underpins them. Examples of commonly labeled sections or chapters in a guideline where this information canbe found include: recommendations and key evidence.

How to Rate:
Item content includes the following CRITERIA:• the guideline describes how the guideline development group linked and used the evidence to inform
recommendations• each recommendation is linked to a key evidence description/paragraph and/or reference list
• recommendations linked to evidence summaries, evidence tables in the results section of the guideline	  Additional CONSIDERATIONS:• Is there congruency between the evidence and recommendations?
• Is the link between the recommendations and supporting evidence easy to find in the guideline?
• When evidence is lacking or a recommendation is informed primarily by consensus of opinion by the guideline
group, rather than the evidence, is this clearly stated and described?

	
  
	
  

25	
  

	
  
	
  

RIGOUR OF DEVELOPMENT
13. The guideline has been externally reviewed by experts prior to its publication.
1
Strongly Disagree

2

3

4

5

6

7
Strongly Agree	
  Comments
	
  
	
  
	
  
	
  

User’s Manual Description:
A guideline should be reviewed externally before it is published. Reviewers should not have been involved in
the guideline development group. Reviewers should include experts in the clinical area as well as somemethodological experts. Target population (patients, public) representatives may also be included. A
description of the methodology used to conduct the external review should be presented, which may include alist of the reviewers and their affiliation.

Where to Look:
Examine the paragraphs/chapters describing the guideline development process and the acknowledgement
section. Examples of commonly labeled sections or chapters in a guideline where this information can be foundinclude: methods, results, interpretation, and acknowledgements.

How to Rate:
Item content includes the following CRITERIA:• purpose and intent of the external review (e.g., to improve quality, gather feedback on draft
recommendations, assess applicability and feasibility, disseminate evidence)• methods taken to undertake the external review (e.g., rating scale, open-ended questions)• description of the external reviewers (e.g., number, type of reviewers, affiliations)• outcomes/information gathered from the external review (e.g., summary of key findings)• description of how the information gathered was used to inform the guideline development process and/or
formation of the recommendations (e.g., guideline panel considered results of review in forming final
recommendations)

Additional CONSIDERATIONS:• Is the item well written? Are the descriptions clear and concise?
• Is the item content easy to find in the guideline?
• Are the external reviewers relevant and appropriate to the scope of the guideline? Was there a rationale given
for choosing the included reviewers?
• How was information from the external review used by the guideline development group?

	
  
	
  

26	
  

	
  
	
  

RIGOUR OF DEVELOPMENT
14. A procedure for updating the guideline is provided.
1
Strongly Disagree

2

3

4

5

6

7
Strongly Agree

Comments
	
  
	
  
	
  
	
  

User’s Manual Description:
Guidelines need to reflect current research. A clear statement about the procedure for updating the guideline
should be provided. For example, a timescale has been given or a standing panel is established who receives
regularly updated literature searches and makes changes as required.

Where to Look:
Examine the introduction paragraph, the paragraphs describing the guideline development process and theclosing paragraphs. Examples of commonly labeled sections or chapters in a guideline where this information
can be found include: methods, guideline update, and date of guideline.

How to Rate:
Item content includes the following CRITERIA:• a statement that the guideline will be updated• explicit time interval or explicit criteria to guide decisions about when an update will occur• methodology for the updating procedure is reported	  Additional CONSIDERATIONS:• Is the item well written? Are the descriptions clear and concise?
• Is the item content easy to find in the guideline?
• Is there enough information provided to know when an update will occur or what criteria would trigger an
update?

	
  
	
  

27	
  

	
  
	
  

DOMAIN 4. CLARITY OF PRESENTATION

15. The recommendations are specific and unambiguous.
16. The different options for management of the condition or health issue
are clearly presented.
17. Key recommendations are easily identifiable.

	
  
	
  

28	
  

	
  
	
  

CLARITY OF PRESENTATION
15. The recommendations are specific and unambiguous.
1
Strongly Disagree

2

3

4

5

6

7
Strongly Agree	
  Comments
	
  
	
  
	
  
	
  

User’s Manual Description:
A recommendation should provide a concrete and precise description of which option is appropriate in whichsituation and in what population group, as informed by the body of evidence.

• An example of a specific recommendation is: Antibiotics should be prescribed in children two years or older
with a diagnosis of acute otitis media if the pain lasts longer than three days or if the pain increases after the
consultation despite adequate treatment with painkillers; in these cases, amoxicillin should be given for 7 days
(supplied with a dosage scheme).

• An example of a vague recommendation is: Antibiotics are indicated for cases with an abnormal orcomplicated course.
It is important to note that in some instances, evidence is not always clear cut and there may be uncertainty
about the best care option(s). In this case, the uncertainty should be stated in the guideline.

Where to Look:
Define and examine the recommendations in the guideline. Examples of commonly labeled sections or
chapters in a guideline where this information can be found include: recommendations and executive summary.

How to Rate:
Item content includes the following CRITERIA:• statement of the recommended action• identification of the intent or purpose of the recommended action (e.g., to improve quality of life, to decreaseside effects)• identification of the relevant population (e.g., patients, public)• caveats or qualifying statements, if relevant (e.g., patients or conditions for whom the recommendations
would not apply)

Additional CONSIDERATIONS:• In the event of multiple recommendations (e.g., management guidelines), is there clarity regarding to whom
each recommendation applies?
• If there is uncertainty in the interpretation and discussion of the evidence, is the uncertainty reflected in therecommendations and explicitly stated?

	
  
	
  

29	
  

	
  
	
  

CLARITY OF PRESENTATION
16. The different options for management of the condition or health issue areclearly presented.
1
Strongly Disagree

2

3

4

5

6

7
Strongly Agree	
  Comments
	
  
	
  
	
  
	
  

User’s Manual Description:
A guideline that targets the management of a disease should consider the different possible options forscreening, prevention, diagnosis or treatment of the condition it covers. These possible options should beclearly presented in the guideline.
For example, a recommendation on the management of depression may contain the following treatment
alternatives:a. Treatment with TCA
b. Treatment with SSRI
c. Psychotherapy
d. Combination of pharmacological and psychological therapy

Where to Look:
Examine the recommendations and their supporting evidence. Examples of commonly labeled sections or
chapters in a guideline where this information can be found include: executive summary, recommendations,discussion, treatment options, and treatment alternatives.

How to Rate:
Item content includes the following CRITERIA:• description of options• description of population or clinical situation most appropriate to each option

Additional CONSIDERATIONS:• Is the item well written? Are the descriptions clear and concise?
• Is the item content easy to find in the guideline?
• Is this pertaining to a guideline broad or narrow in scope? This item may be more relevant to guidelines thatare broad in scope (e.g., covering the management of a condition or issue rather than focusing on a particular
set of interventions for a specific condition/issue).

	
  
	
  

30	
  

	
  
	
  

CLARITY OF PRESENTATION
17. Key recommendations are easily identifiable.
1
Strongly Disagree

2

3

4

5

6

7
Strongly Agree	
  Comments
	
  
	
  
	
  
	
  

User’s Manual Description:
Users should be able to find the most relevant recommendations easily. These recommendations answer themain question(s) that have been covered by the guideline and can be identified in different ways. For example,
they can be summarized in a box, typed in bold, underlined or presented as flow charts or algorithms.

Where to Look:
Examples of commonly labeled sections or chapters in a guideline where this information can be found include:
executive summary, conclusions, and recommendations. Some guidelines provide separate summaries withkey recommendations (e.g., quick reference guide).

How to Rate:
Item content includes the following CRITERIA:• description of recommendations in a summarized box, typed in bold, underlined, or presented as flow charts
or algorithms• specific recommendations are grouped together in one section

Additional CONSIDERATIONS:• Is the item well written? Are the descriptions clear and concise?
• Is the item content easy to find in the guideline?
• Are the key recommendations appropriately selected and do they reflect the key messages of the guideline?
• Are specific recommendations grouped in a section placed near the summary of the key evidence?

	
  
	
  

31	
  

	
  
	
  

DOMAIN 5. APPLICABILITY

18. The guideline describes facilitators and barriers to its application.
19. The guideline provides advice and/or tools on how therecommendations can be put into practice.
20. The potential resource implications of applying the recommendations
have been considered.
21. The guideline presents monitoring and/or auditing criteria.

	
  
	
  

32	
  

	
  
	
  

APPLICABILITY
18. The guideline describes facilitators and barriers to its application.
1
Strongly Disagree

2

3

4

5

6

7
Strongly Agree	
  Comments
	
  
	
  
	
  
	
  

User’s Manual Description:
There may be existing facilitators and barriers that will impact the application of guideline recommendations.
For example:
i. A guideline on stroke may recommend that care should be coordinated through stroke units andstroke services. There may be a special funding mechanism in the region to enable the formation of stroke
units.
ii. A guideline on diabetes in primary care may require that patients are seen and followed up indiabetic clinics. There may be an insufficient number of clinicians available in a region to enable clinics to beestablished.

Where to Look:
Examine the paragraph/chapter on the dissemination/implementation of the guideline or, if available, additional
documents with specific plans or strategies for implementation of the guideline. Examples of commonly labeledsections or chapters in a guideline where this information can be found include: barriers, guideline utilization,and quality indicators.

How to Rate:
Item content includes the following CRITERIA:• identification of the types of facilitators and barriers that were considered• methods by which information regarding the facilitators and barriers to implementing recommendations weresought (e.g., feedback from key stakeholders, pilot testing of guidelines before widespread implementation)• information/description of the types of facilitators and barriers that emerged from the inquiry (e.g.,practitioners have the skills to deliver the recommended care, sufficient equipment is not available to ensure all
eligible members of the population receive mammography)• description of how the information influenced the guideline development process and/or formation of therecommendations	  Additional CONSIDERATIONS:• Is the item well written? Are the descriptions clear and concise?
• Is the item content easy to find in the guideline?
• Does the guideline suggest specific strategies to overcoming the barriers?

	
  
	
  

33	
  

	
  
	
  

APPLICABILITY
19. The guideline provides advice and/or tools on how the recommendations canbe put into practice.
1
Strongly Disagree

2

3

4

5

6

7
Strongly Agree	
  Comments
	
  
	
  
	
  
	
  

User’s Manual Description:
For a guideline to be effective it needs to be disseminated and implemented with additional materials.
For example, these may include: a summary document, a quick reference guide, educational tools, results froma pilot test, patient leaflets, or computer support. Any additional materials should be provided with the guideline.

Where to Look:
Examine the paragraph on the dissemination/implementation of the guideline and, if available, the specificaccompanying materials that have been produced to support the dissemination and implementation of theguideline. Examples of commonly labeled sections or chapters in a guideline where this information can befound include: tools, resources, implementation, and appendices.

How to Rate:
Item content includes the following CRITERIA:• an implementation section in the guideline• tools and resources to facilitate application:
Ø guideline summary documentsØ links to check lists, algorithms
Ø links to how-to manualsØ solutions linked to barrier analysis (see Item 18)
Ø tools to capitalize on guideline facilitators (see Item 18)
Ø outcome of pilot test and lessons learned• directions on how users can access tools and resources	
  Additional CONSIDERATIONS:• Is the item well written? Are the descriptions clear and concise?
• Is the item content easy to find in the guideline?
• Is there information about the development of the implementation tools and validation procedures?

	
  
	
  

34	
  

	
  
	
  

APPLICABILITY
20. The potential resource implications of applying the recommendations havebeen considered.
1
Strongly Disagree

2

3

4

5

6

7
Strongly Agree	
  Comments
	
  
	
  
	
  
	
  

User’s Manual Description:
The recommendations may require additional resources in order to be applied. For example, there may be aneed for more specialized staff, new equipment, and expensive drug treatment. These may have cost
implications for health care budgets. There should be a discussion in the guideline of the potential impact of therecommendations on resources.

Where to Look:
Examine the paragraph(s) on the dissemination/implementation of the guideline or, if available, additional
documents with specific plans or strategies for implementation of the guideline. Some guidelines present cost
implications in the paragraphs that discuss the evidence or decisions behind the recommendations. Examples
of commonly labeled sections or chapters in a guideline where this information can be found include: methods,
cost utility, cost effectiveness, acquisition costs, and implications for budgets.

How to Rate:
Item content includes the following CRITERIA:• identification of the types of cost information that were considered (e.g., economic evaluations, drug
acquisition costs)• methods by which the cost information was sought (e.g., a health economist was part of the guideline
development panel, use of health technology assessments for specific drugs, etc.)• information/description of the cost information that emerged from the inquiry (e.g., specific drug acquisition
costs per treatment course)• description of how the information gathered was used to inform the guideline development process and/or
formation of the recommendations	  Additional CONSIDERATIONS:• Is the item well written? Are the descriptions clear and concise?
• Is the item content easy to find in the guideline?
• Were appropriate experts involved in finding and analyzing the cost information?

	
  
	
  

35	
  

	
  
	
  

APPLICABILITY
21. The guideline presents monitoring and/or auditing criteria.
1Strongly Disagree

2

3

4

5

6

7
Strongly Agree	
  Comments
	
  
	
  
	
  
	
  

User’s Manual Description:
Measuring the application of guideline recommendations can facilitate their ongoing use. This requires clearly
defined criteria that are derived from the key recommendations in the guideline. The criteria may include
process measures, behavioural measures, clinical or health outcome measures. Examples of monitoring andaudit criteria are:
• The HbA1c should be < 8.0%.
• The level of diastolic blood pressure should be < 95 mmHg.
• 80% of the population aged 50 years should receive colorectal cancer screening rates using fecal occult
blood tests.
• If complaints of acute otitis media last longer than three days, amoxicillin should be prescribed.

Where to Look:
Examine the paragraph/chapter on auditing or monitoring the use of the guideline or, if available, additional
documents with specific plans or strategies for evaluation of the guideline. Examples of commonly labeledsections or chapters in a guideline where this information can be found include: recommendations, quality
indicators, and audit criteria.

How to Rate:
Item content includes the following CRITERIA:• identification of criteria to assess guideline implementation or adherence to recommendations• criteria for assessing impact of implementing the recommendations• advice on the frequency and interval of measurement• descriptions or operational definitions of how the criteria should be measured	  Additional CONSIDERATIONS:• Is the item well written? Are the descriptions clear and concise?
• Is the item content easy to find in the guideline?
• Are a range of criteria provided including process measures, behavioural measures, and clinical or healthoutcomes?

	
  
	
  

36	
  

	
  
	
  

DOMAIN 6. EDITORIAL INDEPENDENCE

22. The views of the funding body have not influenced the content of theguideline.
23. Competing interests of guideline development group members havebeen recorded and addressed.

	
  
	
  

37	
  

	
  
	
  

EDITORIAL INDEPENDENCE
22. The views of the funding body have not influenced the content of theguideline.
1
Strongly Disagree

2

3

4

5

6

7
Strongly Agree	
  Comments
	
  
	
  
	
  
	
  

User’s Manual Description:
Many guidelines are developed with external funding (e.g., government, professional associations, charity
organizations, pharmaceutical companies). Support may be in the form of financial contribution for thecomplete development, or for parts of it (e.g., printing of the guidelines). There should be an explicit statementthat the views or interests of the funding body have not influenced the final recommendations.

Where to Look:
Examine the paragraphs/chapters on the guideline development process or acknowledgements section.
Examples of commonly labeled sections or chapters in a guideline where this information can be found include:
disclaimer and funding source.

How to Rate:
Item content includes the following CRITERIA:• the name of the funding body or source of funding (or explicit statement of no funding)• a statement that the funding body did not influence the content of the guideline	  Additional CONSIDERATIONS:• Is the item well written? Are the descriptions clear and concise?
• Is the item content easy to find in the guideline?
• How did the guideline development group address potential influence from the funding body?

	
  
	
  

38	
  

	
  
	
  

EDITORIAL INDEPENDENCE
23. Competing interests of guideline development group members have beenrecorded and addressed.
1Strongly Disagree

2

3

4

5

6

7
Strongly Agree	
  Comments
	
  
	
  
	
  
	
  

User’s Manual Description:
There are circumstances when members of the development group may have competing interests. For
example, this would apply to a member of the development group whose research on the topic covered by theguideline is also funded by a pharmaceutical company. There should be an explicit statement that all groupmembers have declared whether they have any competing interests.

Where to Look:
Examine the paragraphs/chapters describing the guideline development group or acknowledgements section.
Examples of commonly labeled sections or chapters in a guideline where this information can be found include:
methods, conflicts of interest, guideline panel, and appendix.

How to Rate:
Item content includes the following CRITERIA:• description of the types of competing interests considered• methods by which potential competing interests were sought• description of the competing interests• description of how the competing interests influenced the guideline process and development ofrecommendations

Additional CONSIDERATIONS:• Is the item well written? Are the descriptions clear and concise?
• Is the item content easy to find in the guideline?
• What measures were taken to minimize the influence of competing interests on guideline development orformulation of the recommendations?

	
  
	
  

39	
  

	
  
	
  

OVERALL GUIDELINE ASSESSMENT

	
  
	
  

40	
  

	
  
	
  

OVERALL GUIDELINE ASSESSMENT
For each question, please choose the response which best characterizes the guideline
assessed:
1. Rate the overall quality of this guideline.
1
Lowest possible
quality

2

3

4

5

6

7
Highest possible
quality

2. I would recommend this guideline for use.
Yes
Yes, with modifications
No

NOTES

User’s Manual Description:
The overall assessment requires the AGREE II user to make a judgment as to the quality of the guideline,
taking into account the appraisal items considered in the assessment process.

	
  
	
  

41	
  

	
  
	
  

	
  

AGREE II INSTRUMENT

	
  
	
  
	
  
	
  
	
  
	
  

	
  
	
  
	
  

	
  
	
  

42	
  

	
  

	
  
	
  

DOMAIN 1. SCOPE AND PURPOSE

	
  
	
  
1. The overall objective(s) of the guideline is (are) specifically described.
	
  
	
  
1
7
	
  4
2
3
5
6
Strongly Disagree
Strongly Agree
	
  
	
  
Comments
	
  
	
  
	
  
	
  	
  
	
  
	
  
	
  
	
  
	
  
	
  
	
  
2. The health question(s) covered by the guidelineis (are) specifically described.
	
  
	
  
1
7
2
3
5
6
	
  4
Strongly Disagree
Strongly Agree
	
  
	
  
Comments
	
  
	
  
	
  
	
  
	
  
	
  
	
  
	
  
	
  
	
  
	
  
3. The population (patients, public, etc.) to whom
the guideline is meant to apply is	  specifically described.
	
  
	
  
1
7
	
  4
2
3
5
6
Strongly Disagree
Strongly
Agree
	
  
	
  
	
  
Comments
	
  
	
  
	
  
	
  
	
  
	
  
	
  
	
  
	
  
	
  
	
  
	
  
	
  
	
  

43	
  

	
  
	
  

	
  

DOMAIN 2. STAKEHOLDER INVOLVEMENT	  4. The guideline development group includes	
   individuals from all relevant professionalgroups.
views and preferences have been sought.
	
  
	
  
1
7
2
3
4	
  
5
6
Strongly Disagree
Strongly Agree
	
  
	
  
Comments
	
  
	
  
	
  
	
  
	
  
	
  
	
  
	
  
	
  
	
  
5. The views and preferences of the target population(patients, public, etc.) have been	
  sought.
	
  
	
  
1
7
2
3
4	
  
5
6
Strongly Disagree
Strongly Agree
	
  
	
  
Comments
	
  
	
  
	
  
	
  
	
  
	
  
	
  
	
  
6. The target users of the guideline are clearly	
   defined.
	
  
	
  
1
7
2
3
4
5
6
	
  
Strongly Disagree
Strongly Agree
	
  
	
  
Comments
	
  
	
  
	
  
	
  
	
  
	
  
	
  
	
  

	
  
	
  

44	
  

	
  
	
  

	
  

DOMAIN 3. RIGOUR OF DEVELOPMENT
	
  
	
  

	
  

7. Systematic methods were used to search for evidence.
1
Strongly Disagree

2

3

4

5

6

7
Strongly Agree

Comments

	
  
	
  
	
  
	
  

8. The criteria for selecting the evidence are clearly described.
1
Strongly Disagree

2

3

4

5

6

7
Strongly Agree

Comments

	
  
	
  
	
  
	
  

9. The strengths and limitations of the body of evidence are clearly described.
1
Strongly Disagree

2

3

4

5

Comments

	
  
	
  
	
  
	
  

	
  
	
  

45	
  

	
  
	
  

6

7
Strongly Agree	
  

DOMAIN 3. RIGOUR OF DEVELOPMENT continued
	
  
	
  
10. The methods for formulating the recommendations
are clearly described.
	
  
	
  
1
7
2
3
4	
  
5
6
Strongly Disagree
Strongly Agree
	
  
	
  
Comments
	
  
	
  
	
  
	
  
	
  
	
  
	
  
	
  	
  
	
  
11. The health benefits, side effects, and risks	
   have been considered in formulating therecommendations.
	
  
	
  
	
  
1
7
2
3
4	
  
5
6
Strongly Disagree
Strongly Agree
	
  
	
  
Comments
	
  
	
  
	
  
	
  
	
  
	
  
	
  

DOMAIN 3. RIGOUR OF DEVELOPMENT continued
12. There is an explicit link between the recommendations and the supporting	
  evidence.
1
Strongly Disagree
Comments

	
  
	
  
	
  
	
  

	
  
	
  

2

3

	
  
	
  
4	
  
	
  
	
  
	
  
	
  
	
  
	
  
	
  
	
  
	
  
	
  
	
  

46	
  

5

	
  
	
  

6

7
Strongly Agree	
  

DOMAIN 3. RIGOUR OF DEVELOPMENT continued
	
  
	
  
	
  
13. The guideline has been externally reviewed
by experts prior to its publication.
	
  
	
  
1
7
2
3
4	
  
5
6
	
   AgreeStrongly Disagree
Strongly
Comments

	
  
	
  
	
  
	
  

14. A procedure for updating the guideline is provided.
1
Strongly Disagree

2

3

4

5

Comments

	
  
	
  
	
  
	
  

	
  
	
  

47	
  

	
  
	
  

6

7
Strongly Agree	
  

DOMAIN 4. CLARITY OF PRESENTATION
	
  
	
  
15. The recommendations are specific and unambiguous.
	
  
	
  
1
7
	
  
2
3
4
5
6
Strongly Disagree
Strongly
Agree
	
  
	
  
Comments
	
  
	
  
	
  
	
  
	
  
	
  
	
  
	
  
	
  
	
  
16. The different options for management of	
   the condition or health issue are clearly	  presented.
	
  
	
  
1
7
2
3
4
5
6
	
  
Strongly Disagree
Strongly Agree
	
  
	
  
Comments
	
  
	
  
	
  
	
  
	
  
	
  
	
  
	
  

DOMAIN 5. APPLICABILITY
17. Key recommendations are easily identifiable.

	
  

	
  
	
  
4
	
  
	
  
	
  
	
  
	
  
	
  
	
  
	
  
	
  
	
  
	
  
	
  
	
  

	
  
	
  

48	
  

1
Strongly Disagree
Comments

2

3

5

	
  
	
  

6

7
Strongly Agree	
  

DOMAIN 5. APPLICABILITY
	
  
	
  
18. The guideline describes facilitators and barriers
to its application.
	
  
	
  
	
  
1
7
2
3
4
5
6
Strongly Disagree
Strongly Agree
	
  
	
  
	
  
	
  
Comments
	
  
	
  
	
  
	
  

19. The guideline provides advice and/or tools on how the recommendations can be
put into practice.
1
Strongly Disagree

2

3

4

5

6

7
Strongly Agree

Comments

	
  
	
  
	
  
	
  

20. The potential resource implications of applying the recommendations have been
considered.
1
Strongly Disagree

2

3

4

5

Comments

	
  
	
  
	
  
	
  

	
  
	
  

49	
  

	
  
	
  

6

7
Strongly Agree	
  

DOMAIN 5. APPLICABILITY continued
	
  
	
  
21. The guideline presents monitoring and/or auditing criteria.
	
  
	
  
1
	
  
2
3
4
5
6
Strongly Disagree
	
  
	
  
	
  
Comments
	
  
	
  
	
  
	
  
	
  
	
  
	
  
	
  
	
  
	
  
	
  
	
  
	
  
	
  
	
  
	
  
	
  
	
  
	
  
	
  
	
  

7
Strongly Agree

DOMAIN 6. EDITORIAL INDEPENDENCE
	
  
	
  
	
  
	
  
	
  
	
  
	
  
	
  
	
  
	
  
	
  
	
  
	
  

	
  
	
  

50	
  

	
  
	
  

	
  

DOMAIN 6. EDITORIAL INDEPENDENCE
	
  
	
  
22. The views of the funding body have not influenced
the content of the guideline.
	
  
	
  
	
  
1
7
2
3
4
5
6
Strongly Disagree
StronglyAgree	
  Comments

	
  
	
  
	
  
	
  

23. Competing interests of guideline development group members have been recorded
and addressed.
1Strongly Disagree

2

3

4

5

Comments

	
  
	
  
	
  
	
  

	
  
	
  

51	
  

	
  
	
  

6

7
Strongly AgreeOVERALL GUIDELINE ASSESSMENT
For each question, please choose the response which best characterizes the guideline
assessed:
1. Rate the overall quality of this guideline.
1
Lowest possible
quality

2

3

4

5

2. I would recommend this guideline for use.
Yes
Yes, with modifications
No

NOTES

	
  
	
  

52	
  

	
  
	
  

6

7
Highest possible
quality