442 Review of critical appraisal tools for NHPs Two kinds of critical appraisal instruments have been developed specifically to aid the evaluation of the quality of evidence produced by RCTs of conventional medica- tions (5,6). Many of these instruments focus on evaluat- ing the methodological quality of the trial, examining how the design, conduct and analysis of the RCT minimize biases (e.g. selection, measurement) (7). Other instruments focus on the quality of the report itself; examining the information provided regarding the design, conduct and analysis of the results (8). A critical appraisal instrument useful for evaluating the quality of evidence provided by RCTs of NHPs would include all components found in critical appraisal instruments for conventional medications, as well as questions to assess the quality of reporting of data that is specific to NHPs. Medicines from natural sources are predisposed to significant variation in chemical content as a consequence of how they are grown and/or pro- cessed. The resulting qualitative and/or quantitative variation in chemical content of NHPs can have a direct effect on their pharmacological effects and subse- quently on the outcome of the RCT in which they are being evaluated (9,10). Adequate reporting of the identity and chemical content of the specific NHP that is being evaluated in the RCT is critical if the results of the trial are to be applicable to other brands or preparations of the NHP. Other factors that are unique to NHPs and can affect the quality of the results of an RCT, may be the appearance, smell and taste of the NHP. Therefore correctly blinding an RCT by creating a placebo that is identical in every way to the test NHP is very important. Thus, to adequately appraise the results of an RCT of an NHP, the optimal instrument would include items to assess the identity of the NHP (and its placebo) being evaluated. It would also include questions to help the user determine if the results of the RCT could be applied to other patients and to other products made from the same plant or other natural source. This ability to help the user determine the generalizability of the trial results would facilitate the translation of the research results for application to practice situations (11). It is also important that instructions for how to use the critical appraisal instrument are provided to help ensure that the user is interpreting the questions or items in the instrument correctly (12). The purpose of this project was to conduct a systematic review to identify and analyze instruments designed for critically appraising RCTs that evaluate the efficacy of NHPs. Instruments were examined to determine if they adequately addressed four components: quality of methods, details of the identity and content of the NHP tested, generalizability of results to other patients and other preparations of the NHP and instructions for use of the instrument. Additionally they were examined to determine if the instruments had been validated. Thus, the first objective of this project was to identify published critical appraisal instruments designed to assess the quality of RCTs evaluating the efficacy of NHPs. The second objective was to systematically review the appraisal instruments to document the inclusion of items in each instrument that examined the following: the methodological quality of the trial; the identity and content of the NHP and placebo; and the applicability of trial results to clinical practice. The third objective was to determine how each appraisal instrument had been developed, if it had been validated and if instructions for use were provided. Methods Retrieval of Articles Database searches were performed (PubMed, Embase, Web of Science, CINAHL, IPA and Cochrane Library) up to September 2006 to locate potentially relevant articles describing the critical appraisal of RCTs of NHPs. Search terms were: critical appraisal, scale, checklist, instrument, tool, form, randomized controlled trial, natural health product, herbs, botanical, plant preparation, dietary supplement, complementary therapy, evidence based medicine, quality control, quality assess- ment and standards. Searches were limited to English language articles, with no restrictions on publication dates. Textbooks, files of the investigators and websites of evidence-based medicine (13) were examined for additional instruments. Finally, bibliographies of relevant references were examined for additional citations. Relevance Assessment All potentially relevant articles were retrieved and screened to determine relevance to study objectives by two investigators (T.J. and L.L.), using a form developed specifically for this purpose. Articles were deemed relevant if they met at least one of the following inclusion criteria: they contained instruments used for the appraisal of RCTs of NHPs; they detailed how to report RCTs of NHPs; or they discussed the importance of quality assessment of NHP trials and provided suggested items of importance. Articles were not considered relevant if they met at least one of the following exclusion criteria: the instrument was meant for the assessment of meta-analyses, systematic reviews, or any trial design other than RCT; the instrument was not published in a journal, textbook or on a website; or the instrument was designed for the assessment of RCTs of therapies other than NHPs, e.g. surgery. If the same instrument was identified from several resources, it was included only once.