446 Review of critical appraisal tools for NHPs Table 1. Miscellaneous items that appeared in instruments for critical appraisal of RCTs of NHPs Item Enough detail for reanalysis by reader/study replicable Intervention described in detail Groups treated equally Compliance Indication for further intervention Components known to be predictive of clinical effect? Possibility of publication bias Is journal peer-reviewed? Informed consent/ethics approval Syndrome of disease based on Chinese herbal medicine Rationale of Chinese herbal medication composition Selection bias after allocation Number of times item appeared 3 2 1 1 1 1 1 1 1 1 1 1 Number of times cited 6 (38%) 3 (19%) 3 (19%) 2 (13%) 2 (13%) 2 (13%) 1 (6%) 1 (6%) resulted in articles that pertained to implements/tools used for surgery or other procedures and were not at all relevant to our study. Similar difficulties were also encountered by the Agency for Healthcare Research and Quality (7). We followed their suggestions of not relying only on literature searches, and used published reports, such as theirs, to identify existing instruments. In the end, we did find that many articles, not identified through database searching, were located by hand searching the reference lists of other articles. Due to resource constraints we were not able to search non-English literature or contact other researchers/ experts in this field for identification of other potential instruments. It is possible that instruments were missed by not searching in languages other than English. However, we believe we were able to identify many of the commonly used instruments, as the same instru- ments began to appear repeatedly in the databases we searched and in the hand searching of reference lists. These are likely the same instruments that would also be found by any health care provider trying to find a critical appraisal instrument for use in practice or research. Thus, despite the limitations we have described, we believe that we were able to identify and review the most readily available assessment instruments for RCTs of NHPs. Conclusion Although this systematic review was successful in identifying appraisal instruments for evaluating the quality of RCTs of NHPs, none of those identified were widely accepted as the gold standard instrument. The CONSORT Statement for Herbal Interventions comes closest to fulfilling requirements for a quality instrument to critically appraise RCTs of NHPs, however it’s limitation is that it is specific for herbal medicines. There is therefore a need for the development and validation of an instrument that can be used to reliably assess the quality of results of RCTs of any NHPs, whether herbs or single, natural sourced chemicals. Empirical evidence and results of this review suggest that randomization sequence generation, allocation con- cealment, blinding and participant flow are four elements that improve the quality of an RCT and should be included as items in a critical appraisal instrument. Items pertaining specifically to the identity and content of NHPs as well as applicability of results to practice must be included. It remains to be determined if a group of items specific to NHPs could be developed, validated and added to an existing gold standard instrument used for assessing quality of trials of conventional medication or whether an instrument specific for assessing trials of NHPs will need to be developed. Table 2. Referencesa cited most frequently as source for adapting/ modifying for instrument development Reference Jadad et al. (40) Moher et al. (revised CONSORT) (15) JAMA Users’ guides 1993–94 (11,42,43) Schulz et al. (44) Cochrane handbook (45) Juni et al. (46) Meinert et al. (47) Linde et al. (48) aSome articles reported more than one reference used as a source. Although appraisal instruments for other types of study design exist, we limited our review to only instruments designed to assess quality of RCTs as the efficacy of NHPs are being increasingly evaluated using this methodology. Although we followed systematic review methodology as rigorously as possible when conducting this research we did encounter difficulties particularly with regards to literature searching. Selecting search terms to use to identify relevant articles in various databases proved problematic, as indexing terms varied considerably between databases. It is possible that some instruments may not have been identified as search terms were not exhaustive. It also appeared as though MeSH headings were not very effective at identifying desired articles. As a result, text word searching was used, which resulted in many hits that were deemed to be irrelevant to the research. For example, using the term ‘instrument’