May 2, 2011

Sophie M. Regalado
Editor, JCHLA/JABSC

Ms. Regalado,

In the August, 2010, issue of this journal, Bradley [1] described limitations of our methodological search filters that are in place as Clinical Queries limits in OvidSP EMBASE. Bradley compared the retrieval obtained when searching OvidSP EMBASE using our methodologic search filters as published with that stored within the EMBASE Clinical Queries limits. Bradley examined 33 of our Clinical Queries filters and found that 27 gave identical results, five gave different results, and one could not be tested.

We disagree with some of Bradley's observations pertaining to the filters that did not yield identical results. When comparing the retrieval of the diagnosis (two or more terms, minimum difference) filter as published with that stored within the EMBASE Clinical Queries limits, Bradley reported a 27.8% difference in retrieval. However, Bradley incorrectly entered our EMBASE diagnosis filter; the first term should be "sensitiv*.tw." (as reported in our paper [2]) rather than "sensitive*.tw." as shown in Bradley's Table 2. Running the diagnosis (two or more terms, minimum difference) filter as published yields identical results to the search limits provided using EMBASE Clinical Queries.

Bradley makes a similar error when entering the EMBASE reviews (two or more terms, best optimization) filter also shown in Table 2 of her paper; the first term should be "meta-analys*.mp." (as reported in our paper [3]) rather than "meta-analysis*.mp". Running the reviews (two or more terms, best optimization) filter as published once again yields identical results to the search limits provided using EMBASE Clinical Queries.

In Bradley's Table 2, she reports the yield of the OvidSP EMBASE qualitative studies (one term, best optimization) filter but not for the comparison, as Bradley indicates that the filter could not be run as the EMTREE heading was not recognized. Because this EMTREE heading contains the word "and", the heading must be entered in the OvidSP EMBASE search box within quotes, that is, exp "health care facilities and services" rather than exp health care facilities and services as shown in Bradley's Table 2. Exp is used in Ovid to explode a subject heading. Running the qualitative studies (one term, best optimization) filter with the EMTREE heading in quotes yields identical results to the search limits provided using EMBASE Clinical Queries.

The differences in yield reported by Bradley for the remaining three filters can be explained by errors contained within the EMBASE Clinical Queries limits; one for each of the qualitative studies (two or more terms,

best optimization), treatment (two or more terms, high specificity), and treatment (two or more terms, minimum difference). The qualitative studies (two or more terms, best optimization) filter in Ovid limits has "interview.tw." rather than "interview*.tw.", resulting in fewer citations retrieved when using the Ovid limits, as noted by Bradley. The treatment (two or more terms, high specificity) filter in Ovid limits has "placebo:mp." rather than "placebo:.tw.", resulting in more citations retrieved when using the Ovid limits, as noted by Bradley. The treatment (two or more terms, minimim difference) filter in Ovid limits has "double-blind:.mp." rather than "double-blind:.tw.", resulting in more citations retrieved when using the Ovid limits, as noted by Bradley. Ovid made these changes and the EMBASE Clinical Queries were updated in May 2011.

Bradley also described limitations related to the translations of our methodologic search filters in PubMed. When comparing the yield of OvidSP MEDLINE searches with the PubMed translations, Bradley indicated that, although the majority of the searches yielded a similar number of records, there were five that showed discrepancies greater than 10%, two of which (causation/etiology, high sensitivity and high specificity) were built into PubMed Clinical Queries. Bradley also indicated that, because of these discrepancies, the indication by NLM that the sensitivity, specificity, precision, and accuracy are the same as for the Ovid MEDLINE filters is misleading and, in some cases, incorrect. We disagree with Bradley's statement. First, the NLM reports only the sensitivity and specificity on the Clinical Queries and Health Services Research Queries PubMed websites; precision and accuracy are not reported. Second, without an item-by-item determination of whether each item retrieved is a true-positive, false-positive, true-negative, or false-negative, it is impossible to say that the sensitivity, specificity, and accuracy are misleading or incorrect. Tables 1 and 2 show the operating characteristics for two searches that yield differing numbers of records. As can be seen, it is possible that the only figure that differs is precision. We have indicated in all our publications the precision figures are an estimate and, as noted, these figures are not published by NLM on their website.

Bradley proposes modifications to the five PubMed filter translations that show differences (Bradley's Table 4). We agree with Bradley that although the modified filters may appear improved, this would only be determined by a careful study using a gold standard set.

Bradley also described limitations related to the translations of our methodologic search filters in EBSCO MEDLINE. She found that three EBSCO MEDLINE Clinical Queries gave different results. Once again, Bradley proposes modifications to the translations. Once again, as indicated above, only a careful study using a gold standard set could determine that the modified filters are an improvement.
<table><caption>**Table 1.** A search yielding 100 records.</caption><thead><tr><th rowspan="2">Search term</th><th colspan="2">Gold standard</th></tr><tr><th>+</th><th>-</th></tr></thead><tbody><tr><td>+</td><td>10</td><td>40</td></tr><tr><td>-</td><td>10</td><td>40</td></tr></tbody></table>

Note: Sensitivity, 10/20 (50%); Specificity, 40/80 (50%); Accuracy, 50/100 (50%); Precision, 10/50 (20%).

<table><caption>**Table 2.** A search yielding 1000 records.</caption><thead><tr><th rowspan="2">Search term</th><th colspan="2">Gold standard</th></tr><tr><th>+</th><th>-</th></tr></thead><tbody><tr><td>+</td><td>10</td><td>490</td></tr><tr><td>-</td><td>10</td><td>490</td></tr></tbody></table>

Note: Sensitivity, 10/20 (50%); Specificity, 490/980 (50%); Accuracy, 500/1000 (50%); Precision, 10/500, (2%).

Nancy L. Wilczynski*

R. Brian Haynes

K. Ann McKibbon,

Department of Clinical Epidemiology and Biostatistics,
Health Information Research Unit,

McMaster University,

Hamilton, Ontario, Canada

*Corresponding author, e-mail: wilczyn@mcmaster.ca

References

1. Bradley SM. Examination of the Clinical Queries and Systematic Review “hedges” in EMBASE and MEDLINE. *Journal of the Canadian Health Libraries Association*. 2010; 31(2):27–37. doi:10.5596/c10-022.

2. Wilczynski NL, Haynes RB; Hedges Team. EMBASE search strategies for identifying methodologically sound diagnostic studies for use by clinicians and researchers. *BMC Med*. 2005; 3:7. doi:10.1186/1741-7015-3-7.

3. Wilczynski NL, Haynes RB; Hedges Team. EMBASE search strategies achieved high sensitivity and specificity for retrieving methodologically sound systematic reviews. *J Clin Epidemiol*. 2007;60(1):29–33. Epub 2006 Jul 20. PubMed PMID: 17161751.