LETTER / LETTRE

July 26, 2011

Sophie M. Regalado
Editor, JCHLA / JABSC

Ms. Regalado,

I am writing in reply to the letter submitted by Wilczynski et al. regarding my article "Examination of the Clinical Queries and Systematic Review 'hedges' in EMBASE and MEDLINE" that was published in the JCHLA in August 2010 [1].

My response to this letter is complicated by the fact that, on May 19, 2011, Ovid announced in an email to their customers that, in MEDLINE and PsycINFO, the "strategy type labels will be updated from 'sensitivity', 'specificity', and 'optimized' to *maximizes sensitivity, maximizes specificity and best balance of sensitivity and specificity* for each of the categories (for example, Reviews, Therapy, Diagnosis, etc.)" on May 23, 2011 (D. Giustini, personal communication, 2011). They also stated that "the strategies will not change but those using the '1 term' will no longer be available in the Limit menu". No mention was made of changes to EMBASE filter names but, in the June 2011 issue of Ovid's C+T+S Update e-newsletter, the equivalent changes were said to occur to EMBASE filters on May 9, 2011 [2]. As a result, when I refer herein to "old" versus "new" filters, I am referring to the old versus new labels or names.

In the first stage of my search filters research, the plan was to copy and paste the Ovid MEDLINE and EMBASE filter search strings from those on the Health Information Research Unit Hedges webpages [3, 4]. However, I quickly found that several of the MEDLINE filters contained typographical errors and could not be run:

Clinical prediction guides – 2 or more terms high sensitivity,

Clinical prediction guides – 2 or more terms min difference,

Causation (Etiology) – 1 term high sensitivity,

Causation (Etiology) – 1 term min difference.

These websites have since been updated to the new filter names [5, 6], and those consisting of a single term have been removed; however, there are still two on the current MEDLINE page that cannot be run as entered:

Clinical prediction guides – maximizes sensitivity,

Clinical prediction guides – best balance of sensitivity and specificity.

Because of these typographical errors, the methodology was changed to copying and pasting the filters from those published in the individual electronic papers.

Wilczynski et al. are correct that there are typographical errors in the first two EMBASE hedges listed in Table 2 of my publication ("diagnosis (optimized)" and "reviews

(2 or more terms min difference)"). However, on checking my documented (saved) searches, they had been entered correctly when originally run (lines 1 and 2 in Table 1). These were the first two anomalies discovered and I contacted Ovid by email in early 2009 and asked if an explanation of the discrepancies could be provided. No reply was received. In August 2010, the saved searches were rerun and it was found that these two hedges were no longer producing anomalous results. As a result, my assumption was that Ovid had fixed them.

Wilczynski et al. are also correct that the third hedge ("qualitative studies (1 term min difference)") can be run if the phrase is put in quotation marks (line 3 in Table 1). However, they had not included quotation marks in their published version [7] and this hedge was not listed on their HIRU Hedges EMBASE Site [4]. So it was entered as presented in their publication. This single term hedge is no longer available in the EMBASE menu.

For the remaining three hedges ("qualitative studies (2 or more terms min difference)", "treatment (2 or more terms high specificity)", and "treatment (2 or more terms min difference)", Ovid confirmed to Wilczynski et al. that there were errors in the search strings entered and these errors would be fixed by mid-May 2011. In mid-May, they also changed the names of the hedges and removed all 1-term hedges. I have rerun my saved searches of these "old" filters and found that, even in July 2011, these three are still producing anomalous results (lines 4–6 in Table 1). On comparing the results using the "same" filters having new and old labels, I discovered that they are not all equivalent. For example, a search of the EMBASE term "exp diabetes mellitus/" run together with the new and old hedge labels for "treatment (2 or more terms high specificity)" yields different results (search run July 26, 2011, limited to EMBASE).

Lines 3 and 4 are supposedly the same filter, just with new and old names, but obviously they are not the same. Ovid could not simply rename these hedges, as that would mean that saved searches using filters with the old names could no longer be run; rather they created additional sets of hedges. The three filters that Ovid said would be corrected only had the correction made to the newly named filters (lines 4–6 in Table 1). This means that any saved searches that contain any of these three "old" filter labels will still use the wrong search strings. Further, it means that quality control testing is now required for all of the new filters in EMBASE, MEDLINE, and PsycINFO.

<table><thead><tr><td>#</td><td>Searches</td><td>Results</td></tr></thead><tbody><tr><td>1</td><td>exp diabetes mellitus/</td><td>429447</td></tr><tr><td>2</td><td>limit 1 to embase</td><td>330556</td></tr><tr><td>3</td><td>limit 2 to "therapy (maximizes specificity)"</td><td>11058</td></tr><tr><td>4</td><td>limit 2 to "treatment (2 or more terms high specificity)"</td><td>18834</td></tr></tbody></table>
<table><caption>**Table 1.** Discrepancies found in OvidSP EMBASE hedge search results. Search “exp heart infarction/” with limit applied or “ANDed” to search string results (Run July 4, 2011 (1980 – 2009 week of 01 July); limited to EMBASE).</caption><thead><tr><th>Haynes' EMBASE #</th><th>hedge old title</th><th>Haynes' EMBASE hedge search string</th><th>Hits</th><th>EMBASE new hedge title</th><th>Hits</th><th>% difference</th><th>EMBASE old hedge title</th><th>Hits</th><th>% difference</th></tr></thead><tbody><tr><td>1</td><td>Diagnosis (2 or more terms min difference)</td><td>sensitiv:.tw. or diagnostic accuracy.sh. or diagnostic.tw.</td><td>12924</td><td>“diagnosis (best balance of sensitivity and specificity)”</td><td>12924</td><td>0</td><td>“diagnosis (optimized)”</td><td>12924</td><td>0</td></tr><tr><td>2</td><td>Reviews (2 or more terms, best optimization)</td><td>meta-analys:.mp. or search:.tw. or review.pt.</td><td>26033</td><td>“reviews (best balance of sensitivity and specificity)”</td><td>26033</td><td>0</td><td>“reviews (2 or more terms min difference)”</td><td>26033</td><td>0</td></tr><tr><td>3</td><td>Qualitative studies (1 term, best optimization)</td><td>exp “health care facilities and services”/</td><td>45664</td><td>No longer available as menu limit</td><td>N/A</td><td>N/A</td><td>“qualitative studies (1 term min difference)”</td><td>45664</td><td>0</td></tr><tr><td>4</td><td>Qualitative studies (2 or more terms, best optimization)</td><td>interview:.tw. or exp health care organization/ or experiences.tw.</td><td>8686</td><td>“qualitative (best balance of sensitivity and specificity)”</td><td>8686</td><td>0</td><td>“qualitative studies (2 or more terms, min difference)”</td><td>8183</td><td>-5.8%</td></tr><tr><td>5</td><td>Treatment (2 or more terms high specificity)</td><td>double-blind:.mp. or placebo:.tw. or blind:.tw.</td><td>7269</td><td>“therapy (maximizes specificity)”</td><td>7269</td><td>0</td><td>“treatment (2 or more terms high specificity)”</td><td>13974</td><td>92.2%</td></tr><tr><td>6</td><td>Treatment (2 or more terms min difference)</td><td>random:.tw. or placebo:.mp. or double-blind:.tw.</td><td>23304</td><td>“therapy (best balance of sensitivity and specificity)”</td><td>23304</td><td>0</td><td>“treatment (2 or more terms min difference)”</td><td>23659</td><td>1.5%</td></tr></tbody></table>

<table><caption>**Table 2.** HIRU hedges MEDLINE “Therapy – Maximizes sensitivity” filter statistics [5].</caption><thead><tr><th>Therapy</th><th>Ovid Filter</th><th>Sens/spec/prec/acc(%)</th></tr></thead><tbody><tr><td>Filter type</td><td>PubMed Translation</td><td></td></tr><tr><td>Maximizes sensitivity</td><td>clinical trial.mp. or clinical trial.pt. or random:.mp. or tu.xs.<br/>(clinical[Title/Abstract] and trial[Title/Abstract]) or clinical trials[MeSH Terms] or clinical trial[Publication Type] or random*[Title/Abstract] or random allocation[MeSH Terms] or therapeutic use[MeSH Subheading]</td><td>99/70/10/71</td></tr></tbody></table>

The authors contest my statement that the statistics given for the PubMed translations of their Ovid MEDLINE filters are misleading and in some cases incorrect. On the HIRU Hedges MEDLINE website, they list these filters together with the identical sets of statistics (sensitivity/specificity/precision/accuracy (%) (Table 2)). Listing all PubMed translations as having identical statistics to the Ovid strategies implies that search results will be equivalent. Although they are correct that a gold standard test would be needed to prove that any of these translations are incorrect; at the same time, only a gold standard test would prove them to be correct, and to my knowledge, no such tests have been done. So listing the same statistics is misleading at the very least. Although the NLM only lists the sensitivities and specificities as being the same, this has not been proven.

Finally, they say that although my proposed modified filters might appear to be improved, only a gold standard

test could prove this. I agree and emphatically stated so in my paper. However, in some cases, a simple test can be done to compare results. The EBSCO MEDLINE filter “Clinical Prediction Guides – High Specificity” was found to be in error. The Ovid search string consists of only words in the title or abstract (validation.tw. or validate.tw.), with no truncation, so it should have translated perfectly across the Medline platforms. Running the MeSH term “Arterial Occlusive Diseases” with no explode or focus and applying the equivalent built-in limits yields 78 records in Ovid, 78 in PubMed, and 60 in EBSCO. My translation of EBSCO filter yields 78 records (all run on July 26, 2011). However, as I pointed out in my paper, the same numbers of results do not necessarily mean the same results. So I also uploaded all of the records to RefWorks. The Ovid and PubMed results and those obtained by my translated EBSCO filter yielded identical results — same number and same records. So without question, my
translated filter is better than the EBSCO built-in version,
for which the search string was never obtained.

Susan M. Bradley
Consulting medical librarian
Vancouver, B.C.
E-mail: sue.slais@gmail.com

References

1. Bradley SM. Examination of the Clinical Queries and Systematic Review “hedges” in EMBASE and MEDLINE. *Journal of the Canadian Health Libraries Association* 2010; 31(2):27–37. doi:10.5596/c10-022.

2. Ovid C + T + S Update e-newsletter. June 2011. [rev. Jun 2011; cited 14 Jul 2011]. Available from: http://www.ovid.com/site/products/cts_update_june2011.jsp#4.

3. Health Information Research Unit, McMaster University. Search strategies for MEDLINE in Ovid syntax and the PubMed translation [Internet]. June 2007. [rev. 29 Jun 2007; cited 8 Sep 2009]. Available from: http://hiru.mcmaster.ca/hiru/HIRU_Hedges_MEDLINE_Strategies.aspx.

4. Health Information Research Unit, McMaster University. Search Strategies for EMBASE in Ovid Syntax [Internet]. June 2007. [rev. 29 Jun 2007; cited 8 Sep 2009]. Available from: http://hiru.mcmaster.ca/hiru/HIRU_Hedges_EMBASE_Strategies.aspx.

5. Health Information Research Unit, McMaster University. Search strategies for MEDLINE in Ovid syntax and the PubMed translation [Internet]. February 2011. [rev. Jun 2011; cited 14 Jul 2011]. Available from: http://hiru.mcmaster.ca/hiru/HIRU_Hedges_MEDLINE_Strategies.aspx.

6. Health Information Research Unit, McMaster University. Search Strategies for EMBASE in Ovid Syntax [Internet]. February 2011. [rev. Jun 2011; 14 Jul 2011]. Available from: http://hiru.mcmaster.ca/hiru/HIRU_Hedges_EMBASE_Strategies.aspx.

7. Walters LA, Wilczynski NL, Haynes RB, Hedges Team. Developing Optimal Search Strategies for Retrieving Clinically Relevant Qualitative Studies in EMBASE. *Qual Health Res.* 2006;16(1):162–8. doi:10.1177/1049732305284027.