**Evaluating MD on Tap and MEDLINE via Handheld Computers as a Resource
for Mobile Clinicians.**

Final Report

Submitted to the NIH Office of Evaluation

November 2006

By

Dr. Susan E. Hauser

Communications Engineering Branch

Lister Hill national Center for Biomedical Engineering

National Library of Medicine

National Institutes of Health

Bethesda, MD 20894

**Purpose:**

By observing and recording the use of a handheld computer application
and system, MD on Tap, in real clinical scenarios, we evaluate MEDLINE's
role in this environment as a tool for the practice of Evidence Based
Medicine (EBM) and the ability to answer clinical questions at the
bedside using MEDLINE. The study will simultaneously gain an
understanding of which features of the MD on Tap system design are most
useful for quickly finding relevant, high-quality journal article
citations in real time. Specific research questions are:

1\. Is PDA access to MEDLINE a useful tool in the practice of EBM?

2\. Is PDA access to MEDLINE a useful tool for discovering design
principles for other tools for the practice of EBM?

3\. Is searching MEDLINE during rounds in a teaching hospital useful in
teaching EBM ?

4\. What factors contribute to successful searches at the point of care?

5\. What is the quality of the answers obtained by
physicians-in-training using handheld computers?

**Background:**

Published articles about the information needs of clinicians often
include MEDLINE^®^ in a long list of sources of information. Other
recent publications describe the growing use of handheld computers among
clinicians because of their ability to provide diverse information at
the point-of-care. Healthcare providers, especially younger clinicians,
residents and medical students, are increasingly adopting the use of
handheld computers for access to a variety of information sources. The
handheld computer hardware, operating systems, peripherals,
communications and input modalities are all in flux, with each new
generation of device boasting improved performance and usability.
Providers of online information resources, such as MEDLINE, will need to
employ well-founded design principles to effectively serve mobile
healthcare providers over the wide and changing handheld computer
platforms available.

The MD on Tap project seeks to discover those design principles for the
real-time on-line delivery of information to mobile healthcare
professionals via wireless handheld computers. The MD on Tap system
consists of client applications for Palm and Pocket PC wireless PDAs or
smartphones. The application supports real-time search of and retrieval
from MEDLINE. The clients communicate over the Internet with an
intermediate http server that communicates with the National Library of
Medicine's PubMed via the entrez e-utilities to execute MEDLINE searches
and retrieve citation data. The intermediate server records
de-identified transactions received from clients. Through analysis of
these stored transactions we have learned several aspects of aggregate
user behavior regarding searching MEDLINE with a small device. However,
the analysis is limited in the ability to relate the query terms and
retrieved citations to the clinical scenario being addressed. Careful
manual examination can occasionally infer the clinical question being
asked, but with insufficient confidence to determine if the question was
answered by the citations retrieved. This study overcomes that
limitation.

**Methods:**

A contract was awarded to the University Clinical, Education and
Research Associates (UCERA) on Honolulu, Hawaii to conduct the
evaluation at teaching hospitals in Honolulu associated with the John A.
Burns School of Medicine (JABSOM), University of Hawaii. Joshua L.
Jacobs, MD, Director of Medical Education, supervised the Hawaii
activity. Five medical residents conducted the fieldwork as part of
clinical electives for the study of Medical Informatics. Each resident
gathered data for about 4 weeks, with the total period for the 5
residents extending from December 2005 to June 2006, including gaps in
which no data was generated.

*[Training:]{.underline}* Prior to the fieldwork, residents became
familiar with using the MD on Tap client on a Treo 650 PDA/cell phone.
The PDA/cell phone was chosen by UCERA, rather than a WiFi PDA, because
WiFi is not widely available in hospitals in Honolulu. The residents
also learned to use a desktop computer web-based transaction review
program developed by the MD on Tap team to assist in writing daily
summaries.

*[Fieldwork:]{.underline}* Each resident accompanied medical teams
consisting of an attending physician, residents, interns and students on
teaching rounds for approximately 4 weeks, not as a member of the
medical team, but as a trained observer. To maximize the number and
variety of clinical questions, all residents chose to accompany teams
rounding in Intensive Care Units in two Honolulu hospitals. While on
rounds, the residents recorded all clinical scenarios and clinical
questions that arose at the point of care, excluding questions
concerning individual patient data. They used MD on Tap to search for
citations that addressed the questions and identified those citations
that were judged to be relevant to answering the question. The residents
recorded the questions and searched for answers even when the resident
already knew the answer. The residents were instructed to explore all of
the features of the MD on Tap application in order to access the
usefulness of each. The residents used a special MD on Tap client that
includes a UserID tag with each transaction. All transactions are stored
in the database associated with the MD on Tap intermediate server. The
UserID tag allows for identification of transactions to be included in
the evaluation analysis. The UserID tag was also used by a web-based
transaction review program supplied by the MD on Tap team to assist the
residents in writing their daily summaries by reviewing their MD on Tap
transactions at a later time from a desktop computer.

*[Daily Summaries:]{.underline}* Residents submitted by email to the NLM
MD on Tap team daily summaries in Excel format of the scenarios,
clinical questions, EBM task category of each question, who asked each
question, found citations (PMIDs) that were relevant to answering each
question, and comments.

*[Summative Report:]{.underline}* Following the data collection
fieldwork, each resident submitted a final qualitative and quantitative
assessment of the ability to find relevant evidence in MEDLINE using MD
on Tap at the point of care. They were instructed to distill their
experience and discuss features of MEDLINE searching and the features of
MD on Tap that were most useful for quickly finding useful citations,
and any additional features that could facilitate that process. In
practice, the residents only responded with specific answers to a list
of specific questions provided by the MD on Tap team.

*[Second Opinion:]{.underline}* One member of the NLM team, a senior
medical librarian and expert indexer, reviewed all of the citations
selected by the residents and corroborated the relevance of the citation
to answering the clinical question as described in the daily summary.
Only those citations that passed this second level of scrutiny were
classified as Relevant in our final analysis.

*[Site Visit:]{.underline}* Three members of the MD on Tap team
conducted a 3-day site visit of the JABSOM campus and the two teaching
hospitals where the fieldwork was conducted. The team met several JABSOM
faculty and staff and had short meetings with two of the participating
medical residents. The team also presented a seminar on MD on Tap at
Grand Rounds in one of the hospitals and conducted four workshops on MD
on Tap for fellows, students and librarians.

*[Intermediate Server:]{.underline}* All transactions executed by the
residents were parsed by the MD on Tap Intermediate server and stored in
the MD on Tap database.

**Results:**

During training and fieldwork, the five residents executed 3146 MD on
Tap transactions, all stored in the MD on Tap database. They entered
1258 MEDLINE queries and viewed 1686 MEDLINE citations. The transactions
have been collated with the daily summaries from resident evaluators,
with additional data extracted from the database and with the relevancy
scores assigned by our team member. Various discrepancies between what
the residents did and what they said they did have been reconciled. All
these data are assembled into a few large spreadsheets to support
analysis. Tables 1 and 2 show some of the aggregate counts and
percentage results. In these tables, "Relevant citations" are those
judged to be relevant by both the resident and by the medical librarian.

**Table 1. Results from the Daily Summaries and the Medical Librarian**.

+-------------------------+----+----+----+----+----+----+-----+-----+
| **Resident-\>**         | *  | *  | *  | *  | *  | *  | **A | **A |
|                         | *1 | *2 | *3 | *4 | *5 | *T | ggr | ggr |
|                         | ** | ** | ** | ** | ** | ot | .** | .** |
|                         |    |    |    |    |    | al |     |     |
|                         |    |    |    |    |    | ** | **  | **  |
|                         |    |    |    |    |    |    | Avg | %** |
|                         |    |    |    |    |    |    | .** |     |
+-------------------------+----+----+----+----+----+----+-----+-----+
| **\# scenarios**        | 39 | 42 | 65 | 41 | 41 | 2  |     |     |
|                         |    |    |    |    |    | 28 |     |     |
+-------------------------+----+----+----+----+----+----+-----+-----+
| **\# clinical questions | 55 | 53 | 85 | 1  | 61 | 3  |     |     |
| (Q)**                   |    |    |    | 09 |    | 63 |     |     |
+-------------------------+----+----+----+----+----+----+-----+-----+
| **\# days**             | 63 | 75 | 87 | 1  | 82 | 4  |     |     |
|                         |    |    |    | 11 |    | 18 |     |     |
+-------------------------+----+----+----+----+----+----+-----+-----+
| **Avg. Q per day**      | 4  | 3  | 5  | 5  | 4  |    | 4.7 |     |
|                         | .2 | .5 | .3 | .7 | .7 |    |     |     |
+-------------------------+----+----+----+----+----+----+-----+-----+
| **% Q with task =       | 11 | 15 | 34 | 13 | 5  |    |     | 17  |
| Diagnosis**             |    |    |    |    |    |    |     |     |
+-------------------------+----+----+----+----+----+----+-----+-----+
| **% Q with task =       | 35 | 36 | 11 | 48 | 43 |    |     | 34  |
| Etiology**              |    |    |    |    |    |    |     |     |
+-------------------------+----+----+----+----+----+----+-----+-----+
| **% Q with task =       | 47 | 49 | 52 | 32 | 52 |    |     | 45  |
| Therapy**               |    |    |    |    |    |    |     |     |
+-------------------------+----+----+----+----+----+----+-----+-----+
| **% Q with task =       | 7  | 0  | 2  | 0  | 0  |    |     | 2   |
| Prognosis**             |    |    |    |    |    |    |     |     |
+-------------------------+----+----+----+----+----+----+-----+-----+
| **% Q asked by          | 40 | 55 | 22 | 35 | 38 |    |     | 36  |
| Attending**             |    |    |    |    |    |    |     |     |
+-------------------------+----+----+----+----+----+----+-----+-----+
| **% Q asked by          | 51 | 45 | 52 | 67 | 62 |    |     | 57  |
| Resident**              |    |    |    |    |    |    |     |     |
+-------------------------+----+----+----+----+----+----+-----+-----+
| **% Q asked by Intern** | 9  | 0  | 26 | 0  | 0  |    |     | 7   |
+-------------------------+----+----+----+----+----+----+-----+-----+
| **% Q asked by          | 0  | 0  | 0  | 3  | 0  |    |     | 1   |
| Student**               |    |    |    |    |    |    |     |     |
+-------------------------+----+----+----+----+----+----+-----+-----+
| **\# Q for which 1 or   | 48 | 41 | 59 | 72 | 28 | 2  |     | 68  |
| more Relevant citation  |    |    |    |    |    | 48 |     |     |
| found**                 |    |    |    |    |    |    |     |     |
+-------------------------+----+----+----+----+----+----+-----+-----+
| **% Q for which 1 or    | 87 | 77 | 69 | 66 | 46 |    |     | 68  |
| more Relevant citation  |    |    |    |    |    |    |     |     |
| found**                 |    |    |    |    |    |    |     |     |
+-------------------------+----+----+----+----+----+----+-----+-----+
| **\# of Relevant        | 1  | 97 | 1  | 2  | 63 | 7  |     |     |
| citations**             | 74 |    | 60 | 44 |    | 38 |     |     |
+-------------------------+----+----+----+----+----+----+-----+-----+

**Table 2. Results collated from the MD on Tap database and the Daily
Summaries**.

+-------------------------+----+----+----+----+----+----+-----+-----+
| **Resident-\>**         | *  | *  | *  | *  | *  | *  | **A | **A |
|                         | *1 | *2 | *3 | *4 | *5 | *T | ggr | ggr |
|                         | ** | ** | ** | ** | ** | ot | .** | .** |
|                         |    |    |    |    |    | al |     |     |
|                         |    |    |    |    |    | ** | **  | **  |
|                         |    |    |    |    |    |    | Avg | %** |
|                         |    |    |    |    |    |    | .** |     |
+-------------------------+----+----+----+----+----+----+-----+-----+
| **Elapsed time while    | 6  | 8  | 7  | 9  | 8  | 39 |     |     |
| connected**             | h, | h, | h, | h, | h, | h, |     |     |
|                         | 1  | 5  | 6m | 7m | 8m | 2  |     |     |
|                         | 4m | 0m |    |    |    | 5m |     |     |
+-------------------------+----+----+----+----+----+----+-----+-----+
| **Avg. Q per hour       | 8  | 6  | 12 | 12 | 7  |    | 9.2 |     |
| connected**             | .8 | .0 | .0 | .0 | .5 |    |     |     |
+-------------------------+----+----+----+----+----+----+-----+-----+
| **Avg. elapsed time per | 3  | 5  | 1  | 1  | 2  |    | 259 |     |
| Q (sec)**               | 17 | 51 | 46 | 65 | 95 |    |     |     |
+-------------------------+----+----+----+----+----+----+-----+-----+
| **Avg. \# transactions  | 8  | 13 | 4  | 6  | 8  |    | 7.6 |     |
| per Q**                 | .3 | .6 | .8 | .0 | .4 |    |     |     |
+-------------------------+----+----+----+----+----+----+-----+-----+
| **Avg. \# queries per   | 2  | 4  | 2  | 1  | 2  |    | 2.6 |     |
| Q**                     | .7 | .8 | .2 | .8 | .7 |    |     |     |
+-------------------------+----+----+----+----+----+----+-----+-----+
| **Avg. \# viewed        | 5  | 6  | 2  | 4  | 4  |    | 4.3 |     |
| citations per Q**       | .1 | .9 | .3 | .0 | .6 |    |     |     |
+-------------------------+----+----+----+----+----+----+-----+-----+
| **Avg. \# Relevant      | 2  | 1  | 1  | 1  | 0  |    | 1.3 |     |
| citations found per Q** | .5 | .8 | .1 | .1 | .7 |    |     |     |
+-------------------------+----+----+----+----+----+----+-----+-----+
| **Avg. time to find     | 1  | 2  | 1  | 1  | 1  |    | 161 |     |
| Relevant citation from  | 88 | 38 | 06 | 19 | 40 |    |     |     |
| original query (sec)**  |    |    |    |    |    |    |     |     |
+-------------------------+----+----+----+----+----+----+-----+-----+
| **Avg. time to find     | 1  | 1  | 47 | 67 | 78 |    | 84  |     |
| Relevant citation from  | 07 | 11 |    |    |    |    |     |     |
| successful query        |    |    |    |    |    |    |     |     |
| (sec)**                 |    |    |    |    |    |    |     |     |
+-------------------------+----+----+----+----+----+----+-----+-----+

The 1258 queries included continued queries, and related article queries
as well as original queries. The residents frequently repeated the same
query. The residents executed 803 distinct (i.e. unique set of query
terms, limits and search preferences), original (i.e. not a continued
query or a Related Article query) queries. Aggregate characteristics of
these queries are shown in Table 3.

Table 3. Aggregate characteristics of 803 distinct queries.

  ------------------------------- -------------------- ------------------
  **Characteristic**              **Count or Average** **%**

  **Entrez Search Engine**        629                  78

  **Essie Search Engine**         113                  14

  **Google Search Engine**        61                   8

  **"Only with abstract" limit**  259                  32

  **Human limit**                 273                  34

  **English limit**               273                  34

  **Journal Subset limit**        0                    0

  **Publication Type limit**      25                   3

  **Date limit**                  117                  15

  **Clinical Query hedge**        56                   7

  **Cluster by journal subject**  233                  29

  **Cluster by EBM                53                   7
  classification**                                     

  **Auto Spellcheck**             668                  83

  **PICO interface**              80                   7

  **\# non-stop-word terms**      3.3                  

  **\# MeSH (translated) terms**  1.8                  
  ------------------------------- -------------------- ------------------

At Grand Grounds and the workshops conducted in Hawaii, we presented the
possibility of using a PICO (Patient/Problem, Intervention, Comparison,
Outcome) frame as a user interface to assist in formulating MEDLINE
searches. The idea was enthusiastically received by our audiences, and
Dr. Jacobs formally requested that we add that feature to MD on Tap. We
were able to complete the addition of a PICO interface in time for the
last two residents, who conducted their fieldwork at about the same
time, to try it briefly. One of the residents liked the interface and
one did not. Because usage was not extensive, there may be insufficient
data to make a quantitative distinction about performance differences
between PICO searches and non-PICO searches.

The two residents with whom we met during the site visit brought up the
importance of being able to view the whole title of an article as the
first step in deciding if the article might be relevant to the clinical
question. This was especially important in their environment because of
the slower data rate of the cell phone connection, as opposed to WiFi
connection. Fetching a complete citation in order to read the title
represented a considerable investment of time. MD on Tap was modified to
include the option of viewing the complete article title for initial
review of search results.

**Continued Analysis:**

The 803 distinct queries were manually collated with the lists of
relevant articles. Of the 803 queries, 526 resulting in no articles
identified as relevant to the clinical question being addressed and 277
queries returned 1 or more relevant articles. We have begun analyzing
these for characteristics that contribute to the likelihood of returning
relevant results. For example, we started with the number of
non-stop-word terms in the query string and the number of terms in the
query string that PubMed translates to a MeSH term. A summary appears in
Tables 4a through 5b. It appears that queries with 3 or 4 non-stop-word
terms increases the likelihood of finding relevant citations above the
overall likelihood, whereas 5 or more terms decreases the likelihood.
For MeSH terms, the likelihood of finding relevant citations increases
with each added term. The likelihood is greater than the overall
likelihood with each term over 2 terms. These data need additional
study, including statistical tests of the significance of the apparent
differences.

**Table 4a. Initial analysis of non-stop-word query terms (counts).**

  -------------------------------- ----------- --------- --------- --------- ---------
                                   **\#                                      
                                   Relevant                                  
                                   citations                                 
                                   in query                                  
                                   results**                                 

  **\# non-stop-word terms in      **0**       **\>0**   **\>1**   **\>2**   **\>3**
  query**                                                                    

  **1**                            17          5         2         1         1

  **2**                            140         71        29        15        4

  **3**                            179         107       51        25        12

  **4**                            94          54        13        4         0

  **5**                            62          27        9         6         1

  **6**                            19          10        6         5         2

  **7**                            12          3         0         0         0

  **8**                            2           0         0         0         0

  **9**                            1           0         0         0         0

  **Total \# queries -\>**         526         277       110       56        20
  -------------------------------- ----------- --------- --------- --------- ---------

**Table 4b. Initial analysis of non-stop-word query terms
(percentages)**.

  -------------------------------- ------------ --------- --------- --------- ---------
                                   **Relevant                                 
                                   citations in                               
                                   results as %                               
                                   of all                                     
                                   queries**                                  

                                   **0**        **\>0**   **\>1**   **\>2**   **\>3**

  **% of all queries -\>**         66           35        14        7         2

  **\# non-stop-word terms in                                                 
  query**                                                                     

  **1**                            77           23        9         5         0

  **2**                            66           34        14        7         2

  **3**                            63           37        18        9         4

  **4**                            64           36        9         3         0

  **5**                            70           30        10        7         1

  **6**                            66           34        21        17        7

  **7**                            80           20        0         0         0

  **8**                            100          0         0         0         0

  **9**                            100          0         0         0         0
  -------------------------------- ------------ --------- --------- --------- ---------

**Table 5a. Initial analysis of MeSH query terms (counts)**.

  -------------------------------- ----------- --------- --------- --------- ---------
                                   **\#                                      
                                   Relevant                                  
                                   citations                                 
                                   in query                                  
                                   results**                                 

  **\# MeSH terms in query**       **0**       **\>0**   **\>1**   **\>2**   **\>3**

  **0**                            35          12        4         3         2

  **1**                            168         82        38        19        6

  **2**                            236         125       50        24        10

  **3**                            75          47        17        9         1

  **4**                            11          8         1         1         1

  **5**                            1           3         0         0         0

  **Total \# queries -\>**         526         277       110       56        20
  -------------------------------- ----------- --------- --------- --------- ---------

**Table 5b. Initial analysis of MeSH query terms (percentages)**.

  -------------------------------- ------------ --------- --------- --------- ---------
                                   **Relevant                                 
                                   citations in                               
                                   results as %                               
                                   of all                                     
                                   queries**                                  

                                   **0**        **\>0**   **\>1**   **\>2**   **\>3**

  **% of all queries -\>**         66           35        14        7         2

  **\# MeSH terms in query**                                                  

  **0**                            74           26        9         6         4

  **1**                            67           33        15        8         2

  **2**                            65           35        14        7         3

  **3**                            61           39        14        7         1

  **4**                            58           42        5         5         5

  **5**                            25           75        0         0         0
  -------------------------------- ------------ --------- --------- --------- ---------

We anticipate that similar analysis of the several search features will
lead to an understanding of which methods work best for efficient
MEDLINE searching on a handheld computer.

**Publications:**

One session paper and one poster about aspects of this evaluation will
be presented at the 2006 American Medical Informatics Association
Symposium, to be held November 12-15 in Washington, DC:

"MEDLINE as a Source of Just-in-Time Answers to Clinical Questions" by
Dina Demner-Fushman, Susan E. Hauser, Susanne M. Humphrey, Glenn M.
Ford, Joshua L. Jacobs, and George R. Thoma.

"Preliminary Comparison of Three Search Engines for Point of Care Access
to MEDLINE Citations" by Susan E. Hauser, Dina Demner-Fushman, Glenn M.
Ford, Joshua L. Jacobs, and George Thoma.

A copy of the paper and the poster are attached to this report.

We anticipate additional publications as analysis of the data continues.

**Conclusions:**

Searching MEDLINE for references to the primary literature can assist in
answering clinical questions that arise at the point of care. Handheld
computers with Internet access are useful tools for healthcare providers
to access to MEDLINE in real time. The MD on Tap application is an
effective interface to MEDLINE in clinical settings. MD on Tap is easy
to use and allows clinicians to quickly find relevant citations.